@tailcalled @MatthewBJane @henrybemis13 This post. Of course “cognitive tasks” hints we might be in very low reliability turf
@ed_hagen @Jake_Hallam_ https://t.co/Qdx9buclLj This is true, reliable cognitive tasks don't actually show much individual differences
RT @LenaMBlott: @AlonZivony @DrGBuckingham Excellent paper on relatively low reliability of stroop and other commonly used tasks (esp if sc…
@AlonZivony @DrGBuckingham Excellent paper on relatively low reliability of stroop and other commonly used tasks (esp if scored as diff scores btw 2 conditions) for indiv diff research:https://t.co/7hs09b2yAO
@DrGBuckingham @AlonZivony It’s easily replicable but the test-retest reliability isn’t great https://t.co/3tWL4lQhmv
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences https://t.co/U6LhdNSKfy via @HedgeCraig et al https://t.co/BenEPmmn91
@RicardoRey_95 @Cognomada @AliciaFrancoXVE Yo tampoco soy ningún experto en esto jajaja. Pero si tuviera que empezar, yo creo que a partir de este paper de Hedge se ha tomado mucha más conciencia de la problemática de usar medidas de tareas experimentales
@MatthewBJane "Correlational research (and the ICC) relies on the proportion of variance accounted for by individual differences, and the standard subtractions (e.g., to calculate the Stroop RT cost) do not improve this signal-to-noise ratio–if anything, i
A great paper by @HedgeCraig, Georgina Powell and Petroc Sumner! "...robust experimental effects do not necessarily translate to optimal methods of studying individual differences." 🧠⚡️ https://t.co/Vkk9iFhNfE
@bioDGPs_DGPA @igor_dgps Take the reliability paradox (https://t.co/QeT9sgWE3i): Why do statistically significant results not go hand in hand with high reliability? If the data are robust and show a clear pattern, shouldn't this show on both significance a
@ianhussey I completely agree with your point, but a stronger counter-argument may lie in the effect size - reliability trade-off (e.g., https://t.co/QJc6RrJxgt), and if you're sceptical of single-measurement rel measures, both arguments together might be
Excellent paper from the vault. Distinguishing between-subject & within-subject variability: relations with construct reliability & individual differences. https://t.co/7hNF7Vo90K https://t.co/XtHI2piud5
The identification of robust correlates requires precise measures (c Hedge et al.). The goal of this project was hence to probe the reliability of individual rationality measurements. https://t.co/PAv94IphxQ
Yesterday, @Sam_D_Parsons gave a great talk on (un)reliability in cognitive measures! 🔥 Proud that he was on board at our #CRC940 #SFB940 symposium and showed us his splithalf R package and tools to examine #reliability in cognitive tasks! #reproducibility
RT @JosephineZerna: Recommended by @Sam_D_Parsons in his talk about the (un)reliability of cognitive tasks at the #CRC940 Symposium in Dres…
Recommended by @Sam_D_Parsons in his talk about the (un)reliability of cognitive tasks at the #CRC940 Symposium in Dresden today - hopefully there will be some time between sessions tomorrow to read it in full! https://t.co/IsmQacGeyF
@ajshackman …and agree we’re likely to find even stronger relationships if we move beyond classic tasks, which paradoxically often suppress across-sub variability (cf. one of my fave refs, https://t.co/BYGOq2o6kC), and devise nuanced paradigms that better
Surprisingly, not every effect is heterogeneous in the population. Some tasks are actually very homogeneous: https://t.co/KS4tfaEx01. Heterogeneity varies in multi-lab studies but is often swamped by measurement error.
@Jake_Elder52 You might be interested in the work of @HedgeCraig which grapples with measurement reliability issues. E.g., https://t.co/alQ28MK4Yz
@hannah_h_b You can find the datasets from Hedges et al (2018; https://t.co/qyVo1ZhQri) all on the OSF at https://t.co/AwSLUJcDa1
RT @actpredictlab: @MattNeuropsych @ajwills72 this one makes a similar point, when using behavioural tasks to study individual differences.…
@MattNeuropsych @ajwills72 this one makes a similar point, when using behavioural tasks to study individual differences. https://t.co/j5RHKaCGIW
@DerekCBriggs Interesting -- thanks!! An interesting problem, indeed, as it continues to be found in the modern day literature, e.g., https://t.co/7ULUb00MTC
RT @dongwon_oh: The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences [open access] https://t.…
RT @dongwon_oh: The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences [open access] https://t.…
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences [open access] https://t.co/lo3rtNyebc h/t @actpredictlab https://t.co/Sr3tyYqx3r
@PEEP_lab @t_awkr @Sam_D_Parsons Hedge's classic "Reliability paradox" is my favorite paper on this issue https://t.co/c4VCeUCcEw And @JeffRouder and @JuliaHaaf have several excellent papers extending this issue. For example, https://t.co/CCu5nsh80G T
RT @danieljamesyon: @actpredictlab @DrGBuckingham Possibly relevant? https://t.co/nT2opvVwuv
@actpredictlab @DrGBuckingham Possibly relevant? https://t.co/nT2opvVwuv
@lakens @rubenarslan I just cited two papers that show it can be useful. Besides, the problem is not limited to the BART, as I showed in the pictures above. + Another issue: https://t.co/YsJRACMvj3 Similar solution to the issue: https://t.co/mDXWN05BFT
RT @Sam_D_Parsons: @jamesheathers "The Reliability Paradox" should be essential reading for anyone looking to mix cognitive measures with i…
@drcpennington @aggieerin Hedge et al have so much data! added benefit is they used lots of trials, so it's a great dataset for exploring that too. They also have an appendix looking at the stability of reliability estimates with increasing n trials https
@aggieerin In my archive two papers focused on the effect size across cognitive tasks. They may not directly answer your question. You can dig what you want from their data. https://t.co/GMI9eP7Gqa https://t.co/2ncIsRbBuE
This has been nicely described in Hedge et al. (2018) https://t.co/cuODbMQCmF and corresponds to our findings 👇 https://t.co/sLoeCApzfZ
RT @MeridethGattis: Note to self: Add the reliability paradox to my research methods lectures #PS1018 https://t.co/0YrljOZn1u
RT @MeridethGattis: Note to self: Add the reliability paradox to my research methods lectures #PS1018 https://t.co/0YrljOZn1u
Note to self: Add the reliability paradox to my research methods lectures #PS1018 https://t.co/0YrljOZn1u
@vosshbc If you are interested in the translation challenge with experimental (e.g. cognitive psych/neuro) measures, also see Hedge's now classic 2018 paper https://t.co/jpyJHaQgnf (Also our paper with some approaches for bridging the translation gap: h
This isn't necessarily shocking. It is well-known that behavioral measures often sacrifice high between-person variation (key to reliability) for low between-person variability (key to observing robust experimental effects). See: https://t.co/hfRlgMSbdk, h
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences | SpringerLink https://t.co/c7ctdbWOpr
RT @leafs_s: Behavior Research Methods July 2017 The reliability paradox: Why robust cognitive tasks do not produce reliable individual d…
RT @leafs_s: Behavior Research Methods July 2017 The reliability paradox: Why robust cognitive tasks do not produce reliable individual d…
RT @leafs_s: Behavior Research Methods July 2017 The reliability paradox: Why robust cognitive tasks do not produce reliable individual d…
Behavior Research Methods July 2017 The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences https://t.co/SOtEjPdfep
RT @CompBrainBeh: must-read for cognitive scientists: tasks that are very reliable across sessions by definition are poor measures of indiv…
RT @leafs_s: Behavior Research Methods July 2017 The reliability paradox: Why robust cognitive tasks do not produce reliable individual di…
RT @leafs_s: Behavior Research Methods July 2017 The reliability paradox: Why robust cognitive tasks do not produce reliable individual di…
Behavior Research Methods July 2017 The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences https://t.co/RBzLteILyz
RT @Sam_D_Parsons: @jamesheathers "The Reliability Paradox" should be essential reading for anyone looking to mix cognitive measures with i…
RT @Sam_D_Parsons: @jamesheathers "The Reliability Paradox" should be essential reading for anyone looking to mix cognitive measures with i…
@jamesheathers "The Reliability Paradox" should be essential reading for anyone looking to mix cognitive measures with individual differences. Easily one of those papers you could read multiple times and learn something new and important each time https://
RT @TraversEoin: A similar point was made a few years ago by Hedge et al. Tasks that show big, robust effects (like Stroop) usually don't v…
A similar point was made a few years ago by Hedge et al. Tasks that show big, robust effects (like Stroop) usually don't vary much across participants, meaning they're no good for individual differences research. https://t.co/L9uLkckLh9
This fits with activation being a difference score, and similar phenomena have been observed in many fields - that tasks with lower between-person variability produce less reliable estimates has been referred to as the 'reliability paradox' https://t.co/Fa
RT @bhismadev: Sobering messages on the measurement crisis from @russpoldrack at #ReadEmo20. Key papers https://t.co/ztPwTjW1Sv https://t.c…
RT @MaraMather: For fMRI researchers who, like me, had friends concerned about headline 'Duke University researchers say every brain activi…
RT @MaraMather: For fMRI researchers who, like me, had friends concerned about headline 'Duke University researchers say every brain activi…
RT @MaraMather: For fMRI researchers who, like me, had friends concerned about headline 'Duke University researchers say every brain activi…
RT @MaraMather: For fMRI researchers who, like me, had friends concerned about headline 'Duke University researchers say every brain activi…
RT @MaraMather: For fMRI researchers who, like me, had friends concerned about headline 'Duke University researchers say every brain activi…
RT @MaraMather: For fMRI researchers who, like me, had friends concerned about headline 'Duke University researchers say every brain activi…
RT @MaraMather: For fMRI researchers who, like me, had friends concerned about headline 'Duke University researchers say every brain activi…
RT @MaraMather: For fMRI researchers who, like me, had friends concerned about headline 'Duke University researchers say every brain activi…
For fMRI researchers who, like me, had friends concerned about headline 'Duke University researchers say every brain activity study you've ever read is wrong,' please explain to them that reliability at an individual level differs from task reliability htt
RT @bhismadev: Sobering messages on the measurement crisis from @russpoldrack at #ReadEmo20. Key papers https://t.co/ztPwTjW1Sv https://t.c…
RT @bhismadev: Sobering messages on the measurement crisis from @russpoldrack at #ReadEmo20. Key papers https://t.co/ztPwTjW1Sv https://t.c…
RT @bhismadev: Sobering messages on the measurement crisis from @russpoldrack at #ReadEmo20. Key papers https://t.co/ztPwTjW1Sv https://t.c…
RT @bhismadev: Sobering messages on the measurement crisis from @russpoldrack at #ReadEmo20. Key papers https://t.co/ztPwTjW1Sv https://t.c…
RT @bhismadev: Sobering messages on the measurement crisis from @russpoldrack at #ReadEmo20. Key papers https://t.co/ztPwTjW1Sv https://t.c…
RT @bhismadev: Sobering messages on the measurement crisis from @russpoldrack at #ReadEmo20. Key papers https://t.co/ztPwTjW1Sv https://t.c…
Sobering messages on the measurement crisis from @russpoldrack at #ReadEmo20. Key papers https://t.co/ztPwTjW1Sv https://t.co/GUUJviouum https://t.co/moOx1Aya13 https://t.co/7PWNm73Wcj
@tekraynak @annchenruth @PsychScience It's a good question. First I would highly recommend this paper that gets into the details even further https://t.co/YE5VN0g7Qs In short, robustness is definitely not required for reliability and, on average, may even
RT @davidaellis: @EikoFried @max12elliott Reminded me of this paper https://t.co/iSrOEujUys
@max12elliott @ESprooten @MillerLabMIT e.g. in our behavioral work we see that nearly every subject shows a positive Stroop effect, but the test-retest reliability of its magnitude is very low. see here for more: https://t.co/4TT1Ji0f1J
@EikoFried @max12elliott Reminded me of this paper https://t.co/iSrOEujUys
A 2 year-old paper, but an important lesson for anybody who, like me, tries to use cognitive tests in correlations. The reason for null-effects may be that cognitive tests are really not suited to such analyses. #researchmethods #cognitivescience #fmri htt
@micahgallen @jonroiser @DobyRahnev @nicholasdwright yes this is AUROC2 so will include variability due to task performance as well as meta. Hierarchical modeling of covariances probably helps too as in Audrey's recent paper. For context, this is interesti
@tekraynak Ah, didn't realize that. Thanks for letting me know! It's the Hedge 2019 paper -- https://t.co/X8TL6uZxHt -- which is noted in the TiCS 2020. There are caveats to it, but worthy.
@todd_gureckis @behrenstimb @wooyoungahn I would argue that it would be relevant in the context of papers like this https://t.co/pdOK8k2n8T
RT @CoyneoftheRealm: Bombshell conclusion: Implications of our findings that well-established approaches in experimental psychology and neu…
RT @IlmariMttnen: Kognitiivisten testien ongelma on mm. usein siinä, että "parhaiksi" testeiksi on valikoitunut joukko testejä, joissa yksi…
RT @CoyneoftheRealm: Bombshell conclusion: Implications of our findings that well-established approaches in experimental psychology and neu…
RT @CoyneoftheRealm: Bombshell conclusion: Implications of our findings that well-established approaches in experimental psychology and neu…
RT @Paraic_O_S: Not sure how I missed this paper (published last year) but looks like a seriously important one for anyone using experiment…
Bombshell conclusion: Implications of our findings that well-established approaches in experimental psychology and neuropsychology may not directly translate to study of individual differences in brain structure, chemistry, and function, and alternative me
Not sure how I missed this paper (published last year) but looks like a seriously important one for anyone using experimental tasks in the context of individual differences. https://t.co/UWow8Z61eO
https://t.co/ki2D54g0wF for Sam's paper! This field is a hot mess y'all and we need to fix it to move fwd. https://t.co/WYmZrdjwCy by Hedge and colleagues is also a game-changer. Well-written and easily digestible.
@chrisdc77 Yeah, Sanjay made the same point -- see this thread https://t.co/0ySxp6ZG4K
RT @Nate__Haines: 2/7 Biggest critiques include poor test-retest reliability: (1) https://t.co/HNJAcf4gn6 (2) https://t.co/SCPvYKVGP8 An…
2/7 Biggest critiques include poor test-retest reliability: (1) https://t.co/HNJAcf4gn6 (2) https://t.co/SCPvYKVGP8 And also poor criterion validity/convergence with trait measures: (1) https://t.co/XwMHuoyKeD (2) https://t.co/k9t3rfFUxT (3) https://
@ianhussey data from the awesome 'reliability paradox' paper from Hedge et al. https://t.co/veGFqjNdpL
@JimGrange Good question! I usually do not use power analyses (since I do Bayesian analyses) but I think one would need to compute disattenuated correlations for that? Some discussion here: https://t.co/rJj34nDykm
RT @Blair__Saunders: @denisohora @TonyLFreitas This paper on the "reliability paradox" gets at that: https://t.co/vQUp0L8qMO
@VFlaudias Article intéressant à lire également. En accès libre https://t.co/HxPVys44Fn
@actpredictlab @antoniahamilton @eredcay This is a good one: https://t.co/ytECIOhEmh