RT @GSCollins: External validation studies are not immune to poor design (e.g., small sample size), methods (e.g., handling of missing data…
RT @GSCollins: External validation studies are not immune to poor design (e.g., small sample size), methods (e.g., handling of missing data…
External validation studies are not immune to poor design (e.g., small sample size), methods (e.g., handling of missing data, weak evaluation of performance) and incomplete reporting too. - https://t.co/sVatW0yAxR https://t.co/A7YpkFb8iW
@NaudetFlorian @ramspek 2/n ... Also chk out 'External validation of multivariable prediction models: a systematic review of methodological conduct and reporting' https://t.co/CZDHoZVroJ via @GSCollins et al
@MarkSendak @beenwrekt @IAmSamFin @kdpsinghlab @JFutoma @rajiinio @DukeInnovate @TPA_Debray @CarlMoons @TRIPODStatement Indeed, the issue of external validation has long been around in the stats/epi literature in the context of clinical prediction, e.g., H
@Argenscore @rahatheart1 @mmamas1973 @VictorDayan1 @pomyers @dr_benoy_n_shah @GilbertTangMD @DrMoritzWvB @PPibarot @DVervoort94 @gonzaeperez @drjohnm @djc795 @ovidiogarciav @UniofOxford Prof @GSCollins et al have been banging on this sore point for donkey'
@tunguz However, if you want go beyond covid I agree there is a lot of low quality prediction model studies (e.g., https://t.co/FTyHaQfDNm, https://t.co/CVGZ4a6kHB, https://t.co/H31DUoCmTA, https://t.co/a90fmleV8Y, https://t.co/WC9HQUGJae and many many mor
@toates_19 @DrXiaoLiu @MattFenech83 @WHO @ITU @naomiclarelee @shubsupdoc If only people would read the @TRIPODStatement (https://t.co/4REkAJO1lZ - that would be a reasonably good start on performance measures to examine 😉 - and go beyond the ROC (https://t
@kdpsinghlab @DrLukeOR @DrHughHarvey @Meddev_guy It's rarely 'done right' both model development and more importantly model validation/evaluation (https://t.co/keWV5tUDbB). I probably have more concerns with validation than I do with development.
but don't worry, it's not a problem unique to psychology/psychiatry: https://t.co/DiYOf2Zjca prediction modeling efforts everywhere are cringeworthy in terms of evaluation and reporting...
RT @GSCollins: @OAlmilaji some papers here on external validation (https://t.co/EdrovSHPw2, https://t.co/u0vaWEAnoq, https://t.co/VVhf8CqVf…
RT @GSCollins: @OAlmilaji some papers here on external validation (https://t.co/EdrovSHPw2, https://t.co/u0vaWEAnoq, https://t.co/VVhf8CqVf…
RT @GSCollins: @OAlmilaji some papers here on external validation (https://t.co/EdrovSHPw2, https://t.co/u0vaWEAnoq, https://t.co/VVhf8CqVf…
RT @GSCollins: @OAlmilaji some papers here on external validation (https://t.co/EdrovSHPw2, https://t.co/u0vaWEAnoq, https://t.co/VVhf8CqVf…
RT @GSCollins: @OAlmilaji some papers here on external validation (https://t.co/EdrovSHPw2, https://t.co/u0vaWEAnoq, https://t.co/VVhf8CqVf…
@OAlmilaji some papers here on external validation (https://t.co/EdrovSHPw2, https://t.co/u0vaWEAnoq, https://t.co/VVhf8CqVfg, https://t.co/keWV5tUDbB, https://t.co/nyn0RYFM20 https://t.co/uYW6j3iFIg, https://t.co/seU2MgmzBO, https://t.co/BgvrdmWTpd) + man
@GaelVaroquaux @danilobzdok The research into sample size considerations for evaluating prediction models in tests sets is sparse (https://t.co/Y9mZVKrcit, https://t.co/VVhf8CqVfg, https://t.co/i7g3UlMrHp, https://t.co/5iq445F5Rg) and typically overlooked
RT @GSCollins: @Richard_D_Riley @drbobphillips @Damian_Roland @riseup_sperg c-index can improve sometimes (e.g., Figure 2 in https://t.co/k…
@Richard_D_Riley @drbobphillips @Damian_Roland @riseup_sperg c-index can improve sometimes (e.g., Figure 2 in https://t.co/keWV5tUDbB), and we observed it when we validated (for example, QCancer Colorectal https://t.co/5bPuQ4oMUu). But calibration won't be
I started to write down this explicitly as a "limitation of a study", when any predictive model I fit and its c-index are involved 👇🙏 (the study I'm thinking of will make sense anyway)
RT @gscollins: 👇Important work carrying out an independent evaluation of 22 covid prediction models. The plot below (from https://t.co/keWV…
RT @gscollins: 👇Important work carrying out an independent evaluation of 22 covid prediction models. The plot below (from https://t.co/keWV…
RT @gscollins: 👇Important work carrying out an independent evaluation of 22 covid prediction models. The plot below (from https://t.co/keWV…
What's interesting is the heteroscedasticity, the better performing training models delivered... Probably followed good development guidelines as well 😄
RT @gscollins: 👇Important work carrying out an independent evaluation of 22 covid prediction models. The plot below (from https://t.co/keWV…
RT @gscollins: 👇Important work carrying out an independent evaluation of 22 covid prediction models. The plot below (from https://t.co/keWV…
RT @gscollins: 👇Important work carrying out an independent evaluation of 22 covid prediction models. The plot below (from https://t.co/keWV…
...and hopefully we'll see studies collecting multiple (different) datasets to evaluate these covid prediction models (and possibly update/recalibrate).
RT @gscollins: 👇Important work carrying out an independent evaluation of 22 covid prediction models. The plot below (from https://t.co/keWV…
👇Important work carrying out an independent evaluation of 22 covid prediction models. The plot below (from https://t.co/keWV5tUDbB) shows why such validation studies are important - as performance will nearly always better in the development data. More stu
@VickersBiostats @TRIPODStatement (1) Nah, we weren't wrong and (2) yes more people should know about it (given how widespread it is still used - to allegedly assess calibration [whenever calibration is assessed - which is still disappointingly uncommon -
@SaudenoBR @MaartenvSmeden @flavioclesio Other reviews have also concluded that ML studies are often poorly done, poorly reported and inconclusive (https://t.co/TdDcGMIzwn, https://t.co/NaI3yKYLZB), as are statistical models (https://t.co/hZTKuT51la, http
@flavioclesio @BenVanCalster @zacharylipton Paper is highlighting methodological flaws in these ML/stat comparisons, and once you account for that, claims of superiority disappears. Plenty of reviews highlight models developed using stats also poor (https:
RT @GSCollins: @MaartenvSmeden @rmachado235 @Richard_D_Riley @statsepi ...and some additional papers: https://t.co/8Eg7abThaH, https://t.co…
RT @GSCollins: @MaartenvSmeden @rmachado235 @Richard_D_Riley @statsepi ...and some additional papers: https://t.co/8Eg7abThaH, https://t.co…
RT @GSCollins: @MaartenvSmeden @rmachado235 @Richard_D_Riley @statsepi ...and some additional papers: https://t.co/8Eg7abThaH, https://t.co…
@ebhcmedstats @Richard_D_Riley @ESteyerberg @TPA_Debray @BenVanCalster The number of papers assessing calibration is tiny, (https://t.co/keWV5tUDbB) and many of those unfortunately use the HL test. We should be encouraging presentation of calibration curve
@ShalitUri @kdpsinghlab @ampanmdagaba @HerrDoktorFunk @venkmurthy @MaartenvSmeden @Richard_D_Riley @ADAlthousePhD @f2harrell @ESteyerberg @NavTangri If your test set is suitably large (https://t.co/BO63TPJbgg, https://t.co/0SvHzm8gH9). They're often not la
RT @GSCollins: @Richard_D_Riley @f2harrell @aminadibi @ESteyerberg @cecilejanssens External validation studies are often poorly done/report…
@Richard_D_Riley @f2harrell @aminadibi @ESteyerberg @cecilejanssens External validation studies are often poorly done/reported (from a review we did a few years ago https://t.co/keWV5tUDbB, that could do with updating). Many are done using convenience samp
RT @GSCollins: @Farzad_MD It's worth highlighting the non-AI/ML often also suffer methodological/reporting shortcomings (https://t.co/hZTKu…
RT @GSCollins: @Farzad_MD It's worth highlighting the non-AI/ML often also suffer methodological/reporting shortcomings (https://t.co/hZTKu…
RT @GSCollins: @IAmSamFin @georgemsavva @MaxALittle @benthebray @Farzad_MD @AledadeACO Absolutely, there are plenty of systematic reviews e…
@IAmSamFin @georgemsavva @MaxALittle @benthebray @Farzad_MD @AledadeACO Absolutely, there are plenty of systematic reviews evaluating prediction models using regression highlighting major flaws in methodology and reporting (e.g., https://t.co/hZTKuT51la, h
@Farzad_MD It's worth highlighting the non-AI/ML often also suffer methodological/reporting shortcomings (https://t.co/hZTKuT51la, https://t.co/S3m4hovf7j, https://t.co/keWV5tUDbB). Lessons are not being learn't, existing literature often being ignored. (2
@aminadibi @MaartenvSmeden Most external validations are done poorly (https://t.co/keWV5tUDbB) + they require large sample sizes (https://t.co/VVhf8CqVfg, https://t.co/aWtpmUJu5t). Don't waste data at model development, use all to build a better model and
@benthebray @jessRmorley @Dominic1King @Moorfields @uhbtrust @scrippsresearch @DeepMindAI @pearsekeane Plenty of papers on validation in the stats/epi literature (https://t.co/9N262yyTgg, https://t.co/FGUKpF4ecJ, https://t.co/PPnSgGRmqA). A good internal (
@adamkvonende Well, yes I agree to some extent, providing the external validation data is representative of your target population and not merely a convenience sample. Not all external validations are useful - many are too small and poorly done (https://t
RT @ESteyerberg: @MaartenvSmeden @mikejohansen2 Validation is important, and as @GSCollins has stated clearly, e.g. https://t.co/ZDU1mt2CQz…
RT @ESteyerberg: @MaartenvSmeden @mikejohansen2 Validation is important, and as @GSCollins has stated clearly, e.g. https://t.co/ZDU1mt2CQz…
RT @ESteyerberg: @MaartenvSmeden @mikejohansen2 Validation is important, and as @GSCollins has stated clearly, e.g. https://t.co/ZDU1mt2CQz…
@MaartenvSmeden @mikejohansen2 Validation is important, and as @GSCollins has stated clearly, e.g. https://t.co/ZDU1mt2CQz, it refers to assessment of model performance; it does not lead to claims of a model being valid. See @BenVanCalster for the utopia o
RT @GSCollins: @arjunmanrai (1/2) Good discussion. Main point was to emphasise that studies comparing ML with LR often have methodological…
@arjunmanrai (1/2) Good discussion. Main point was to emphasise that studies comparing ML with LR often have methodological flaws/poorly reported which hampers a fair comparison. Crucial to address (problem also observed in non-ML prediction literature htt
RT @GSCollins: @ParpiaSameer Cheers. #ML per se is not the concern but rather the poor way they are often implemented, evaluated and report…
@ParpiaSameer Cheers. #ML per se is not the concern but rather the poor way they are often implemented, evaluated and reported (similar concerns for regression models https://t.co/PPnSgGRmqA) along with the unfair comparisons. Spin and overinterpretation t
@EricTopol @NatureMedicine Validation of non-AI prediction (from established prediction field) are often fraught with methodological shortcomings (https://t.co/PPnSgGRmqA). AI field should learn from these (not ignore). Standards / guidance needed (https:/
RT @GSCollins: @Richard_D_Riley @dennislendrem Validation studies often carried using existing data that are convenient (often too small),…
RT @GSCollins: @Richard_D_Riley @dennislendrem Validation studies often carried using existing data that are convenient (often too small),…
RT @GSCollins: @Richard_D_Riley @dennislendrem Validation studies often carried using existing data that are convenient (often too small),…
RT @GSCollins: @Richard_D_Riley @dennislendrem Validation studies often carried using existing data that are convenient (often too small),…
RT @GSCollins: @Richard_D_Riley @dennislendrem Validation studies often carried using existing data that are convenient (often too small),…
RT @GSCollins: @Richard_D_Riley @dennislendrem Validation studies often carried using existing data that are convenient (often too small),…
RT @GSCollins: @Richard_D_Riley @dennislendrem Validation studies often carried using existing data that are convenient (often too small),…
RT @GSCollins: @Richard_D_Riley @dennislendrem Validation studies often carried using existing data that are convenient (often too small),…
@Richard_D_Riley @dennislendrem Validation studies often carried using existing data that are convenient (often too small), not reflective of the target population and poorly conducted (https://t.co/PPnSgGRmqA). A good internal validation is often as good
RT @drbobphillips: An academic version of 'Why you shouldn't believe anyone who pretends to have a crystal ball' here: https://t.co/xg3D26d…
External validation of multivariable prediction models: a systematic review of methodological... https://t.co/9cPKfPBUOm #bmcmedresmethodol
RT @drbobphillips: An academic version of 'Why you shouldn't believe anyone who pretends to have a crystal ball' here: https://t.co/xg3D26d…
An academic version of 'Why you shouldn't believe anyone who pretends to have a crystal ball' here: https://t.co/xg3D26dXRN
RT @GSCollins: @drbobphillips More details on some problems with published validation studies here….https://t.co/nneomKr8Pa
From the harrower of self important self validators ... proper link: https://t.co/xAE8dNlBUh
@drbobphillips More details on some problems with published validation studies here….https://t.co/nneomKr8Pa
And those that are validated are poorly done and poorly reported https://t.co/ZqbIAjAqeS https://t.co/1fAh1ohrIr
Improvements needed in conduct and reporting of clinical risk prediction models http://t.co/CWFoNeTxET
SR of studies that carried out ext validation of prediction models: http://t.co/as2RNQe0st - most studies poorly reported + poorly designed.
Review finds cornerstone of translational research poorly done; almost never tested against physician judgement. http://t.co/UZEBiT15hp
Reporting external validation of multivariable prediction models http://t.co/qRkdEUQjcJ #bmcmedicalresearchmethodology
External validation of multivariable prediction models: systematic review of methodological conduct & reporting, http://t.co/99ra9WDLfC
External validation multivariable prediction model syst rev methods conduct &reporting http://t.co/5auBLMqDBq #bmcmedicalresearchmethodology
Room for improvement in conduct and reporting of clinical prediction models. http://t.co/DflV1hFeZP Good tips for best practice outlined.
Room for improvement in conduct and reporting of clinical prediction models. http://t.co/DflV1hFeZP Good tips for best practice outlined.
External validation of prediction models: SR of conduct and reporting http://t.co/YqxyLmNG3N #bmcmedicalresearchmethodology #TRIPODstatment