layout: true --- name: xaringan-title class: inverse, left, middle .pull-left[ # .center[Bad statistics in medical research] ## .center[Adrian Barnett, QUT] ### .center[September 2022] [<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg> @aidybarnett](http://twitter.com/aidybarnett) [<svg aria-hidden="true" role="img" viewBox="0 0 496 512" style="height:1em;width:0.97em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg> @agbarnett](http://github.com/agbarnett) [<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M172.2 226.8c-14.6-2.9-28.2 8.9-28.2 23.8V301c0 10.2 7.1 18.4 16.7 22 18.2 6.8 31.3 24.4 31.3 45 0 26.5-21.5 48-48 48s-48-21.5-48-48V120c0-13.3-10.7-24-24-24H24c-13.3 0-24 10.7-24 24v248c0 89.5 82.1 160.2 175 140.7 54.4-11.4 98.3-55.4 109.7-109.7 17.4-82.9-37-157.2-112.5-172.2zM209 0c-9.2-.5-17 6.8-17 16v31.6c0 8.5 6.6 15.5 15 15.9 129.4 7 233.4 112 240.9 241.5.5 8.4 7.5 15 15.9 15h32.1c9.2 0 16.5-7.8 16-17C503.4 139.8 372.2 8.6 209 0zm.3 96c-9.3-.7-17.3 6.7-17.3 16.1v32.1c0 8.4 6.5 15.3 14.8 15.9 76.8 6.3 138 68.2 144.9 145.2.8 8.3 7.6 14.7 15.9 14.7h32.2c9.3 0 16.8-8 16.1-17.3-8.4-110.1-96.5-198.2-206.6-206.7z"/></svg> Median Watch](https://medianwatch.netlify.app) [<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg> a.barnett@qut.edu.au](mailto:a.barnett@qut.edu.au) ] .pull-right[ <!--- ---> <img src="figures/nature_cover_statistics.jpg" width="150%" style="display: block; margin: auto;" /> (Illustration by David Parkins) ] --- background-image: url(figures/AcknowledgementTraditionalOwners.jpg) background-size: cover --- <img src="figures/lancet_paper.jpg" width="63%" style="display: block; margin: auto;" /> <!--- * Incorrectly used logistic regression instead of survival analysis ---> * 813 hospitalised, 613 (75%) excluded because they were still hospitalised or not confirmed as infected -- ### Two implications: * Bad research is not restricted to "low impact" journals * Has been cited 25,000+ times: Many researchers do not recognise the serious statistical flaw --- <img src="figures/lancet_impact.jpg" width="69%" style="display: block; margin: auto;" /> <!--- https://twitter.com/TheLancet/status/1542088311702962177 ---> <!--- Emphasis on publishing in top-tier journals is a huge driver of the reproducibility crisis ---> --- <img src="figures/scrabble.jpg" width="52%" style="display: block; margin: auto;" /> From [NPR](https://www.npr.org/sections/thetwo-way/2015/07/21/424980378/winner-of-french-scrabble-title-does-not-speak-french) --- class: inverse <img src="figures/gauss.jfif" width="44%" style="display: block; margin: auto;" /> (Image from _Wikipedia_) --- # Continuous data and the normal assumption <img src="slides_files/figure-html/unnamed-chunk-7-1.png" style="display: block; margin: auto;" /> --- # Continuous data and the normal assumption <img src="slides_files/figure-html/unnamed-chunk-8-1.png" style="display: block; margin: auto;" /> --- class: center # P-values <img src="https://imgs.xkcd.com/comics/p_values.png" width="32%" style="display: block; margin: auto;" /> (From _xkcd_) --- <img src="figures/gelman_age.png" width="58%" style="display: block; margin: auto;" /> Hat-tip to [Andrew Gelman](https://statmodeling.stat.columbia.edu/) --- # Quiz question <!--- Add a menti with options ---> #### https://www.menti.com/5624oqzron #### Go to www.menti.com and use the code 3205 7080 The P value is the probability ... * that the null hypothesis is true * that the alternative hypothesis is true * that the study worked * that the results happened by chance * of obtaining the observed or more extreme results if the alternative hypothesis is true --- class:inverse ## The evidence-base is horribly biased <img src="figures/Z_plot.png" width="60%" style="display: block; margin: auto;" /> Z-values extracted from confidence intervals in _Medline_ between 1976 and 2019 --- ## Bad practice in Australia .pull-left[ In the past three years how often have you: * Not attempted to publish a valid "negative" or "neutral" study = 25% * Selected the statistical method that provided the desired result = 8% ... * ... Rises to 30% when asking about witnessing this approach in others From NHMRC [2019 Survey of research culture in Australian NHMRC-funded institutions](https://www.nhmrc.gov.au/file/15321/download?token=PivJ5dGE) ] .pull-right[ <!--- from https://www.craigmarker.com/file-drawer-problem/ --->  ] --- ## Abandon p-values and statistical significance <img src="figures/pval_journey1.jpg" width="50%" style="display: block; margin: auto;" /> --- ## Abandon p-values and statistical significance <img src="figures/pval_journey2.jpg" width="50%" style="display: block; margin: auto;" /> --- ## Abandon p-values and statistical significance <img src="figures/pval_journey3.jpg" width="50%" style="display: block; margin: auto;" /> --- ## Abandon p-values and statistical significance <img src="figures/pval_journey4.jpg" width="50%" style="display: block; margin: auto;" /> False positive probability = 9 / (9 + 12) = 43%. See [DOI: 10.1098/rsos.140216](https://royalsocietypublishing.org/doi/10.1098/rsos.140216) for details. --- class:inverse ## Abandon p-values and statistical significance .pull-left[ ## "Significance tests are popular with non-statisticians, who like to feel certainty where no certainty exists" (Yates and Healy 1964) DOI: [10.2307/2344003](https://shibbolethsp.jstor.org/start?entityID=https%3A%2F%2Fidp.qut.edu.au%2Fentity&dest=https://www.jstor.org/stable/2344003&site=jstor) ] .pull-right[ <img src="figures/Yates_Fisher_Cochran.jpg" width="60%" style="display: block; margin: auto;" /> Yates, Fisher and Cochran (Source: Barry Eagel, _Wikimedia Commons_) ] --- ## Most studies are under-powered <img src="figures/power_histogram.jpg" width="52%" style="display: block; margin: auto;" /> * Actual power estimated from 20,000 randomised controlled trials in the Cochrane database DOI: [10.1111/1740-9713.01587](https://rss.onlinelibrary.wiley.com/doi/full/10.1111/1740-9713.01587) --- <!--- registered reports ---> <img src="figures/pbio.3000246.g001.png" width="61%" style="display: block; margin: auto;" /> DOI: [https://doi.org/10.1371/journal.pbio.3000246.g001](https://journals.plos.org/plosbiology/article/figure?id=10.1371/journal.pbio.3000246.g001) --- class: inverse, center, middle  --- ## Stop developing new diagnostic or prognostic models <img src="figures/Wynants.jpg" width="47%" style="display: block; margin: auto;" /> DOI: [10.1136/bmj.m1328](https://www.bmj.com/content/369/bmj.m1328) <!--- Stop developing prediction models (hard), talk about standard text in justification, talk about stochastic variability, rule out tests might still be useful, e.g. Troponin. Appreciate that data changes. Most predictive value comes from a few variables.---> --- class: inverse <img src="figures/prediction_flow1.png" width="62%" style="display: block; margin: auto;" /> --- class: inverse <img src="figures/prediction_flow2.png" width="68%" style="display: block; margin: auto;" /> From [Maarten Van Smeden](https://twitter.com/MaartenvSmeden/status/1561276708304609281/) --- class:center, inverse, middle ## Medical research funders should no longer fund projects that plan to develop a prediction model unless a qualified statistician is a principal investigator --- class: top, center, inverse background-image: url(figures/podium.jpg) background-size: cover ### .left[Worst ever statistical methods section] -- ## .left[t-test] -- ## .right[SPSS] <!--- http://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?ACTRN=12617001415392 ---> -- ## .center[SSPS] <!--- https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=373697 ---> --- class: inverse .pull-left[  ] .pull-right[ ## "Many people think that all you need to do statistics is a computer and appropriate software." ## Doug Altman ] --- class: center # Figures <img src="https://imgs.xkcd.com/comics/scientific_paper_graph_quality.png" width="65%" style="display: block; margin: auto;" /> (From _xkcd_) --- class: center, middle  --- class: center, middle  --- class: center, middle <img src="figures/awful_plot3.jfif" width="98%" style="display: block; margin: auto;" /> --- class: center, middle <img src="figures/awful_plot4.jfif" width="88%" style="display: block; margin: auto;" /> --- ## Stop using pie charts <img src="figures/pie_charts.jfif" width="62%" style="display: block; margin: auto;" /> From https://twitter.com/MaxCRoser/status/857389434756505600 --- class:center, inverse, middle .pull-left[ # This is everybody's problem and nobody's problem ] .pull-right[ ### Grayrigg derailment  By Lawrence Clift - http://www.lawrencecliftphotography.co.uk/ ] <!--- pathetic amount spent on quality control ---> --- class: center, middle <img src="figures/nature_cover.jpg" width="90%" style="display: block; margin: auto;" /> <!--- journals too often promote publishing “mansions of straw rather than houses of brick.” Bill Kaelin ---> (Illustration by David Parkins) --- # Quality control .pull-left[ ### Randomly audit 100 papers per year from each university & research institute ### Statisticians check the analysis and see if they can reproduce the results <!--- ### Reviewers read the papers and assess originality, reproducibility and translation into practice ---> ] .pull-right[  ] DOI: [10.1371/journal.pone.0195613](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0195613) --- class:inverse, center # Statistical robots are coming  This is not AI <!--- given up trying to fix things myself ---> --- ### Yuhji Saitoh <img src="figures/saitoh_tables.png" width="75%" style="display: block; margin: auto;" /> --- class:center, middle <img src="figures/smyth.png" width="70%" style="display: block; margin: auto;" /> (from [ABC news](https://www.abc.net.au/news/2022-01-11/qld-cancer-researcher-mark-smyth-fabricated-data-review-finds/100750208)) --- <img src="figures/saitoh_simple.jpg" width="70%" style="display: block; margin: auto;" /> --- class: inverse # The continuing scandal of bad medical research .pull-left[ * 1994: Doug Altman wrote about ‘The scandal of poor medical research’ * 2015: Doug: Wish I'd said 'bad' * 2022: 1013 out of 1659 at high risk of bias. Estimated cost of bad trials was £726 million to £8 billion, DOI: [10.1186/s13063-022-06415-5](https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-022-06415-5) ] .pull-right[  ##### By Oxford Clinical Trials Research Unit ]