Presentation is loading. Please wait.

Presentation is loading. Please wait.

Non-replicating comments on replication Steven Goodman, MD, PhD Johns Hopkins University SAMSI Workshop July 10-12, 2006.

Similar presentations


Presentation on theme: "Non-replicating comments on replication Steven Goodman, MD, PhD Johns Hopkins University SAMSI Workshop July 10-12, 2006."— Presentation transcript:

1 Non-replicating comments on replication Steven Goodman, MD, PhD Johns Hopkins University SAMSI Workshop July 10-12, 2006

2 Things identified as cancer risks (SImon and Altman, JNCI, 1994) Electric Razors Broken Arms (in women) Fluorescent lights Allergies Breeding reindeer Being a waiter Owning a pet bird Being short Being tall Hot dogs Having a refrigerator!!

3 Outline Glaring examples P-value/replication misconceptions Ioannidis methods/conclusions Evidence of selective reporting Reproducible research

4 We have no idea how or why the magnets work. A real breakthrough… …the [study] must be regarded as preliminary…. But…the early results were clear and... the treatment ought to be put to use immediately.

5

6 FDA Discussion (Fisher, CCT, 20:16-39,1999) L. Moyé, MD, PhD What we have to wrestle with is how to interpret p-values for secondary endpoints in a trial which frankly was negative for the primary. …In a trial with a positive endpoint…you havent spent all of the alpha on that primary endpoint, and so you have some alpha to spend on secondary endpoints….In a trial with a negative finding for the primary endpoint, you have no more alpha to spend for the secondary endpoints.

7 FDA Discussion, cont. (Fisher, CCT, 20:16-39,1999) Dr. Lipicky: What are the p-values needed for the secondary endpoints? …Certainly were not talking 0.05 anymore. …Youre out of this 0.05 stuff and I would have like to have seen what you thought was significant and at what level… What p-value tells you that its there study after study? Dr. Konstam: …what kind of statistical correction would you have to do that survival data given the fact that its not a specified endpoint? I have no idea how to do that from a mathematical viewpoint.

8 Replication probability, as a function of the p-value Goodman, SN, A Comment on Replication, P-values and Evidence, Stat Med, 11: , 1992.

9 Why the replication probability is low

10 What do we mean by replication? Statistical significance? Same results/concslusions from same original data? Same results/conclusions from same analytic data? Same R/C in ostensibly identical study? Same R/C in similar but non-identical study? Surrogate for whether underlying hypothesis is true? Is combinability/heterogeneity a more profitable concept to explore?

11 Reasons for non-replication Hypothesis not true. {Prior / Posterior probability} Misrepresented evidence. {Improper/selective analysis, improper/selective reporting} Different balance of unmeasured covariates across studies/designs {Quality of design, reliability of mechanistic knowledge} Different handling/measurement of measured covariates across studies/designs. {Combinability / heterogeneity} Fundamentally different question asked, i.e. new study is not a replicate of previous one. {Combinability / mechanistic knowledge}

12

13

14 JAMA, 2005

15

16 Ioannidis findings 45 original articles claiming effectiveness w/ > 1000 citations in NEJM, Lancet, JAMA, (16%) subsequently contradicted 7 (16%) exaggerated effects 20 (44%) replicated 11 (24%) unchallenged 5/6 nonrandomized studies contradicted + exag. vs. 9/39 RCTs.

17 Unit of analysis? StudyConditionAgent NHS CAD preventionEstrogen / Progestin NHS CAD preventionVit. E (women) HPFS CAD preventionVit. E (men) Zutphen CAD preventionFlavonoids Case series LeukemiaTrans retinoic acid Case series Resp. distressNitric Oxide

18

19

20 Effect of bias on Bayes factor As -->0, the LR -->

21 LR (bias, ) p0.05p-value=0 Bias Power =80% 90%80%90%

22

23

24 , Including this one

25 DIfference for which we have 80% power will have a Z= = 2.8, p=0.005.

26 Max LR (bias, ) POWER BIAS80%90%

27

28

29 Our findings are likely underestimates due to underreporting of omitted outcomes by trialists, with 86% of survey responders initially denying the existence of unreported outcomes despite clear evidence to the contrary. This surprisingly high percentage suggests that contacting trialists for information about unreported outcomes is unreliable, even despite our simply worded questionnaire. We also reviewed all primary and secondary published articles for a trial; if only the primary article had been reviewed, more trial outcomes would have been classified as unreported.

30 Abstract conclusions Design Cohort study using protocols and published reports of randomized trials approved by the Scientific-Ethical Committees for Copenhagen and Frederiksberg, Denmark, in The number and characteristics of reported and unreported trial outcomes were recorded from protocols, journal articles, and a survey of trialists…. Results One hundred two trials with 122 published journal articles and 3736 outcomes were identified. Overall, 50% of efficacy and 65% of harm outcomes per trial were incompletely reported. Statistically significant outcomes had a higher odds of being fully reported compared with nonsignificant outcomes for both efficacy (pooled odds ratio, 2.4; 95% confidence interval [CI], ) and harm (pooled odds ratio, 4.7; 95% CI, ) data. In comparing published articles with protocols, 62% of trials had at least 1 primary outcome that was changed, introduced, or omitted. Eighty-six percent of survey responders (42/49) denied the existence of unreported outcomes despite clear evidence to the contrary. Conclusions The reporting of trial outcomes is not only frequently incomplete but also biased and inconsistent with protocols. Published articles, as well as reviews that incorporate them, may therefore be unreliable and overestimate the benefits of an intervention. To ensure transparency, planned trials should be registered and protocols should be made publicly

31 Reproducible Research Roger Peng, F. Dominici, S. Zeger AJE, 2006

32

33 A Research Pipeline

34 What is Reproducible Research? Data: Analytic dataset is available Methods: Computer code underlying figures, tables, and other principal results is available Documentation: Adequate documentation of the code, software environment, and data is available Distribution: Standard methods of distribution are employed for others to access materials

35

36

37 A Research Pipeline (reprise)

38 Jon Claerbout on Reproducible Research Our experience shows that it is only slightly more difficult to give birth to a living document than a dead one…. Authors who wish to communicate will produce live documents. Authors who merely wish to advertise their scholarship without really sharing it will continue to produce dead ones.

39 Example of Live Documents: Sweave, LaTeX, and R A system for mingling statistical code with text Figures, tables, and other numerical results are generated on-the-fly Code for figures, tables, results can be extracted separately Knowledge of LaTeX and R required; complex build environment needed Basic principles not wedded to any particular statistical language/document prep. system

40

41

42

43 A Licensing Spectrum for Data Full access: Data can be used for any purpose Attribution: Data can be used for any purpose so long as a specific citation is used Share-alike: Data can be used to produce new findings --- any modifications/linkages must be made available under the same terms Reproduction: Data can only be used for reproducing results and commenting on those results via a letter to the editor (No Data Available)

44 Issues to Consider Making datasets available What is code? Does it exist? Separating content from presentation Technical sophistication of authors, publishers, readers; requirements? Protecting authors original ideas Logistics – data storage, accessibility

45 RR Options considered at medical journal Assign and advertise RR Score depending on how much info author makes available. Do we ask everyone? Do we penalize those who dont/cant share data? How do we prioritize between components of the score? Do we treat differently sophisticated and unsophisticated analysts? Divulge data sharing policy of author, including code- sharing, like roles on manuscript and conflict of interest.

46 What do we mean by replication? Statistical significance? Same results/concslusions from same original data? Same results/conclusions from same analytic data? Same R/C in ostensibly identical study? Same R/C in similar but non-identical study? Surrogate for whether underlying hypothesis is true? Is combinability/heterogeneity a more profitable concept to explore?

47 Reasons for non-replication Hypothesis not true. {Prior / Posterior probability} Misrepresented evidence. {Improper/selective analysis, selective reporting} Different balance of unmeasured covariates across studies/designs {Quality of design, reliability of mechanistic knowledge} Different handling/measurement of measured covariates across studies/designs. {Combinability / heterogeneity} Fundamentally different question asked, i.e. new study is not a replicate of previous one. {Combinability / mechanistic knowledge}


Download ppt "Non-replicating comments on replication Steven Goodman, MD, PhD Johns Hopkins University SAMSI Workshop July 10-12, 2006."

Similar presentations


Ads by Google