1. 1 Moderators of Treatment Effects in the General Medicine Literature: Looking for Improvement Nicole Bloser, MHA, MPH University of California, Davis June 5, 2007

2. 2 Background Parallel group randomized controlled trials (RCTs) are the cornerstone of evidence based medicine Parallel group randomized controlled trials (RCTs) are the cornerstone of evidence based medicine Result: Average of usually immeasurable individual effects Result: Average of usually immeasurable individual effects Some benefit, some harmed Some benefit, some harmed Heterogeneity of treatment effects Heterogeneity of treatment effects

3. 3 Background Examining treatment impact in similar individuals or subgroups Examining treatment impact in similar individuals or subgroups N-of-1 clinical trial N-of-1 clinical trial Prospective stratification with a multivariable risk index Prospective stratification with a multivariable risk index Subgroup analysis Subgroup analysis

4. 4 Background Subgroup analysis – problems Subgroup analysis – problems Low power Low power Multiple testing Multiple testing Not all subgroup analysis is the same Not all subgroup analysis is the same P-value within subgroup ≠ interaction analysis P-value within subgroup ≠ interaction analysis Interaction analysis is the correct way to examine moderators of treatment effects (MTEs) Interaction analysis is the correct way to examine moderators of treatment effects (MTEs)

5. 5 Background MTEs necessary to maximize benefit and minimize harm MTEs necessary to maximize benefit and minimize harm MTEs often not examined (<50% of RCTs report interaction analysis) MTEs often not examined (<50% of RCTs report interaction analysis) Studies that examine MTE reporting: Studies that examine MTE reporting: Majority in cardiovascular literature Majority in cardiovascular literature None since revised CONSORT statement (2001) None since revised CONSORT statement (2001)

6. 6 Research Objective We sought to identify current practice in evaluating moderators of treatment effects (MTEs) and to elucidate trends We sought to identify current practice in evaluating moderators of treatment effects (MTEs) and to elucidate trends Persistent low rate of analyses would suggest Persistent low rate of analyses would suggest Missed opportunities Missed opportunities Slower progress towards personalized medicine Slower progress towards personalized medicine

7. 7 Study Design Systematic review Systematic review Annals, BMJ, JAMA, Lancet, NEJM Annals, BMJ, JAMA, Lancet, NEJM Odd months, 1994, 1999, 2004 Odd months, 1994, 1999, 2004 Randomized controlled trials Randomized controlled trials Unit of randomization as the individual Unit of randomization as the individual All independently reviewed and coded by two investigators All independently reviewed and coded by two investigators Adjudication by a third Adjudication by a third

8. 8 Study Selection All articles from Annals, BMJ, JAMA, Lancet, and NEJM for odd months of 1994, 1999, and 2004. N = 4863 4863 articles from initial search Exclude: All articles that were not clinical trials. N=4,322 N=541, Random sample of N=379 selected IncludeExclude N=303 articlesN=76 articles 319 trials included 9 trials excluded 77 trials excluded Reasons for exclusion: Not RCT (N=61 trials) Unit of randomization not individual patient (N=25 trials)

9. 9 Methods Trials were coded as having Trials were coded as having MTE analysis (utilizing a formal test for heterogeneity) MTE analysis (utilizing a formal test for heterogeneity) Subgroup analysis only (no formal test) Subgroup analysis only (no formal test) Neither Neither Chi-square test used for bivariate comparisons Chi-square test used for bivariate comparisons Multiple logistic regression used to identify predictors of MTE analysis Multiple logistic regression used to identify predictors of MTE analysis

10. 10 Trial Characteristics

11. 11 Trial Characteristics 38% of trials had authors from North America (US and Canada) 38% of trials had authors from North America (US and Canada) 95% utilized a parallel group design 95% utilized a parallel group design Study sample size ranged from 6-41,000 Study sample size ranged from 6-41,000 (Median: 262, IQR: 101-708) (Median: 262, IQR: 101-708)

12. 12 MTE and Subgroup Reporting For those trials reporting MTE analysis: For those trials reporting MTE analysis: 43 (47%) reported on one covariate 43 (47%) reported on one covariate 24 (26%) reported on 2-4 covariates 24 (26%) reported on 2-4 covariates 17 (18%) reported on 5-10 covariates 17 (18%) reported on 5-10 covariates 7 (7%) reported on 11-19 covariates 7 (7%) reported on 11-19 covariates

13. 13 Among those trials that reported MTE, major covariates examined included: Only one trial reported MTE analysis using a composite multivariable risk index. Covariates examined for MTE

14. 14 Bivariate analysis Journals published in North America (Annals, JAMA) and first authors writing from North America were more likely to publish trials with MTE analysis Journals published in North America (Annals, JAMA) and first authors writing from North America were more likely to publish trials with MTE analysis Sample size (p<0.0001 for trend) Sample size (p<0.0001 for trend) Quintile 1: 14%, Quintile 5: 52% Quintile 1: 14%, Quintile 5: 52% More MTE analysis reported in each of the three successive time periods (p=0.047) More MTE analysis reported in each of the three successive time periods (p=0.047)

15. 15 Logistic Regression Prediction of MTE analysis Prediction of MTE analysis Included in model: study year, journal clinical condition, first author’s region, and sample size Included in model: study year, journal clinical condition, first author’s region, and sample size Journal and sample size were significant Journal and sample size were significant Reference categories: BMJ, Quintile 1 Reference categories: BMJ, Quintile 1 JAMA: 4.4 (1.4, 13.5), Annals: 4.2 (1.2, 15.1) JAMA: 4.4 (1.4, 13.5), Annals: 4.2 (1.2, 15.1) Quintile 5: 7.5 (2.9, 19.3) Quintile 5: 7.5 (2.9, 19.3)

16. 16 Limitations Limited number of trials reviewed Limited number of trials reviewed MTE results are published elsewhere MTE results are published elsewhere MTE examined, but not reported due to non- significance MTE examined, but not reported due to non- significance

17. 17 Conclusions Missed opportunities for MTE analysis abound Missed opportunities for MTE analysis abound Conservative reaction that stifles hypothesis generation Conservative reaction that stifles hypothesis generation Impairs recognition of patient strata Impairs recognition of patient strata Impedes future research Impedes future research When subgroups are reported, the reporting is not done correctly in many (~50%) cases. This could lead to erroneous conclusions. When subgroups are reported, the reporting is not done correctly in many (~50%) cases. This could lead to erroneous conclusions.

18. 18 Conclusions NIH guidelines regarding subgroup specific results recommends reporting both significant and non-significant results, yet only half of the trials reported any MTE analysis. NIH guidelines regarding subgroup specific results recommends reporting both significant and non-significant results, yet only half of the trials reported any MTE analysis. In the face of broad NIH mandates for inclusion of subjects by race/ethnicity, the low proportion of trials examining race/ethnicity as a treatment effect modifier is puzzling. In the face of broad NIH mandates for inclusion of subjects by race/ethnicity, the low proportion of trials examining race/ethnicity as a treatment effect modifier is puzzling.

19. 19 Policy Implications MTE analysis critical to future research MTE analysis critical to future research The genomic revolution is only going to increase the desire to individualize treatment effects The genomic revolution is only going to increase the desire to individualize treatment effects Kraemer et al. argue that exploratory moderator analysis is critical for designing future confirmatory studies Kraemer et al. argue that exploratory moderator analysis is critical for designing future confirmatory studies Significant exploratory effects are later used as guidance for future stratification Significant exploratory effects are later used as guidance for future stratification Kraemer, HC, E Frank, and DJ Kupfer, Moderators of treatment outcomes: clinical, research, and policy importance. Jama, 2006. 296(10): p. 1286-9.

20. 20 Policy Implications Rigorous and routine exploratory MTE analysis is necessary and should be encouraged Rigorous and routine exploratory MTE analysis is necessary and should be encouraged Standards are essential for developing practice guidelines that are appropriate to the needs of complex patients Standards are essential for developing practice guidelines that are appropriate to the needs of complex patients

21. 21 Research Team University of California, Davis Richard Kravitz, MD, MSPH - PI Elizabeth Yakes, MS University of California, Los Angeles Naihua Duan, PhD - PI Diana Liao, MPH Kiavash Nikkhou

22. 22 Thank you