1. 1 Eloise E. Kaizar The Ohio State University Combining Information From Randomized and Observational Data: A Simulation Study June 5, 2008 Joel Greenhouse Howard Seltman Carnegie Mellon University TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAA

2. 2 Outline Motivating Example Motivating Example –Association between suicidality and antidepressant use in pediatric population Trying to answer the right question Trying to answer the right question Exploiting strengths of different data Exploiting strengths of different data Simulation Study Simulation Study

3. 3 Pediatric Antidepressant Use Problem: Antidepressant use may cause suicide for some children/adolescents Problem: Antidepressant use may cause suicide for some children/adolescents Goal: Estimate the average treatment effect for use in regulatory decision making Goal: Estimate the average treatment effect for use in regulatory decision making

4. 4 Randomized Controlled Trials Hammad, et al. (2006) Archives of General Psychiatry Hammad, et al. (2006) Archives of General Psychiatry

5. 5 The Right Question Study population average treatment effect Study population average treatment effect Population average treatment effect Population average treatment effect

6. 6 Heterogeneity Variation due to differences in population (True) Variation due to differences in population (True) Variation due to differences in study design (Artifactual) Variation due to differences in study design (Artifactual)

7. 7 Evidence for Weak External Validity Administrative data Administrative data –Show no significant association between antidepressant use and suicidal actions (Valuck et al. 2004, Jick et al. 2004) (Valuck et al. 2004, Jick et al. 2004) Epidemiological data Epidemiological data –Suggest inverse relationship between antidepressant use and completed suicide Geographically (Gibbons et al. 2006, Isacsson 2000, Ludwig and Marcotte 2005) Geographically (Gibbons et al. 2006, Isacsson 2000, Ludwig and Marcotte 2005) Temporally (Gibbons et al. 2007, Olfson, et al 1998) Temporally (Gibbons et al. 2007, Olfson, et al 1998)

8. 8 Assessing External Validity Compare the RCT patients with a nationally representative probability sample of adolescents Compare the RCT patients with a nationally representative probability sample of adolescents –Youth Risk Behavior Survey (YRBS) –Representative of adolescents attending school (aged 12-18) –Basic demographic information –Self-report depression –Self-report suicidality

9. 9 Match RCTs and YRBS Consider only MDD RCTs of ages 12-18 Consider only MDD RCTs of ages 12-18 Consider only YRBS respondents reporting depression Consider only YRBS respondents reporting depression YRBSRCTs Average Age 16.14 (0.04) 14.76 (2.99) % Female 62.6 (1.8) 63.8 (1.4) % White 54.1 (3.5) 80.1 (1.2) Poststratify YRBS to match RCTs Poststratify YRBS to match RCTs

10. 10 Compare Outcomes 8-week suicidality 8-week suicidality –RCTs 3.6% –YRBS 7.1% Suicide attempt Suicide attempt –RCTs 5.4% (lifetime) –YRBS 19.9% (12-month)

11. 11 Randomized Controlled Trials Hammad, et al. (2006) Archives of General Psychiatry Hammad, et al. (2006) Archives of General Psychiatry

12. 12 Generalizing RCT Data Low Risk High Risk Reduce the size of the excluded population Reduce the size of the excluded population –Practical Clinical Trial Estimate the effect size in the excluded population Estimate the effect size in the excluded population

13. 13 Current Approaches to Estimating Average Effect Size Use meta-analysis to combine RCT data Use meta-analysis to combine RCT data –Assume effect is not systematically heterogeneous by exclusion criteria Use multi-level meta-analysis to combine RCT and observational data Use multi-level meta-analysis to combine RCT and observational data –Partial exchangeability –Assumes the mean is of interest Include bias parameters Include bias parameters

14. 14 Proposed Approaches Confidence Profile Method [Eddy, et al., 1988, 1989] Confidence Profile Method [Eddy, et al., 1988, 1989] –Model the biases in observational and RCT data Response Surface Approach [Rubin, 1990, 1991] Response Surface Approach [Rubin, 1990, 1991] –Create a response surface that incorporates design variables –Extrapolate to the ideal design Cross Design Synthesis [US GAO, 1992] Cross Design Synthesis [US GAO, 1992] –Stratify data based on design variables –Extrapolate to empty cells

15. 15 Usefulness of Evidence External Validity Internal Validity RCT Obs. Ideal Stronger Weaker Stronger Weaker

16. 16 Framework Self-Selection Variable Randomized (Strong Internal Validity) Self-Selected (Weak Internal Validity) Generalizability Variable Eligible for Randomization Ineligible for Randomization

17. 17 Framework Self-Selection Variable Randomized (Strong Internal Validity) Self-Selected (Weak Internal Validity) Generalizability Variable Eligible for Randomization Ineligible for Randomization

18. 18 Self-Selection Variable Randomized (Strong Internal Validity) Self-Selected (Weak Internal Validity) Generalizability Variable Eligible for Randomization Ineligible for Randomization Framework Randomized Experiments Observational Studies

19. 19 Self-Selection Variable Randomized (Strong Internal Validity) Self-Selected (Weak Internal Validity) Generalizability Variable Eligible for Randomization Ineligible for Randomization Linear Bias Model

20. 20 Simulation Study Goal: Goal: –Use simulation study to investigate effectiveness of different methods for combining information from diverse sources in a realistic setting Key characteristics: Key characteristics: –24 high-quality experiments with complete compliance and uniform randomization eligibility 200 subjects, individual data unavailable 200 subjects, individual data unavailable –1 high-quality observational study with no generalizability bias 25,000 subjects, individual data available 25,000 subjects, individual data available

21. 21 Simulation Study: Implementation Generate 1000 data sets Generate 1000 data sets Fit models using Bayesian approach Fit models using Bayesian approach Compare on MSE, bias and coverage Compare on MSE, bias and coverage Scenario01234 Effect Size 0.80.70.70.80.7 Generalizability Bias 00.40.400.4 Self-Selection Bias 00.400.45-0.4

22. 22

23. 23

24. 24

25. 25

26. 26

27. 27 Self-Selection Variable Randomized (Strong Internal Validity) Self-Selected (Weak Internal Validity) Generalizability Variable Eligible for Randomization Ineligible for Randomization Linear Bias Model

28. 28 Summary Even RCTs that have no heterogeneity may not be estimating the effect of interest. Even RCTs that have no heterogeneity may not be estimating the effect of interest. Observational data may be used to assess the extent of the generalizability problem Observational data may be used to assess the extent of the generalizability problem The Cross Design Synthesis approach can potentially be effective for estimating average effect size The Cross Design Synthesis approach can potentially be effective for estimating average effect size Still at the beginning of this work Still at the beginning of this work –More fair comparisons –Extend to real settings