1. JSM 2012, San Diego1 Caution should be used in applying propensity scores estimated in a full cohort to adjust for confounding in subgroup analyses Sue M. Marcus, Columbia University Robert D. Gibbons, University of Chicago

2. JSM 2012, San Diego2 Testimony of Andrew Leon: Medication and Veteran Suicide ‘All of us here today share a common goal: to do the very best for our veterans’ ‘doing the best requires the discipline to use empirical methods to understand optimal mental health care and prevention of suicide.’

3. JSM 2012, San Diego3 Outline: Caution should be used… Context: automated propensity score analyses of large observational databases for drug safety surveillence When to use caution (Rosenbaum and Rubin 1983; Marcus and Gibbons 2012) Illustration: Do antiepileptic drugs cause suicide?

4. JSM 2012, San Diego4 Drug Safety Spontaneous reports collected through FDA’s Adverse Event Reporting System Analysis of large-scale integrated medical claims data Large potential for bias

5. JSM 2012, San Diego5 Propensity scores estimated in full cohort for subgroup? If so, one step closer to automated drug safety system for which separate analysis for each subgroup is unnecessary A correctly specified propensity score should (at least in expectation) remain valid in a subgroup population (Rosenbaum and Rubin 1983) When can this go wrong?

6. JSM 2012, San Diego6 Illustration: Do AEDs cause suicide? 1/2008 FDA alert: AEDs can increase suicidal thoughts and behaviors 7/2008 FDA scientific advisory committee: association between AEDs and suicidality American Epilepsy Society: unintended dire consequences, do not want to discontinue effective seizure medication if it does not cause suicide

7. JSM 2012, San Diego7 Causal question? AEDs given for bipolar disorder, major depression, epilepsy, pain disorders, migraines, alcohol craving, others Do AEDs cause suicide or do people with higher propensity for suicide tend to have higher propensity to take AEDs? Goal: disentangle who takes AEDs from the biological effect of the drugs

8. JSM 2012, San Diego8 Conflicting conclusions following FDA alert for two propensity–score adjusted analyses PaperGibbons et al 2009 Patorno et al 2010 PopulationBipolar DisorderBD, epilepsy, migraine, pain ComparisonAED vs no AEDEach AED vs topiramate ConclusionAEDs do not increase SA Some AEDs may have increased risk

9. JSM 2012, San Diego9 AED A (↑BP) vs AED B (↑epilepsy) Answers public health question: more suicide among those who take A vs B? Does not address whether cause of suicide is biological effect of drug or reflects who is taking drug Higher suicide rate for A reflects higher suicide rate for BP compared to epilepsy

10. JSM 2012, San Diego10 Correct specification for full vs subgroup Propensity to use drug depends on different characteristics for different disorders (eg bipolar disorder vs epilepsy) Can we correctly specify propensity for each subgroup using full cohort? Propensity to use AED vs Topiramate does not balance comparison of AED vs no treatment for particular disorder

11. JSM 2012, San Diego11 Potential Outcomes Framework r 1 = response if AED, r 0 = response if no AED Z = 1 for AED, = 0 for no AED in general, E (r 1 - r 0 ) is not equal to E (r 1 | Z = 1) – E (r 0 | Z = 0) E (r 1 - r 0 ) may be equal to E (r 1 | Z = 1, x) – E (r 0 | Z = 0, x)

12. JSM 2012, San Diego12 What is being estimated? Gibbons et al E (r 1 | Z = 1, x, BP) – E (r 0 | Z = 0, x, BP) Patorno et al E (r 1 | Z = particular AED, x, BP or epilepsy or pain) – E (r 0 | Z = Topiramate, x, BP or epilepsy or pain ) Patorno et al estimate reflects who takes each AED, rather than biologic effect of each AED

13. JSM 2012, San Diego13 Correctly specified PS? Generally more difficult to correctly specify PS for full cohort when many subgroups have different processes related to confounding by indication Those with epilepsy have different reason for choosing particular AED compared to those with BP and also have different underlying suicide rates Better to analyze each subgroup separately?

14. JSM 2012, San Diego14 Covariance adjustment on PS Known to perform poorly when PS is poorly estimated (Rosenbaum and Rubin, 1983; Marcus and Gibbons 2011) Can happen when the variance in the PS for the treatment group is smaller than for control (those who receive new treatment more homogeneous) Univariate covariance adjustment can greatly increase bias (Rubin, 1973)

15. JSM 2012, San Diego15 Conclusions Potential outcome framework can help to clarify whether what is being estimated makes sense AED vs no AED for single disorder better than AED 1 vs AED 2 for many disorders Goal is to ‘add efficiency to studies with many subgroups’ which could greatly facilitate automatic large-scale drug safety screening Is this worth the cost of increased bias: ‘stopping or refusing to start AEDs in epilepsy may result in serious harm, including death’ Fountoulakis et al 2012