1. Sharon@hcp.med.harvard.edu (ARM 2004) 1 INNOVATIVE STATISTICAL APPROACHES IN HSR: BAYESIAN, MULTIPLE INFORMANTS, & PROPENSITY SCORES Thomas R. Belin, UCLA Nicholas J. Horton, Smith College Sharon-Lise T. Normand, Harvard University

2. Sharon@hcp.med.harvard.edu (ARM 2004) 2 A REVIEW OF CONCEPTS & APPROACHES FOR CAUSAL INFERENCE: PROPENSITY SCORES Sharon-Lise T. Normand (sharon@hcp.med.harvard.edu) Harvard Medical School & Harvard School of Public Health Thanks: Laura A. Petersen

3. Sharon@hcp.med.harvard.edu (ARM 2004) 3 REGIONALIZING CARDIAC SERVICES Under FFS, incentive for hospitals to duplicate profitable cardiac service capabilities. Under non-FFS, incentive to regionalize costly services to referral institutions. Non-FFS systems provide lower rates of cardiac procedures than FFS systems. Availability of on-site cardiac procedures affects the use of such procedures.

4. Sharon@hcp.med.harvard.edu (ARM 2004) 4 Should we regionalize invasive cardiac services? Examine use of clinically needed angiography between AMI patients treated within a regionalized system (VA) and those treated within a non- regionalized system (FFS). (Petersen, Normand, Leape, McNeil, NEJM: 2002)

5. Sharon@hcp.med.harvard.edu (ARM 2004) 5 OUTLINE OF TALK What is a causal effect? –Role of Randomization. Statistical Adjustments. –Regression –Stratification –Both Concluding remarks.

6. Sharon@hcp.med.harvard.edu (ARM 2004) 6 WHAT IS A CAUSAL EFFECT? Let Y i denote the “outcome” for the i th patient. The causal effect is:  i = Y i(treatment) – Y i(control) Receipt of Angiography if treated in a regionalized system (VA) Receipt of Angiography if treated in a non- regionalized system (FFS) Outcomes under treatments not assigned are missing

7. Sharon@hcp.med.harvard.edu (ARM 2004) 7 ROLE OF RANDOMIZATION Properties of an RCT: 1.Experimenter determines the assignment of treatments to patients using a known mechanism. 2.Every participant has a non-zero “chance” of being assigned the treatment. 3.Outcome and treatment assignment are independent given the covariates.

8. Sharon@hcp.med.harvard.edu (ARM 2004) 8 ROLE OF RANDOMIZATION So what? Randomization (treatment assignment mechanism) permits: Treatment Effect = Y for those assigned to treatment - Y for those assigned to control  i = Y i(treatment) – Y i(control)

9. Sharon@hcp.med.harvard.edu (ARM 2004) 9 OBSERVATIONAL STUDY Properties of an Observational Study: 1.Investigator has no control over the assignment of treatments to patients so that the mechanism is unknown. 2.No longer true that every participant has a non-zero “chance” of being assigned the treatment. 3.Outcome and treatment assignment may be dependent given the covariates.

10. Sharon@hcp.med.harvard.edu (ARM 2004) 10 TYPES OF BIASES 1.Non-representative sample of target population. 2.Treatment assignment is non-ignorable, can not assume treatment assignment depends only on observed covariates. 3.All Measured Confounders: Inexact matching on basis of covariates. Incomplete matches (discarding treated).

11. Sharon@hcp.med.harvard.edu (ARM 2004) 11 TYPES OF BIASES RCT has random assignment of treatments but not random selection of subjects from the target population. Observational study has random sampling of target population but not random assignment of treatments.

12. Sharon@hcp.med.harvard.edu (ARM 2004) 12 ANALYTIC ADJUSTMENTS 1.Model-Based (Statistical) Adjustment: regress outcome on the confounders and estimate adjusted outcomes. 2.Matched Sampling: create categories from the confounding variables (X) such that within each category, there are both treated and control. 3.Adjust & Stratify: Replace confounders by a single summary, e(X), and construct strata/matches/weight. Estimate treatment differences within each stratum/pair etc.

13. Sharon@hcp.med.harvard.edu (ARM 2004) 13 CHARACTERISTICS OF ACC/AHA CLASS I HOSPITALIZED AMI PATIENTS Characteristic VA (N = 1104) FFS (N = 10464) Stand. Diff. H x Hypertension67%59%18% Black13%4%35% Prior MI38%35%7% Asthma/COPD31%23%17% ST-Elevation49%43%10% Ischemia29%12%44% Angiography44%51%

14. Sharon@hcp.med.harvard.edu (ARM 2004) 14 SO THEY ARE DIFFERENT! (USUAL) REGRESSION logit(P(Y=1|X)) =  0 +  (VA) +  T X Performs poorly when the variances of the confounders in the treated and control groups are unequal. Unequal variances of confounders are not uncommon in observational studies..

15. Sharon@hcp.med.harvard.edu (ARM 2004) 15 REGRESSION Difficult to assess whether treated and control groups overlap enough on X to allow sensible estimation of an effect. If treatment groups have different covariate distributions, model-based adjustments depend on the form of model. But, dist n of X may differ for treated and control  must impose linearity and extrapolate over different regions of covariates.

16. Sharon@hcp.med.harvard.edu (ARM 2004) 16 HOW TO MAKE VALID CASUAL INFERENCE? Infer the assignment mechanism. Estimate: e(X) = P(VA = 1| X) Ensure everyone has a chance of being assigned the treatment. 0 < e(X) < 1 Guarantee we are comparing similar patients. Want comparable e(X) between treatment groups. Estimate effect of treatment (VA) on Y (angiography). E(Y| e(X),VA = 1) – E(Y| e(X),VA = 0)

17. Sharon@hcp.med.harvard.edu (ARM 2004) 17 PROPENSITY SCORE: TREATMENT ASSIGNMENT MECHANISM P(VA = 1 | X) = e(X) = demographics (age, race) + severity on entry (ST-Elevation, chestpain duration, etc) + comorbidity (H x of AMI, CHF, Diabetes, etc).

18. 18 CLASS I VA PATIENTS (N = 1104) CLASS I FFS PATIENTS (N = 10464) Estimated Propensity Score: e(X) = P(VA) TREATMENT ASSIGNMENT MECHANISM

19. Sharon@hcp.med.harvard.edu (ARM 2004) 19 REGRESS & STRATIFY (OR REGRESS & MATCH) Stratify (or match) patients based on the estimated propensity score. Within each stratum, distribution of observed confounders differs only randomly between treated and control subjects. Regardless: matching towards a population who looks like those who received the treatment.

20. Sharon@hcp.med.harvard.edu (ARM 2004) 20 1090 MATCHED PAIRS CLASS I VA (N = 1090) Estimated Propensity Score: P(VA = 1) CLASS I FFS (N = 1090) Estimated Propensity Score: P(VA = 1)

21. Sharon@hcp.med.harvard.edu (ARM 2004) 21 CHARACTERISTICS OF ACC/AHA CLASS I POST MATCHING (1090 PAIRS) CharacteristicVAFFS Stand. Diff. H x Hypertension67%66%2% Black12% 1% Prior MI38%42%-7% Asthma/COPD30% 1% ST-Elevation49%47%4% Ischemia27%26%2% ROC Area = 0.80

22. Sharon@hcp.med.harvard.edu (ARM 2004) 22 ADJUSTED COMPARISON USING 1090 MATCHED PAIRS OF CLASS I PATIENTS Adjusted for Patient Characteristics Only No. (%) of Discordant Pairs No. (%) of Discordant Pairs in which VA Received Angiography P-Value 553 (51)242 (44)0.0038 Not adjusted for hospital clustering. OR from regression adjustment [95% CI] using all patients: 0.77 [0.66, 0.88].

23. Sharon@hcp.med.harvard.edu (ARM 2004) 23 ADJUSTED COMPARISON USING 1085 MATCHED PAIRS OF CLASS I PATIENTS: Adjusted for Patient Characteristics and Hospital Availability of Angiography No. (%) of Discordant Pairs No. (%) of Discordant Pairs in which VA Received Angiography P-Value 505 (47)255 (50)0.86 Not adjusted for hospital clustering. OR from regression adjustment [95% CI] using all patients: 1.04 [0.89, 1.21].

24. Sharon@hcp.med.harvard.edu (ARM 2004) 24 CONCLUSIONS FROM EXAMPLE Under-use of angiography regardless of system of care. Under-use is higher within the regionalized system [VA]. Differences in under-use between VA and FFS associated with on-site availability of cardiac procedure technology.

25. Sharon@hcp.med.harvard.edu (ARM 2004) 25 CONCLUDING REMARKS Role for propensity score approaches in HSR. Provides a principled and practical approach for assessing appropriateness and robustness of assumptions. No Panacea – doesn’t overcome hidden bias problems. Should you always use?