1. The Application of Propensity Score Analysis to Non-randomized Medical Device Clinical Studies: A Regulatory Perspective
Lilly Yue, Ph.D.*
CDRH, FDA, Rockville MD 20850
*No official support or endorsement by the Food and Drug Administration of this presentation is intended or should be inferred.

2. Outline Randomized clinical trials
Non-randomized studies and a potential problem
Propensity scores methods for bias reduction
Practical issues with the application of propensity score methodology
Limitations of propensity score methods
Conclusions

3. Randomized Trials
All patients have a specified chance of receiving each treatment.
Treatments are concurrent.
Data collection is concurrent, uniform, and high quality.
Expect that all patient covariates, measured or unmeasured, e.g., age, gender, duration of disease, …, are balanced between the two treatment groups.

4. Randomized Trials
Assumptions underlying statistical comparison tests are met.
So, the two trt groups are comparable and observed treatment difference is an unbiased estimate of true treatment difference.
But, the above advantages are not guaranteed for small, poorly designed or poorly conducted randomized trials.

5. Nonrandomized Studies and a Potential Problem
None of advantages provided by randomized trials is available in non-randomized studies.
A potential problem:
Two treatment groups were not comparable before the start of treatment.
i.e., not comparable due to imbalanced covariates between two treatment groups.
So, direct treatment comparisons are invalid.

6. Adjustments for Covariates
Three common methods of adjusting for confounding covariates:
Matching
Subclassification (stratification)
Regression (Covariate) adjustment

7. Each covariate: 2 categories 5 covariates: 32 subclasses
Question: When there are many confounding covariates needed to adjust for, e.g., age, gender, …
Matching based on many covariates is not practical.
Subclassification is difficulty: As the number of covariates increases, the number of subclasses grows exponentially:
Each covariate: 2 categories 5 covariates: 32 subclasses
Regression adjustment may not be possible: Potential problem: over-fitting

8. Propensity Score Methodology
Replace the collection of confounding covariates with one scalar function of these covariates: the propensity score.
Age
Gender
Duration
…….
1 composite covariate:
Propensity Score
Balancing score

9. Propensity Score Methodology (cont.)
Propensity score (PS): conditional prob. of receiving Trt A rather than Trt B, given a collection of observed covariates.
Purpose: simultaneously balance many covariates in the two trt groups and thus reduce the bias.

10. Propensity scores construction
Statistical modeling of relationship between treatment membership and covariates
Statistical methods: multiple logistic regression or others
Outcome: event -- actual trt membership: A or B
Predictor variables: all measured covariates, some interaction terms or squared terms, e.g.,
age, gender, duration of disease,…, age*duration,…

11. Propensity scores construction
Clinical outcome variable, e.g., major complication event, is NOT involved in the modeling
No concern of over-fitting
Obtain a propensity score model: a math equation
PS = f (age, gender, …)
Calculate estimated propensity scores for all patients

12. Properties of propensity scores
A group of patients with the same propensity score are equally likely to have been assigned to trt A.
Within a group of patients with the same propensity score, e.g., 0.7, some patients actually got trt A and some got trt B, just as they had been randomly allocated to whichever trt they actually received.

13. “Randomized After the Fact”
PS=0.7
Trt A
Trt B

14. When the propensity scores are balanced across two treatment groups, the distribution of all the covariates are balanced in expectation across the two groups.
Use the propensity scores as a diagnostic tool to measure treatment group comparability.
If the two treatment groups overlap well enough in terms of the propensity scores, we compare the two treatment groups adjusting for the PS.

15. Compare treatments adjusting for propensity score
Matching
Subclassification (stratification)
Regression (Covariate) adjustment

16. PS Trt A vs. Trt B Compare treatments based on matched pairs
Matching based on propensity scores (PS)
PS Trt A vs. Trt B
Compare treatments based on matched pairs
Problem: may exclude unmatched patients
PS1
PS2
PSm

17. Stratification PS All patients are sorted by propensity scores.
Divide into equal-sized subclasses.
Compare two trts within each subclass, as in a randomized trial; then estimate overall trt effect as weighted average.
It is intended to use all patients.
But, if trial size is small, some subclass may contain patients from only one treatment group. PS 1 2
……. 5

18. Regression (covariate) adjustment
Treatment effect estimation model fitting:
the relationship of clinical outcome and treatment
Outcome: Clinical outcome, e.g., adverse events
Predictor variables: trt received, propensity score, a
subset of important covariates
Statistical method: e.g., regression or logistical
regression

19. Propensity Score Methods
Summary
Fit propensity score (PS) model
using all measured covariates
Estimate PS for all patients
using PS model
Compare treatments
adjusting for propensity scores

20. Practical Issues Issues in propensity score estimation
How to handle missing baseline covariate values
What terms of covariates should be included
Evaluation of treatment group comparability
Assessment of the resulting balance of the distributions of covariates
Issues in treatment comparison:
Which method: matching, stratification, regression
Issues in study design with PS analysis
Pre-specified vs. post hoc PS analysis
Pre-specify the covariates needed to collect in the study and then included in PS estimation
Sample size estimation adjusting for the propensity scores

21. Example – Device A Non-concurrent, two-arm, multi-center study
Control: Medical treatment without device,
N=65, hospital record collection
Treatment: Device A, N = 130
Primary effectiveness endpoint: Treatment success
Hypothesis testing: superiority in success rate
20 imbalanced clinically important baseline covariates, e.g., prior cardiac surgery
22% patients with missing baseline covariate values

22. Enrollment Time

23. Two treatment groups are not comparable
Imbalance in multiple baseline covariates
Imbalance in the time of enrollment
So, any direct treatment comparisons on the effectiveness endpoint are inappropriate.
And, p-values from direct treatment comparisons are un-interpretable.
What about treatment comparisons adjusting for the imbalanced covariates?
Traditional covariate analysis
Propensity score analysis

24. Performed propensity score (PS) analysis Handed missing values
MI: generate multiple data sets for PS analysis
Generate one data set: generalized PS analysis
Others
Included all statistically significant and/or clinically important baseline covariates in PS modeling.
Checked comparability of two trt groups through estimated propensity score distributions.
Found that the two trt groups did not overlap well.

25. Estimated Propensity Scores (with time)

26. Estimated Propensity Scores (w/o time)

27. Patients in Propensity Score Quintile
Total
Ctl
(w/time) % 28% 12% % %
Trt
1% % % 29% %
Ctl
(w/o time) % 37% % % %
Trt
8% % % % %

28. Treatment Success
Total
Crl S N
Trt S N
Tried Cochran-Mantel-Haenszel test controlling for PS quintile, Logistic regression using PS as a continuous covariate
However, the sig. p-values are un-interpretable

29. Conclusion:
The two treatment groups did not overlap enough to allow a sensible treatment comparison.
So, any treatment comparisons adjusting for imbalanced covariates are problematic.

30. Example: Device B New vs. control in a non-randomized study
Primary endpoint: MACE incidence rate at 6-month after treatment
Non-inferiority margin: 7%, in this study
Sample size: new: 290, control: 560
14 covariates were considered.

31. propensity score stratification adjustment
Covariate balance checking before and after
propensity score stratification adjustment
Mean p-value
New Control Before After
Mi <
Diab
CCS
Lesleng <
Preref
Presten <

32. Model Building
The PS is conditional Prob. that a patient would have been assigned to new device, based on his or her baseline covariates.
A hierarchical logistic regression model with a stepwise selection process was used to build the propensity score model.
The final propensity score model includes all covariates as well as a quadratic term.

33. Table 2. Distribution of patients at five strata
Subclass Control New Total
Total

34. Estimated Propensity Scores N(new)=560, N(control)=290

35. propensity score stratification adjustment
Covariate balance checking before and after
propensity score stratification adjustment
Mean p-value
New Control Before After
Mi <
Diab
CCS
Lesleng <
Preref
Presten <

36. After adj. balance check:
Prior Mi rate:
Overall: Group % patients with prior Mi
New
Control
Diff
After:
Quintile
Group
New
Control

37. Percentage of patients with prior Mi

38. Adjusted Difference: Mew – Control:
Point estimate: -1.5%
2-sided 95% C.I. : (-6.6%, 3.6%)
Non-inferiority margin: 7%
Claim: Non-inferiority w.r.t. Mace 6-month

39. Study Design Plan in advance
Pre-specify clinically relevant baseline covariates: as many as possible
Sample size estimation:
Ignore the propensity score adjustment?
Could be inappropriate

40. Limitations
Propensity score methods can only adjust for observed confounding covariates and not for unobserved ones.
Propensity score is seriously degraded when important variables influencing selection have not been collected.
Propensity score may not eliminate all selection bias.

41. Limitations Propensity score methods work better in larger samples.
Propensity score is not only way of adjusting for covariates. And, it may or may not be helpful in a particular comparison study.
Randomized trials are considered the highest level of evidence for trt comparison. Propensity score methods lack the discipline and rigor of randomized trials, and not as definitive as randomized trials.

42. Conclusions
Propensity score methods generalize technique with one confounding covariate to allow simultaneous adjustment for many covariates and thus reduce bias.
Propensity score methodology is an addition to, not a substitute of traditional covariate adjustment methods.
Plan ahead and carefully consider the practical issues discussed above.
Randomized studies are still preferred and strongly encouraged whenever possible!

43. References
Rubin, DB, Estimating casual effects from large data sets using propensity scores. Ann Intern Med 1997; 127:
Rosenbaum, PR, Rubin DB, Reducing bias in observational studies using subclassification on the propensity score. JASA 1984; 79:
D’agostino, RB, Jr., Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group, Statistics in medicine, 1998,17:

44. References
Blackstone, EH, Comparing apples and oranges, J. Thoracic and Cardiovascular Surgery, January 2002; 1:8-15
Grunkemeier, GL and et al, Propensity score analysis of stroke after off-pump coronary artery bypass grafting, Ann Thorac Surg 2002; 74:
Wolfgang, C. and et al, Comparing mortality of elder patients on hemodialysis versus peritoneal dialysis: A propensity score approach, J. Am Soc Nephrol 2002; 13:

45. Thanks!