THE USE OF HISTORICAL CONTROLS IN DEVICE STUDIES Vic Hasselblad Duke Clinical Research Institute

HISTORICAL CONTROLS: THE SETTING New trial will have a single experimental arm The endpoint is dichotomous Comparison will be to either summary rates from other studies another single arm using patient level analyses

HISTORICAL CONTROLS NO PATIENT LEVEL DATA Each historical arm has to be treated as a sample Results are usually calculated from a random effects model The distribution for the next sample is estimated This requires that the between study variance be estimated specifically

AN EXAMPLE FROM DISTAL PROTECTION DEVICES A new distal protection device proposed using data from existing distal protection devices as historical controls The endpoint was major adverse cardiac events (MACE) The results from these three arms appeared to be very consistent The estimation proved difficult (as we shall see)

DISTAL PROTECTION DEVICES

FilterWire (FIRE Trial) GuardWire (FIRE Trial) GuardWire (SAFER Trial) MACE Rate at 30 Days DISTAL PROTECTION DEVICES

The prior for the mean rate was a non- informative (Jeffries) prior The prior for the study-to-study variation (τ 2 ) was assumed to be non- informative (1/τ 2 ) The expected distribution of the log-odds of the event rate was assumed to be normal CONSTRUCTING A HIERARCHICAL BAYES RANDOM EFFECTS MODEL

POSTERIOR FOR VARIANCE (τ 2 )

POSTERIOR FOR MEAN RATE

In order to use the results from a small number of arms, one has to assume that the variation between arms is quite small. In other words, one has to add subjective information to the prior. CONCLUSIONS FOR HISTORICAL CONTROLS NO PATIENT LEVEL DATA

HISTORICAL CONTROLS WITH PATIENT LEVEL DATA Propensity scores are used to correct for the fact that the two populations are not guaranteed to be similar Patients are stratified by their propensity to get a particular treatment Patients within a given propensity score are compared and the results are pooled across propensity categories

AN EXAMPLE WITH STENTS The object was to compare two different stent methodologies, one of which was a historical one The safety endpoint was MACE at 30 days The comparison was based on non-inferiority Propensity scores were used to make the comparison: two different models were used as a sensitivity analysis

FIRST PROPENSITY SCORE Used vessel diameter, lesion length, and presence of diabetes as predictors. SECOND PROPENSITY SCORE Used vessel diameter, lesion length, presence of diabetes plus several others factors including smoking and EF as predictors.

AN EXAMPLE WITH STENTS First propensity score Second propensity score Delta

Percent Bias in the Estimated Treatment Effect Based on a Stratified Propensity Score (from Lunceford and Davidian, 2004) STRATIFIED PROPENSITY SCORES CAN HAVE DIFFICULTIES

Ratio of Means Squared Errors Stratified Propensity Score Versus Doubly Robust Estimator (from Lunceford and Davidian, 2004) STRATIFIED PROPENSITY SCORES CAN HAVE DIFFICULTIES

HISTORICAL CONTROLS WITH PATIENT LEVEL DATA Even with the use of propensity scores, the results of a historical control analysis may not be definitive The use of stratified propensity scores is not always the solution In certain situations doubly robust estimators are better as long as they have the correct model (for propensity or risk) If the models are wrong, all bets are off

SUMMARY Historical control analyses are fraught with difficulties In many cases you dont know if problems exist until after the data have been collected