Application of Futility Testing in Vaccine Outcome Studies (with a Recent Example) Aiying Chen, Scott Patterson, Fabrice Bailleux and Ehab Bassily BIOS in Sanofi Pasteur, US & FR JSM July 30, 2018

Outline Introduction Methods for Futility Testing Methods for Futility Testing Applied on VE Example Study Conclusion Disclaimer: These are the authors’ views and do not represent the views of our employers and any affiliates. JSM 30 JUL2018

Introduction Vaccine efficacy (VE) trial To evaluate whether the vaccine can reduce incidence rate / severity of the disease Randomized, Controlled trials conducted In many cases, large scale VE trial required, long time period to complete the study with considerable costs/resources. N = C/(r*(2-VE)), Example: VE= 60%, r=1.5%, C=100, N= 4800 / per group, or 9600 in the trial. It is beneficial to have interim monitoring/analyses before the end of the trial Early demonstration of efficacy (stop for efficacy) Safety reason to avoid more patients exposure to unsafe treatments Little chance to be statistically significant at the end (stop for futility) JSM 30JUL2018

Introduction Futility aims: Available methods for futility testing: To avoid the patients exposure to ineffective treatment Preserve limited resources (time and money) on more promising projects Available methods for futility testing: Hypothesis testing for futility based on confidence interval of the observed vaccine efficacy Based on calculation of probability to reject the null hypothesis at the end ---- conditional power, predictive power Interim analysis for futility can be added in the design including interim analysis to stop for efficacy JSM 30JUL2018

Hypothesis Testing --- Confidence Interval The tested hypothesis for futility: 𝐻 0 :𝑉𝐸≥ 𝑉𝐸 𝑓 𝐻 𝐴 :𝑉𝐸< 𝑉𝐸 𝑓 Hypothesis testing for vaccine efficacy : 𝐻 0 :𝑉𝐸≤ 𝑉𝐸 0 𝐻 𝐴 :𝑉𝐸> 𝑉𝐸 0 Statistical hypothesis to evaluate VE are based on the confidence interval of the point estimate of the observed VE JSM 30JUL2018

Conditional Power The probability that the final study result will be statistically significant, given the observed at interim, and a specific assumption about the future data Frequentist analysis method Stop the trial if the conditional probability of rejecting the null hypothesis is too low (below a pre-specified threshold, 0.2 for example, (see Ellenberg,S. etc ) The calculations involves: The observed data and Assumption regarding the distributions of future data in the two treatment groups JSM 30JUL2018

Predictive Power The probability of achieving significant/successful result at a future analysis, given the current observed data Averaging the conditional power function over a range of treatment differences The weights are based on the posterior density of the treatment effect given the observed data (Bayesian Calculation) 𝑃 𝑡 = 𝑝 𝑡 𝜃 𝜋 𝜃 𝐷 𝑡 𝑑𝜃 If this probability is too small and lower than the pre-specified criteria, then stop the trial for futility. If this probability is reasonable, we can continue the trial as pre-planned JSM 30JUL2018

Type I Error Consideration No type I error adjusted for futility needed – no false positive risk to regulators if we stop the study early Interim analysis testing for efficacy increase the overall type I error rate Type II error – (false negative) should be considered Futility analysis increase the overall beta, (overall power decreased, especially when the vaccine is really efficacious) ------------ Sponsor risk JSM 30JUL2018

Futility Testing Methods Applied on Vaccine Efficacy JSM 30JUL2018

Vaccine Efficacy (VE) 𝑉𝐸=1−𝑅= 1 − 𝑝 𝑇 𝑝 𝐶 𝑉𝐸=1−𝑅= 1 − 𝑝 𝑇 𝑝 𝐶 𝑝 𝑇 , 𝑝 𝐶 : true incidence rates of the 𝑛 𝑇 (vaccine), 𝑛 𝐶 (control) 𝑅 : the incidence ratio between the two groups VE=1 (i.e., 𝑝 𝑇 =0), completely efficacious VE=0 (i.e., 𝑝 𝑇 = 𝑝 𝐶 ), no efficacy Note: acceptable if 𝑉𝐸 0 > 0 JSM 30JUL2018

Confidence Interval Method Applied on VE For extremely low incidence disease, the number of cases from each group follows Possion Distr. The number of cases in vaccine group is binomially distributed conditionally to the total number of cases. 𝑥 𝑇 ≈𝑃𝑜𝑖𝑠s𝑜𝑛(  𝑇 ),  𝑇 ≈ 𝑛 𝑇 𝑝 𝑇 ; 𝑥 𝐶 ≈𝑃𝑜𝑖𝑠s𝑜𝑛(  𝐶 ),  𝐶 ≈ 𝑛 𝐶 𝑝 𝐶 𝑥 𝑇 | (𝑥 𝑇 + 𝑥 𝐶 =𝑆) ≈ b(S, ), with 𝜃=  𝑇  𝑇 +  𝐶 = 𝑅 𝑅+1/𝑢 = 1−𝑉𝐸 1−𝑉𝐸+1/𝑢 , with 𝑢= 𝑛 𝑇 𝑛 𝐶 A confidence interval for θ is determined; using exact method for example; and then it is derived to give a confidence interval for VE JSM 30JUL2018

Conditional Power Applied on VE n: target number of events ; n1: observed number of events at interim; n2: number of events at the remaining stages; n1+n2 = n X :the number of cases from vaccine group, X  Binomial(n, 𝜃 0 ); 𝑋 1 :observed number of cases from vaccine group at interim stage; 𝑋 2 : the number of cases from vaccine group at the remaining stages; Assume 𝑋 2 ~𝐵𝑖𝑛(𝑛2, 𝜃 1 ); 𝜃 1 is the estimated value of  at interim Conditional power: P( 𝑋 2 ≤𝑋− 𝑋 1 |𝑛2, 𝜃 1 ) If the conditional power is too low (e.g. < 0.2) then the trial stops for futility, otherwise the trial continues. JSM 30JUL2018

Predictive Power Applied on VE Assume uniform prior distribution with 𝑓 𝜃 =1 Posterior distribution of 𝜃: 𝑓 𝜃| 𝑥 𝑣 1 ,𝑠 = 𝑓 𝑥 𝑣 1 𝜃,𝑆 𝑓(𝜃) 𝑓( 𝑥 𝑣 1 |𝑆) =𝑏𝑒𝑡𝑎( 𝑥 𝑣 1 +1, 𝑠− 𝑥 𝑣 1 + 1) 𝜃 𝑥 𝑣 1 (1−𝜃) 𝑠− 𝑥 𝑣 1 +1 Predictive distribution: 𝑃 𝑋 1 𝑛 = 𝑥 1 𝑛 𝑥 𝑣 1 ,𝑠 = 0 1 𝑃 𝑋 1 𝑛 = 𝑥 1 𝑛 𝜃 𝑓 𝜃 𝑥 𝑣 1 ,𝑠 𝑑𝜃 Predictive power: x11 and x01 cases at interim for futility (x11 + x01 = xT1) Final analysis is performed with xTf cases xTn new cases are needed (xTn =xTf-xT1) Future cases are split into the vaccine and placebo (x1n, x0n) and calculation of the associated probability from predictive distribution Each set of future cases is combined with cases collected at interim and the VE is calculated with the associated CI, and the lower bound (LB) compared to VE0. The Bayesian predictive power is the Sum of the probability over the set of cases for which LB > VE0. If the Bayesian predictive power is too low (e.g. < 0.2) then the trial stops for futility, otherwise the trial continues. JSM 30JUL2018

Example Study (1/2) Purpose: to evaluate the efficacy of the candidate vaccine to prevent some infection in subjects at risk Primary outcome measure: number of events/cases confirmed by lab test Type I error: 0.025; Power: 90% Sample size: 225 events; >= 12,000 subjects enrolled, around 7 years to complete the trial Randomization ratio: 2:1 Factors affecting the vaccine efficacy: sensitivity, specificity Note: all numbers in the example study are disguised JSM 30JUL2018

Example Study (2/2) Futility hypothesis testing H0: VE >= 50% vs H1: VE < 50% Efficacy hypothesis testing: H0: VE <= 20% vs H1: VE > 20% Interim design Futility testing at 60, 100 cases, and efficacy interims Futility Interim Case split threshold Observed VE 95% CI Predictive power Conditional power 60 cases 38:22 13.6% (-53.2%, 50.5%) 6.4% 0.1% 100 cases 60:40 25.0% (-15%, 50.6%) 8.6% 1.9% JSM 30JUL2018

Example Study Results Interim Case Split Observed VE 95% CI Predictive Power Conditional Power 60 cases 30:30 50% (14.2, 70.9) 81.4% 95.7% 33:27 38.9% (-5.7, 64.4) 49.7% 48.3% 34:26 34.6% (-13.5, 61.9) 37.9% 26.4% 35:25 30.0% (-21.8, 59.3) 27.1% 11.1% 36:24 25.0% (-31.2, 56.5) 18.1% 3.5% 37:23 19.6% (-41.6, 53.5) 11.2% 0.8% 38:22 13.6% (-53.2, 50.5) 6.4% 0.1% Observational Result method Result Observed VE 39/21, VE=7.1% 95% CI (-66.5, 46.7) Predictive power 3.4 % JSM 30JUL2018

Conclusion We apply three methods for futility testing to vaccine efficacy trials Type I error rate is not impacted Confidence interval method is more elegant Predictive power and conditional power methods based on the observed VE provide similar results, more chance to stop early for futility. It is possible to have different decision for futility with different methods, but this have to be investigated before, and to select the method before the start of the trial. Alpha-spending methods should be used for efficacy interim analyses JSM 30JUL2018

Some References Chan ISF and NR Bohidar, Exact power and sample size for vaccine efficacy studies. Comm. in Stat.-Theory and Methods 27:6,1305- 1322, 1998. Chou, S., et al., Sample Size Calculation in Clinical Research 2003 Ellenberg, S. et al., Data Monitoring Committees in Clinical Trials: A Practical Perspective 2002. Nauta, J., Statistics in Clinical Vaccine Trials. JSM 30JUL2018

Thank You ! JSM 30JUL2018