1. Application of Futility Testing in Vaccine Outcome Studies (with a Recent Example)
Aiying Chen, Scott Patterson, Fabrice Bailleux and Ehab Bassily
BIOS in Sanofi Pasteur, US & FR
JSM
July 30, 2018

2. Outline Introduction Methods for Futility Testing
Methods for Futility Testing Applied on VE
Example Study
Conclusion
Disclaimer: These are the authors’ views and do not represent the views of our employers and any affiliates.
JSM
30 JUL2018

3. Introduction Vaccine efficacy (VE) trial
To evaluate whether the vaccine can reduce incidence rate / severity of the disease
Randomized, Controlled trials conducted
In many cases, large scale VE trial required, long time period to complete the study with considerable costs/resources.
N = C/(r*(2-VE)),
Example: VE= 60%, r=1.5%, C=100, N= 4800 / per group, or 9600 in the trial.
It is beneficial to have interim monitoring/analyses before the end of the trial
Early demonstration of efficacy (stop for efficacy)
Safety reason to avoid more patients exposure to unsafe treatments
Little chance to be statistically significant at the end (stop for futility)
JSM
30JUL2018

4. Introduction Futility aims: Available methods for futility testing:
To avoid the patients exposure to ineffective treatment
Preserve limited resources (time and money) on more promising projects
Available methods for futility testing:
Hypothesis testing for futility based on confidence interval of the observed vaccine efficacy
Based on calculation of probability to reject the null hypothesis at the end ---- conditional power, predictive power
Interim analysis for futility can be added in the design including interim analysis to stop for efficacy
JSM
30JUL2018

5. Hypothesis Testing --- Confidence Interval
The tested hypothesis for futility: 𝐻 0 :𝑉𝐸≥ 𝑉𝐸 𝑓 𝐻 𝐴 :𝑉𝐸< 𝑉𝐸 𝑓
Hypothesis testing for vaccine efficacy : 𝐻 0 :𝑉𝐸≤ 𝑉𝐸 0 𝐻 𝐴 :𝑉𝐸> 𝑉𝐸 0
Statistical hypothesis to evaluate VE are based on the confidence interval of the point estimate of the observed VE
JSM
30JUL2018

6. Conditional Power
The probability that the final study result will be statistically significant, given the observed at interim, and a specific assumption about the future data
Frequentist analysis method
Stop the trial if the conditional probability of rejecting the null hypothesis is too low (below a pre-specified threshold, 0.2 for example, (see Ellenberg,S. etc )
The calculations involves:
The observed data and
Assumption regarding the distributions of future data in the two treatment groups
JSM JUL2018

7. Predictive Power
The probability of achieving significant/successful result at a future analysis, given the current observed data
Averaging the conditional power function over a range of treatment differences
The weights are based on the posterior density of the treatment effect given the observed data (Bayesian Calculation)
𝑃 𝑡 = 𝑝 𝑡 𝜃 𝜋 𝜃 𝐷 𝑡 𝑑𝜃
If this probability is too small and lower than the pre-specified criteria, then stop the trial for futility. If this probability is reasonable, we can continue the trial as pre-planned
JSM JUL2018

8. Type I Error Consideration
No type I error adjusted for futility needed – no false positive risk to regulators if we stop the study early
Interim analysis testing for efficacy increase the overall type I error rate
Type II error – (false negative) should be considered
Futility analysis increase the overall beta, (overall power decreased, especially when the vaccine is really efficacious) Sponsor risk
JSM JUL2018

9. Futility Testing Methods Applied on Vaccine Efficacy
JSM
30JUL2018

10. Vaccine Efficacy (VE) 𝑉𝐸=1−𝑅= 1 − 𝑝 𝑇 𝑝 𝐶
𝑉𝐸=1−𝑅= 1 − 𝑝 𝑇 𝑝 𝐶
𝑝 𝑇 , 𝑝 𝐶 : true incidence rates of the 𝑛 𝑇 (vaccine), 𝑛 𝐶 (control)
𝑅 : the incidence ratio between the two groups
VE=1 (i.e., 𝑝 𝑇 =0), completely efficacious
VE=0 (i.e., 𝑝 𝑇 = 𝑝 𝐶 ), no efficacy
Note: acceptable if 𝑉𝐸 0 > 0
JSM JUL2018

11. Confidence Interval Method Applied on VE
For extremely low incidence disease, the number of cases from each group follows Possion Distr.
The number of cases in vaccine group is binomially distributed conditionally to the total number of cases.
𝑥 𝑇 ≈𝑃𝑜𝑖𝑠s𝑜𝑛(  𝑇 ),  𝑇 ≈ 𝑛 𝑇 𝑝 𝑇 ; 𝑥 𝐶 ≈𝑃𝑜𝑖𝑠s𝑜𝑛(  𝐶 ),  𝐶 ≈ 𝑛 𝐶 𝑝 𝐶
𝑥 𝑇 | (𝑥 𝑇 + 𝑥 𝐶 =𝑆) ≈ b(S, ), with 𝜃=  𝑇  𝑇 +  𝐶 = 𝑅 𝑅+1/𝑢 = 1−𝑉𝐸 1−𝑉𝐸+1/𝑢 , with 𝑢= 𝑛 𝑇 𝑛 𝐶
A confidence interval for θ is determined; using exact method for example; and then it is derived to give a confidence interval for VE
JSM
30JUL2018

12. Conditional Power Applied on VE
n: target number of events ; n1: observed number of events at interim; n2: number of events at the remaining stages; n1+n2 = n
X :the number of cases from vaccine group, X  Binomial(n, 𝜃 0 );
𝑋 1 :observed number of cases from vaccine group at interim stage; 𝑋 2 : the number of cases from vaccine group at the remaining stages;
Assume 𝑋 2 ~𝐵𝑖𝑛(𝑛2, 𝜃 1 ); 𝜃 1 is the estimated value of  at interim
Conditional power: P( 𝑋 2 ≤𝑋− 𝑋 1 |𝑛2, 𝜃 1 )
If the conditional power is too low (e.g. < 0.2) then the trial stops for futility, otherwise the trial continues.
JSM
30JUL2018

13. Predictive Power Applied on VE
Assume uniform prior distribution with 𝑓 𝜃 =1
Posterior distribution of 𝜃: 𝑓 𝜃| 𝑥 𝑣 1 ,𝑠 = 𝑓 𝑥 𝑣 1 𝜃,𝑆 𝑓(𝜃) 𝑓( 𝑥 𝑣 1 |𝑆) =𝑏𝑒𝑡𝑎( 𝑥 𝑣 1 +1, 𝑠− 𝑥 𝑣 1 + 1) 𝜃 𝑥 𝑣 1 (1−𝜃) 𝑠− 𝑥 𝑣 1 +1
Predictive distribution: 𝑃 𝑋 1 𝑛 = 𝑥 1 𝑛 𝑥 𝑣 1 ,𝑠 = 0 1 𝑃 𝑋 1 𝑛 = 𝑥 1 𝑛 𝜃 𝑓 𝜃 𝑥 𝑣 1 ,𝑠 𝑑𝜃
Predictive power:
x11 and x01 cases at interim for futility (x11 + x01 = xT1)
Final analysis is performed with xTf cases
xTn new cases are needed (xTn =xTf-xT1)
Future cases are split into the vaccine and placebo (x1n, x0n) and calculation of the associated probability from predictive distribution
Each set of future cases is combined with cases collected at interim and the VE is calculated with the associated CI, and the lower bound (LB) compared to VE0.
The Bayesian predictive power is the Sum of the probability over the set of cases for which LB > VE0.
If the Bayesian predictive power is too low (e.g. < 0.2) then the trial stops for futility, otherwise the trial continues.
JSM
30JUL2018

14. Example Study (1/2)
Purpose: to evaluate the efficacy of the candidate vaccine to prevent some infection in subjects at risk
Primary outcome measure: number of events/cases confirmed by lab test
Type I error: 0.025; Power: 90%
Sample size: 225 events; >= 12,000 subjects enrolled, around 7 years to complete the trial
Randomization ratio: 2:1
Factors affecting the vaccine efficacy: sensitivity, specificity
Note: all numbers in the example study are disguised
JSM JUL2018

15. Example Study (2/2) Futility hypothesis testing
H0: VE >= 50% vs H1: VE < 50%
Efficacy hypothesis testing:
H0: VE <= 20% vs H1: VE > 20%
Interim design
Futility testing at 60, 100 cases, and efficacy interims
Futility Interim
Case split threshold
Observed VE
95% CI
Predictive power
Conditional power
60 cases
38:22
13.6%
(-53.2%, 50.5%)
6.4%
0.1%
100 cases
60:40
25.0%
(-15%, 50.6%)
8.6%
1.9%
JSM JUL2018

16. Example Study Results Interim Case Split Observed VE 95% CI
Predictive Power
Conditional Power
60 cases
30:30
50%
(14.2, 70.9)
81.4%
95.7%
33:27
38.9%
(-5.7, 64.4)
49.7%
48.3%
34:26
34.6%
(-13.5, 61.9)
37.9%
26.4%
35:25
30.0%
(-21.8, 59.3)
27.1%
11.1%
36:24
25.0%
(-31.2, 56.5)
18.1%
3.5%
37:23
19.6%
(-41.6, 53.5)
11.2%
0.8%
38:22
13.6%
(-53.2, 50.5)
6.4%
0.1%
Observational Result
method
Result
Observed VE
39/21, VE=7.1%
95% CI
(-66.5, 46.7)
Predictive power
3.4 %
JSM JUL2018

17. Conclusion
We apply three methods for futility testing to vaccine efficacy trials
Type I error rate is not impacted
Confidence interval method is more elegant
Predictive power and conditional power methods based on the observed VE provide similar results, more chance to stop early for futility.
It is possible to have different decision for futility with different methods, but this have to be investigated before, and to select the method before the start of the trial.
Alpha-spending methods should be used for efficacy interim analyses
JSM JUL2018

18. Some References
Chan ISF and NR Bohidar, Exact power and sample size for vaccine efficacy studies. Comm. in Stat.-Theory and Methods 27:6, , 1998.
Chou, S., et al., Sample Size Calculation in Clinical Research 2003
Ellenberg, S. et al., Data Monitoring Committees in Clinical Trials: A Practical Perspective 2002.
Nauta, J., Statistics in Clinical Vaccine Trials.
JSM JUL2018

19. Thank You !
JSM JUL2018