1. Prospective Subset Analysis in Therapeutic Vaccine Studies Richard Simon, D.Sc. Chief, Biometric Research Branch National Cancer Institute http://linus.nci.nih.gov/brb

2. Not Statistical Credibility of Subset Analyses in Phase 3 Trials: Is There Room for a Different Perspective for Cancer Vaccines?

3. Clinical trial designs for early clinical development of therapeutic cancer vaccines, JCO 19:1848-54, 2001 R Simon, S Steinberg, M Hamilton, A Hildesheim, S Khleif, L Kwak, C Mackall, J Schlom, S Topalian and J Berzofsky

4. Why focus on early clinical development? Principles for phase III trials apply equally to vaccines –Randomized control group –Endpoint reflecting clinical benefit –Defined target population Differences between vaccines and chemotherapeutic agents have important implications for early clinical trials

5. Phase III Clinical Trials of Therapeutic Cancer Vaccines should conform to good statistical principles –Vaccine studies should be performed professionally –There are no perfect studies Adaptive prospective analysis plans are consistent with good statistical principles. Some aspects of therapeutic vaccine studies make adaptive analysis plans more important than for other therapeutic areas

6. Objectives of Phase II Trials Determine whether regimen is sufficiently promising to warrant phase III trial Optimize regimen –Generally using non-clinical endpoint Identify the right population of patients to include in phase III trial

7. Differences Between Therapeutic Vaccines and Chemotherapeutic Agents Many vaccines are incapable of causing immediate serious or life threatening toxicity at doses feasible to manufacture Appropriate target population may not have measurable tumor Vaccination strategies often combine multiple agents and components (adjuvants, cytokines, costimulatory molecules)

8. Trying to determine whether there is a dose-response relationship is a phase III objective. Using more than two dose levels to determine an OBD is even more ambitious.

9. Optimization of vaccine regimen by comparing results of single arm studies using immunological response is problematic –Much time can be wasted on uninterpretable or unreliable phase I or phase II trials Randomized screening studies can be used to optimize immunogenicity. Phase II studies of time to progression should have randomized controls

10. Phase 2.5 Trials Randomized control group  = 0.10 type 1 error rate Endpoint without established clinical benefit Detect relatively large treatment effect E.g. Power 0.8 for detecting 40% reduction in 12 month median time to recurrence with  =0.10 requires 44 patients per arm

11. Prototypic Phase III Clinical Trial One hypothesis tested Randomized treatment assignment –One investigational treatment arm –One control arm Single pre-defined target population Single pre-defined primary endpoint –Measure of clinical benefit Evaluate statistical significance with type I error 5% –Interim analyses consume the type I error

12. Test Multiple Hypotheses Multiple treatment regimens –Doses, schedules, adjuvants, etc Multiple target populations –All eligible patients –Pre-defined subsets –Non-pre-defined subsets Multiple endpoints

13. Number of comparisonsProbability of at least one p<.05 false positive 1.05 2.097 3.14 4.18 5.23 10.40 20.64

14. Testing Multiple Hypotheses Specify hypotheses to be tested in advance Share the type I error among the tests so that if all of the null hypotheses are true the probability of rejecting any of them is no more than.05

15. Principle Hypotheses to be tested must not be determined by the same data used to do the testing –Unless the way that the data determines the hypothesis to be tested is specified in advance, we cannot control the type I error

16. Developmental Strategy (II) Develop Predictor of Response to New Rx Predicted Non- responsive to New Rx Predicted Responsive To New Rx Control New RXControl New RX

17. Developmental Strategy (II) Do not use the diagnostic to restrict eligibility, but to structure a prospective analysis plan. Compare the new drug to the control overall for all patients ignoring the classifier. –If p overall  0.04 claim effectiveness for the eligible population as a whole Otherwise perform a single subset analysis evaluating the new drug in the classifier + patients –If p subset  0.01 claim effectiveness for the classifier + patients.

18. The purpose of the RCT is to evaluate the new treatment overall and for the pre-defined subset The purpose is not to re-evaluate the components of the classifier, or to modify or refine the classifier The purpose is not to demonstrate that repeating the classifier development process on independent data results in the same classifier

19. Developmental Strategy III Do not use the diagnostic to restrict eligibility, but to structure a prospective analysis plan. Compare the new drug to the control for classifier positive patients –If p + >0.05 make no claim of effectiveness –If p +  0.05 claim effectiveness for the classifier positive patients and Continue accrual of classifier negative patients and eventually test treatment effect at 0.05 level

20. The Roadmap 1.Develop a completely specified genomic classifier of the patients likely to benefit from a new drug 2.Establish reproducibility of measurement of the classifier 3.Use the completely specified classifier to design and analyze a new clinical trial to evaluate effectiveness of the new treatment with a pre-defined analysis plan.

21. Guiding Principle The data used to develop the classifier must be distinct from the data used to test hypotheses about treatment effect in subsets determined by the classifier –Developmental studies are exploratory And not closely regulated by FDA FDA should not regulate classifier development –Studies on which treatment effectiveness claims are to be based should be definitive studies that test a treatment hypothesis in a patient population completely pre-specified by the classifier

22. Adaptive Clinical Trials Modify some aspect of the trial based on data developed in the same trial

23. Seamless Phase II/III Trial Futility Analysis Randomized comparison of vaccine based regimen to non-vaccine based control Size trial as phase III study with clinical endpoint Perform interim analysis using immunogenicity –If results are not “promising”, terminate trial –Otherwise, continue randomizing additional patients and do analysis of clinical endpoint at end of trial Interim futility analysis does not consume any of the type I error

24. Seamless Phase II/III Trial Optimize Treatment Arm Randomized comparison of 2 vaccine based regimens to non-vaccine based control Size trial as phase III study with clinical endpoint Perform interim analysis using immunologic response –select vaccine arm with most promising immunologic response data –Continue accrual as 2-arm phase III trial of the selected vaccine arm and the control arm Do analysis of clinical endpoint at end of trial using.05/2 level of significance

25. Adaptive Methods PF Thall, R Simon, SS Ellenberg, (1989) “A two- stage design for choosing among several experimental treatments and a control in clinical trials”, Biometrika 75:303-310. –1 st phase of trial randomize patients among k investigational arms and a control arm –Select the most promising investigational arm to complete the trial –The final analysis comparing the selected investigational arm to control takes into account that the first phase of the trial was used to select the arm

26. Adaptive Signature Design An adaptive design for generating and prospectively testing a gene expression signature for sensitive patients Boris Freidlin and Richard Simon Clinical Cancer Research 11:7872-8, 2005

27. Adaptive Signature Design End of Trial Analysis Compare E to C for all patients at significance level 0.04 –If overall H 0 is rejected, then claim effectiveness of E for eligible patients –Otherwise

28. Otherwise: –Using only the first half of patients accrued during the trial, develop a binary classifier that predicts the subset of patients most likely to benefit from the new treatment E compared to control C –Compare E to C for patients accrued in second stage who are predicted responsive to E based on classifier Perform test at significance level 0.01 If H 0 is rejected, claim effectiveness of E for subset defined by classifier

29. Treatment effect restricted to subset. 10% of patients sensitive, 10 sensitivity genes, 10,000 genes, 400 patients. TestPower Overall.05 level test46.7 Overall.04 level test43.1 Sensitive subset.01 level test (performed only when overall.04 level test is negative) 42.2 Overall adaptive signature design85.3

30. Overall treatment effect, no subset effect. 10,000 genes, 400 patients. TestPower Overall.05 level test74.2 Overall.04 level test70.9 Sensitive subset.01 level test1.0 Overall adaptive signature design70.9

31. Biomarker Adaptive Threshold Design Wenyu Jiang, Boris Freidlin & Richard Simon JNCI 99:1036-43, 2007 http://linus.nci.nih.gov/brb

32. Biomarker Adaptive Threshold Design Randomized pivotal trial comparing new treatment E to control C Survival or DFS endpoint Have identified a univariate biomarker index B thought to be predictive of patients likely to benefit from E relative to C Eligibility not restricted by biomarker No threshold for biomarker determined Biomarker value scaled to range (0,1)

33. Global Test for Treatment Effect Optimized Threshold Compare E vs C for patients with B > b S(b)=log likelihood ratio statistic for treatment versus control comparison in subset of patients with B  b Compute S(b) for all possible threshold values Determine b* value for which S(b) is maximum –T=S(b*) Compute null distribution of T by permuting treatment labels –Re-analyze the data Compute T for the permuted data –Repeat for 10,000 permutations

34. If the data value of T is significant at 0.05 level using the permutation null distribution of T, then reject null hypothesis that E is ineffective Compute point and interval estimates of the threshold b

36. Adaptive Methods The algorithm for adapting should be specified in advance in the protocol In assessing statistical significance, the analysis should take into account the adaptiveness algorithm used The rejection region should be calibrated to limit the experiment-wise type I error (probability of making any false positive claim from a study) to 5%, taking into account the adaptiveness algorithm used

37. But I’m a Bayesian Bayesian inference usually does not control type I error Bayesian methods can lead to a high frequency of erroneous conclusions if prior distributions are not selected carefully Dumb things are dumb to careful Bayesians –Bayesian methods are sharp scalpels, and need to be used carefully by experts with knowledge of anatomy

38. Bayesian analysis is based on specifying a model for all aspects of the data and a “prior distribution” for all parameters of the model –eg True hazard ratio of treatment effect on survival is a parameter –The true hazard ratio is regarded as a random sample from the specified prior distribution True values of parameters are not really random samples from a distribution Prior distributions represent one’s subjective beliefs about the unknown parameters before doing the study Different individuals have different prior distributions Bayesian methods provide a way of updating the subjective prior based on the data obtained from the trial –The Bayesian “posterior distribution” mixes the prior with the data Bayesian methods are more easily used in phase I/II trials where you don’t need to convince others of your prior distributions and your conclusions

39. Conclusions Phase III trials of therapeutic vaccines should meet the standards of phase III trials in other therapeutic areas Prospectively planned subset analyses are consistent with good statistical practice so long as the subsets are pre-defined and the study-wise type I error is preserved at 5% Therapeutic vaccine trials can benefit from adaptive analysis plans –The adaptivity algorithms should be completely pre-defined Regulators should be receptive to innovative clinical trial designs that are consistent with sound statistical principles The standard paradigm of broad eligibility clinical trials with no subset analysis unless the overall treatment effect is significant is often inappropriate in cancer therapeutics