1. Personalized Predictive Medicine and Genomic Clinical Trials Richard Simon, D.Sc. Chief, Biometric Research Branch National Cancer Institute http://brb.nci.nih.gov

2. Biometric Research Branch Website brb.nci.nih.gov Powerpoint presentations Reprints BRB-ArrayTools software Web based Sample Size Planning

3. Personalized Oncology is Here Today and Rapidly Advancing Key information is in tumor genome, not in inherited genetics Personalization is based on limited stratification of traditional diagnostic categories based on key treatment-specific predictive biomarkers

4. Although the randomized clinical trial remains of fundamental importance for predictive genomic medicine, some of the conventional wisdom of how to design and analyze rct’s requires re-examination The paradigm of doing a broad eligibility rct of thousands of patients to answer a single question about average treatment effect for a target population presumed homogeneous with regard to the direction of treatment efficacy no longer has a scientific basis in oncology

5. Standard Approach is Based on Assumptions Qualitative treatment by subset interactions are unlikely i.e. if new treatment T is better than control C on average, it is better for all subsets of patients “Costs” of over-treatment are less than “costs” of under-treatment

6. Cancers of a primary site often represent a heterogeneous group of diverse molecular diseases which vary fundamentally with regard to – the oncogenic mutations that cause them, – their responsiveness to specific drugs

7. How Can We Develop New Drugs in a Manner More Consistent With Modern Tumor Biology and Obtain Reliable Information About What Regimens Work for What Kinds of Patients?

8. Prospective Co-Development of Drugs and Companion Diagnostics 1.Develop a completely specified genomic classifier of the patients likely to benefit from a new drug Based on drug target, pre-clinical, phase 1/2 2.Establish analytical validity of the classifier 3.Use the completely specified classifier to design and analyze a focused clinical trial to evaluate effectiveness of the new treatment and how it relates to the candidate biomarker

9. Targeted (Enrichment) Design Restrict entry to the phase III trial based on the binary predictive classifier

10. Using phase II data, develop predictor of response to new drug Develop Predictor of Response to New Drug Patient Predicted Responsive New Drug Control Patient Predicted Non-Responsive Off Study

11. Applicability of Targeted Design Primarily for settings where the classifier is based on a single gene whose protein product is the target of the drug and there is substantial biological evidence that the drug will not be effective for classifier negative patients Because most cancer drugs have serious side effects and limit the doses at which other drugs can be administered, it is ethically problematic to ask patients to participate in a clinical trial of a regimen from which they are not expected to benefit – eg trastuzumab – Parp inhibitors

12. Evaluating the Efficiency of Targeted Design Simon R and Maitnourim A. Evaluating the efficiency of targeted designs for randomized clinical trials. Clinical Cancer Research 10:6759- 63, 2004; Correction and supplement 12:3229, 2006 Maitnourim A and Simon R. On the efficiency of targeted clinical trials. Statistics in Medicine 24:329-339, 2005.

13. Relative efficiency of targeted design depends on – proportion of patients test positive – effectiveness of new drug (compared to control) for test negative patients Specificity of treatment Sensitivity of test When less than half of patients are test positive and the drug has little or no benefit for test negative patients, the targeted design requires dramatically fewer randomized patients

14. Developmental Strategy (II) Develop Predictor of Response to New Rx Predicted Non-responsive to New Rx Predicted Responsive To New Rx Control New RXControl New RX

15. Do not use the test to restrict eligibility, but to structure a prospective analysis plan Having a prospective analysis plan is essential. “Stratifying” (balancing) the randomization is useful to ensure that all randomized patients have test performed but is not required for valid inference and is not a substitute for a prospective analysis plan Size the study for adequate evaluation of T vs C separately by marker status

16. R Simon. Using genomics in clinical trial design, Clinical Cancer Research 14:5984-93, 2008 R Simon. Designs and adaptive analysis plans for pivotal clinical trials of therapeutics and companion diagnostics, Expert Opinion in Medical Diagnostics 2:721-29, 2008

18. Analysis Plan C Limited Confidence in Classifier Test for difference (interaction) between treatment effect in test positive patients and treatment effect in test negative patients at an elevated level (e.g..10) If interaction is significant at that level, then compare treatments separately for test positive patients and test negative patients Otherwise, compare treatments overall

19. Sample Size Planning for Analysis Plan C 88 events in test + patients needed to detect 50% reduction in hazard at 5% two-sided significance level with 90% power If 25% of patients are positive, when there are 88 events in positive patients there will be about 264 events in negative patients – 264 events provides 90% power for detecting 33% reduction in hazard at 5% two-sided significance leve

20. Futility Analysis Interim futility analyses separately in test – and test + Conservative futility analysis – After observing 132 (264/2) events in test – patients, if hazard ratio of new treatment vs control is < 1, then terminate accrual of test – patients but continue accrual of test + patients till planned 88 events Futility analysis may be performed using a “conditional surrogate” intermediate endpoint to protect the test - patients

21. Does the RCT Need to Be Significant Overall for the T vs C Treatment Comparison? No – That requirement has been traditionally used to protect against data dredging. It is inappropriate for focused trials of a treatment with a companion test.

22. Because of the complexity of cancer biology, it is often difficult to have the right completely defined predictive biomarker identified and analytically validated by the time the pivotal trial of a new drug is ready to start accrual

23. Multiple Biomarker Design Have identified K candidate binary classifiers B 1, …, B K thought to be predictive of patients likely to benefit from T relative to C Eligibility not restricted by candidate classifiers

24. Fallback Analysis Plan Compare outcomes of T to C overall If p < 0.01, claim effectiveness of T overall Otherwise, conduct planned, type I error protected subset analysis

25. Compute p k comparing T vs C restricted to patients positive for B k. Do this for k=0,1,…,K Let p* = min {p k }, k* = argmin{p k } For a global test of significance – Compute null distribution of p* by permuting treatment labels – If the data value of p* is less than the 4’th percentile of the null distribution, then claim effectiveness of T for patients positive for B k*

26. Repeating the analysis for bootstrap samples of cases provides – an estimate of the stability of k* (the indication) – an interval estimate of the size of treatment effect for the size of treatment effect in the target population

27. Adaptive Signature Design Boris Freidlin and Richard Simon Clinical Cancer Research 11:7872-8, 2005

28. Adaptive Signature Design End of Trial Analysis Compare T to C for all patients at significance level α 0 (eg 0.01) – If overall H 0 is rejected, then claim effectiveness of T for eligible patients – Otherwise

29. Otherwise: – Using a randomly selected training set consisting of a pre- specified proportion of patients accrued during the trial, develop a binary classifier that predicts the subset of patients most likely to benefit from the new treatment T compared to control C – Compare T to C for patients accrued in second stage who are predicted responsive to T based on classifier Perform test at significance level 1- α 0 (eg 0.04)

30. Treatment effect restricted to subset. 10% of patients sensitive, 10 sensitivity genes, 10,000 genes, 400 patients. TestPower Overall.05 level test 46.7 Overall.04 level test 43.1 Sensitive subset.01 level test (performed only when overall.04 level test is negative) 42.2 Overall adaptive signature design 85.3

31. Cross-Validated Adaptive Signature Design Freidlin B, Jiang W, Simon R Clinical Cancer Research 16(2) 2010

32. 70% Response to T in Sensitive Patients 25% Response to T Otherwise 25% Response to C 20% Patients Sensitive ASDCV-ASD Overall 0.05 Test 0.4860.503 Overall 0.04 Test 0.4520.471 Sensitive Subset 0.01 Test 0.2070.588 Overall Power 0.5250.731

33. Prediction Based Analysis of Clinical Trials Using cross-validation we can evaluate our methods for analysis of clinical trials, including complex subset analysis algorithms, in terms of their effect on improving patient outcome via informing therapeutic decision making This approach can be used with any set of candidate predictor variables

34. Define an algorithm A for developing a classifier of whether patients benefit preferentially from a new treatment T relative to C For patients with covariate vector x, the algorithm predicts preferred treatment Applying A to a training dataset D provides a classifier model M(A, D) – R(x |M(A, D) ) = T – R(x | D ) = C

35. At the conclusion of the trial randomly partition the patients into K approximately equally sized sets P 1, …, P 10 Let D -i denote the full dataset minus data for patients in P i Using K-fold complete cross-validation, omit patients in P i Apply the defined algorithm to analyze the data in D -i to obtain a classifier M -i For each patient j in P i record the treatment recommendation i.e. R j =T or R j =C

36. Repeat the above for all K loops of the cross- validation All patients have been classified as what their optimal treatment is predicted to be

37. Let S T denote the set of patients for whom treatment T is predicted optimal i.e. S T = {j : R j =T} Compare outcomes for patients in S T who actually received T to those in S T who actually received C – Let z T = standardized log-rank statistic – Let HR T denote the estimated hazard ratio in S T Compute statistical significance of z T by randomly permuting treatment labels and repeating the entire procedure – Do this 1000 or more times to generate the permutation null distribution of treatment effect for the patients in subset

38. The significance test based on comparing T vs C for S T = {j : R j =T} is the basis for demonstrating that T is more effective than C for some patients.

39. By applying the analysis algorithm to the full RCT dataset D, recommendations are developed for how future patients should be treated R(x|D) for all x vectors. The cross-validated estimate HR T of treatment effect in S T provides a conservative estimate of the treatment effect in the subset for which R(x|D)=T

40. Identification of the subset of patients who benefit from T vs C, although imperfect, will generally be substantially greater than for the standard clinical trial in which all patients are classified based on results of testing the single overall null hypothesis

42. Prediction Based Clinical Trials New methods for determining from RCTs which patients, if any, benefit from new treatments can be evaluated directly using the actual RCT data in a manner that separates model development from model evaluation, rather than basing treatment recommendations on the results of a single hypothesis test or on exploratory subset analyses of the full dataset.

43. Prediction Based Clinical Trials Hypothesis testing has value for ensuring that ineffective treatments are not approved Hypothesis testing is not an effective paradigm for identifying which patients benefit from a new treatment – The current paradigm results in over-treatment of populations of patients; with many patients not benefiting – Conventional post-hoc subset analysis is also unsatisfactory as it provides no internal validation of classification accuracy

44. Using a combination of hypothesis testing of a global null hypothesis of no treatment effect and cross-validated prediction analysis based on a pre-specified algorithm for prognostic classification, we can accomplish both objectives: – Preserve type I error to ensure that most ineffective treatments are not approved – Provide an internally validated classifier of which kinds of patients benefit from the new treatment

45. Acknowledgements Boris Freidlin Yingdong Zhao Wenyu Jiang Aboubakar Maitournam