1. Gatekeeping Testing Strategies in Clinical Trials Alex Dmitrienko, Ph.D. Eli Lilly and Company FDA/Industry Statistics Workshop September 2004

2. Slide 2 Outline Gatekeeping strategies account for the hierarchical structure of multiple analyses (comparisons) and preserve the overall false positive rate Clinical trial examples registration trials with multiple primary and secondary endpoints (trials in patients with depression and acute lung injury) dose-finding trials (trial in patients with hypertension)

3. Slide 3 Primary versus secondary findings Findings with respect to secondary endpoints provide much useful information useful to prescribing physicians and patients FDA guidance Clinical studies section of labeling for prescription drugs and biologics The CLINICAL STUDIES section should present those endpoints that are essential to establishing the effectiveness of the drug (or that show the limitations of effectiveness) and those that provide additional useful and valid information about the activities of the drug

4. Slide 4 Primary versus secondary findings Dilemma regulatory agencies and pharmaceutical companies have long debated what secondary findings should be included in the product label regulatory agencies are concerned that pharmaceutical companies tend to present favorable data and ignore unfavorable data Gatekeeper strategies offer one potential solution to the dilemma

5. Slide 5 Types of gatekeeping strategies Two types of gatekeeping strategies sequential strategies have been used for 10 years parallel strategies were introduced by Dmitrienko, Offen and Westfall (2003) Introduce general gatekeeping framework based on the closed testing principle (Marcus et al, 1976) focus on strategies derived using Bonferronis test easily extended to more powerful tests that account for the correlation among the endpoints (Dunnetts test, resampling tests)

6. Slide 6 Example 1: One primary endpoint Depression trial Experimental drug is compared to placebo Single primary endpoint –17-item Hamilton depression rating scale (HAMD 17 score) Trial is declared successful if the drug is superior to placebo Two important secondary endpoints –response rate based on the HAMD 17 score –remission rate based on the HAMD 17 score Can the secondary findings be included in the product label?

7. Slide 7 Sequential gatekeeping strategy Step 1: Perform the primary analysis Step 2: Perform the secondary analyses with an adjustment for multiplicity if the primary analysis yielded a significant result A1: HAMD17 A2: Response rate A3: Remission rate Family 2Family 1 (gatekeeper)

8. Slide 8 Sequential gatekeeping strategy Primary analysis: No adjustment for multiplicity Secondary analyses: Stepwise Holms test All primary and secondary findings are significant at 5% level

9. Slide 9 Example 2: Multiple primary endpoints Clinical trial in patients with acute lung injury Experimental drug is compared to placebo Two primary endpoints –number of days patients are off mechanical ventilation (vent-free days) –28-day all-cause mortality rate Trial is declared successful if the drug is superior to placebo with respect to either endpoint Two important secondary endpoints –number of days patients are out of ICU (ICU-free days) –overall quality of life at the end of the study Can the secondary findings be included in the product label?

10. Slide 10 Parallel gatekeeping strategy Step 1: Perform the primary analyses with an adjustment for multiplicity Step 2: Perform the secondary analyses with an adjustment for multiplicity if at least one primary analysis yielded a significant result A1: Vent-free days A2: Mortality A3: ICU-free days A4: Quality of life Family 1 (gatekeeper) Family 2

11. Slide 11 Parallel gatekeeping strategy Step 1: Perform the primary analyses with an adjustment for multiplicity Step 2: Perform the secondary analyses with an adjustment for multiplicity if at least one primary analysis yielded a significant result A1: Vent-free days A2: Mortality A3: ICU-free days A4: Quality of life Family 1 (gatekeeper) Family 2

12. Slide 12 Parallel gatekeeping strategy Step 1: Perform the primary analyses with an adjustment for multiplicity Step 2: Perform the secondary analyses with an adjustment for multiplicity if at least one primary analysis yielded a significant result A1: Vent-free days A2: Mortality A3: ICU-free days A4: Quality of life Family 1 (gatekeeper) Family 2

13. Slide 13 Statistical details Sequential families of null hypotheses Null hypotheses H 11, H 12, H 21, H 22 grouped into two families F 1 ={H 11, H 12 } and F 2 ={H 21, H 22 } Parallel gatekeeping testing procedure 1. Overall Type I error rate is no greater than (e.g., 0.05) 2. The null hypotheses in F 2 can be tested if at least one hypothesis in F 1 has been rejected [parallel testing] 3. Adjusted p-values for the hypotheses in F 1 do not depend on the p-values associated with the hypotheses in F 2 [primary analyses cannot depend on secondary analyses]

14. Slide 14 Closed testing Closed family of hypotheses Consider all 15 intersections of the null hypotheses –{H 11 or H 12 or H 21 or H 22 }, {H 11 or H 12 or H 21 }, etc Specify a test for each intersection hypothesis that controls the Type I error rate, e.g., Bonferroni test Establish implication relationships –Intersection hypothesis {H 11 or H 12 } implies H 11 and H 12, etc Tests for original hypotheses –Reject a null hypothesis if all intersection hypotheses implying it have been rejected

15. Slide 15 Decision matrix

16. Slide 16 Parallel gatekeeping strategy Two scenarios significant improvement in the mean number of ventilator- free days and 28-day all-cause mortality significant improvement in 28-day all-cause mortality but not in mean number of ventilator-free day Weighted analyses primary endpoints are unequally weighted to reflect their relative importance (likelihood of success) –weight=0.9 for number of ventilator-free days –weight=0.1 for 28-day all-cause mortality

17. Slide 17 Parallel gatekeeping strategy Scenario 1 significant improvement in the mean number of ventilator- free days and 28-day all-cause mortality Primary analyses: Bonferronis test Secondary analyses: Stepwise Holms test All primary and secondary findings are significant at 5% level

18. Slide 18 Parallel gatekeeping strategy Scenario 2 significant improvement in 28-day all-cause mortality but not in mean number of ventilator-free day Primary analyses: Bonferronis test Secondary analyses: Stepwise Holms test The primary mortality analysis and secondary quality of life analysis are significant at 5% level

19. Slide 19 Example 3: Dose-finding study Clinical trial in patients with hypertension Four doses of an experimental drug are compared to placebo –doses are labeled as D1, D2, D3 and D4 Primary endpoint –reduction in diastolic blood pressure Objectives of the study –find the doses with a significant reduction in diastolic blood pressure compared to placebo –study the shape of the dose-response curve

20. Slide 20 Example 3: Dose-finding study Step 1: Compare doses D3 and D4 to placebo Step 2: Compare doses D1 and D3 to placebo if at least one comparison at Step 1 is significant Step 3: Perform various pairwise dose comparisons if at least one comparison at Step 2 is significant A1: D4 vs. P A2: D3 vs. P A3: D2 vs. P A4: D1 vs. P Pairwise comparisons Family 1 (gatekeeper) Family 2 (gatekeeper) Family 3

21. Slide 21 Example 3: Dose-finding study Step 1: Compare doses D3 and D4 to placebo Step 2: Compare doses D1 and D3 to placebo if at least one comparison at Step 1 is significant Step 3: Perform various pairwise dose comparisons if at least one comparison at Step 2 is significant A1: D4 vs. P A2: D3 vs. P A3: D2 vs. P A4: D1 vs. P Pairwise comparisons Family 1 (gatekeeper) Family 2 (gatekeeper) Family 3

22. Slide 22 Example 3: Dose-finding study Step 1: Compare doses D3 and D4 to placebo Step 2: Compare doses D1 and D3 to placebo if at least one comparison at Step 1 is significant Step 3: Perform various pairwise dose comparisons if at least one comparison at Step 2 is significant A1: D4 vs. P A2: D3 vs. P A3: D2 vs. P A4: D1 vs. P Pairwise comparisons Family 1 (gatekeeper) Family 2 (gatekeeper) Family 3

23. Slide 23 Example 3: Dose-finding study

24. Slide 24 Parallel gatekeeping strategy Doses D2, D3 and D4 are significantly different from placebo at 5% level

25. Slide 25 Summary Gatekeeping strategies can be successfully used in pivotal trials with multiple primary and secondary endpoints dose-finding studies Registration trials a priori designation of gatekeeping strategy allows additional data useful to physician and patient to be presented in the product label Dose-finding studies efficient tests of dose-response relationship

26. Slide 26 Extensions More powerful gatekeeping tests based on more powerful tests, e.g., Simes test based on tests accounting for the correlation among the endpoints (exact parametric tests such as Dunnetts test and approximate resampling-based Westfall-Young tests) Software implementation SAS programs for gatekeeping tests can be found in Dmitrienko, Molenberghs, Chuang-Stein, Offen. (2004). Analysis of Clinical Trials: A Practical Guide. SAS Publishing, Cary, NC.

27. Slide 27 References Papers Dmitrienko, Offen, Westfall. (2003). Gatekeeping strategies for clinical trials that do not require all primary effects to be significant. Statistics in Medicine. 22, 2387- 2400. Marcus R, Peritz E, Gabriel KR. (1976). On closed testing procedure with special reference to ordered analysis of variance. Biometrika. 63, 655-660. Westfall, Krishen. (2001). Optimally weighted, fixed sequence, and gatekeeping multiple testing procedures. Journal of Statistical Planning and Inference. 99, 25-40.