ODAC May 3, Subgroup Analyses in Clinical Trials Stephen L George, PhD Department of Biostatistics and Bioinformatics Duke University Medical Center

2ODAC May 3, 2004 Definition of Subgroup Analysis An analysis of treatment effects within subgroups of patients enrolled on a clinical trial An analysis of treatment effects within subgroups of patients enrolled on a clinical trial

3ODAC May 3, 2004 Frequency of Subgroup Analyses Approximately 50% of reports of randomized clinical trials contain at least one subgroup analysis (Pocock et al 1987) Approximately 50% of reports of randomized clinical trials contain at least one subgroup analysis (Pocock et al 1987) Deciding on analysis after looking at the data is “dangerous, useful, and often done” (Good 1983) Deciding on analysis after looking at the data is “dangerous, useful, and often done” (Good 1983)

4ODAC May 3, 2004 Problems with Subgroup Analyses  Increased probability of type I error when H 0 true  Decreased power (increased type II error) in individual subgroups when H 1 true  Difficulty in interpretation

5ODAC May 3, 2004 General Assumptions in Clinical Trials Hypotheses tested usually address an overall or ‘average’ treatment effect in the study population Hypotheses tested usually address an overall or ‘average’ treatment effect in the study population No assumption of homogeneity of effect across subgroups No assumption of homogeneity of effect across subgroups Direction, not magnitude, of the treatment effect is expected be the same in subgroups Direction, not magnitude, of the treatment effect is expected be the same in subgroups

6ODAC May 3, 2004 Implications Overall treatment comparisons are of primary interest Overall treatment comparisons are of primary interest Stratification or regression techniques can be used to adjust the overall comparison for subgroups or covariates Stratification or regression techniques can be used to adjust the overall comparison for subgroups or covariates Subgroup analyses are generally of secondary interest as “hypothesis generating” techniques for future studies Subgroup analyses are generally of secondary interest as “hypothesis generating” techniques for future studies

7ODAC May 3, 2004 Pre-planned vs Unplanned Subgroup Analyses Pre-planned analyses (hypothesis driven) Pre-planned analyses (hypothesis driven) Subgroup hypotheses specified in advance Subgroup hypotheses specified in advance Control of error rates can, in principle, be addressed Control of error rates can, in principle, be addressed Unplanned analyses (exploratory) Unplanned analyses (exploratory) Analyses suggested by the data Analyses suggested by the data Exhaustive search for differential treatment effects by subgroups (data dredging) Exhaustive search for differential treatment effects by subgroups (data dredging) Inflated, and generally unknown, error rates Inflated, and generally unknown, error rates

8ODAC May 3, 2004 ICH Guideline E3 Statistical Considerations (Appendix) “… it is essential to consider the extent to which the analyses were planned prior to the availability of data…This is particularly important in the case of any subgroup analyses, because if such analyses are not preplanned they will ordinarily not provide an adequate basis for definitive conclusions.”

9ODAC May 3, 2004 ICH Guideline E9 5.7 Subgroups, Interactions and Covariates “In most cases…subgroup or interaction analyses are exploratory and should be clearly identified as such;…these analyses should be interpreted cautiously;…any conclusion of treatment efficacy (or lack thereof) or safety based solely on exploratory subgroup analyses are unlikely to be accepted.”

10ODAC May 3, 2004 Error Rates in Subgroup Analyses With k independent subgroups and no difference in treatments, the probability With k independent subgroups and no difference in treatments, the probability of at least one ‘significant’ subgroup is: of at least one ‘significant’ subgroup is: 1- (1- α) k For example, α = 0.05, k = 10 yields For example, α = 0.05, k = 10 yields 1- ( ) 10 = 0.40

11ODAC May 3, 2004

12ODAC May 3, 2004 Control of Error Rates in Subgroup Analyses For planned subgroup analyses, the overall type I error rate can be controlled. One conservative way is to use α * = α/k in each of the subgroup analyses For planned subgroup analyses, the overall type I error rate can be controlled. One conservative way is to use α * = α/k in each of the subgroup analyses In this case, the power (probability of detecting real differences when present) is sharply reduced in individual subgroups In this case, the power (probability of detecting real differences when present) is sharply reduced in individual subgroups For unplanned subgroup analyses, k is unknown so the error rates are unknown For unplanned subgroup analyses, k is unknown so the error rates are unknown

13ODAC May 3, 2004 Hypothetical Example Treatments: Experimental (E) and Control (C) Treatments: Experimental (E) and Control (C) Outcome: Overall survival Outcome: Overall survival Null median: 12 months Null median: 12 months Alt medians: 16 months (E) and 12 months (C) Alt medians: 16 months (E) and 12 months (C) 36 month accrual, 12 month followup, N = month accrual, 12 month followup, N = 500 α = 0.05, 1- β = 0.80 α = 0.05, 1- β = 0.80 Subgroups: 350 males (70%), 150 females Subgroups: 350 males (70%), 150 females

14ODAC May 3, 2004 Subgroup Tests (no α adjustment) Use α * = 0.05 in each subgroup Use α * = 0.05 in each subgroup Overall Type I error rate =.0975 Overall Type I error rate =.0975 Power in males ≈ 0.64, females ≈ 0.33 Power in males ≈ 0.64, females ≈ 0.33 Probability that correct conclusion is reached in both subgroups (males, females) under the alternative hypothesis ≈ (0.64)(0.33) ≈ 0.21 Probability that correct conclusion is reached in both subgroups (males, females) under the alternative hypothesis ≈ (0.64)(0.33) ≈ 0.21

15ODAC May 3, 2004 Subgroup Tests (adjusted α) Use α * = 0.05/2 = in each subgroup Use α * = 0.05/2 = in each subgroup Overall Type I error rate = Overall Type I error rate = Power in males ≈ 0.54, females ≈ 0.24 Power in males ≈ 0.54, females ≈ 0.24 Probability that correct conclusion is reached in both subgroups (males, females) under the alternative hypothesis ≈ (0.54)(0.24) ≈ 0.13 Probability that correct conclusion is reached in both subgroups (males, females) under the alternative hypothesis ≈ (0.54)(0.24) ≈ 0.13

16ODAC May 3, 2004 Aspirin Example A randomized trial of aspirin and sulfinpyrazone in threatened stroke. The Canadian Cooperative Study Group. N Engl J Med 299: 53-59, A randomized trial of aspirin and sulfinpyrazone in threatened stroke. The Canadian Cooperative Study Group. N Engl J Med 299: 53-59, “Among men the risk reduction for stroke or death was 48 per cent … whereas no significant trend was observed among women…We conclude that aspirin is an efficacious drug for men with threatened stroke.” “Among men the risk reduction for stroke or death was 48 per cent … whereas no significant trend was observed among women…We conclude that aspirin is an efficacious drug for men with threatened stroke.”

17ODAC May 3, 2004 Strokes or Deaths: Aspirin Study Aspirin No Aspirin Total Events Total Subjects Males Females Total Events

18ODAC May 3, 2004 Risk Reduction: Aspirin Study O/E Risk Reduction Χ2Χ2Χ2Χ2P-value Males % Females % %

19ODAC May 3, 2004 Antiplatelet Meta-analysis (1988) Secondary prevention of vascular disease by prolonged antiplatelet treatment. Antiplatelet Trialists' Collaboration. British Medical Journal 296: , Secondary prevention of vascular disease by prolonged antiplatelet treatment. Antiplatelet Trialists' Collaboration. British Medical Journal 296: , “Overall, allocation to antiplatelet treatment …reduced vascular mortality by 15% … and non-fatal vascular events (stroke or myocardial infarction) by 30% …” “Overall, allocation to antiplatelet treatment …reduced vascular mortality by 15% … and non-fatal vascular events (stroke or myocardial infarction) by 30% …”

20ODAC May 3, 2004 Guidelines for Assessing Reported Subgroup Differences (Oxman and Guyatt 1992) A priori hypotheses stated A priori hypotheses stated Clinical importance of the difference Clinical importance of the difference Proper assessment of statistical significance Proper assessment of statistical significance Consistency across studies Consistency across studies Indirect supporting evidence Indirect supporting evidence

21ODAC May 3, 2004 Treatment-Covariate Interactions: A Generalization of Subgroup Concepts A treatment-covariate interaction exists when the treatment effect is not the same for all values of a covariate (e.g., gender, age, etc.) A treatment-covariate interaction exists when the treatment effect is not the same for all values of a covariate (e.g., gender, age, etc.) Quantitative interactions: Treatment effects in the same direction, but of different magnitude in some subgroups (common and even expected) Quantitative interactions: Treatment effects in the same direction, but of different magnitude in some subgroups (common and even expected) Qualitative interactions: Treatment effects in opposite direction (rare) Qualitative interactions: Treatment effects in opposite direction (rare)

22ODAC May 3, 2004 Treatment-covariate Interactions Treatment X (0 for control, 1 for experimental) Treatment X (0 for control, 1 for experimental) Covariate Z (e.g., Z = 0 for female, 1 for male) Covariate Z (e.g., Z = 0 for female, 1 for male) Outcome Y = β 0 + β 1 X + β 2 Z + β 3 XZ Outcome Y = β 0 + β 1 X + β 2 Z + β 3 XZ ControlExperimental Trt Effect Female β0β0β0β0 β 0 + β 1 β1β1β1β1 Male β 0 + β 2 β 0 + β 1 + β 2 +β 3 β 1 + β 3 Gender Effect β2β2β2β2 β 2 + β 3

23ODAC May 3, 2004 Some Strategies Design for overall hypotheses but test within pre-defined subgroups: Design for overall hypotheses but test within pre-defined subgroups: High overall error rates High overall error rates Low power in subgroups Low power in subgroups Biased estimates Biased estimates Design for overall hypotheses but test for pre- specified treatment-covariate interactions: Design for overall hypotheses but test for pre- specified treatment-covariate interactions: Low power to detect interactions Low power to detect interactions

24ODAC May 3, 2004 Some Strategies (continued) Design for overall hypotheses and conduct unplanned (exploratory) analyses of subgroup differences: Design for overall hypotheses and conduct unplanned (exploratory) analyses of subgroup differences: Higher, but unknown, error rates Higher, but unknown, error rates Hypothesis generating exercise for future study Hypothesis generating exercise for future study Design for pre-specified subgroups or interactions: Design for pre-specified subgroups or interactions: Control of error rates Control of error rates Large sample sizes Large sample sizes

25ODAC May 3, 2004 Conclusions Pre-planning is key Pre-planning is key Larger studies required for proper subgroup analyses Larger studies required for proper subgroup analyses Exploratory analyses are good for hypothesis generating but are not convincing alone Exploratory analyses are good for hypothesis generating but are not convincing alone More than one study important for validation More than one study important for validation