1. 1 Interim Analysis in Clinical Trials Professor Bikas K Sinha [ ISI, KolkatA ] e-mail : sinhabikas@yahoo.comsinhabikas@yahoo.com RU Workshop : April18, 2012

2. Books/Papers/Lecture Notes….. Sample Size Calculations in Clinical Research By Shein-Chung Chow/Jun Shao/ Hansheng Wang Interim Analysis deals with Early Stopping Rules in Testing of Hypotheses problems involving Comparison of Efficacies of Test & Standard Drugs 2

3. 3 An Example of “Multiple Looks:” Consider planning a comparative trial in which two treatments are being compared for efficacy (response rate). H 0 : p 2 = p 1 H 1 : p 2 > p 1 A standard design says that for 80% power and with alpha of 0.05, you need about 100 patients per arm based on the assumption p 2 = 0.50, p 1 = 0.30 which results in 0.20 for the difference. So what happens if we find p < 0.05 before all patients are enrolled ? Why can’t we look at the data a few times in the middle of the trial and conclude that one treatment is better if we see p < 0.05?

4. 4 The plots to the right show simulated data where p 1 = 0.40 and p 2 = 0.50 In our trial, looking to find a difference between 0.30 to 0.50, we would not expect to conclude that there is evidence for a difference. However, if we look after every 4 patients, we get the scenario where we would stop at 96 patients and conclude that there is a significant difference.

5. 5 If we look after every 10 patients, we get the scenario where we would not stop until all 200 patients were observed and would conclude that there is not a significant difference (p =0.40)

6. 6 If we look after every 40 patients, we get the scenario where we would not stop either. If we wait until the END of the trial (N = 200), then we estimate p 1 to be 0.45 and p 2 to be 0.52. The p-value for testing that there is a significant difference is 0.40.

7. 7 Would we have messed up if we looked early on? Every time we look at the data and consider stopping, we introduce the chance of falsely rejecting the null hypothesis. In other words, every time we look at the data, we have the chance of a type 1 error. If we look at the data multiple times, and we use alpha of 0.05 as our criterion for significance, then we have a 5% chance of stopping each time. Under the true null hypothesis and just 2 looks at the data, then we “approximate” the error rates as:  Probability stop at first look: 0.05  Probability stop at second look: 0.95*0.05 = 0.0475 Total probability of stopping is 0.0975

8. Illustrative Examples :Interim Analysis Example 1. It is desired to carry out an experiment to examine the superiority, or otherwise, of a thera- peutic drug over a standard drug with 5% level and 90% power for detection of 10% difference in the proportions ‘cured’. ‘C’ : Standard Drug ‘T’ : Therapeutic Drug H_0 : P_C - P_T = 0 H_1 : P_C # P_T Size = 0.05, Power = 0.90 for ∆=P_T – P_C = 0.10. IT IS A BOTH-SIDED TEST. 8

9. Sample Size Determination…. Reference : Pages 22/23/30 of Last Reference Ref: Lachin, Controlled Clinical Trials 2:93-113, 1981. 9

10. Determination of Sample Size for Full Analysis 10 Two-sided Test α = 0.05; Z_ α /2 = 1.96 Power = 0.90; β = 0.10, Z_β = 1.282, ∆ =0.10 N = 2(Z_ α /2 + Z_ β)^2 pbar(1-pbar)/∆^2 Assume pbar = 0.35 [suggestive cure rate] N = 2(1.96 + 1.282)^2 (0.35)(0.65)/(0.10)^2 = 21.021128 x 22.75= 478.23……480 Conclusion: Each arm involves 480 subjects.

11. Full Experiment vs. Interim Analysis For Full Experiment : Needed 480 subjects in each ‘arm’. At the end of the entire experiment, suppose we observe : ‘C’ : # cured = 156 out of 480 i.e., 32.5% ‘T’ : # cured = 190 out of 480 i.e., 39.6% Therefore, p^_C = 0.325 and p^_T = 0.396. Hence, pbar = [p^_C + p^_T]/2 = 0.3605. Finally, we compute the value of z given by 11

12. Full Analysis….. Z_obs. = [p^_C – p^_T]/sqrt[pbar(1-pbar)2/N] =[.325-.396]/sqrt[.36x.64x2/480] = -[.071]/sqrt[0.00192] = -2.29 In absolute value, z_obs. is computed as 2.29 which is more than the ‘critical’ value of z given by 1.96 [for a both-sided test with size 5%]. Hence, we conclude that the Null Hypothesis is ‘not tenable’, given the experimental outputs. 12

13. Interim Analysis : 2 ‘Looks’ First Look : use 50% of data 2nd Look : At the end, if continued after 1st. Q. What is the size of the test at 1 st look ? Also, what is the size at the 2 nd look so that on the whole the size is 5 % ? Ans. If we use 5% for the size at each of 1 st and 2 nd looks, then the over-all size becomes 0.08 approx. [Proof ?] Hence……both can NOT be taken at 5%. Start with 5%..... 13

14. Interim Analysis : 2 Looks Defining Equation : α = P[│Z_I │> z*] + P[│Z_I│ z**] where Z_I and Z_II are based on 50% data in two identical and independent segments so that their distributions are identical. Further, Z_{I,II} = [z_I + z_II]/sqrt(2) is based on combined evidence of I & II and hence Z_I and Z_{I,II} are dependent. Choices of z* and z** : intricate formulae. 14

15. Interim Analysis : 2 Looks Z-computation…. z_I obs. is to be based on 50% data upto the 1 st look for each of ‘C’ and ‘T’. Data : C (90/240) & T(120/240) & n = 240. p^_C = 90/240 = 0.375; p^_T = 120/240=0.50 pbar = (0.375 + 0.50)/2 = 0.4375. z_I obs. = [p^_C – p^_T]/sqrt[pbar(1-pbar)2/n] = - [ 0.125 ]/sqrt{.4375x.5625x2/240} = - (0.125)/sqrt{0.002050} = - 2.76 implies ??? 15

16. Interim Analysis : 2 Looks 16 Suggested cut-off points for 2 Looks [approx.] z_c Hebittle-Peto Pocock O’Brien-Fleming z* 3.0 2.178 2.796 z** 2.0 2.178 1.977 z_I obs. in absolute value = 2.76 Conclusion ? Reject H_0 ….suggested by Pocock’s Rule Continue …suggested by other two. Finally, z = - 2.29 suggests rejection of H_0 by all the three rules

17. Interim Analysis : 3 looks Cut-off points : Suggested Rules [approx.] z_c Hebittle-Peto Pocock O’Brien- Fleming z* 3.0 2.289 3.471 z** 3.0 2.289 2.454 z*** 2.0 2.289 2.004 : 1 st look; ** : 2 nd look; *** : 3 rd look 17

18. Interim Analysis : 4 Looks Cut-off points : Suggested Rules z_c Hebittle-Peto Pocock O’Brien-Fleming z* 3.0 2.361 4.106 z** 3.0 2.361 2.903 z*** 3.0 2.361 2.370 z**** 2.0 2.361 2.053 : 1 st look; ** : 2 nd look; *** : 3 rd look and **** : last [4 th ] look 18

19. Interim Analysis : 4 Looks Details of data sets : C : 48/120; 42/120; 30/120; 36/120 …Total 156/480 T : 54/120; 66/120; 32/120; 38/120 …Total 190/480 Progressive proportions for ‘C’ : 48/120=0.40; (48+42)/240= 0.375; (48+42+30)/360=0.333; 156/480=0.325 Progressive proportions for ‘T’ : 54/120=0.45; (54+66)/240= 0.50; (54+ 66+32)/360=0.422; 190/480=0.396 19

20. Interim Analysis : 4 Looks Progressive computations of pbar…… 1 st Look : pbar = (0.40 + 0.45)/2 = 0.425 2 nd Look : pbar = (0.375 + 0.50)/2 = 0.4375 3 rd Look : pbar = ( 0.333 + 0.422)/2 = 0.3639 4 th Look : pbar = (0.325 + 0.396)/2 = 0.3605 20

21. Interim Analysis : 4 Looks Progressive Computations of z-statistic Generic Formula : z-obs. for ‘Look # i’ is the ratio of (a) [p^_C(i)– p^_T(i)] for i-th Look (b) sqrt[pbar(i)(1-pbar(i))2/n(i)] where pbar(i) corresponds to Look # i and also ‘n(i) ’ corresponds to size of each arm of Look # i for each i = 1, 2, 3,4. Note : n(1)=120; n(2)=240; n(3)=360, n(4)=480 21

22. Interim Analysis : 1st Look z_(Look I) obs. = [p^_C – p^_T]/sqrt[pbar(1-pbar)2/n*] = [ 0.40-0.45 ]/sqrt{.425x.575x2/120} = - (0.05)/sqrt{0.004073} = -0.7835 Conclusion : All Rules are suggestive of Continuation to 2 nd Look 22

23. Interim Analysis : 2 nd Look z_(Look II) obs. = [p^_C – p^_T]/sqrt[pbar(1-pbar)2/n**] = [0.375-0.50 ]/sqrt{.4375x.5625x2/240} = - (0.125)/sqrt{0.002050} = - 2.76 Conclusion : Reject H_0 by Pocock’s Rule However, continue to 3 rd Look according to the other two rules. 23

24. Interim Analysis : 3 rd Look z_(Look III) obs. = [p^_C – p^_T]/sqrt[pbar(1-pbar)2/n***] = [0.333-0.422 ]/sqrt{.3639x.6361x2/360} = - (0.089)/sqrt{0.001286} = - 2.48 Conclusion : Reject H_0 by Pocock & OBF Rules but Continue by H-P Rule Last Look : z_obs. = -2.29 Accept H_0 by Pocock’s Rule only 24

25. Data Analysis….Interpretations Decision Rules : Pocock’s Rule : Maintains uniformity in critical values ….so …apparently ‘conservative’ at the start…slowly turns into ‘liberal’ ! Other Rules : Liberal at the start and conservative at the end….. All Rules have to maintain the ‘averaging principle’ to meet alpha at the end. No Rule can be strict/liberal all through the Looks. 25

26. Interim Analysis : Example 2 Continuous data : Testing for equality of mean effects of two treatments : ’C’ & ’T’. As before, we have Null and Alt. Hypotheses and we have a specified value of DELTA = Mean of T – Mean of C and a specified power, say 90% to detect this. Taking size equal to 5%, we solve for the sample size in each arm. This is routine computation and we take sample size N = 525 in each arm.

27. Full Analysis : Sample Size Computation Assume normal distribution with sigma = 5. Two-sided Test α = 0.05; Z_ α /2 = 1.96 Power = 0.90; β = 0.10, Z_β = 1.282, ∆ = 0.20 times sigma = 20% of sigma = 1.0 N = 2(Z_ α /2 + Z_ β)^2 x sigma^2 /∆^2 = 2(1.96 + 1.282)^2 / 0.04 = 525 [approx.] We can think of 5 Looks altogether…at equal Steps…..each with approx. 105 observations.

28. Interim Analysis…Example contd. Details of data sets : (mean, sample size) C : (30.5,105); (31.8, 105); (29.7, 105); (30.2, 105); (31.3, 105) T : (31.7,105); (32.0, 105); (30.8, 105); (33.7, 105); (32.8, 105) Progressive sample means for ‘C’ : 30.5, 31.15, 30.67, 30.55, 30.70 Progressive sample means for ‘T’ : 31.7, 31.85, 30.83, 32.55, 32.60

29. Interim Analysis : Example contd…. Progressive Computations of z-statistic Generic Formula : z-obs. for ‘Look # i’ is the ratio of (a) [mean_C(i)– mean_T(i)] for i-th Look (b) sigma times Sqrt 2/n(i)] where mean refers to sample mean for and also ‘n(i) ’ corresponds to size of each arm of Look # i for each i = 1, 2, 3,4, 5. Note : n(1)=105; n(2)=210; n(3)=315, n(4)=420 and n(5) = 525.

30. Interim Analysis : Example contd. Cut-off points : Suggested Rules z_c Hebittle-Peto Pocock O’Brien-Fleming z* 3.0 2.413 4.562 z** 3.0 2.413 3.225 z*** 3.0 2.413 2.634 z**** 3.0 2.413 2.281 z***** 2.0 2.413 2.040 : 1 st look; ** : 2 nd look; *** : 3 rd look; **** : 4 th look & ***** : Last [5th] look

31. Interim Analysis…Example contd. z_(Look I) obs. = [mean_C – mean_T]/sigma x sqrt[2/n*] = - [ 1.2] / 5 x sqrt{2/105} = - 1.74 Conclusion : Continue to 2 nd Look

32. Interim Analysis : Example contd. z_(Look II) obs. = [mean_C – mean_T]/sigma x sqrt[2/n**] = - [ 0.7 ] / 5 x sqrt{2/210} = - 1.43 Conclusion : Continue to 3rd Look

33. Interim Analysis : Example contd. z_(Look III) obs. = [mean_C – mean_T]/sigma x sqrt[2/n***] = - [ 0.16 ] / 5 x sqrt{2/315} = - 0.40 Conclusion : Continue to 4th Look

34. Interim Analysis : Example contd. z_(Look IV) obs. = [mean_C – mean_T]/sigma x sqrt[2/n****] = - [ 2.0 ] / 5 x sqrt{2/420} = - 5.80 Conclusion : Stop and Reject H_0. Strong evidence against H_0 and yet 105 observations per arm are left to be studied. What if the expt was continued till the end anyway ?

35. Interim Analysis : Example contd. z_(Look V) obs. = [mean_C – mean_T]/sigma x sqrt[2/n*****] = - [ 1.90 ] / 5 x sqrt{2/525} = - 6.16 Conclusion : Reject H_0. Quite a strong evidence against H_0

36. Open Problems….. 1. Computations of Boundaries for Interim Analysis [IA] :Theory & Computational Algorithms for Different ‘Looks’ 2. Comparison of Different IA Rules ? 3. New Simpler Rules ? 4. Power Performance Within/Between Rules ? 5. Sample Size Adjustment : Why & How ? 6. Miscellaneous Topics [Alpha Spending Functions etc] 36