1. Stopping Trials for Futility Ranjit Lall (May 2009)

2. OVERVIEW Summary of a 1 day workshop on “Stopping Trial for Futility” Funded by the RSS/NIHR HTA/MRC 11 th November 2008 in London Speakers:  Jon Nicholl (ScHARR, University of Sheffield, Chair person)  John Whitehead (Lancaster University)  Andy Grieve (King’s College London)  Karl-Fredrik Burman (Director of Statistical Science, Astra Zeneca)

3. WHAT IS FUTILITY? The term ‘futility’ is used to refer to the inability of the trial to achieve its objectives On-going monitoring of trials for safety, treatment efficacy or futility Early interims – review will focus more on safety, quality of conduct and trial integrity rather than efficacy. Later interims - may include formal efficacy or futility analysis

4. WHAT IS FUTILITY? A futility analysis causes a clinical trial to be stopped as soon as it becomes clear that a negative outcome is inevitable and thus it is no longer worthwhile continuing the trial to its completion.

5. PUBLICLY FUNDED TRIALS Early phase trials are supported by MRC Later phase trials supported by HTA HTA budget = £88 million pa Nearly all spent on trials Typical design of phase III trials – parallel group RCT, multi-centre, 3-5 years, £1million - £2.5 million, N = 400- 4000

6. FUTILITY DESIGNS Futility analyses are common in pharmaceutical trials NO HTA funded trials have planned a futility analysis Questions: - Why is this? - Should futility analyses be designed into HTA trials? - If so, which trials? - And when and how should they be done?

7. Should HTA do futility analyses? YES – (i) As pointed out by Ware, Muller and Braunwald (1985), “……………..early termination for futility could reduce enormous expenditure of resources, human and financial, involved in the conduct of trials that ultimately provide a negative answer regarding the value of the medical innovation….”. (ii) It is ethically wrong to continue to recruit patients to trials with little hope of achieving helpful results

8. Should HTA do futility analyses? NO – (i) All well conducted trials provide valuable evidence (ii) The cost of designing trials with a planned futility analysis would outweigh any savings

9. Should HTA do futility analyses? IT DEPENDS – (i)In placebo controlled trials, futility suggests no worthwhile benefit (i.e. little chance of finding evidence that the intervention is better than placebo) (ii)In head-to-head trials, treatment will be far from futile (so is futility analyses inappropriate in head-to –head trials)? (iii)Futility analyses should be conducted in large expensive trials in which early stopping could save substantial resources (iv)Futility analysis most useful where there are several treatments being compared, and there is a desire to eliminate one (for better resource use)

10. Extension grants 54% of MRC/HTA trials seek extensions Usually due to long set up/slow recruitment 55% of trials with extensions still don’t achieve N Cost of extensions for HTA = £50K - £1 million Current HTA ask for a ‘futility analysis’ in terms of conditional power on application. Almost never done as intended or refused

11. Fixed Interim analyses Focus Type I error (i.e. Rejecting the H 0 when its true) Methods used include - Pocock (1977) - O’Brien and Fleming (1979) - Haybrittle and Peto (1976) - Lans and DeMets alpha spending function (1983, 1989)

12. Futility Analysis (cont’d) Two fundamental approaches: (i)Using the above group sequential methods to compute the stopping boundaries for futility (in a similar way to computing stopping boundaries for efficacy) The boundaries can be defined to allow for early stopping for futility only, or to allow for early stopping for either futility or efficacy

13. Futility Analysis (cont’d) (ii) Conditional power The power of a trial tells whether a clinical trial is likely to have high probability to detect a pre-defined treatment effect of interest. Very low power implies that a trial is unlikely to reach statistical significance even if there is a true effect. One should never begin a trial with low power. However, sometimes low power becomes apparent only after a trial is well under way.

14. Futility Analysis (cont’d) (ii) Conditional power: based on the probability of a getting a statistically significant final result, conditional on the data collected so far (in the interim) Stochastic curtailment – a frequentist method Predictive power – partially Bayesian methods Predictive probability – fully Bayesian methods

15. Futility Analysis (cont’d) (a) Stochastic curtailment: At planning stages, sample size based on (a) Difference - d; (b) power (80% or 90%); (c) type I error rate At the interim, recalculate the power to detect a difference d. Calculations involve the observed data and an assumption regarding the distribution of the future data This method assumes that the observed difference between the treatments at the interim stage is the true treatment difference, d.

16. Futility Analysis (cont’d) (a) Stochastic curtailment: Detect e.g.<20% power no point continuing; Detect e.g.>50% power then continue. This cut-off is pre-specified (choice would depend on the risk/benefit considerations) Take other information into account before deciding on stopping or not extending.

17. Futility Analysis (cont’d) (b) Predictive power: Conditional power calculations use the observed data and an assumed future treatment effect Not sure what assumptions to make Predictive power approach addresses this by averaging the conditional power function over a range of treatment difference, with weights based on the observed treatment effect

18. Futility Analysis (cont’d) (b) Predictive probability: Prior probability: based on the interim data and the prior distribution of the parameter for the treatment effect; Statistical model: mathematical model which expresses the relationship of the parameters and the data Posterior probability: Probability of a clinically important treatment effect

19. Futility Analysis (cont’d) If early results show: Intervention is better than expected conditional power will be high Intervention is worse that expected conditional power will be low (unless your sample size increases)

20. Futility Analysis (cont’d) The two most challenging aspects: the selection of optimal stopping thresholds; the timing of the analysis both of which require the balancing of various risks.

21. Ask the audience (questions for the workshop)

22. Should we plan to do futility analysis in publicly- funded trials ? 0 5 1.Yes – for all trials (built into design) 2.Yes – but only for extension requests 3.Never 4.Undecided

23. Should we plan to do futility analysis in publicly- funded trials? 1.Yes – for all trials (built into design) 1.Yes – but only for extension requests 1.Never 2.Undecided

24. Should we plan to do futility analysis in publicly- funded trials? May not be appropriate for all trials, but justification for doing/not doing should be clearly laid out in the application Should be made as a requirement by funders (and the funders should clearly lay out what is required by them) Clear distinction needs to be made between trial futility (e.g. lack of recruitment-operational aspect) and treatment futility (lack of efficacy - statistical)

25. In what circumstances are they appropriate? 1. Only in placebo- controlled trials 2.In both placebo- controlled and head- to head trials 3.Undecided 0 5

26. In what circumstances are they appropriate? 1.Only in placebo-controlled trials 2.In both placebo- controlled and head to-head trials 3.Undecided

27. In what circumstances are they appropriate? It may be difficult to predict in advance Futility analyses may not be very appropriate for very large pragmatic trials, where other information (other than efficacy) is important

28. Should funding bodies insist upon futility analyses before granting extensions? 1.Yes 2.No 3.Undecided 0 5

29. Should funding bodies insist upon futility analyses before granting extensions? 1. Yes 2. No 3. Undecided

30. Should funding bodies insist upon futility analyses before granting extensions? Should not be insisted upon, but if investigator refuses to do futility then justification should be given

31. What is futile? How low can the conditional power be before stopping a trial 1.1% 2.5% 3.10% 4.15% 5.20% 6.30% 7.40% 8.50% 9.It depends 0 5

32. What is futile? How low can the conditional power be before stopping a trial? 1.1% 2.5% 3.10% 4.15% 5.20% 6.30% 7. 40% 8.50% 9.It depends

33. What is futile? How low can the conditional power be before stopping a trial? Futility analysis should be focused on effect size (rather than conditional power) A trial may be considered as futile in its ability to recruit patients, but would add information to a evidence based data

34. Thank you !!