Stopping Trials for Futility RSS/NIHR HTA/MRC 1 day workshop 11 Nov 2008

Programme Welcome Introduction Andy Grieve Carl-Fredrik Burman John Whitehead Questions Lunch Workshop discussions Feedback and discussions Summary and final voting

Workshops There are 4 workshop groups –2 in the main lecture theatre –1 in the Nightingale room –1 in the Council chamber All should have been allocated to a workshop group and location These discussions will be facilitated by –Jenny Freeman (University of Sheffield) –Jon Nicholl (University of Sheffield) –Louise Dent (HTA) –Andrew Cook (HTA)

Workshops Specific questions to be addressed are: –Should we plan to do futility analyses in publicly funded trials? –In what circumstances are they appropriate? –Should funding bodies insist upon them before granting trial extensions? Three case studies have been supplied to facilitate discussions

Futility analysis Jon Nicholl ScHARR, University of Sheffield

Publicly funded trials Publicly funded trials are supported by MRC/NIHR Early phase trials are supported by MRC, later phases by NIHR Phase I – IIa DPFS (MRC), Phase IIb – EME (MRC/NIHR), phase III/IV – HTA Trials are also supported by NIHR through RfPB, applied programme grants, etc

HTA HTA budget = £88m pa at maturity Nearly all to be spent on trials Typical trial design –Parallel group RCT of cost-effectiveness –2 arm head-to-head or 3 arm with placebo –Multi-centre –3-5 years (t s =0.5yr, t r =2yr, t f =1yr, t a =0.5yr) –£1m - £2.5m –N=

Futility designs Futility analyses are not uncommon in Pharma trials But no HTA funded trials have planned a futility analysis Why is this? Should futility analyses be designed into HTA trials? If so, in which trials? And when and how should they be done?

What is a futility analysis? A futility analysis is a calculation made during the course of a trial of the probability that the trial will produce helpful results A trial is judged futile if the probability is too small Helpful results are usually taken to mean ‘statistically significant’ results for the primary endpoint So the calculation usually takes the form of a power calculation conditional on the results observed.

Should HTA do futility analyses? YES Opportunity cost - spending public money on futile trials is wrong because the money could be spent on other trials It is ethically wrong to continue to recruit patients to trials with little hope of achieving helpful results NO All well conducted trials provide valuable evidence The costs of designing trials with a planned futility analysis would outweigh any savings

Futility analyses in HTA trials Assuming they are not ruled out in principle –In what types of trial –In what circumstances –When and how should we do them?

1. Types of trial In placebo controlled trials, futility suggests no worthwhile benefit (ie little or no chance of finding evidence that the intervention is better than placebo) –ie the treatment is futile as well as the trial In head-to-head trials this isn’t true –The treatment may be far from futile So are futility analyses inappropriate in head-to-head trials?

2. Extensions 54% of MRC/HTA trials seek extensions Usually due to long set up time and/or slow recruitment 55% of trials with extensions still don’t achieve N Cost of extensions = £50k - £1m

Extensions HTA currently asks for a ‘futility’ calculation “If the aim of the extension is to detect a difference between the interventions, please state the conditional power (two- sided 5% level) given the data so far, of detecting a currently plausible treatment effect”. Almost never done as intended or refused Is it appropriate to ask for a futility analysis in extension requests? Is this the right form for the futility calculation? What level of conditional power should be taken to mean its futile to continue?

3. When and how Futility analyses should be conducted in large expensive trials in which t r >> t f and in which early stopping could save substantial resources. A futility analysis should be conducted at the time that the expected resource savings are maximised Expected resource savings = (resources not spent – shut down costs) x (prob shut down) If a conditional power calculation is used, use a conservative rule for futility, eg conditional power < 20% Take other information into account before deciding on stopping or not extending

4. Unconditional power calculations Achieved sample sizes in 122 MRC/HTA trials (from Campbell et al, HTA 2007, 11 (48)) Achieved sample sizeProportion >= 100%31.1% 80% -23.8% <80%45.1%

We could design the trial with fixed T (and £C) and target sample size N, but variable achieved size n So instead of choosing N so that Prob (Z ≥ U | H ₁, N) = 1 - β N is chosen so that ∑ Prob (Z ≥ U | H ₁, n).Prob(n | N) = 1 - β Extensions for sample size would then be redundant Problem is this removes any incentive to maximise recruitment and increases N, T, and £C

Possible recommendations Build in a futility analysis when: N and £C are large There is a placebo arm T r >> T f Carry it out when the expected resources saved are maximised Use a conservative criterion for deciding futility, eg conditional probability <20% Always carry out futility analysis in extension requests unless potential savings < £?

Ask the audience

a.m. results: Should we plan to do futility analyses in publicly-funded trials? 1.Yes – for all appropriate trials (built into design) 2.Yes – but only for extension requests 3.Never 4.Undecided

a.m. results: In what circumstances are they appropriate? 1.Only in appropriate placebo-controlled trials 2.In both placebo- controlled and head-to- head trials 3.Undecided

a.m.results: Should funding bodies insist upon futility analyses before granting trial extensions? 1.Yes 2.No 3.Undecided

a.m. results: What is futile? How low can the conditional power be before stopping a trial? 1.1% 2.5% 3.10% 4.15% 5.20% 6.30% 7.40% 8.50% 9.It depends

p.m. results Should we plan to do futility analyses in publicly-funded trials? 1.Yes – for all appropriate trials (built into design) 2.Yes – but only for extension requests 3.Never 4.Undecided The percentage in the first category increased to about 72% (from 53%), as a result of the undecided making up their minds

p.m. results: In what circumstances are they appropriate? 1.Only in appropriate placebo-controlled trials 2.In both placebo- controlled and head-to- head trials 3.Undecided There was a slight shift in the undecided category into the first category

p.m.results: Should funding bodies insist upon futility analyses before granting trial extensions? 1.Yes 2.No 3.Undecided There was a shift from the undecided into the no category so that the numbers answering ‘yes’ and ‘no’ were both in the low 40s

p.m.results: What is futile? How low can the conditional power be before stopping a trial? 1.1% 2.5% 3.10% 4.15% 5.20% 6.30% 7.40% 8.50% 9.It depends The ‘It depends’ category increased from 72% to 84%

Thanks Jon Nicholl, for suggesting the idea The speakers: Andy Grieve, Carl-Fredrik Burman and John Whitehead for agreeing to speak Paul Gentry and the RSS for organising things here and providing the venue Louise Dent and Andrew Cook for facilitating the workshops MRC and HTA for funding the event