1. Cost-effectiveness models to inform trial design: Calculating the expected value of sample information Alan Brennan and J Chilcott, S Kharroubi, A O’Hagan

2. Overview Principles of economic viability 2 level Monte-Carlo algorithm & Mathematics Calculating EVSI (Bayesian Updating) case studies –Normal, Beta, Gamma distributions –Others – WinBUGS, and approximations. Illustrative and real example Implications Future Research

3. Example:- Economic viability of a proposed oil reservoir Some information suggesting there is oil Could do further sample drilling to “size” the oil reservoir Decision = “Go / No go” Criterion = expected profit (net present value or NPV) Is the sampling worthwhile? … that depends on … Costs of collecting the data Current uncertainty in reservoir size Expected gain from sampling = (P big reservoir*Big profits)+(P small reservoir*Big loss)–(Sample cost)

4. Analogies Drug Development Project –Go / No go decisions –Trial supports consideration of next decision (Phases to launch) –Criterion = Expected profit (NPV) –Correct decision  profit if good drug, avoided financial loss if not a good drug NICE / NCCHTA decision –Approval or not –Is additional research required before decision can be made –Criterion = Cost per QALY…. i.e. net health benefits –Correct decision  better health (efficiently) if good drug, avoided poor health investment if not a good drug

5. Principles Strategy options with uncertainty about their performance Decision to make Sampling is worthwhile if Expected gain from sampling - expected cost of sampling > 0 Expected gain from sampling = Function (Probability of changing the decision|sample,. amount of gain made / loss avoided) Applies to all decisions

6. Algortihm

7. 2 Level EVSI - Research Design 4, 5 0)Decision model, threshold, priors for uncertain parameters 1) Simulate data collection: sample parameter(s) of interest once ~ prior decide on sample size (n i ) (1st level) sample a mean value for the simulated data | parameter of interest 2) combine prior + simulated data --> simulated posterior 3) now simulate 1000 times parameters of interest ~ simulated posterior unknown parameters ~ prior uncertainty (2 nd level) 4) calculate best strategy = highest mean net benefit 5) Loop 1 to 4 say 1,000 times Calculate average net benefits 6) EVSI parameter set = (5) - (mean net benefit | current information)

8. Mathematics

9. 2 Level EVSI - Mathematics 4, 5 Mathematical Formulation: EVSI for Parameters  = the parameters for the model (uncertain currently). d = set of possible decisions or strategies. NB(d,  ) = the net benefit for decision d, and parameters  Step 1: no further information (the value of the baseline decision) Given current information chose decision giving maximum expected net benefit. Expected net benefit (no further info) =(1)  i = the parameters of interest for partial EVPI  -i = the other parameters (those not of interest, i.e. remaining uncertainty) 4 Brennan et al Poster SMDM 2002 5 Brennan et al Poster SMDM 2002

10. 2 Level EVSI - Mathematics 4, 5 Step 6: Sample Information on  i Expected Net benefit, sample on  i =(6) Step 7: Expected Value of Sample Information on  i (6) – (1) Partial EVSI = (7) This is a 2 level simulation due to 2 expectations 4 Brennan et al Poster SMDM 2002 5 Brennan et al Poster SMDM 2002

11. Bayesian Updating Normal Beta Gamma

12. Normal Distribution

13.  0 = prior mean for the parameter  0 = prior uncertainty in the mean (standard deviation) = precision of the prior mean  2 pop = patient level uncertainty from a sample ( needed for Bayesian update formula) = sample mean (further data collection from more patients / clinical trial study entrants). = precision of the sample mean. = sample variance 4 Brennan et al Poster SMDM 2002 5 Brennan et al Poster SMDM 2002

14. Normal Distribution = implied posterior mean (the Bayesian update of the mean following the sample information) = implied posterior standard deviation (the Bayesian update of the std dev following the sample information)

15. Normal Distribution - Implications Implied posterior variance will always be smaller than the prior variance because the denominator of the adjustment term is always larger than the numerator. If the sample size is very small then the adjustment term will almost be equal to 1 and posterior variance is almost identical to the prior variance. If the sample size is very large, the numerator of the adjustment term tends to zero, the denominator tends to the prior variance and so, posterior variance tends towards zero.

16. Normal Distribution

17. Beta Distribution

18. Beta / Binomial Distribution e.g. % responders Suppose prior for % of responders is ~ Beta (a,b) If we obtain a further n cases, of which y are successful responders then Posterior ~ Beta (a+y,b+n-y)

19. Gamma Distribution

20. Gamma / Poisson Distribution e.g. no. of side effects a patient experiences in a year Suppose prior for mean number of side effects per person is ~ Gamma (a,b) If we obtain a further n samples, (y 1, y 2, … y n ) from a Poisson distribution then Posterior for mean number of side effects per person ~ Gamma (a+  y i, b+n)

21. Bayesian Updating Other Distributions

22. Other Distributions

23. Bayesian Updating without a Formula WinBUGS Put in prior distribution Put in data (e.g. sample of patients or parameter) Use MCMC to generate posterior (‘000s of iterations) Use posterior in model to generate new decision Loop round and put in a next data sample Other approximation methods (talk to Samer!)

24. Illustrative Model

25. First (Illustrative) Model 2 treatments – T1 versus T0 Criterion = Cost per QALY < £10,000 Uncertainty in …… % responders to T1 and T0 Utility gain of a responder Long term duration of response Other cost parameters

28. Illustrative Model Results Baseline strategy = T1 Cost per QALY = £5,267 Overall EVPI = £1,351 per person

29. EVSI for Parameter Subsets

31. Illustrative data collection cost = £100k fixed plus £500 marginal Expected Net Benefit of Sampling

32. Real Example

33. Second Example Pharmaco-genetic Test to predict response Rheumatoid Arthritis Up to 20 strategies of sequenced treatments U.S. - 2 year costs and benefits perspective Criterion = Cost per additional year in response Range of thresholds ($10,000 to $30,000) Real uncertainty (modelled by Beta’s)

34. “Biologics” Anakinra ($12,697), Etanercept ($18,850), Infliximab ($24,112)* Is Response Genetic? 91 patients, 150mg Anakinra, 24 week RCT 1,2, gene = IL-1A +4845 Positive response = reduction of at least 50% in swollen joints 1 Camp et al. American Human Genetics Conf abstract 1088, 1999 2 Bresnihan Arthritis & Rheumatism, 1998 * Costs include monitoring Anakinra 100mg Etanercept 25mg eow Infliximab 3mg/kg 8 weekly 50% 100%

35. A Pharmaco-Genetic Strategy Strategy 1 Strategy 2

36. Partial EVSI: PGt Research only Caveat: Small No.of Simulations on 1st Level

37. Doing Fewer Calculations?

38. Properties of the EVSI curve Fixed at zero if no sample is collected Bounded above by EVPI Monotonic Diminishing return Suggests perhaps exponential form? Tried with 2 examples – fitted curve is exponential function of the square root of n

39. Fitting an Exponential Curve to EVSI: Illustrative Model - % response to T0

40. Fitting an Exponential Curve to EVSI: Pharmaco-genetic Test response

41. Unresolved Question Does the following formula always provide a good fit? EVSI (n) = EVPI * [1 – exp -a*sqrt(n) ] The 2 examples are Normal and Beta Is it provable by theory?

42. Discussion Issues Phase III trials Future Research Agenda

43. Discussion Issues – Phase III trials Based on proving a clinical DELTA Implication is that if clinical DELTA is shown then adoption will follow i.e. it is a proxy for economic viability Often FDA requires placebo control (lower sample size), which implies DELTA versus competitors is unproven Could consider economic DELTA …….

44. Discussion Issues – Phase III trials Early “societal” economic models provide a tool for assessing: 1.What would be an economic DELTA? 2.Implied sample needed in efficacy trial for cost-effectiveness 3.What other information is needed to prove cost-effectiveness? 4.Will proposed clinical DELTA be enough for decision makers Similar commercial economic models could link proposed data collection with probability of re-imbursement and hence with expected profit (NPV)

45. Discussion Issues – Problems & Development Agenda 1.Technical - Bayesian Updating for other distributions 2.Partnership and case studies - to develop Bayesian tools for researchers who currently use frequentist only sample size calculation 3.Methods for complexity in Bayesian updating - e.g. the new trial will have slightly different patient group to the previous trial (meta-analysis and adjustment)

46. Conclusions Can now do EVSI calculations from a societal perspective using the 2 level Monte-Carlo algorithm Bayesian Updating works for case studies –Normal, Beta, Gamma distributions –Others need – WinBUGS, and/or approximations. Future Research Issues –Bayesian Technical –Collaborative Issues with Frequentist Sample Size

47. Thankyou