1. Evidence-based decision making. How would you like that done, Minister ? Tony Ades RSS Avon Group, February 7 2008 Academic Unit of Primary Health Care, Department of Community Based Medicine

2. Outline If you were a Minister of Health, and DoH had a decision-making machine (DMM), what properties would you want it to have ? How is evidence used to make decisions ? Some examples (good and bad) Obstacles to building a DMM.

3. Evidence-Based Medicine response to increasing amounts of evidence, arbitrary / selective use of evidence. formal protocol inclusion / exclusion formal synthesis (meta-analysis) of studies Hierarchy of evidence Objective: to help clinicians make decisions

4. Evidence-based decision-making ? Decision Evidence

5. Evidence-based decision-making ? Decision Evidence ?

6. Evidence-based decision-making Model Costs Benefits Net Benefit Decision Evidence

7. Evidence-based decision-making …. Model Costs Benefits Net Benefit Decision Evidence … needs an evidence-based model

8. How are health care decisions made? 1. Estimate the Costs C j of each strategy, j costs of intervention, downstream treatment costs 2. Estimate the Benefits L j of each strategy quality-adjusted life years 3. Decide the value of an additional Life Year £20,000 - £30,000 4. Work out the Net Benefit of each strategy j NB j = L j. 20000 – C j 5. Choose strategy j*, with the highest Net Benefit

9. Desirable properties of a decision making machine Makes complete use of evidence Checks consistency of evidence Propagates uncertainty

10. Complete use of evidence 1.EBM-style Protocol for study inclusion / exclusion. Aim: all relevant evidence 2.Repeatability. ? > 95% overlap between different reviewers would help guarantee completeness 3.Wider inclusion: If parameters a and b are in a model, data on the product ab is relevant 4.But what counts as relevant ? Anything that reduces decision uncertainty is relevant

11. Consistency of evidence 1.All the studies informing the parameter a should agree. Pooling 2.The combined information on a, together with the combined information on b, should agree with the combined (independant) information on the product ab. External validation, triangulation, backward-propagation of evidence

12. Uncertainty propagation Uncertainty in parameters (or data) is propagated through the model to be reflected in uncertainty in the decision Average Net Benefit over the joint distribution of parameters θ: Choose strategy j* with highest Expected Net benefit. Probabilistic Decision Analysis, usually done by forward Monte Carlo Tells Decision Maker the probability the optimal decision is wrong Should be joint posterior distribution of parameters θ

13. What kind of machine can do this ? … 1.Complete use of evidence achieved through Bayesian or Likelihood methods. Likelihood quite difficult computationally 2.Consistency of evidence: detecting inconsistency is normal practice in statistics – but less so in decision modelling. Validation considered important, but formal inference seldom brought to bear 3.Uncertainty propagation : essentially Bayesian idea. Monte Carlo (MC) framework very convenient for decision modelling: hence simulation from a Bayesian posterior distribution. Or try to mimic through bootstrap ? Bayesian Markov chain MC, eg WinBUGS …very convenient

14. Completeness, consistency, uncertainty propagation: hardly controversial ! So, whats the problem ? Some modellers are not following this script Script not always easy to follow Understanding the data at all Responding to inconsistencies Inference for computationally complex models Next: Some examples of what is done badly, what can be done that is good, and what is still difficult to do well

15. TOPIC 1: Chlamydia screening (… not following the script) 1.Most common STI. Treatable, but often silent 2.Complications of chlamydia: Pelvic Inflammatory Disease, Ectopic Pregnancy, Infertility 3.=> Prima facie case for screening, But is it cost- effective ? 4.Two UK studies: ClaSS (pro-active), HPA (opportunistic). Both UK Government funded. 5.Each produced models for CEA. Epidemiology / transmission model, and a natural history model, embedded in a cost-effectiveness analysis.

16. Chlamydia complication rates – ClaSS study Pr(PID | chlamydia) based on Uppsala Study. GP registers with screening information linked with hospital registers of PID. Pr(PID | chlamydia) estimate based on all PID, not just those screened +ve. Therefore not linked to chlamydia incidence / prevalnce both Over –estimates as includes PID not caused by chlamydia And under –estimates as misses PID treated by GPs Use of this estimate assumes (i) chlamydia incidence same in UK and Sweden (ii) the over- and under-estimation biases are known to exactly cancel each other out. About 3.6%

17. Chlamydia complication rates – HPA study No single, evidence-based, estimate 3 Alternative scenarios: 1%, 10%, 30% But authors observe that 10% is around the maximum credible UK complication rate, based on 1.7% pa PID incidence observed in GPRD A deterministic sensitivity analysis. Useful to know Screening not cost-effective if complication rate < 10%....? ….. but not if we have no idea what the true rate is ! Ignores correlations between parameters …

18. Chlamydia complication rates: What is the evidence ? Prospective UK chlamydia prevalence p = 1.5 % (Women aged 15-44, NATSAL 2000) Duration of an episode d: 120 days (recent meta-analysis) => UK chlamydia Incidence h = p/d = 1.5 x 365.25/120 = 4.6 % pa Probability Pelvic Inflammatory Disease (PID) per episode of chlamydia : c = 0.15 (meta-analysis of prospective studies) => incidence of PID due to chlamydia: ch = 0.69% pa Retrospective UK Incidence of all PID (GPRD): ch/B = 1.7% pa, but not all PID caused by chlamydia. Proportion of All PID due to chlamydia (retrospective study) : B = 0.39 => incidence of PID due to chlamydia: ch = 0.66 % pa

19. Towards complete use of evidence 1.Meta-analyses of prevalence, duration. 2.Meta-analyses of prospective studies of PID 3.Several retrospective studies of PID 4.Similar prospective & Retrospective studies on other complications: ectopic pregnancy and infertility 5.Further prospective studies following PID forward to infertility and further retrospective studies looking back from infertility to PID. 6.Large network of evidence, to be assembled, synthesised, checked for consistency 7.BUT some difficulties interpreting literature, adjusting for biases, understanding recruitment into studies, taking test sensitivity / specificity into account …. etc, etc etc

20. TOPIC 2: How it should be done ! Antenatal screening for HIV: universal or targeted ? a = Proportion Sub-Saharan African (SSA) b = Proportion Injecting Drug Users (IDU) 1-a-b = Proportion Low-Risk c = HIV prevalence SSA d = HIV prevalence IDU e = HIV prevalence Rest (Low Risk) No direct evidence

21. Complete use of evidence: HIV in London, 1998 % Num Denom 1.Proportion Sub-Saharan African (SSA a 10.6 11044 104577 2.Proportion Injecting Drug Users (IDU) b 1.36 12 882 3.HIV prevalence SSA c 1.63 252 15428 4.HIV prevalence IDU d 2.11 10 473 5.HIV prevalence non-SSA [db + e(1-a-b)]/(1-a) 0.054 74 136139 Mixture of Low-risk & IDU: HIV prevalence is a weighted average of Low-Risk prevalence & IDU prevalence … provides information on e the Low-Risk prevalence

22. Complete use of evidence: another example value (%) Num Denom 1.Proportion Sub-Saharan African (SSA) a 10.6 11044 104577 2.Proportion Injecting Drug Users (IDU) b 1.36 12 882 3.HIV prevalence SSA c 1.63 252 15428 4.HIV prevalence IDU d 2.11 10 473 5.HIV prevalence non-SSA [db + e(1-a-b)]/(1-a) 0.054 74 136139 6.Overall HIV prevalence ac +db +e(1-a-b) 0.187 254 102287 Weighted average of all 3 risk groups …. Possibility of INCONSISTENCY

23. … including proportion already diagnosed in the HIV model…

24. … the larger data set … 12 independent data items : 9 parameters 1. proportion Sub-Saharan African (SSA) a 2. proportion Injecting Drug Users (IDU) b 3. HIV prevalence SSA c 4. HIV prevalence IDU d 5. HIV prevalence non-SSA [db + e(1-a-b)]/(1-a) 6. Overall HIV prevalence ac +db +e(1-a-b) 7. Diagnosed HIV in SSA : all diag HIV fca/[fca +gdb +he(1-a-b)] 8. Diagnosed HIV in IDU : non-SSA diag HIV gdb/[gdb + h(1-a-b)] 9. Overall proportion diagnd: [fca+gdb+ he(1-a-b)] / [ca+db+e(1-a-b)] 10. Proportion infected IDUs diagnosed g 11. Proportion serotype B in infected women (SS) w 12. Proportion serotype B in infected women (Non-SSA) [db/[db+e(1-a-b)]]+we(1-a-b)/[db+e(1-a-b)]

25. Uncertainty propagation from posterior Uncertainty in the data is propagated back to the model parameters, and then through the model to the decision. The decision is based on Expected Incremental Net Benefit, difference between Exp NB Universal and Targeted Screening. inb = 105000*(1-a-b) * (M * e * (1-h) - T*(1-e*h)) If Exp[INB] is +ve, choose Universal; if –ve, Targeted. M is the Net Benefit of diagnosing an Infected Mother T is the cost of an HIV test Number of pregnant women in London per year Proportion Low-Risk

26. Uncertainty propagation Probability INB +ve = 0.974 Choose universal with confidence e h inb

27. TOPIC 3: Inconsistency - its not so easy after all. 1. proportion Sub-Saharan African (SSA) a 2. proportion Injecting Drug Users (IDU) b 3. HIV prevalence SSA c 4. HIV prevalence IDU d 5. HIV prevalence non-SSA [db + e(1-a-b)]/(1-a) 6. Overall HIV prevalence ac +db +e(1-a-b) 7. Diagnosed HIV in SSA : all diag HIV fca/[fca +gdb +he(1-a-b)] 8. Diagnosed HIV in IDU : non-SSA diag HIV gdb/[gdb + h(1-a-b)] 9. Overall proportion diagnd: [fca+gdb+ he(1-a-b)] / [ca+db+e(1-a-b)] 10. Proportion infected IDUs diagnosed g 11. Proportion serotype B in infected women (SS) w 12. Proportion serotype B in infected women (Non-SSA) [db/[db+e(1-a-b)]]+we(1-a-b)/[db+e(1-a-b)] Highlighted items fit poorly

28. Information on b and information on d conflicting with information on bd 1. proportion Sub-Saharan African (SSA) a 2. proportion Injecting Drug Users (IDU) b 3. HIV prevalence SSA c 4. HIV prevalence IDU d 5. HIV prevalence non-SSA [db + e(1-a-b)]/(1-a) 6. Overall HIV prevalence ac +db +e(1-a-b) 7. Diagnosed HIV in SSA : all diag HIV fca/[fca +gdb +he(1-a-b)] 8. Diagnosed HIV in IDU : non-SSA diag HIV gdb/[gdb + h(1-a-b)] 9. Overall proportion diagnd: [fca+gdb+ he(1-a-b)] / [ca+db+e(1-a-b)] 10. Proportion infected IDUs diagnosed g 11. Proportion serotype B in infected women (SS) w 12. Proportion serotype B in infected women (Non-SSA) [db/[db+e(1-a-b)]]+we(1-a-b)/[db+e(1-a-b)]

29. How to deal with inconsistency ? DETECTION of inconsistency based on statistical methods But Identifying the SOURCE of inconsistency NOT a statistical issue: 1.Synthesise, check, detect, reconsider evidence. 2.then drop or bias adjust suspect data item. Post-hoc: danger of abuse. Final estimates will depend on which item is dropped or adjusted 1.Before synthesis, critically examine every item of information. 2.Elicit expert opinion on bias and relevance of each item 3.Devise bias priors for each item 4.Overall evidence synthesis Works well as a two-stage process in simple pair-wise meta-analysis (Turner & Spiegelhalter) Identifiability problems, convergence problems ……

30. How often is this done ? Real examples of Bayesian synthesis based on all available evidence ? 1.DoH estimates of HIV prevalence 2.DoH estimates of HCV prevalence 3.NHS / HTA report on antenatal screening for Early Onset neonatal Group B Strep 4.Several NICE analyses: mixed treatment comparisons. (Sort of)

31. TOPIC 4: Things we cannot yet do: Uncertainty propagation in STI models Infectious disease models have dynamic elements. (Lower incidence => lower prevalence => lower incidence) Compartmental models. Susceptible, Infected, Immune, states. Differential equations. Typically, no formal parameter estimation by ML or Bayesian methods. Instead iterative recalibration. ie tweaking. Scenerio-type sensitivity analysis. No propagation of uncertainty, no formal inference …

32. TOPIC 5: Things we cannot yet do: Uncertainty propagation in STI models STI models ESPECIALLY difficult. As well as the compartmental model, ALSO need to capture rate of formation / break-up of partnerships, numbers of concurrent partners. ClaSS and HPA used microsimulation models. For each set of parameter values, several thousand individuals form relationships, acquire chlamydia, develop complications etc, and are tracked over a period of years until they stabilise. Computer intensive. Neither ML nor Bayes solutions currently available: so no uncertainty propagation

33. Why Does This Matter Heroic & time consuming deterministic SA required. Conclusions like Screening not CE if PID rate < 10% not secure. (a) we need to know Pr(PID rate <10%) (b) Given data on PID incidence, same PID progression rate may be high if chlamydia prevalence incidence assumed high, or low if chlamydia incidence is high. - Given (indirect) data in chlamydia incidence, chalmydia prevalence highly correlated with chlamydia duration. But chlamydia duration parameter highly correlated with transmission rate (per sexual contact)…. Correlated with concurrency rate ….etc etc etc

34. Methods research needed How to formulate STI models in a form where they are tractable enough to use Bayesian MCMC OR Develop Bayesian computation methods so that they can accommodate current formulations of STI models.

35. Summary 1.We do know how what counts as evidence- based decision making : (completeness, consistency, uncertainty). eg NICE 2.Some people have not been trying hard enough ! 3.Not always easy. More research on: How to respond to inconsistent evidence How to do proper statistical inference (and so uncertainty propagation) with infectious disease models, especially STI models

36. Slides, papers, programs: http://www.bristol.ac.uk/http://www.bristol.ac.uk/cobm/ research/mpes Multi-Parameter Evidence Synthesis page: