John Edmunds, Jess Metcalf, Justin Lessler john.edmunds@lshtm.ac.uk Challenges in using mathematical modelling for public health decision-making John Edmunds, Jess Metcalf, Justin Lessler john.edmunds@lshtm.ac.uk

Why use models? Observational studies of impact are retrospective, and difficult to interpret Intervention studies (trials) relatively small & key endpoints rarely observed, therefore models used for extrapolation Other endpoints (e.g. deaths) Other population groups (perhaps excluded from study) Over time To others in the population Synthesise the results of different studies in a unified framework Improve our understanding of key drivers Identify and quantify knowledge gaps

Why use models? Observational studies of impact are retrospective, and difficult to interpret Intervention studies (trials) relatively small & key endpoints rarely observed, therefore models used for extrapolation Other endpoints (e.g. deaths) Other population groups (perhaps excluded from study) Over time To others in the population Synthesise the results of different studies in a unified framework Improve our understanding of key drivers Identify and quantify knowledge gaps Synthesis & prediction Understanding, research and surveillance

Main role of mathematical modelling in UK infectious disease policy Epidemics Contingency planning Risk assessment Epidemic forecasting (& “now-casting”) Impact of interventions “Endemic” diseases Vaccine policy Other control policy (e.g. Screening and treatment) Burden of disease assessment (priority setting)

Main role of mathematical modelling in UK infectious disease policy Epidemics Contingency planning Risk assessment Epidemic forecasting (& “now-casting”) Impact of interventions “Endemic” diseases Vaccine policy Other control policy (e.g. Screening and treatment) Burden of disease assessment (priority setting) Impact of alternative policies (scenarios) Transmission models Transmission models

Challenges Nature of the problem: Multiple agencies: “It is difficult to make predictions, particularly of the future” (Niels Bohr, Yogi Berra, Somebody..) Multiple agencies: Communication, co-ordination, “translation” Responding to the needs of policy-makers

Nature of the problem: dynamics & uncertainty Varicella incidence Cost-effectiveness of a combined schedule over time Zoster incidence Van Hoek et al. Vaccine (2011)

Nature of the problem: unexpected events Epidemic Deaths 1918 200,000 1957 18,000 1968 46,000 2009 600 Pathogen emergence very difficult to predict Drivers (mechanisms) poorly understood Low probability high impact events E.g. Characteristics of next flu pandemic What is the probability of an epidemic worse than 1918? Other infections? The next HIV or vCJD? Remain vigilant but surveillance is retrospective

Nature of the problem: unexpected events Fraser et al. Science (2009) Models rapidly employed during an outbreak Characterise novel pathogen Reproduction numbers, case fatality, etc Inference: can “make sense” of limited data Though early analyses must be treated with caution Delays and censoring of data Bias Chance Can be employed (later) to forecast the epidemic progress and evaluate interventions

Nature of the problem: unexpected events Baguelin et al. Vaccine 2010 September predictions Peak height second wave estimated to be similar to first Peak early Nov., end Jan. Vaccination has limited impact on cases October Further data reduced uncertainty but did not alter central estimate

Multi-agency: communication & co-ordination Modelling is a component of the advice Advisory body (e.g. JCVI, NICE) made up of individuals from diverse backgrounds Lack of practical modelling experience Advice is distilled Serves different purposes E.g. different outcomes may be critical for different “audiences” Presented in different ways DH advises Ministers JCVI PHE Academia MHRA NIBSC

Communicating the limits of modelling Public health officials might want clear quantitative statements about the impact of an intervention “Single version of the truth” Precise quantitative predictions highly likely to be misleading Uncertainty Parametric Structural “Rules” Representing uncertainty properly, presenting it to decision-makers. Can clear policy advice still be made? Sensitivity analysis What happens if it can’t? Clear guidance of further research Value of information Assessing the quality of other research

Communicating the limits of modelling: structural uncertainty Mathematical models Predictive power comes from appropriate model structure (mechanism) As well as parameter values If key drivers are omitted then results will be unreliable Review and assess other work Second opinion initially infected R0 Cases at 30 days 365 days 10 1.5 31 224,000 3 64 774 billion 1,000 3,094 22 million 6,387 77 trillion Meltzer et al. 2001 EID 7(6)

Limits of modelling: predictions vs scenarios “things that could happen” Predictions “things we think will happen” Khazeni et al. Annal Intern Med 2009 Ong et al. PLoS One 2010

Limits of public health: Optimal vaccine allocation Vaccine given before outbreak Optimised coverage by age class For different outcomes Compare with CDC recommended strategy Medlock & Galvani Science (2009)

Limits of public health: Optimal vaccine allocation Vaccine given before outbreak Optimised coverage by age class For different outcomes Compare with CDC recommended strategy Vaccine coverage cannot be optimised Not under the control of central decision-maker Many other examples of modelling interventions that are not likely to be considered Medlock & Galvani Science (2009)

Responding to the needs of policy-makers Often want information on whether an intervention is worth implementing when compared to alternatives Economic analysis Explicitly weigh costs and benefits A number of countries have recommended methodologies that should be followed Also journals and societies (e.g. ISPOR/SMDM) However, many of these “rules of the game” reflect ethical stances that are not universally shared E.g. Discounting of health benefits May be very influential

Responding to the needs of policy-makers: Vaccines in UK Epidemic models and associated economic models used heavily by JCVI in formulating their advice and recommendations NHS Constitution Minister (England) bound to accept recommendation of JCVI if the programme is cost-effective CEA is therefore critical Transmission model is not required (but usually done) Critical to be done correctly Large scale programmes Difficult to terminate

Responding to the needs of policy-makers: Date Milestone Activity Sept ‘06 JCVI subgroup Review existing models Refine questions (e.g age at vaccination and delivery route) Feb ‘07 Review model assumptions Preliminary results July ‘07 Papers submitted to JCVI secretariat Sent for peer review Sept ‘07 Peer review comments received Revised analysis Oct ‘07 JCVI recommendation Many layers of approval Require different outputs, or results presented in different forms Can be time consuming “Mission creep” Multiple revisions Comments from committees, referees etc. Short deadlines Academics rarely incentivised to participate in these processes

Responding to the needs of policy-makers: Many layers of approval Require different outputs, or results presented in different forms Can be time consuming “Mission creep” Multiple revisions Comments from committees, referees etc. Short deadlines Academics rarely incentivised to participate in these processes Choi et al. Vaccine (2010)

Responding to the needs of policy-makers: Many layers of approval Require different outputs, or results presented in different forms Can be time consuming “Mission creep” Multiple revisions Comments from committees, referees etc. Short deadlines Academics rarely incentivised to participate in these processes Jit et al. BMJ (2008)

Tackling challenges: UK vaccines and flu Relatively simple decision-making framework Centralised, clear, limited number of actors Relatively healthy (multiple) modelling groups Mechanisms in place for these to feed into decisions E.g. Modelling representative on JCVI Peer review and scrutiny Strong modelling team within the public health agency Ensure right questions are addressed Outputs are appropriate Standing resource (core funding leads to flexibility and rapid response) Optimises use of all relevant data – adding value Inputs into design of surveillance system Long-term relationships & working patterns built-up Sophisticated “interpreters” and users of modelling within decision-making framework Clear role for economic analyses and clear (?) rules for conducting these Limited role of industry and lobby groups

Tackling challenges: UK vaccines and flu Relatively simple decision-making framework Top down Relatively healthy (multiple) modelling groups Wasteful Strong modelling team within the public health agency Research best done by academics No role for un-publishable quick and dirty analyses Long-term relationships & working patterns built-up Club Sophisticated “interpreters” and users of modelling within decision-making framework Results “digested” into summary statements where qualifiers & context lost Clear role for economic analyses and clear (?) rules for conducting these Economists know the price of everything and the value of nothing Limited role of industry and lobby groups Have a legitimate role to play in a healthy democracy

Integrating modelling with public health decision-making There are technical challenges that will always remain Highly non-linear systems Imperfectly understood natural history Uncertainty in parameter values Imperfectly observed process, bias in data sets unethical to measure, too expensive to design the experiment, etc Other challenges that will always remain due to: Complex decision-making environment May be subject to press or political scrutiny Compromises & time pressures that will inevitably occur Inadequate resources Enormously rewarding to play a part in improving the health of the nation

Integrating modelling with public health decision-making There are technical challenges that will always remain Highly non-linear systems Imperfectly understood natural history Uncertainty in parameter values Imperfectly observed process, bias in data sets unethical to measure, too expensive to design the experiment, etc Other challenges that will always remain due to: Complex decision-making environment May be subject to press or political scrutiny Compromises & time pressures that will inevitably occur Inadequate resources Enormously rewarding to play a part in improving the health of the nation

Integrating modelling with public health decision-making There are technical challenges that will always remain Highly non-linear systems Imperfectly understood natural history Uncertainty in parameter values Imperfectly observed process, bias in data sets unethical to measure, too expensive to design the experiment, etc Other challenges that will always remain due to: Complex decision-making environment May be subject to press or political scrutiny Compromises & time pressures that will inevitably occur Inadequate resources Enormously rewarding to play a part in improving the health of the nation

Integrating modelling with public health decision-making There are technical challenges that will always remain Highly non-linear systems Imperfectly understood natural history Uncertainty in parameter values Imperfectly observed process, bias in data sets unethical to measure, too expensive to design the experiment, etc Other challenges that will always remain due to: Complex decision-making environment May be subject to press or political scrutiny Compromises & time pressures that will inevitably occur Inadequate resources Enormously rewarding to play a part in improving the health of the nation IPD incidence E&W, HPA Serotypes in PCV7, <2 yrs