1. John Edmunds, Jess Metcalf, Justin Lessler john.edmunds@lshtm.ac.uk
Challenges in using mathematical modelling for public health decision-making
John Edmunds, Jess Metcalf, Justin Lessler

2. Why use models?
Observational studies of impact are retrospective, and difficult to interpret
Intervention studies (trials) relatively small & key endpoints rarely observed, therefore models used for extrapolation
Other endpoints (e.g. deaths)
Other population groups (perhaps excluded from study)
Over time
To others in the population
Synthesise the results of different studies in a unified framework
Improve our understanding of key drivers
Identify and quantify knowledge gaps

3. Why use models?
Observational studies of impact are retrospective, and difficult to interpret
Intervention studies (trials) relatively small & key endpoints rarely observed, therefore models used for extrapolation
Other endpoints (e.g. deaths)
Other population groups (perhaps excluded from study)
Over time
To others in the population
Synthesise the results of different studies in a unified framework
Improve our understanding of key drivers
Identify and quantify knowledge gaps
Synthesis &
prediction
Understanding, research and surveillance

4. Main role of mathematical modelling in UK infectious disease policy
Epidemics
Contingency planning
Risk assessment
Epidemic forecasting (& “now-casting”)
Impact of interventions
“Endemic” diseases
Vaccine policy
Other control policy (e.g. Screening and treatment)
Burden of disease assessment (priority setting)

5. Main role of mathematical modelling in UK infectious disease policy
Epidemics
Contingency planning
Risk assessment
Epidemic forecasting (& “now-casting”)
Impact of interventions
“Endemic” diseases
Vaccine policy
Other control policy (e.g. Screening and treatment)
Burden of disease assessment (priority setting)
Impact of alternative policies (scenarios)
Transmission models
Transmission
models

6. Challenges Nature of the problem: Multiple agencies:
“It is difficult to make predictions, particularly of the future” (Niels Bohr, Yogi Berra, Somebody..)
Multiple agencies:
Communication, co-ordination, “translation”
Responding to the needs of policy-makers

7. Nature of the problem: dynamics & uncertainty
Varicella incidence
Cost-effectiveness of a combined schedule
over time
Zoster incidence
Van Hoek et al. Vaccine (2011)

8. Nature of the problem: unexpected events
Epidemic
Deaths
1918
200,000
1957
18,000
1968
46,000
2009
600
Pathogen emergence very difficult to predict
Drivers (mechanisms) poorly understood
Low probability high impact events
E.g. Characteristics of next flu pandemic
What is the probability of an epidemic worse than 1918?
Other infections?
The next HIV or vCJD?
Remain vigilant but surveillance is retrospective

9. Nature of the problem: unexpected events
Fraser et al. Science (2009)
Models rapidly employed during an outbreak
Characterise novel pathogen
Reproduction numbers, case fatality, etc
Inference: can “make sense” of limited data
Though early analyses must be treated with caution
Delays and censoring of data
Bias
Chance
Can be employed (later) to forecast the epidemic progress and evaluate interventions

10. Nature of the problem: unexpected events
Baguelin et al. Vaccine 2010
September predictions
Peak height second wave estimated to be similar to first
Peak early Nov., end Jan.
Vaccination has limited impact on cases
October
Further data reduced uncertainty but did not alter central estimate

11. Multi-agency: communication & co-ordination
Modelling is a component of the advice
Advisory body (e.g. JCVI, NICE) made up of individuals from diverse backgrounds
Lack of practical modelling experience
Advice is distilled
Serves different purposes
E.g. different outcomes may be critical for different “audiences”
Presented in different ways
DH advises
Ministers
JCVI
PHE
Academia
MHRA NIBSC

12. Communicating the limits of modelling
Public health officials might want clear quantitative statements about the impact of an intervention
“Single version of the truth”
Precise quantitative predictions highly likely to be misleading
Uncertainty
Parametric
Structural
“Rules”
Representing uncertainty properly, presenting it to decision-makers.
Can clear policy advice still be made?
Sensitivity analysis
What happens if it can’t?
Clear guidance of further research
Value of information
Assessing the quality of other research

13. Communicating the limits of modelling: structural uncertainty
Mathematical models
Predictive power comes from appropriate model structure (mechanism)
As well as parameter values
If key drivers are omitted then results will be unreliable
Review and assess other work
Second opinion
initially infected R0
Cases at
30 days
365 days 10
1.5 31
224,000 3 64
774 billion
1,000
3,094
22 million
6,387
77 trillion
Meltzer et al EID 7(6)

14. Limits of modelling: predictions vs scenarios
“things that could happen”
Predictions
“things we think will happen”
Khazeni et al.
Annal Intern Med
2009
Ong et al.
PLoS One 2010

15. Limits of public health: Optimal vaccine allocation
Vaccine given before outbreak
Optimised coverage by age class
For different outcomes
Compare with CDC recommended strategy
Medlock & Galvani Science (2009)

16. Limits of public health: Optimal vaccine allocation
Vaccine given before outbreak
Optimised coverage by age class
For different outcomes
Compare with CDC recommended strategy
Vaccine coverage cannot be optimised
Not under the control of central decision-maker
Many other examples of modelling interventions that are not likely to be considered
Medlock & Galvani Science (2009)

17. Responding to the needs of policy-makers
Often want information on whether an intervention is worth implementing when compared to alternatives
Economic analysis
Explicitly weigh costs and benefits
A number of countries have recommended methodologies that should be followed
Also journals and societies (e.g. ISPOR/SMDM)
However, many of these “rules of the game” reflect ethical stances that are not universally shared
E.g. Discounting of health benefits
May be very influential

18. Responding to the needs of policy-makers: Vaccines in UK
Epidemic models and associated economic models used heavily by JCVI in formulating their advice and recommendations
NHS Constitution
Minister (England) bound to accept recommendation of JCVI if the programme is cost-effective
CEA is therefore critical
Transmission model is not required (but usually done)
Critical to be done correctly
Large scale programmes
Difficult to terminate

19. Responding to the needs of policy-makers:
Date
Milestone
Activity
Sept ‘06
JCVI subgroup
Review existing models
Refine questions (e.g age at vaccination and delivery route)
Feb ‘07
Review model assumptions
Preliminary results
July ‘07
Papers submitted to JCVI secretariat
Sent for peer review
Sept ‘07
Peer review comments received
Revised analysis
Oct ‘07
JCVI recommendation
Many layers of approval
Require different outputs, or results presented in different forms
Can be time consuming
“Mission creep”
Multiple revisions
Comments from committees, referees etc.
Short deadlines
Academics rarely incentivised to participate in these processes

20. Responding to the needs of policy-makers:
Many layers of approval
Require different outputs, or results presented in different forms
Can be time consuming
“Mission creep”
Multiple revisions
Comments from committees, referees etc.
Short deadlines
Academics rarely incentivised to participate in these processes
Choi et al. Vaccine (2010)

21. Responding to the needs of policy-makers:
Many layers of approval
Require different outputs, or results presented in different forms
Can be time consuming
“Mission creep”
Multiple revisions
Comments from committees, referees etc.
Short deadlines
Academics rarely incentivised to participate in these processes
Jit et al. BMJ (2008)

22. Tackling challenges: UK vaccines and flu
Relatively simple decision-making framework
Centralised, clear, limited number of actors
Relatively healthy (multiple) modelling groups
Mechanisms in place for these to feed into decisions
E.g. Modelling representative on JCVI
Peer review and scrutiny
Strong modelling team within the public health agency
Ensure right questions are addressed
Outputs are appropriate
Standing resource (core funding leads to flexibility and rapid response)
Optimises use of all relevant data – adding value
Inputs into design of surveillance system
Long-term relationships & working patterns built-up
Sophisticated “interpreters” and users of modelling within decision-making framework
Clear role for economic analyses and clear (?) rules for conducting these
Limited role of industry and lobby groups

23. Tackling challenges: UK vaccines and flu
Relatively simple decision-making framework
Top down
Relatively healthy (multiple) modelling groups
Wasteful
Strong modelling team within the public health agency
Research best done by academics
No role for un-publishable quick and dirty analyses
Long-term relationships & working patterns built-up
Club
Sophisticated “interpreters” and users of modelling within decision-making framework
Results “digested” into summary statements where qualifiers & context lost
Clear role for economic analyses and clear (?) rules for conducting these
Economists know the price of everything and the value of nothing
Limited role of industry and lobby groups
Have a legitimate role to play in a healthy democracy

24. Integrating modelling with public health decision-making
There are technical challenges that will always remain
Highly non-linear systems
Imperfectly understood natural history
Uncertainty in parameter values
Imperfectly observed process, bias in data sets
unethical to measure, too expensive to design the experiment, etc
Other challenges that will always remain due to:
Complex decision-making environment
May be subject to press or political scrutiny
Compromises & time pressures that will inevitably occur
Inadequate resources
Enormously rewarding to play a part in improving the health of the nation

25. Integrating modelling with public health decision-making
There are technical challenges that will always remain
Highly non-linear systems
Imperfectly understood natural history
Uncertainty in parameter values
Imperfectly observed process, bias in data sets
unethical to measure, too expensive to design the experiment, etc
Other challenges that will always remain due to:
Complex decision-making environment
May be subject to press or political scrutiny
Compromises & time pressures that will inevitably occur
Inadequate resources
Enormously rewarding to play a part in improving the health of the nation

26. Integrating modelling with public health decision-making
There are technical challenges that will always remain
Highly non-linear systems
Imperfectly understood natural history
Uncertainty in parameter values
Imperfectly observed process, bias in data sets
unethical to measure, too expensive to design the experiment, etc
Other challenges that will always remain due to:
Complex decision-making environment
May be subject to press or political scrutiny
Compromises & time pressures that will inevitably occur
Inadequate resources
Enormously rewarding to play a part in improving the health of the nation

27. Integrating modelling with public health decision-making
There are technical challenges that will always remain
Highly non-linear systems
Imperfectly understood natural history
Uncertainty in parameter values
Imperfectly observed process, bias in data sets
unethical to measure, too expensive to design the experiment, etc
Other challenges that will always remain due to:
Complex decision-making environment
May be subject to press or political scrutiny
Compromises & time pressures that will inevitably occur
Inadequate resources
Enormously rewarding to play a part in improving the health of the nation
IPD incidence E&W, HPA
Serotypes in PCV7, <2 yrs