New Research in Economic Modeling and Simulation Greg Samsa PhD

Organization of the talk 3 questions  How does your work contribute to economic and comparative effectiveness modeling?  What’s new in economic modeling and simulation?  What’s changing in how these models are applied? 3 questions  How does your work contribute to economic and comparative effectiveness modeling?  What’s new in economic modeling and simulation?  What’s changing in how these models are applied?

Question 1 How does your work contribute to economic and comparative effectiveness modeling? Short answer: You provide the inputs How does your work contribute to economic and comparative effectiveness modeling? Short answer: You provide the inputs

Question 2 What’s new in economic modeling and simulation? Short answer: Very complex models are now computationally feasible – the debate is shifting from what is possible to what is desired What’s new in economic modeling and simulation? Short answer: Very complex models are now computationally feasible – the debate is shifting from what is possible to what is desired

Question 3 What’s changing in how these models are applied? Short answer: Decision makers are starting to take these models more seriously, and to embed them within more general strategies for learning What’s changing in how these models are applied? Short answer: Decision makers are starting to take these models more seriously, and to embed them within more general strategies for learning

Background The previous speakers have provided definitions and examples of economic models I’ll discuss “complex decision and cost-effectiveness models” – sufficiently complex to require simulation to implement The previous speakers have provided definitions and examples of economic models I’ll discuss “complex decision and cost-effectiveness models” – sufficiently complex to require simulation to implement

Observation 1 A CEA model is a peculiar thing  It is a counting machine intended to clarify trade-offs among things that users value (e.g., survival, quality of life, costs)  Inputs are obtained from different sources – “the model” isn’t something that can be directly observed  The usual principles of validation don’t apply –I’ll discuss what makes a good model later A CEA model is a peculiar thing  It is a counting machine intended to clarify trade-offs among things that users value (e.g., survival, quality of life, costs)  Inputs are obtained from different sources – “the model” isn’t something that can be directly observed  The usual principles of validation don’t apply –I’ll discuss what makes a good model later

Observation 2 A dirty little secret:  A CEA model is no stronger than its weakest link  Advocates focus on a model’s strengths, but the weaknesses are also of importance A dirty little secret:  A CEA model is no stronger than its weakest link  Advocates focus on a model’s strengths, but the weaknesses are also of importance

A personal confession “The Duke Stroke Policy Model combines data from the best sources – natural history from Framingham, costs from national claims, utilities from a large survey developed specifically for this purpose…”

All true, but… Its conclusions depend on how survival, quality of life and costs vary by disability level  These parameters were derived by extrapolating from small studies of inconsistent quality  The user is essentially relying on (a) the face validity of the parameter estimates; and (b) sensitivity analyses Its conclusions depend on how survival, quality of life and costs vary by disability level  These parameters were derived by extrapolating from small studies of inconsistent quality  The user is essentially relying on (a) the face validity of the parameter estimates; and (b) sensitivity analyses

N=? Natural history n=5,000 Costs n=500,000 Utilities n=1,500  Natural history, costs, utilities by disease state – n is variable (e.g., only 100 hemorrhagic strokes)  Impact of disability level n=20, mostly interview rather than observation Natural history n=5,000 Costs n=500,000 Utilities n=1,500  Natural history, costs, utilities by disease state – n is variable (e.g., only 100 hemorrhagic strokes)  Impact of disability level n=20, mostly interview rather than observation

Question 1 How does your work contribute to economic and cost-effectiveness models?

Data sources for model inputs Traditional efficacy trials Effectiveness trials Registries Administrative data Surveys Observational studies Literature reviews Traditional efficacy trials Effectiveness trials Registries Administrative data Surveys Observational studies Literature reviews

Study planning Design studies to improve estimates of parameters that are:  Important (e.g., using sensitivity analysis)  Currently estimated with bias or imprecision Design studies to improve estimates of parameters that are:  Important (e.g., using sensitivity analysis)  Currently estimated with bias or imprecision

Implication CEA models can not only organize thinking, decision making and communication about a topic, but can also be used to help set an agenda for research

Question 2 What’s new in economic modeling and simulation?

Bayesian approach A general approach to CEA modeling  All parameter estimates are based on prior distributions  Ideally, correlations among parameters are considered  Ideally, these distributions reflect the impact of covariates  The output – posterior distributions – reflects the impact of uncertainty A general approach to CEA modeling  All parameter estimates are based on prior distributions  Ideally, correlations among parameters are considered  Ideally, these distributions reflect the impact of covariates  The output – posterior distributions – reflects the impact of uncertainty

Example output “Uncertainty in all the model parameters was addressed using (a) prior distributions; and (b) resampling – in >95% of replications of the simulation the ICER was <$20,000/QALY…”

Advantages This is a general, intellectually coherent way of modeling Now computationally feasible Europeans and analysts like it A more sophisticated treatment of uncertainty than 1- and multi-way sensitivity analysis This is a general, intellectually coherent way of modeling Now computationally feasible Europeans and analysts like it A more sophisticated treatment of uncertainty than 1- and multi-way sensitivity analysis

Disadvantages Possible loss of transparency Parameter estimates might not be possible / practical to obtain Easy for the model to take on a life of its own Possible loss of transparency Parameter estimates might not be possible / practical to obtain Easy for the model to take on a life of its own

What makes a good model? Model structure focuses on core of the issue As simple as possible, but not too simple Model is transparent Model inputs can be collected at the required level of precision / quality Model structure focuses on core of the issue As simple as possible, but not too simple Model is transparent Model inputs can be collected at the required level of precision / quality

Opinion Are more structurally, technically and computationally complex models such as Bayesian CEA models “good”?  My opinion: sometimes Are more structurally, technically and computationally complex models such as Bayesian CEA models “good”?  My opinion: sometimes

Question 3 What’s changing in how CEA models are applied?

Back in the day “Your health care organization should place our acute stroke drug on the formulary because its ICER indicates that it is good value for the money…”

Problems The decision maker doesn’t have the same societal perspective as the analyst The analysis ignores silos Even with discounting, lifetime impact is less important to the decision maker than short term impacts Unless accompanied by a back-of-the envelope calculation, the result isn’t transparent The decision maker doesn’t have the same societal perspective as the analyst The analysis ignores silos Even with discounting, lifetime impact is less important to the decision maker than short term impacts Unless accompanied by a back-of-the envelope calculation, the result isn’t transparent

Example A back of the envelope model  Suppose that an acute stroke drug keeps 2 people per 100 out of nursing homes. If they survive 3 years at $50,000 per year, the excess cost is $300,000 per 100 patients, or $3,000 per patient. So long as it costs less than $3,000, an acute stroke treatment that is even marginally effective is likely to be cost-effective as well. A back of the envelope model  Suppose that an acute stroke drug keeps 2 people per 100 out of nursing homes. If they survive 3 years at $50,000 per year, the excess cost is $300,000 per 100 patients, or $3,000 per patient. So long as it costs less than $3,000, an acute stroke treatment that is even marginally effective is likely to be cost-effective as well.

Current trends With calculations becoming less burdensome, it is easier to produced customized models (e.g., including only costs of interest to the decision maker) Model results are embedded within more realistic frameworks such as comparative effectiveness With calculations becoming less burdensome, it is easier to produced customized models (e.g., including only costs of interest to the decision maker) Model results are embedded within more realistic frameworks such as comparative effectiveness

Ideal framework Transparent Includes as many of the elements of interest to the decision maker as possible CEA model is descriptive, not prescriptive Transparent Includes as many of the elements of interest to the decision maker as possible CEA model is descriptive, not prescriptive

Example As an example of a formal decision making process intended to satisfy these criteria, I’ll describe how the oncology clinics at Duke systematically learn  CEA models are one (albeit not the only) tool that we use As an example of a formal decision making process intended to satisfy these criteria, I’ll describe how the oncology clinics at Duke systematically learn  CEA models are one (albeit not the only) tool that we use

Oncology modeling at Duke Rapid learning cancer clinics Combine sound data collection with an explicit mechanism for learning Rapid learning cancer clinics Combine sound data collection with an explicit mechanism for learning

Data A data warehouse is used to generate multiple views, typically derived from linked files (e.g., cancer type, treatments, outcomes) The lynchpin is a data set of patient-reported outcomes (derived from the PCM) A data warehouse is used to generate multiple views, typically derived from linked files (e.g., cancer type, treatments, outcomes) The lynchpin is a data set of patient-reported outcomes (derived from the PCM)

Data quality The PCM contains 70+ items on a 0-10 scale (e.g., level of nausea during last 7 days) Filled out in waiting room using e- tablets – migrating to web Results are reported to clinicians in real time – for example, highlighting issues to discuss during the visit The PCM contains 70+ items on a 0-10 scale (e.g., level of nausea during last 7 days) Filled out in waiting room using e- tablets – migrating to web Results are reported to clinicians in real time – for example, highlighting issues to discuss during the visit

Incentives Patients: confident that their concerns won’t be overlooked Physicians: saves time in performing a review of symptoms  Principle: (Sufficiently) valid data are produced by design, not by accident Patients: confident that their concerns won’t be overlooked Physicians: saves time in performing a review of symptoms  Principle: (Sufficiently) valid data are produced by design, not by accident

Formal learning structure Learning from the databases occurs within a formal PDCA cycle  Relevant stakeholders are represented  The stakeholders determine the level of accuracy / precision required to make decisions  The stakeholders determine study design (e.g., interventional, observational) Learning from the databases occurs within a formal PDCA cycle  Relevant stakeholders are represented  The stakeholders determine the level of accuracy / precision required to make decisions  The stakeholders determine study design (e.g., interventional, observational)

Types of designs Observational designs with undirected machine learning Observational designs with pre- specified hypotheses Pre-post designs with interventions Randomized trials Observational designs with undirected machine learning Observational designs with pre- specified hypotheses Pre-post designs with interventions Randomized trials

Example Is it “worth it” to refer patients with high levels of psychological distress to specialized counseling?

Inputs Observational data – natural history of outcomes by level of distress CEA model to estimate what level of improvement would justify use of specialized counseling resources Literature review on expected impact of counseling Pre-post design to assess impact of counseling in our setting Observational data – natural history of outcomes by level of distress CEA model to estimate what level of improvement would justify use of specialized counseling resources Literature review on expected impact of counseling Pre-post design to assess impact of counseling in our setting

Criteria for learning A practice is worth changing if the alternative is cost-effective We use CEA models that are simple to moderately complex A practice is worth changing if the alternative is cost-effective We use CEA models that are simple to moderately complex

Comment Our goal is to systematically and explicitly embed learning into our usual procedures

Final thoughts Health economics has always been quantitative – now, it is becoming more explicitly “statistical” as well A statistically-inspired literature on CEA is rapidly developing – a distinguishing characteristic is the ability to accommodate increasingly complex models through advances in computation Health economics has always been quantitative – now, it is becoming more explicitly “statistical” as well A statistically-inspired literature on CEA is rapidly developing – a distinguishing characteristic is the ability to accommodate increasingly complex models through advances in computation

Final thoughts The danger in this literature is that, if its perspective is entirely statistical, it can become divorced from reality A particular area of promise lies in integrating CEA modeling with modern systematic approaches to learning The danger in this literature is that, if its perspective is entirely statistical, it can become divorced from reality A particular area of promise lies in integrating CEA modeling with modern systematic approaches to learning