Presentation on theme: "Decision and Cost-Effectiveness Analysis: Understanding Sensitivity Analysis Training in Clinical Research DCEA Lecture 5 UCSF Dept. of Epidemiology &"— Presentation transcript:
Decision and Cost-Effectiveness Analysis: Understanding Sensitivity Analysis Training in Clinical Research DCEA Lecture 5 UCSF Dept. of Epidemiology & Biostatistics Jose Luis Burgos, MD, MPH, AAHIVS March 4, 2009
Project Map Think through your research question Sketch your analysis Collect data for your model Adjust model Run-test your model Conduct Sensitivity Analysis Write it up.
Objectives To understand the purposes of sensitivity analysis. To understand techniques used for sensitivity analysis.
Why do sensitivity analyses? All CEAs have substantial uncertaintyAll CEAs have substantial uncertainty Sensitivity analyses deal with that uncertainty systematicallySensitivity analyses deal with that uncertainty systematically Convince audience results are ‘robust’ – qualitative findings don’t change with small changes in inputsConvince audience results are ‘robust’ – qualitative findings don’t change with small changes in inputs
Sensitivity analysis Prior lectures reviewed how inputs are determined, plus a few simple sensitivity/threshold analyses. This lecture will cover four topics: This lecture will cover four topics: 1. Types of uncertainty 2. Deterministic sensitivity analyses (one-way, multi-way, scenario) 3. Probabilistic sensitivity analysis (Monte Carlo) 4. Uses of sensitivity analysis
Types of uncertainty 1. Parameter Uncertainty What’s are the correct input values?What’s are the correct input values? 2.Methodological uncertainty a)Model Structure How values are combinedHow values are combined or modeled or modeled b)Model Process a)Implicit decisions made by the analyst such as viewpoints or effects considered viewpoints or effects considered
Addressing Methodological Uncertainty Sensitivity Analysis – Scenario analysis with different models to combine costs and estimate effects Statistical Analysis – Where multiple parameter sets available, can test the fit of different models
Addressing Model Process Uncertainty Standardized CE analysis – difficult and different panels give different recommendations, but key common components are (from Drummond): 1.The background of the question 2.The viewpoint for the analysis 3.The reason for selecting a particular form of analysis 4.The population to which the analysis applies 5.The comparators being assessed 6.The source of the medical evidence and its quality 7.The range of costs considered and their measurement 8.The measure of benefit in the economic study (e.g. LY gained, QALYs gained) 9.The methods for adjusting costs and benefits according to their timing 10.The methods dealing with uncertainty 11.The incremental analysis of costs and benefits 12.The overall results of the study and its limitations
Deterministic Sensitivity Analyses How does assigning specific different values to inputs change output? One-way (‘univariate’): Vary 1 input at a time Multi-way (‘multivariate’): Vary 2+ inputs at a time Scenario (variant of multi-way): Tests set of relevant conditions. Threshold analysis (one-way or multi-way): Input values beyond which cost-effectiveness achieved (or lost).
Univariate Sensitivity Analysis Examine robustness of ICERs to changes in a single parameter: – ‘best’, ‘high’ and ‘low’ estimates (but experts consistently underestimate true variability) – Value +/- 1 SD – 95% CI (based on observed or assumed distribution) – ‘clinical meaningful range – Extremes – Threshold analysis
One-way SA – Aneurysm management
One-way SA Muennig, 2008
13 One-way SA – LTBI program
Peginterferon Model Inputs: Estimated “base case” and range European Journal of Gastroenterology & Hepatology. 2007;19:
LTBI Model Inputs: Estimated “base case” and range Int Jour Tuberc Lung Dis 2008
Univariate Sensitivity Analyses: Base case and range of outcomes for 1,000 IDUs Burgos JL, Kahn JG, et al. Int Jour Tuberc Lung Dis 2008
Automating one-way SAs: Male circumcision for HIV prevention in South Africa Kahn JG, et al. PLoS Med 3(12): e517. doi: /journal.pmed
Male circumcision for HIV prevention in South Africa Considering HAART cost averted Kahn JG, et al. PLoS Med 3(12): e517. doi: /journal.pmed
Two-way SA: CE of Empowerment program
Two-way sensitivity analysis for changes in HIV risk and average cost for managing active TB cases Burgos JL, Kahn JG, et al. Int Jour Tuberc Lung Dis 2008
Three-Way SA Kahn JG, et al. PLoS Med 3(12): e517. doi: /journal.pmed
Scenario SA Burgos JL, Kahn JG, et al. Int Jour Tuberc Lung Dis 2008
Low HIV prevalence setting: 25% in CSWs; 16.4% in clients Low HIV/AIDS treatment cost: $1,433Med. HIV/AIDS treatment cost: $2,507High HIV/AIDS treatment cost: $3,582 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 ($3,864)($1,824) $2,076 [$509] ($7,426)($5,386)($1,486)($10,989)($8,949)($5,059) Medium HIV prevalence setting: 50.3% in CSWs; 33.0% in clients Low HIV/AIDS treatment cost: $1,433Med. HIV/AIDS treatment cost: $2,507High HIV/AIDS treatment cost: $3,582 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 ($6,023)($3,983)($83)($11,203($9,163)($5,263) ($16,386) ($14,346)($10,446) Multivariate SA on female condom promotion: Net costs by HIV prevalence and key cost inputs for 1,000 CSWs
Threshold Analysis: NVP for prevention of vertical transmission of HIV in sub-Saharan Africa Input values needed for $50/DALY 15% HIV prevalence 30% HIV prevalence Regimen efficacy (47%) 18.0%10.6% VCT cost ($7.30)$18.50$36.00 HIV transmission (25.1%)9.6%5.6% HIV prevalence for $50/DALY 4.5%
Threshold Analysis: NVP $ for prevention of vertical transmission of HIV in sub-Saharan Africa Marseille E, Kahn JG, Saba J. AIDS 1998; 12:
Addressing Parameter Uncertainty: Multivariate Sensitivity Analysis Types of Multivariate Sensitivity Analysis: Repeat bivariate Maximize / minimize CE ratio for different parameter combinations Scenario analysis Monte Carlo simulations under different assumed distributions for parameters (probabilistic sensitivity analysis’ Gold)
Probabilistic sensitivity analysis What is it? What is it good for?
The problem with deterministic SAs No estimate of the probability of achieving a particular outcome (Probabilistic SAs are the remedy)
Probabilistic sensitivity analyses Value Returns the likelihood of attaining particular outcome or outcome range. Everything known about each input expressed all at once. Particularly valuable when many inputs important. Drawback Need to know, or be able to make decent estimates of, the underlying probability distribution.
From empirical data to PD (1) # of clients# of Subjects 1 to to to to to to to to 1592 ≥1601 Frequency distribution of # of clients reported by 101 FSWs Variable | Obs Mean Std. Err. SD [95% Conf. Interval] # of Clients |
From empirical data to PD (2) Graphical Representation of the # of clients reported by 101 FSWs
# of clients# of Subjects 1 to to to to to to to to ≥ Probability distribution of # of clients reported by 100 FSWs From empirical data to PD (3)
From empirical data to PD (4) Graphical Representation of the Prob dist. of clients reported by 101 FSWs
From empirical data to PD (5) Probability Distribution of # of clients among 10,000 FSWs Variable | Obs Mean Std. Err. SD [95% Conf. Interval] # clients | 10,
Triangular Distribution Muennig, 2008
Published distributions Henson SJ. Estimating costs of acute gastrointestinal Illness in BC. Int Jou Food Microb. 2008; 127: 43-45
Published distributions Tafazzoli et al., 2009
Monte Carlo simulation output Crystal Ball output
$226-$504 SA For QALYs Gained Treeage output
(the inner teachings) Other uses of sensitivity analysis (the inner teachings) Planning the analysis Debugging the model Documenting relationships between inputs and outputs Identifying thresholds Influencing policy
Other uses: Planning the analysis Program software to permit SAs on likely SA variables. SA curves provide a check on integrity of model. Identify candidates for more data collection early.
Other uses: Debugging the model Tricks of the trade One-ways best because simple and intuitive. Plug in extreme values. Separate diagnosis of numerator from denominator. Break outputs down further if necessary (intervention versus control arms).
Other uses: Documenting relationships between inputs and outputs Distinguish between ‘bugs’ and insights. Examples of insights: Slowing disease progression can increase costs. Higher disease prevalence can mean lower benefits. Benefits decrease with age - competing mortality risks.
Unexpected dynamic uncovered by SA: Female condoms study Marseille et. al. Soc Sci Med Jan;52(1):
Other uses: Identify thresholds – Influence Policy Preventing HIV vertical transmission in sub-Saharan Africa Cost of ARVs to prevent vertical transmission. Universal versus targeted provision of NVP.
Cost per DALY of HIVNET 012 NVP regimen as function of HIV seroprevalence and type of counseling/testing regimen
Pegynterferon CE acceptability curve European Journal of Gastroenterology & Hepatology. 2007;19:
Summary SA is a set of techniques for the explicit management of uncertainty. Essential part of establishing key findings. Indispensable for convincing your audience that your results are technically sound and policy-relevant.