1. Analysis Alongside A Randomized Trial Todd Wagner, PhD May 2009

2. Objectives At the end of the class, you should At the end of the class, you should –Understand how to set up your datasets –Be familiar with analytical methods –Want to hear the class on decision modeling

3. Dominance No need to calculate an incremental cost effectiveness ratio if: No need to calculate an incremental cost effectiveness ratio if: –Intervention is more effective and less expensive than control –Intervention is more expensive and less effective than control But: But: –Check whether dominance exists within subgroups –Does dominance persists after including uncertainty

4. Cost EXP - Cost CONTROL _____________________ QALY EXP -QALY CONTROL Incremental Cost-Effectiveness Ratio (ICER) Calculate in the absence of dominance Calculate in the absence of dominance

5. Cost Data Have costs consistent with the stated perspective Have costs consistent with the stated perspective Societal Societal –Health care utilization –Patient costs –Caregiver costs –Intervention costs (direct plus indirect)

6. Common Hurdle Many of the parameters in the analysis are based on assumptions (e.g., wage rate, mileage costs) Many of the parameters in the analysis are based on assumptions (e.g., wage rate, mileage costs) Consider whether these assumptions are biased toward/against the intervention Consider whether these assumptions are biased toward/against the intervention Ideally want to include uncertainty Ideally want to include uncertainty –One way sensitivity analysis –Statistical methods (bootstrapping, Cooks D)

7. Another hurdle Include disease-related utilization or all health care utilization? Include disease-related utilization or all health care utilization? How do you define disease-related? How do you define disease-related? Recommend: look at all utilization for the CEA Recommend: look at all utilization for the CEA

8. Labor Outcomes Productivity employment is not in the cost estimate Productivity employment is not in the cost estimate Anyone remember why? Anyone remember why? If labor outcomes are important, still collect them but report them separately. If labor outcomes are important, still collect them but report them separately.

9. Dataset You need to create a “long” dataset You need to create a “long” dataset You need to have an group indicator for experiment (1) and control (0) You need to have an group indicator for experiment (1) and control (0) You need to have cost and outcome information You need to have cost and outcome information You need subgroup identifiers You need subgroup identifiers

10. Data

11. Analysis. tabstat totcost followup if exp==1, by(papres) stat(mean) format(%3.2f) Summary statistics: mean by categories of: papres (initial pap results) by categories of: papres (initial pap results) papres | totcost followup papres | totcost followup-----------+-------------------- ASCUS/AGUS | 347.31 0.57 LGSIL | 373.97 0.64 LGSIL | 373.97 0.64 HGSIL | 404.72 0.87 HGSIL | 404.72 0.87-----------+-------------------- Total | 355.45 0.61 Total | 355.45 0.61--------------------------------. tabstat totcost followup if exp==0, by(papres) stat(mean) format(%3.2f) Summary statistics: mean by categories of: papres (initial pap results) by categories of: papres (initial pap results) papres | totcost followup papres | totcost followup-----------+-------------------- ASCUS/AGUS | 74.92 0.32 LGSIL | 73.98 0.30 LGSIL | 73.98 0.30 HGSIL | 104.94 0.43 HGSIL | 104.94 0.43-----------+-------------------- Total | 77.28 0.32 Total | 77.28 0.32--------------------------------

12. Hypothesis Testing. reg totcost exp Source | SS df MS Number of obs = 348 -------------+------------------------------ F( 1, 346) = 292.64 Model | 6728200.2 1 6728200.2 Prob > F = 0.0000 Residual | 7954890.66 346 22991.0135 R-squared = 0.4582 -------------+------------------------------ Adj R-squared = 0.4567 Total | 14683090.9 347 42314.3829 Root MSE = 151.63 ------------------------------------------------------------------------------ totcost | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- exp | 278.1664 16.26051 17.11 0.000 246.1845 310.1483 _cons | 77.27982 11.62933 6.65 0.000 54.40675 100.1529 ------------------------------------------------------------------------------ Logistic regression Number of obs = 348 LR chi2(1) = 28.42 Prob > chi2 = 0.0000 Log likelihood = -226.30785 Pseudo R2 = 0.0591 ------------------------------------------------------------------------------ followup | Odds Ratio Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- exp | 3.225974.7243906 5.22 0.000 2.077419 5.009538 ------------------------------------------------------------------------------

13. Conclusion Intervention is more costly and more effective Intervention is more costly and more effective Next Steps: Next Steps: –Include uncertainty –Check for subgroups and –calculate the ICER

14. ICER Calculation Intervention Group papres | totcost followup papres | totcost followup-----------+-------------------- ASCUS/AGUS | 347.31 0.57 LGSIL | 373.97 0.64 LGSIL | 373.97 0.64 HGSIL | 404.72 0.87 HGSIL | 404.72 0.87-----------+-------------------- Total | 355.45 0.61 Total | 355.45 0.61-------------------------------- Control Group papres | totcost followup papres | totcost followup-----------+-------------------- ASCUS/AGUS | 74.92 0.32 LGSIL | 73.98 0.30 LGSIL | 73.98 0.30 HGSIL | 104.94 0.43 HGSIL | 104.94 0.43-----------+-------------------- Total | 77.28 0.32 Total | 77.28 0.32-------------------------------- ICER=(355.45-77.28) (0.61-0.32) (0.61-0.32)ICER=982.1837 Is that good or bad?

15. Uncertainty You need to calculate the confidence regions around this parameter estimate You need to calculate the confidence regions around this parameter estimate Variable | Reps Observed Bias Std. Err. [95% Conf. Interval] ---------+------------------------------------------------------------------- cerp2 | 1000 982.1837 24.04897 152.2913 683.336 1281.031 (N) cerp2 | 1000 982.1837 24.04897 152.2913 683.336 1281.031 (N) | 787.438 1369.206 (P) | 787.438 1369.206 (P) | 787.3935 1366.588 (BC) | 787.3935 1366.588 (BC)----------------------------------------------------------------------------- N = normal, P = percentile, BC = bias-corrected N = normal, P = percentile, BC = bias-corrected

16. Confidence Regions Ratios are complex to interpret Ratios are complex to interpret

17. More effective and more expensive More effective and less expensiveLess effective and less expensive Less effective and more expensive

18. Acceptability Curves Acceptability curves show the information based on willingness to pay for the outcome. Acceptability curves show the information based on willingness to pay for the outcome. Shape of the curve is dependent on the bootsrapped estimates Shape of the curve is dependent on the bootsrapped estimates Allows decision makers with different thresholds to interpret the data Allows decision makers with different thresholds to interpret the data Source: Henry Glick

19. Threshold Value for Cost-Effectiveness Ratio Rule of thumb: Rule of thumb: U.S. health care system adopts interventions with ratio of less than $50,000 ($100,000) per Quality Adjusted Life Year. Variable | Reps Observed Bias Std. Err. [95% Conf. Interval] ---------+------------------------------------------------------------------- cerp2 | 1000 982.1837 24.04897 152.2913 683.336 1281.031 (N) cerp2 | 1000 982.1837 24.04897 152.2913 683.336 1281.031 (N) | 787.438 1369.206 (P) | 787.438 1369.206 (P) | 787.3935 1366.588 (BC) | 787.3935 1366.588 (BC)----------------------------------------------------------------------------- N = normal, P = percentile, BC = bias-corrected N = normal, P = percentile, BC = bias-corrected

20. Adopt this Pap Smear Intervention? Poll Poll Yes Yes No No Don’t know Don’t know Don’t care Don’t care

21. Limitations with RCTs Proxy outcomes Proxy outcomes Length of follow-up Length of follow-up Generalizability Generalizability

22. Ideal Study RCT comparing behavioral intervention and usual care control RCT comparing behavioral intervention and usual care control Follow participants for life Follow participants for life Advantages: know all costs and all benefits Advantages: know all costs and all benefits Disadvantages: expensive and possibly useless results at end of study Disadvantages: expensive and possibly useless results at end of study

23. Usual Study Outcomes measured at end of study Outcomes measured at end of study Use models to estimate lifetime costs and benefits in the CEA Use models to estimate lifetime costs and benefits in the CEA

24. Effectiveness Preferred metric for CEA is quality adjusted life years (QALYs) Preferred metric for CEA is quality adjusted life years (QALYs) Most behavioral interventions use an “ intermediate outcome ” Most behavioral interventions use an “ intermediate outcome ” –e.g., receipt of mammography Few behavioral interventions use QALYs because the study would have to be huge and/or very long in duration Few behavioral interventions use QALYs because the study would have to be huge and/or very long in duration

25. CEA with Intermediate Outcomes Easy and sufficient for publication Easy and sufficient for publication Hard to interpret ICER Hard to interpret ICER –Can’t compare two CEAs with different intermediate outcomes –Can’t compare CEA to other CEA from another clinical area Sometimes only feasible approach Sometimes only feasible approach

26. Intermediate to QALYs “Translate” intermediate outcome to QALYs “Translate” intermediate outcome to QALYs Either build a model de novo or use an existing model Either build a model de novo or use an existing model Requires a lot of resources Requires a lot of resources Most useful, but most challenging Most useful, but most challenging

27. Lost in Translation Gap between ideal and usual study Gap between ideal and usual study Models fill the gap Models fill the gap Behavioral models have unique challenges Behavioral models have unique challenges –Partial behavior change is missing from current models –Definition: people who progressed in their “ stage of change ” but did not successfully change their behavior at the end of the study Wagner TH, Goldstein MK. Behavioral interventions and cost-effectiveness analysis. Prev Med. Dec 2004;39(6):1208-1214.

28. Outcome Responsiveness Many trials use indirect QALY assessments (HUI, QWB, EQ-5D) Many trials use indirect QALY assessments (HUI, QWB, EQ-5D) These measures are often not responsive to the intervention effect at the end of the trial. These measures are often not responsive to the intervention effect at the end of the trial. How do you interpret the results when the main outcome shows an effect and the QALYs do not? How do you interpret the results when the main outcome shows an effect and the QALYs do not?

29. QALY Responsiveness Try to understand why QALYs were not sensitive Try to understand why QALYs were not sensitive Analyze data for subgroup effects Analyze data for subgroup effects Analyze other outcomes that may be sensitive to change. Analyze other outcomes that may be sensitive to change. –Disease specific quality of life –Willingness to pay If you think there is an effect, ignore the p-value on the QALYs and model the data using the mean and variance estimates If you think there is an effect, ignore the p-value on the QALYs and model the data using the mean and variance estimates

30. Another Approach Our CEA did not leverage the trial data Our CEA did not leverage the trial data We extracted parameters, created confidence regions and then (hopefully) put these parameters into a decision model We extracted parameters, created confidence regions and then (hopefully) put these parameters into a decision model

31. Net Monetary Benefits What we did – –Collect data to estimate ΔC and ΔE – –Calculate an estimate: – –ΔC / ΔE (ICER = incremental cost- effectiveness ratio) Another option: – –λ ΔE – ΔC (INB = incremental net-benefit) Hoch JS, Briggs AH, Willan AR. Something old, something new, something borrowed, something blue: a framework for the marriage of health econometrics and cost-effectiveness analysis. Health Econ. Jul 2002;11(5):415-430.

32. Data

33. NMB set up You have cost data and effectiveness data for each person You have cost data and effectiveness data for each person Use this information in a regular regression framework Use this information in a regular regression framework You need to estimate You need to estimate λ – –λ is the WTP for the outcome – –Rerun the analysis for different λ values

34. NMB Limitations Based on a linear model Based on a linear model Does not necessarily translate into non- linear or latent models Does not necessarily translate into non- linear or latent models I suspect this is possible if you estimate predicted probabilities for groups and then carry out the analysis for the groups. I suspect this is possible if you estimate predicted probabilities for groups and then carry out the analysis for the groups.

35. Generalizability Recall that RCTs may not enroll a generalizable population Recall that RCTs may not enroll a generalizable population For many RCTs in VA, you can compare participants to non-participants For many RCTs in VA, you can compare participants to non-participants –Generate propensity scores –Use the propensity score as a weight in the RCT analysis– place more weight on people who are a lot like the non-participants

36. Next talk May 27, 2009 How Can Cost Effectiveness Analysis Be Made More Relevant to US Health Care? Paul Barnett, Ph.D. May 27, 2009 How Can Cost Effectiveness Analysis Be Made More Relevant to US Health Care? Paul Barnett, Ph.D. Paul Barnett, Ph.D.