Presentation is loading. Please wait.

Presentation is loading. Please wait.

INCREASING THE TRANSPARENCY OF CEA MODELING ASSUMPTIONS: A SENSITIVITY ANALYSIS BASED ON STRENGTH OF EVIDENCE RS Braithwaite MS Roberts AC Justice.

Similar presentations


Presentation on theme: "INCREASING THE TRANSPARENCY OF CEA MODELING ASSUMPTIONS: A SENSITIVITY ANALYSIS BASED ON STRENGTH OF EVIDENCE RS Braithwaite MS Roberts AC Justice."— Presentation transcript:

1 INCREASING THE TRANSPARENCY OF CEA MODELING ASSUMPTIONS: A SENSITIVITY ANALYSIS BASED ON STRENGTH OF EVIDENCE RS Braithwaite MS Roberts AC Justice

2 Introduction Tragicomic anecdote Tragicomic anecdote

3 Introduction Policy makers/clinicians reluctant to use CEA because assumptions difficult to understand Policy makers/clinicians reluctant to use CEA because assumptions difficult to understand Using Cost-Effectiveness Analysis to Improve Health Care: Opportunities and Barriers. Neumann PJ 2005 Using Cost-Effectiveness Analysis to Improve Health Care: Opportunities and Barriers. Neumann PJ 2005 CMS (26th National meeting of SMDM, 2004) CMS (26th National meeting of SMDM, 2004) CEA modelers may base parameter estimates on studies that have limited evidence. CEA modelers may base parameter estimates on studies that have limited evidence. Modelers may not consider all studies with comparable evidence and applicability Modelers may not consider all studies with comparable evidence and applicability

4 Objective To develop a method to clarify the tradeoff between strength of evidence and precision of CEA results. To develop a method to clarify the tradeoff between strength of evidence and precision of CEA results.

5 Methods Proof of concept based on hypothetical data and simplified model of HIV natural history. Proof of concept based on hypothetical data and simplified model of HIV natural history. Question: What is the cost-effectiveness of Directly Observed Therapy (DOT) for HIV patients? Question: What is the cost-effectiveness of Directly Observed Therapy (DOT) for HIV patients?

6 Methods Basic idea Basic idea When data sources have insufficient strength of evidence, we should no longer use them to estimate model parameters. When data sources have insufficient strength of evidence, we should no longer use them to estimate model parameters. Instead, we should assume that little is known and specify them using wide probability distributions with the fewest embedded assumptions Instead, we should assume that little is known and specify them using wide probability distributions with the fewest embedded assumptions Uniform distribution Uniform distribution

7 Methods Assess strength of evidence based on USPTF guidelines which specify three valuation domains Assess strength of evidence based on USPTF guidelines which specify three valuation domains Study design Study design Extent to which design differs from controlled experiment Extent to which design differs from controlled experiment Level 1 = best (RCT) Level 1 = best (RCT) Level 3=worst (expert opinion, anecdotal evidence) Level 3=worst (expert opinion, anecdotal evidence) Internal validity Internal validity Extent to which results represent truth in study population Extent to which results represent truth in study population Good = best (little LTFU, objective assessment) Good = best (little LTFU, objective assessment) Poor = worst (large or diverging LTFU, subjective assessment) Poor = worst (large or diverging LTFU, subjective assessment) External validity External validity Extent to which results represent truth in target population Extent to which results represent truth in target population High = best (similar pt characteristics, care settings) High = best (similar pt characteristics, care settings) Low = worst (dissimilar pt characteristics, care settings) Low = worst (dissimilar pt characteristics, care settings)

8 Methods Vary evidence criteria in 3 domains from most to least inclusive Vary evidence criteria in 3 domains from most to least inclusive Individually and in aggregate Individually and in aggregate If evidence meets or exceeds criteria, use it to estimate parameter input distribution If evidence meets or exceeds criteria, use it to estimate parameter input distribution If evidence does not meet criteria, do not use it If evidence does not meet criteria, do not use it Use uniform distribution over plausible range sufficiently wide to be acceptable to all CEA users Use uniform distribution over plausible range sufficiently wide to be acceptable to all CEA users

9 Methods For natural history parameters that can only be observed rather than determined experimentally observational studies eligible for Level 1 design For natural history parameters that can only be observed rather than determined experimentally observational studies eligible for Level 1 design Overall mortality rate due to age-, sex-, and race-related causes Overall mortality rate due to age-, sex-, and race-related causes When more than one source of evidence met criteria, we used that source with greatest statistical precision When more than one source of evidence met criteria, we used that source with greatest statistical precision Alternative: pool weighting by inverse of variance Alternative: pool weighting by inverse of variance When substituting uniform distribution make sure that direction of aggregate effect is neutral When substituting uniform distribution make sure that direction of aggregate effect is neutral Maximizes conservatism of approach Maximizes conservatism of approach

10 Methods Model: extremely simple 10-parameter probabilistic simulation of DOT in HIV Model: extremely simple 10-parameter probabilistic simulation of DOT in HIV 17 data sources considered 17 data sources considered

11 Results Base Case: No evidence criteria Base Case: No evidence criteria All 17 data sources eligible for parameter estimation All 17 data sources eligible for parameter estimation Study Design = High Study Design = High 13 out of 17 sources were eligible 13 out of 17 sources were eligible Internal Validity = Good Internal Validity = Good 9 out of 17 sources were eligible 9 out of 17 sources were eligible External Validity = High External Validity = High 5 out of 17 sources were eligible 5 out of 17 sources were eligible All three criteria All three criteria Only 3 out of 17 sources were eligible Only 3 out of 17 sources were eligible

12 Results: All Evidence

13 Results: Design = 1

14 Results: Internal Validity = Good

15 Results: External Validity = High

16 Results: All Evidence

17 Results: Design = 1

18 Results: Internal Validity = Good

19 Results: External Validity = High

20 Results – Overall No evidence criteria $78,000/QALY No evidence criteria $78,000/QALY Study Design = 1 $227,000/QALY Study Design = 1 $227,000/QALY Internal Validity = Good $158,000/QALY Internal Validity = Good $158,000/QALY External Validity = High >$6,000,000/QALY External Validity = High >$6,000,000/QALY All three criteria > $6,000,000/QALY All three criteria > $6,000,000/QALY

21 Limitations Incorporates a simple model of HIV that was constructed solely for the purpose of illustrating proof of concept. Incorporates a simple model of HIV that was constructed solely for the purpose of illustrating proof of concept. Method is likely to need further refinement before it could be used on more complex and realistic simulations. Method is likely to need further refinement before it could be used on more complex and realistic simulations. Method only addresses parameter uncertainty, leaving other determinates of modeling uncertainty unexplored. Method only addresses parameter uncertainty, leaving other determinates of modeling uncertainty unexplored.

22 Conclusions Strength of evidence may have profound impact on the precision and estimates of CEAs Strength of evidence may have profound impact on the precision and estimates of CEAs With all evidence was permitted results similar to previously published DOT CEA (Goldie03) With all evidence was permitted results similar to previously published DOT CEA (Goldie03) $40,000 to $75,000/QALY $40,000 to $75,000/QALY Little uncertainty Little uncertainty With stricter evidence criteria our results differed markedly With stricter evidence criteria our results differed markedly > $ 150,000/QALY > $ 150,000/QALY Great uncertainty Great uncertainty

23 Implications Sensitivity analysis by strength of evidence concept can be linked to any desired ranking method for strength of evidence, and therefore can be customized to facilitate its use by expert panels and organizations. Sensitivity analysis by strength of evidence concept can be linked to any desired ranking method for strength of evidence, and therefore can be customized to facilitate its use by expert panels and organizations. Advance of this work does not lie in its specification of particular hierarchy of strength of evidence Advance of this work does not lie in its specification of particular hierarchy of strength of evidence Advance lies in showing how any hierarchy can be implemented within CEA model. Advance lies in showing how any hierarchy can be implemented within CEA model.

24 Implications Users who think any data is better than no data will likely base inferences on model results that incorporate all data sources, regardless of strength of evidence Users who think any data is better than no data will likely base inferences on model results that incorporate all data sources, regardless of strength of evidence Users who think my judgment supersedes all but the best data would likely only base inferences on model results that reflect only highest grades of evidence. Users who think my judgment supersedes all but the best data would likely only base inferences on model results that reflect only highest grades of evidence. Many models may fail to provide conclusive results when validity criteria are stringent. Many models may fail to provide conclusive results when validity criteria are stringent. Nonetheless, in the long run this may help CEA to become a more essential decision making tool. Nonetheless, in the long run this may help CEA to become a more essential decision making tool.

25


Download ppt "INCREASING THE TRANSPARENCY OF CEA MODELING ASSUMPTIONS: A SENSITIVITY ANALYSIS BASED ON STRENGTH OF EVIDENCE RS Braithwaite MS Roberts AC Justice."

Similar presentations


Ads by Google