3 IntroductionPolicy makers/clinicians reluctant to use CEA because assumptions difficult to understandUsing Cost-Effectiveness Analysis to Improve Health Care: Opportunities and Barriers. Neumann PJ 2005CMS (26th National meeting of SMDM, 2004)CEA modelers may base parameter estimates on studies that have limited evidence.Modelers may not consider all studies with comparable evidence and applicability
4 ObjectiveTo develop a method to clarify the tradeoff between strength of evidence and precision of CEA results.
5 MethodsProof of concept based on hypothetical data and simplified model of HIV natural history.Question: What is the cost-effectiveness of Directly Observed Therapy (DOT) for HIV patients?
6 MethodsBasic ideaWhen data sources have insufficient strength of evidence, we should no longer use them to estimate model parameters.Instead, we should assume that little is known and specify them using wide probability distributions with the fewest embedded assumptionsUniform distribution
7 MethodsAssess strength of evidence based on USPTF guidelines which specify three valuation domainsStudy designExtent to which design differs from controlled experimentLevel 1 = best (RCT)Level 3=worst (expert opinion, anecdotal evidence)Internal validityExtent to which results represent truth in study populationGood = best (little LTFU, objective assessment)Poor = worst (large or diverging LTFU, subjective assessment)External validityExtent to which results represent truth in target populationHigh = best (similar pt characteristics, care settings)Low = worst (dissimilar pt characteristics, care settings)
8 MethodsVary evidence criteria in 3 domains from most to least inclusiveIndividually and in aggregateIf evidence meets or exceeds criteria, use it to estimate parameter input distributionIf evidence does not meet criteria, do not use itUse uniform distribution over plausible range sufficiently wide to be acceptable to all CEA users
9 MethodsFor natural history parameters that can only be observed rather than determined experimentally observational studies eligible for Level 1 designOverall mortality rate due to age-, sex-, and race-related causesWhen more than one source of evidence met criteria, we used that source with greatest statistical precisionAlternative: pool weighting by inverse of varianceWhen substituting uniform distribution make sure that direction of aggregate effect is neutralMaximizes conservatism of approach
10 MethodsModel: extremely simple 10-parameter probabilistic simulation of DOT in HIV17 data sources considered
11 Results Base Case: No evidence criteria Study Design = High All 17 data sources eligible for parameter estimationStudy Design = High13 out of 17 sources were eligibleInternal Validity = Good9 out of 17 sources were eligibleExternal Validity = High5 out of 17 sources were eligibleAll three criteriaOnly 3 out of 17 sources were eligible
20 Results – Overall No evidence criteria $78,000/QALY Study Design = $227,000/QALYInternal Validity = Good $158,000/QALYExternal Validity = High >$6,000,000/QALYAll three criteria > $6,000,000/QALY
21 LimitationsIncorporates a simple model of HIV that was constructed solely for the purpose of illustrating proof of concept.Method is likely to need further refinement before it could be used on more complex and realistic simulations.Method only addresses parameter uncertainty, leaving other determinates of modeling uncertainty unexplored.
22 ConclusionsStrength of evidence may have profound impact on the precision and estimates of CEAsWith all evidence was permitted results similar to previously published DOT CEA (Goldie03)$40,000 to $75,000/QALYLittle uncertaintyWith stricter evidence criteria our results differed markedly> $ 150,000/QALYGreat uncertainty
23 ImplicationsSensitivity analysis by strength of evidence concept can be linked to any desired ranking method for strength of evidence, and therefore can be customized to facilitate its use by expert panels and organizations.Advance of this work does not lie in its specification of particular hierarchy of strength of evidenceAdvance lies in showing how any hierarchy can be implemented within CEA model.
24 ImplicationsUsers who think “any data is better than no data” will likely base inferences on model results that incorporate all data sources, regardless of strength of evidenceUsers who think “my judgment supersedes all but the best data” would likely only base inferences on model results that reflect only highest grades of evidence.Many models may fail to provide conclusive results when validity criteria are stringent.Nonetheless, in the long run this may help CEA to become a more essential decision making tool.