1. Can decisionmakers rely on health economic evaluations
Can decisionmakers rely on health economic evaluations? A review of studies assessing the quality of published health economic evaluations
Nathalie Dourdin PhD, Monika Wagner PhD, Peter Melnyk PhD and Donna Rindress PhD
BioMedCom Consultants inc., Montréal, QC, Canada
Issues pertaining to the quality of published HE evaluations were identified by determining how many QA studies reported a given issue as a problem in 20% or more of the HE evaluations they reviewed.
When a study mentioned a quality criterion as an issue, the study was included in the percentage calculation regardless of the reason for mention (e.g. not clear, not stated, not justified, etc.,).
The proportions of QA studies that identified a specific quality issue were then calculated.
Quality issues were grouped in clusters as follows:
Outcome measures, valuation of health outcomes
Types of costs and resources included and year of costing
Methodologies for modeling, validation of model, assumptions
Sensitivity analyses and CEAC (cost effectiveness acceptability curve)
Incremental analyses, calculation errors, wrongful use of efficacy/effectiveness
Potential biases and limitations
Sources of funding and disclosure of potential conflicts of interest
BACKGROUND & OBJECTIVE
Health economic (HE) evaluations are intended to provide decisionmakers with information on the comparative efficiency of medical technologies.
The quality of an HE study is critical to the useful application of its findings.
There are a number of different guidelines and checklists available in the public domain on the conduct and reporting of HE and outcomes research.
Adoption of guidelines should help ensure a high degree of rigour and standardization among HE evaluations.
There are currently 74 guidelines for the conduct and reporting of HE evaluations (published and public domain).
The objective of this work is to review studies assessing the quality of published HE evaluations (QA studies) and to investigate whether quality has improved over time with the use of guidelines.
The most commonly reported quality criteria with issues cited by QA studies were:
Perspective
Sensitivity analyses/uncertainty
Discounting
Types of costs included
Source of funding/conflict of interest
Among 68 studies included, only 15 reported on the change in quality of HE evaluations following guideline publication.
11 found evidence that the quality of studies is improving
4 found no improvement over time and sub-optimal adherence to guidelines
CONCLUSION
Despite a wealth of guidelines, evidence for their impact on the quality of published health economic studies is mixed.
Some reviews of health economic evaluations found evidence that the quality of studies is improving.
Other reviews found no or only modest improvement over time and sub-optimal adherence to guidelines.
Outcome measures for quality of HE studies varied significantly across all these studies and were usually qualitative in nature.
FUTURE DIRECTIONS
We need standardized guidelines to promote more consistent level of quality of HE publications.
This is not sufficient, use and reinforcement by journals publishing HE evaluations (for both reviewers and authors) is needed.
We also need more standardized and quantitative measures of quality.
METHODS
Literature searches
Full-text published HE guidelines and studies assessing the quality of HE evaluations were collected from available public domain sources, including PubMed.
The PubMed database was searched from inception using the search terms “health economic guidelines, quality assessment, cost-benefit analysis/standards, cost-effectiveness analysis/standards”
Limits: Meta-Analysis, Practice Guideline, Review, Consensus Development Conference, NIH, Guideline
English, French, German studies selected
Abstracts from recent conferences of ISPOR, SMDM, HTAi, ASHE and iHEA were searched for any relevant abstracts and presentations
Inclusion criteria: reviews of economic evaluations
Data extraction
Information from retrieved QA studies was systematically extracted using a data extraction template
Data extracted included HE study inclusion/exclusion criteria, type of quality assessment tool(s) used, as well as quality assessment results.
REFERENCES
1. Campbell H, Briggs A, Buxton M, Kim L, Thompson S. J Health Serv Res Policy. 2007;12(1): Campbell JD, Spackman DE, Sullivan SD. Allergy. 2008;63(12): Colmenero F, Sullivan SD, Palmer JA, Brauer CA, Bungay K, Watkins J, et al. Am J Manag Care. 2007;13(7): Greenberg D, Earle C, Fang CH, Eldar-Lissai A, Neumann PJ. J Natl Cancer Inst. 2010;102(2): Jefferson T, Demicheli V, Vale L. JAMA ;287(21): Kaplan RM, Groessl EJ. J Consult Clin Psychol. 2002;70(3): Manuel MR, Chen LM, Caughey AB, Subak LL. Gynecol Oncol. 2004;93(1): Neumann PJ, Greenberg D, Olchanski NV, Stone PW, Rosen AB. Value Health. 2005;8(1): Neumann PJ, Fang CH, Cohen JT. Pharmacoeconomics. 2009;27(10): Otero HJ, Rybicki FJ, Greenberg D, Neumann PJ. Radiology. 2008;249(3): Phillips KA, Chen JL. Am J Prev Med. 2002;22(2): Schwappach DL, Boluarte TA, Suhrcke M. Cost Eff Resour Alloc. 2007;5: Schwappach DL, Boluarte TA. BMC Health Serv Res. 2007;7: Spiegel BM, Targownik LE, Kanwal F, Derosa V, Dulai GS, Gralnek IM, et al. Gastroenterology. 2004;127(2): Stone PW, Schackman BR, Neukermans CP, Olchanski N, Greenberg D, Rosen AB, et al. Lancet Infect Dis. 2005;5(6):
RESULTS
68 QA studies met the inclusion criteria; 50 published after 2000
Studies used various tools for assessing quality as well as different ways of reporting the quality assessment
Given this heterogeneous approach to QA, some issues may not have been identified in specific assessments
Of the studies published after 2000, 39 different approaches were used to measure quality
BioMedCom, Pure and Applied Intelligence
BioMedCom, L’intelligence en action