RTI International is a trade name of Research Triangle Institute Nancy Berkman, PhDMeera Viswanathan, PhD Development of a Tool to Evaluate the Quality of Non-randomized Studies of Interventions or Exposures Presented by Nancy D Berkman, PhD & Meera Viswanathan, PhD Presented at AHRQ 2009 Annual Conference Bethesda, Maryland September 15, 2009

Acknowledgements Project funding provided by –Phase 1: Grant from RTI Independent Research and Development (IR&D) funds –Phase 2: Contract from Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services, through the Evidence-based Practice Centers program (EPC)

Context for the Project Increasing demand to include non-randomized studies in systematic literature reviews and comparative effectiveness reviews to capture –The effects of interventions or exposures on a more broadly defined population than can be observed through RCTs –Topics where RCTs would be logistically or ethically inappropriate –Longer term outcomes and harms (side effects) The trade-off for wider applicability of findings among observational studies, compared with RCTs, is a potentially wider range of sources of bias, including in selection, performance, detection of effects, and attrition.

Background: Rating the Quality of Non-randomized Studies The quality (internal validity) of each study included in a review needs to be evaluated: Well-established criteria and instruments exist for evaluating the quality of RCTs, but not non-randomized (observational) studies PIs conducting systematic reviews generally lack access to validated and adaptable instruments for evaluating the quality of observational studies. Each new review often develops its own quality rating tool, “reinvents the wheel”, leading to inconsistent standards within and across reviews.

Project Goals To create a practical and validated tool for evaluating the quality of non-randomized studies of interventions or exposures that is: Reflects a comprehensive theoretical framework: captures all relevant domains Broad applicability: can be used "off the shelf" by different PIs Modifiable: can be adapted to different topic areas Easy to use and understand: can be used by reviewers with varying levels of expertise or experience Validated: users can be confident of their evaluation of study quality Advances the methodology in the field Disseminated widely

Methods: Phase 1 Item development Reviewed the literature on the evaluation of the quality of observational studies Collected quality review items used in early tools to evaluate non-RCTs through –Published literature –90 AHRQ-sponsored EPC reviews Categorized all potential items into the 12 quality domains identified in Evaluating non-randomized intervention studies (Deeks et al., 2003)

Methods: Phase 1 (continued) Item Bank development Selected the best items for measuring each of the included domains Modified selected items where necessary to ensure that critical domains were included and to improve readability Developed a pre-specified set of responses Developed explanatory text to be used by PIs and abstractors to individualize as well as standardize interpretation

Methods: Phase 2 Technical Expert Panel input –Conceptual framework to ensure that we included all relevant domains –Face validity Cognitive interviews with potential users –Readability –Conceptualization Validation –Content/face validity –Inter-rater reliability testing

Conceptual Underpinnings of the Instrument Evaluation of quality can rely on either a description of methods or an assessment of validity and precision Methods description approach –Follows the reporting structure of many manuscripts –Relies less on judgment than on reporting Validity and precision approach –What we really care about –More challenging to evaluate –Greater reliance on judgment

Domains for quality evaluation approaches Methods description approach Background/context Sample definition and selection Intervention/exposure Creation of treatment groups Follow-up Specification of outcomes Analysis: comparability of groups Analysis: outcomes Interpretation Validity and precision approach Selection bias Performance bias Information bias Detection bias Attrition bias Reporting bias Precision

Tool Results Comprehensive: bank of 39 questions Modifiable: includes relevant items appropriate for all non-randomized study types Easy to use: instructions for PIs and abstractors to assist in appropriate interpretation of questions. Example: What is the level of detail in describing the intervention or exposure? [PI: specify which details need to be stated, e.g., intensity, duration, frequency, route, setting, and timing of intervention/exposure. For case-control studies, consider if the condition, timing, frequency, and setting of symptoms is provided in the case definition]

Next Steps Finalize inter-rater reliability results Publish findings and disseminate the tool Proposed Phase III: –Design specific validation including inter-rater reliability testing by study type –Reduce the number of questions needed to address specific domains –Develop a web-based platform for generating design and topic-specific instruments from the item bank.