Conducting a Sound Systematic Review: Balancing Resources with Quality Control Eric B. Bass, MD, MPH Johns Hopkins University Evidence-based Practice Center

Learning Objectives  Identify steps in a systematic review that have greatest implications for resource needs  Explain how those steps are vulnerable to error/bias  Identify strategies for quality control when resources are limited

Steps in the systematic review Search literature Review citations Extract data Evaluate study quality & applicability Summarize & synthesize evidence Establish eligibility criteria for each question

Focus the Question(s)  Formulate in specific measurable terms  Clarify priorities  Be realistic about time & effort  Negotiate with sponsor

Set Eligibility Criteria  Study design –RCT only? –Comparison group? –Any observational study?  Study population  Study setting  Size of study  Year of publication  Peer-reviewed?  Language

Search Literature  Select highest yield sources –MEDLINE ® –EMBASE ® –Cochrane database of reviews & trials  Consider other sources –Use sampling to estimate incremental yield  Use hand searches for quality control –Citations in eligible articles –Table of contents of relevant journals –Query experts

Review Citations  Titles –Many citations obviously not relevant & can be excluded quickly  Abstracts –Usually sufficient to determine eligibility of citations  Full text of articles –For some studies, eligibility can only be determined by reading full text

Review Citations: An Example Title review (n=10,475) Title review: 6,863 excluded Abstract review (n=3,612) Abstract review: 3,163 excluded Full-text review (n=449) Full-text review: 386 excluded 63 included articles

Review Citations: Quality Control  Use independent dual reviewers  Assess samples early  Discuss discrepancies

Extract Data  Set priorities for data extraction –Anticipate content of final evidence tables –Resist temptation to extract everything  Choose data management method –Systematic review software vs. Access vs. tables –Consider complexity, consistency, completeness of data  Establish quality control –Dual review vs. solo review with random checks –Independent vs. sequential dual review –Discuss discrepancies

Evaluate Study Quality & Applicability  Decide how evaluation will be used –To determine eligibility of studies –To give more or less weight to studies –To improve future research  Focus on most important aspects of quality & applicability  Use established instrument(s) –Jadad criteria for RCTs  Establish quality control –Independent dual review –Decide how to reconcile discrepancies

Summarize & Synthesize Evidence  Assemble evidence tables –Reassess priorities –Exclude duplicative data  Prepare summary of evidence on each question  Assess strength of evidence –Use 2 or more reviewers –Decide how to reconcile differences

Summary  To conduct a sound systematic review with limited resources: –Pay attention to priorities when defining questions, eligibility criteria & elements of evidence tables –Establish quality control measures for each step that could introduce error or bias reviewing citations extracting data evaluating quality, applicability & overall strength of evidence