1. Clinical Epidemiology Boot Camp: Systematic Reviews
Selina Liu MD MSc FRCPC
Research Fellow
Resident Research Career Development Program
September 19, 2012

2. Outline Introduction – Evidence-Based Medicine (EBM)
Levels of evidence
To discuss the definition of a systematic review
vs. traditional/narrative reviews
The process of conducting a systematic review
Strengths & limitations of systematic reviews
To describe how to critically appraise a systematic review
Example of a systematic review

3. Evidence-Based Medicine
What is Evidence-Based Medicine?
“…the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients”
“ It’s about integrating individual clinical expertise and the best external evidence”
philosophical origins – date back to mid-19th century Paris (or possibly earlier)
Sackett DL et al. BMJ. 1996;312(7023):71-2
Application of clinical epidemiology to patient care

4. Evidence-Based Medicine
Five Steps of Evidence-Based Medicine
1. Asking Focused Questions
Translation of uncertainty to an answerable question
2. Finding the Evidence
Systematic retrieval of the best evidence available
3. Critical Appraisal
Testing evidence for validity, clinical relevance, and applicability
4. Making a Decision
Application of results in practice
5. Evaluating Performance
Auditing evidence-based decisions
Oxford Centre for Evidence-Based Medicine (CEBM)

5. Evidence-Based Medicine
Why Evidence-Based Medicine?
clinical decision making is complex!
Why EBM? Need to evaluate risk vs. benefits (i.e. of diagnostic test, intervention etc.)
Mulrow CD, Cook DJ, Davidoff F. Ann Intern Med. 1997;126(5):389-91

6. Evidence-Based Medicine
How do we practice Evidence-Based Medicine? Can be difficult:
“information overload”  difficult for clinicians to “keep up” with all of the latest evidence
often there are multiple studies examining the same or similar questions
may be of variable quality, generalizability
estimated time required for reading (general medicine):
19 articles per day, 365 days per year
Davidoff F et al. BMJ. 1995;310(6987):1085-6
Why EBM? Need to evaluate risk vs. benefits (i.e. of diagnostic test, intervention etc.)

7. Evidence-Based Medicine
Weighing the evidence - “Levels of Evidence”
OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence”, Oxford Centre for Evidence-Based Medicine.

8. Evidence-Based Medicine
of RCTs
Systematic reviews of cohort studies

9. Systematic Reviews What is a Systematic Review?
the application of strategies that limit bias in the assembly, critical appraisal, and synthesis of all relevant studies on a specific topic
use rigorous, standardized methods for selecting & assessing articles
Oxford Centre for Evidence-Based Medicine OR
a report that summarizes all evidence that can be drawn from research (or other sources), that is relevant to a specific clinical question

10. Systematic Reviews Systematic Reviews vs. Traditional Review Articles
written by senior expert in the field, summarizes evidence and recommendations
usually address broad areas/questions (i.e. “management of T2DM”)
often lack structure
may include personal experience/anecdotal evidence
Fletcher RH & Fletcher SW Clinical Epidemiology: The Essentials

11. Systematic Reviews Systematic Review vs. Traditional/Narrative Review
Both retrospective observational studies
Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med. 1997;126(5):

12. Systematic Reviews
Guyatt G et al Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

13. Systematic Reviews Process of Conducting a Systematic Review
1. Define the question
2. Conduct literature search
3. Apply inclusion and exclusion criteria
4. Create data abstraction
5. Conduct analysis
Guyatt G et al Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

14. Systematic Reviews - Process
1. Define the Question
single, focused question
i.e. What is the effect of cinnamon on glycemic control in diabetes?
specify inclusion and exclusion criteria
Population, Intervention or Exposure, Outcome, Methodology
For the systematic review to be useful:
strong studies of the question should be available, but their results should not be so much in agreement that the question is already answered!
there should not be so few studies of the question that each individual study could be fully critiqued directly
Methodology – time, language, publication restrictions

15. Systematic Reviews - Process
2. Conduct literature search
need to ensure that all of the appropriate studies are included
NOT just a biased sample of studies
decide on information sources
i.e. MEDLINE, recent reviews, textbooks, experts in the field, articles cited by references already found by other approaches, databases of articles, clinical trial registries etc.
identify titles and abstracts

16. Systematic Reviews - Process
3. Apply inclusion and exclusion criteria
Apply inclusion and exclusion criteria to titles and abstracts
obtain full articles for eligible titles and abstracts
Apply inclusion and exclusion criteria to full articles
Select final eligible articles
Assess agreement on study selection
of the initial titles and abstracts retrieved, usually only a small proportion of articles are selected

17. Systematic Reviews - Process
4. Create data abstraction
Assess methodologic quality of each article
Assess agreement on validity assessment
Data abstraction
Participants
Interventions and Comparison Interventions
Study Design
Results

18. Systematic Reviews - Process
5. Conduct analysis
Summarize data
If appropriate: meta-analysis – statistical technique to combine quantitative data
usually combine studies vs. combine patients
Describe results – often graphically
Forest Plot – shows point estimate and confidence interval (for RCTs, observational studies)
Summary Receiver-Operator Curves (for studies of diagnostic tests)
Explore heterogeneity, conduct subgroup analysis (if appropriate)
Explore possibility of publication bias (and other biases)
Combining STUDIES vs. combining PATIENTS

19. Systematic Reviews - Process
How to decide if appropriate to perform a meta-analysis?
Two general approaches:
1. statistical test for homogeneity
BUT – even if fail to reject H0 (i.e. no evidence of a statistically significant difference between studies), usually have high risk of false-negative (saying studies are homogeneous when they really are not)
Limited power - meta-analyses are usually of few number of studies, affected also by number of patients/study, distribution of patients among studies
2. informed judgement

20. Systematic Reviews - Process
Meta-analysis – mathematical models:
Fixed-Effect Model
Assumes that studies are of exactly the same question, so results differ only by chance
Confidence intervals calculated may imply more precision (i.e. are narrower) than in reality (since studies usually differ somewhat)
Random-Effects Model
Assumes that the studies address somewhat different questions, but that they form a closely related family of studies of a similar question
Studies taken to be a random sample of all studies bearing on the question
Produces WIDER confidence intervals (more “realistic”)
Fletcher RH & Fletcher SW Clinical Epidemiology: the Essentials (4th Edition)

21. Systematic Reviews – Forest Plot
# of studies included, pattern of effect sizes (Tx vs Pbo), # stat sign studies, large vs small studies

22. Systematic Reviews - Bias
Several types of bias:
publication bias
published studies may be systematically different than unpublished studies (“positive” studies vs. “negative” studies?)
language bias
i.e. if only English-language articles are selected
size bias
large studies that result in several publications may be more readily noticed than smaller studies
bias related to funding?
industry-sponsored studies

23. How to detect publication bias?
Funnel plots – plot effect vs. study size/precision
symmetrical,
peaked distribution
(inverted funnel)
Guyatt G et al Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

24. How to detect publication bias?
Funnel plots
asymmetrical
distribution
Guyatt G et al Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

25. Systematic Reviews - Strengths
provide an efficient way to become familiar with the best available research evidence for a focused clinical question
can establish whether results are consistent, generalizable across populations/settings, treatment variations, and whether findings vary by certain subgroups
can extend the available literature (if the review team has obtained unpublished information from the primary authors)
meta-analyses – may provide a more precise estimate of the underlying “true effect” than any individual study
Garg AX, Hackam D, Tonelli M Clin J Am Soc Nephrol. 3(1):

26. Systematic Reviews - Limitations
summarized results are limited by the quality of the primary studies
“garbage in garbage out”
results dependent on selection of included articles
quality threshold, publication bias, language bias etc.
meta-analyses - may be inappropriate to mathematically combine primary study results if the primary studies differ in design, quality, population, intervention etc.
subjectivity involved in deciding whether to pool or not
subjectivity in interpretation of summarized results (“over- interpretation)
Garg AX, Hackam D, Tonelli M Clin J Am Soc Nephrol. 3(1):

27. Systematic Reviews – Critical Appraisal
Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA. 1994;272(17):
Tonelli M, Hackam D, Garg AX. Methods Mol Biol. 2009;473:217-33

28. Critical Appraisal – Tools
Oxford Centre for Evidence-Based Medicine

29. Critical Appraisal – Tools
Oxford Centre for Evidence-Based Medicine

30. Critical Appraisal – Tools
AMSTAR – 2007
Assessment of Multiple SysTemAtic Reviews
Shea BJ, Grimshaw JM, Wells GA et al.
11 item tool
developed via exploratory factor analysis of a 37-item assessment tool
Shea BJ, Grimshaw JM, Wells GA et al. BMC Med Res Methodol. 2007;7:10

31. Critical Appraisal - Tools
Shea BJ, Grimshaw JM, Wells GA et al. BMC Med Res Methodol. 2007;7:10

32. Reporting Systematic Reviews - Tools
The PRISMA Statement – 2009
Preferred Reporting Items for Systematic reviews and Meta- Analyses
Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group
PLoS Medicine
Annals of Internal Medicine
BMJ
Journal of Clinical Epidemiology
Open Medicine
International Journal of Surgery

33. The PRISMA Statement
Aim – to help authors improve the reporting of systematic reviews and meta-analyses
**NOT intended to be a quality assessment tool
27 item checklist
four-phase flow diagram
update and expansion of prior QUOROM statement
QUality Of Reporting Of Meta-analyses
focused on reporting of meta-analyses of randomized controlled trials

34. Table 1. Checklist of items to include when reporting a systematic review or meta-analysis.
Moher D, Liberati A, Tetzlaff J, Altman DG, et al. (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e doi: /journal.pmed

35. Figure 1. Flow of information through the different phases of a systematic review.
Moher D, Liberati A, Tetzlaff J, Altman DG, et al. (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e doi: /journal.pmed

36. Example of a Systematic Review
Cochrane Database of Systematic Reviews 2012, Issue 9:CD007170

37. Cinnamon for Diabetes Mellitus
Objective – to evaluate the effects of cinnamon in patients with diabetes mellitus
Selection criteria – all RCTs comparing the effects of orally administered monopreparations of cinnamon (Cinnamomum spp.) to placebo, active medication or no treatment in persons with either type 1 or type 2 diabetes mellitus
Primary outcomes – fasting blood glucose, post-prandial glucose, adverse events
Secondary outcomes – HbA1c, serum insulin, HOMA-IR, HRQoL, morbidity, costs
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

38. Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

39. Risk of Bias Summary
Review authors’ judgements about each “risk of bias” item for each included study
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

40. Risk of Bias Graph
Review authors’ judgements about each “risk of bias” item presented as percentages across all included studies
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

41. Results – Cinnamon vs. Placebo
Primary outcome – fasting blood glucose
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

42. Results – Cinnamon vs. Placebo
Primary outcome – adverse events
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

43. Results – Cinnamon vs. Placebo
Secondary outcome – HbA1c
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

44. Results Other primary outcomes: Post-prandial glucose – 1trial
Other secondary outcomes:
Serum insulin – 2 trials
HOMA-IR – 2 trials
HRQoL, morbidity, costs – no trials
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

45. Conclusions
Insufficient evidence to support the use of cinnamon for type 1 or type 2 diabetes
Further trials required:
To address methodologic issues in current studies
i.e. allocation concealment, blinding
To include other important endpoints
HRQOL, diabetes complications, cost
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

46. Useful Resources
Guyatt G, Rennie D, Meade MO, Cook DJ Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition). New York NY, McGraw-Hill
available online via Western Libraries
Oxford Centre for Evidence-Based Medicine (CEBM)
Cochrane Database of Systematic Reviews

47. References
Sackett DL, Rosenberg WMC, Muir Gray JA, Haynes RB, Richardson WS. BMJ. 1996;312(7023):71-2
Mulrow CD, Cook DJ, Davidoff F. Ann Intern Med. 1997;126(5):389-91
Davidoff F, Haynes B, Sackett D, Smith R. BMJ. 1995;310(6987):1085-6
Oxford Centre for Evidence-Based Medicine (CEBM)
OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence”, Oxford Centre for Evidence-Based Medicine.
Fletcher RH & Fletcher SW Clinical Epidemiology: the Essentials (4th Edition). Baltimore MD, Lippincott Williams & Wilkins
Guyatt G, Rennie D, Meade MO, Cook DJ Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition). New York NY, McGraw-Hill
Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med. 1997;126(5):
Garg AX, Hackam D, Tonelli M Clin J Am Soc Nephrol. 3(1):
Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA ;272(17):
Tonelli M, Hackam D, Garg AX. Methods Mol Biol. 2009;473:217-33
Cochrane Database of Systematic Reviews