Presentation is loading. Please wait.

Presentation is loading. Please wait.

Clinical Epidemiology Boot Camp: Systematic Reviews Selina Liu MD MSc FRCPC Research Fellow Resident Research Career Development Program September 19,

Similar presentations


Presentation on theme: "Clinical Epidemiology Boot Camp: Systematic Reviews Selina Liu MD MSc FRCPC Research Fellow Resident Research Career Development Program September 19,"— Presentation transcript:

1 Clinical Epidemiology Boot Camp: Systematic Reviews Selina Liu MD MSc FRCPC Research Fellow Resident Research Career Development Program September 19, 2012

2 Outline  Introduction – Evidence-Based Medicine (EBM)  Levels of evidence  To discuss the definition of a systematic review  vs. traditional/narrative reviews  The process of conducting a systematic review  Strengths & limitations of systematic reviews  To describe how to critically appraise a systematic review  Example of a systematic review

3 Evidence-Based Medicine  What is Evidence-Based Medicine?  “…the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients”  “ It’s about integrating individual clinical expertise and the best external evidence”  philosophical origins – date back to mid-19 th century Paris (or possibly earlier) Sackett DL et al. BMJ. 1996;312(7023):71-2

4 Evidence-Based Medicine  Five Steps of Evidence-Based Medicine  1. Asking Focused Questions  Translation of uncertainty to an answerable question  2. Finding the Evidence  Systematic retrieval of the best evidence available  3. Critical Appraisal  Testing evidence for validity, clinical relevance, and applicability  4. Making a Decision  Application of results in practice  5. Evaluating Performance  Auditing evidence-based decisions Oxford Centre for Evidence-Based Medicine (CEBM)

5 Evidence-Based Medicine  Why Evidence-Based Medicine?  clinical decision making is complex! Mulrow CD, Cook DJ, Davidoff F. Ann Intern Med. 1997;126(5):389-91

6 Evidence-Based Medicine  How do we practice Evidence-Based Medicine? Can be difficult:  “ information overload”  difficult for clinicians to “keep up” with all of the latest evidence  often there are multiple studies examining the same or similar questions  may be of variable quality, generalizability  estimated time required for reading (general medicine):  19 articles per day, 365 days per year Davidoff F et al. BMJ. 1995;310(6987):1085-6

7 Evidence-Based Medicine  Weighing the evidence - “Levels of Evidence” OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence”, Oxford Centre for Evidence-Based Medicine.

8 Evidence-Based Medicine Systematic reviews of cohort studies of RCTs

9 Systematic Reviews  What is a Systematic Review?  the application of strategies that limit bias in the assembly, critical appraisal, and synthesis of all relevant studies on a specific topic  use rigorous, standardized methods for selecting & assessing articles Oxford Centre for Evidence-Based Medicine OR  a report that summarizes all evidence that can be drawn from research (or other sources), that is relevant to a specific clinical question

10 Systematic Reviews  Systematic Reviews vs. Traditional Review Articles  traditional review articles  written by senior expert in the field, summarizes evidence and recommendations  usually address broad areas/questions (i.e. “management of T2DM”)  often lack structure  may include personal experience/anecdotal evidence Fletcher RH & Fletcher SW Clinical Epidemiology: The Essentials

11 Systematic Reviews  Systematic Review vs. Traditional/Narrative Review Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med. 1997;126(5):

12 Systematic Reviews Guyatt G et al Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2 nd Edition)

13 Systematic Reviews  Process of Conducting a Systematic Review 1. Define the question 2. Conduct literature search 3. Apply inclusion and exclusion criteria 4. Create data abstraction 5. Conduct analysis Guyatt G et al Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2 nd Edition)

14 Systematic Reviews - Process  1. Define the Question  single, focused question  i.e. What is the effect of cinnamon on glycemic control in diabetes?  specify inclusion and exclusion criteria  Population, Intervention or Exposure, Outcome, Methodology  For the systematic review to be useful:  strong studies of the question should be available, but their results should not be so much in agreement that the question is already answered!  there should not be so few studies of the question that each individual study could be fully critiqued directly

15 Systematic Reviews - Process  2. Conduct literature search  need to ensure that all of the appropriate studies are included  NOT just a biased sample of studies  decide on information sources  i.e. MEDLINE, recent reviews, textbooks, experts in the field, articles cited by references already found by other approaches, databases of articles, clinical trial registries etc.  identify titles and abstracts

16 Systematic Reviews - Process  3. Apply inclusion and exclusion criteria  Apply inclusion and exclusion criteria to titles and abstracts  obtain full articles for eligible titles and abstracts  Apply inclusion and exclusion criteria to full articles  Select final eligible articles  Assess agreement on study selection  of the initial titles and abstracts retrieved, usually only a small proportion of articles are selected

17 Systematic Reviews - Process  4. Create data abstraction  Assess methodologic quality of each article  Assess agreement on validity assessment  Data abstraction  Participants  Interventions and Comparison Interventions  Study Design  Results

18 Systematic Reviews - Process  5. Conduct analysis  Summarize data  If appropriate: meta-analysis – statistical technique to combine quantitative data  usually combine studies vs. combine patients  Describe results – often graphically  Forest Plot – shows point estimate and confidence interval (for RCTs, observational studies)  Summary Receiver-Operator Curves (for studies of diagnostic tests)  Explore heterogeneity, conduct subgroup analysis (if appropriate)  Explore possibility of publication bias (and other biases)

19 Systematic Reviews - Process  How to decide if appropriate to perform a meta-analysis?  Two general approaches:  1. statistical test for homogeneity  BUT – even if fail to reject H 0 (i.e. no evidence of a statistically significant difference between studies), usually have high risk of false-negative (saying studies are homogeneous when they really are not)  Limited power - meta-analyses are usually of few number of studies, - affected also by number of patients/study, distribution of patients among studies  2. informed judgement

20 Systematic Reviews - Process  Meta-analysis – mathematical models:  Fixed-Effect Model  Assumes that studies are of exactly the same question, so results differ only by chance  Confidence intervals calculated may imply more precision (i.e. are narrower) than in reality (since studies usually differ somewhat)  Random-Effects Model  Assumes that the studies address somewhat different questions, but that they form a closely related family of studies of a similar question  Studies taken to be a random sample of all studies bearing on the question  Produces WIDER confidence intervals (more “realistic”) Fletcher RH & Fletcher SW Clinical Epidemiology: the Essentials (4 th Edition)

21 Systematic Reviews – Forest Plot

22 Systematic Reviews - Bias  Several types of bias:  publication bias  published studies may be systematically different than unpublished studies (“positive” studies vs. “negative” studies?)  language bias  i.e. if only English-language articles are selected  size bias  large studies that result in several publications may be more readily noticed than smaller studies  bias related to funding?  industry-sponsored studies

23 How to detect publication bias?  Funnel plots – plot effect vs. study size/precision symmetrical, peaked distribution (inverted funnel) Guyatt G et al Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2 nd Edition)

24 How to detect publication bias?  Funnel plots asymmetrical distribution Guyatt G et al Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2 nd Edition)

25 Systematic Reviews - Strengths  provide an efficient way to become familiar with the best available research evidence for a focused clinical question  can establish whether results are consistent, generalizable across populations/settings, treatment variations, and whether findings vary by certain subgroups  can extend the available literature (if the review team has obtained unpublished information from the primary authors)  meta-analyses – may provide a more precise estimate of the underlying “true effect” than any individual study Garg AX, Hackam D, Tonelli M Clin J Am Soc Nephrol. 3(1):

26 Systematic Reviews - Limitations  summarized results are limited by the quality of the primary studies  “garbage in garbage out”  results dependent on selection of included articles  quality threshold, publication bias, language bias etc.  meta-analyses - may be inappropriate to mathematically combine primary study results if the primary studies differ in design, quality, population, intervention etc.  subjectivity involved in deciding whether to pool or not  subjectivity in interpretation of summarized results (“over- interpretation) Garg AX, Hackam D, Tonelli M Clin J Am Soc Nephrol. 3(1):

27 Systematic Reviews – Critical Appraisal Tonelli M, Hackam D, Garg AX. Methods Mol Biol. 2009;473: Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA. 1994;272(17):

28 Oxford Centre for Evidence-Based Medicine Critical Appraisal – Tools

29 Oxford Centre for Evidence-Based Medicine

30 Critical Appraisal – Tools Shea BJ, Grimshaw JM, Wells GA et al. BMC Med Res Methodol. 2007;7:10  AMSTAR – 2007  Assessment of Multiple SysTemAtic Reviews  Shea BJ, Grimshaw JM, Wells GA et al.  11 item tool  developed via exploratory factor analysis of a 37-item assessment tool

31 Critical Appraisal - Tools Shea BJ, Grimshaw JM, Wells GA et al. BMC Med Res Methodol. 2007;7:10

32 Reporting Systematic Reviews - Tools  The PRISMA Statement – 2009  Preferred Reporting Items for Systematic reviews and Meta- Analyses  Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group  PLoS Medicine  Annals of Internal Medicine  BMJ  Journal of Clinical Epidemiology  Open Medicine  International Journal of Surgery

33 The PRISMA Statement  Aim – to help authors improve the reporting of systematic reviews and meta-analyses  **NOT intended to be a quality assessment tool  27 item checklist  four-phase flow diagram  update and expansion of prior QUOROM statement  QUality Of Reporting Of Meta-analyses  focused on reporting of meta-analyses of randomized controlled trials

34 Table 1. Checklist of items to include when reporting a systematic review or meta-analysis. Moher D, Liberati A, Tetzlaff J, Altman DG, et al. (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e doi: /journal.pmed

35 Figure 1. Flow of information through the different phases of a systematic review. Moher D, Liberati A, Tetzlaff J, Altman DG, et al. (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e doi: /journal.pmed

36 Example of a Systematic Review Cochrane Database of Systematic Reviews 2012, Issue 9:CD007170

37 Cinnamon for Diabetes Mellitus  Objective – to evaluate the effects of cinnamon in patients with diabetes mellitus  Selection criteria – all RCTs comparing the effects of orally administered monopreparations of cinnamon (Cinnamomum spp.) to placebo, active medication or no treatment in persons with either type 1 or type 2 diabetes mellitus  Primary outcomes – fasting blood glucose, post-prandial glucose, adverse events  Secondary outcomes – HbA1c, serum insulin, HOMA-IR, HRQoL, morbidity, costs Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

38

39  Review authors’ judgements about each “risk of bias” item for each included study Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD Risk of Bias Summary

40  Review authors’ judgements about each “risk of bias” item presented as percentages across all included studies Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD Risk of Bias Graph

41  Primary outcome – fasting blood glucose Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD Results – Cinnamon vs. Placebo

42  Primary outcome – adverse events Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD Results – Cinnamon vs. Placebo

43  Secondary outcome – HbA1c Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD Results – Cinnamon vs. Placebo

44  Other primary outcomes:  Post-prandial glucose – 1trial  Other secondary outcomes:  Serum insulin – 2 trials  HOMA-IR – 2 trials  HRQoL, morbidity, costs – no trials Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD Results

45  Insufficient evidence to support the use of cinnamon for type 1 or type 2 diabetes  Further trials required:  To address methodologic issues in current studies  i.e. allocation concealment, blinding  To include other important endpoints  HRQOL, diabetes complications, cost Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD Conclusions

46 Useful Resources  Guyatt G, Rennie D, Meade MO, Cook DJ Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2 nd Edition). New York NY, McGraw-Hill  available online via Western Libraries  Oxford Centre for Evidence-Based Medicine (CEBM)   Cochrane Database of Systematic Reviews 

47 References  Sackett DL, Rosenberg WMC, Muir Gray JA, Haynes RB, Richardson WS. BMJ. 1996;312(7023):71-2  Mulrow CD, Cook DJ, Davidoff F. Ann Intern Med. 1997;126(5):  Davidoff F, Haynes B, Sackett D, Smith R. BMJ. 1995;310(6987):  Oxford Centre for Evidence-Based Medicine (CEBM)  OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence”, Oxford Centre for Evidence-Based Medicine.  Fletcher RH & Fletcher SW Clinical Epidemiology: the Essentials (4 th Edition). Baltimore MD, Lippincott Williams & Wilkins  Guyatt G, Rennie D, Meade MO, Cook DJ Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2 nd Edition). New York NY, McGraw-Hill  Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med. 1997;126(5):  Garg AX, Hackam D, Tonelli M Clin J Am Soc Nephrol. 3(1):  Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA. 1994;272(17):  Tonelli M, Hackam D, Garg AX. Methods Mol Biol. 2009;473:  Cochrane Database of Systematic Reviews


Download ppt "Clinical Epidemiology Boot Camp: Systematic Reviews Selina Liu MD MSc FRCPC Research Fellow Resident Research Career Development Program September 19,"

Similar presentations


Ads by Google