1. Author(s): Joel J. Gagnier M.Sc., Ph.D., 2011 License: Unless otherwise noted, this material is made available under the terms of the Creative Commons Attribution–Noncommercial–Share Alike 3.0 License: http://creativecommons.org/licenses/by-nc-sa/3.0/ We have reviewed this material in accordance with U.S. Copyright Law and have tried to maximize your ability to use, share, and adapt it. The citation key on the following slide provides information about how you may share and adapt this material. Copyright holders of content included in this material should contact open.michigan@umich.edu with any questions, corrections, or clarification regarding the use of content. For more information about how to cite these materials visit http://open.umich.edu/education/about/terms-of-use. Any medical information in this material is intended to inform and educate and is not a tool for self-diagnosis or a replacement for medical evaluation, advice, diagnosis or treatment by a healthcare professional. Please speak to your physician if you have questions about your medical condition. Viewer discretion is advised: Some medical content is graphic and may not be suitable for all viewers.

2. Attribution Key for more information see: http://open.umich.edu/wiki/AttributionPolicy Use + Share + Adapt Make Your Own Assessment Creative Commons – Attribution License Creative Commons – Attribution Share Alike License Creative Commons – Attribution Noncommercial License Creative Commons – Attribution Noncommercial Share Alike License GNU – Free Documentation License Creative Commons – Zero Waiver Public Domain – Ineligible: Works that are ineligible for copyright protection in the U.S. (USC 17 § 102(b)) *laws in your jurisdiction may differ Public Domain – Expired: Works that are no longer protected due to an expired copyright term. Public Domain – Government: Works that are produced by the U.S. Government. (USC 17 § 105) Public Domain – Self Dedicated: Works that a copyright holder has dedicated to the public domain. Fair Use: Use of works that is determined to be Fair consistent with the U.S. Copyright Act. (USC 17 § 107) *laws in your jurisdiction may differ Our determination DOES NOT mean that all uses of this 3rd-party content are Fair Uses and we DO NOT guarantee that your use of the content is Fair. To use this content you should do your own independent analysis to determine whether or not your use will be Fair. { Content the copyright holder, author, or law permits you to use, share and adapt. } { Content Open.Michigan believes can be used, shared, and adapted because it is ineligible for copyright. } { Content Open.Michigan has used under a Fair Use determination. }

3. LECTURE 5: INTERPRETING RESULTS, CRITICAL APPRAISAL OF SYSTEMATIC REVIEWS Joel J. Gagnier MSc, PhD

4. Interpreting results  You need to describe how you are going to go about summarizing and interpreting the results you end up with  This should be a general discussion of the things you will be considering when making this interpretation

5. When interpreting results 1.Consider limitations  ROB, publication bias 2.Consider strength of evidence  Effect size, variance  Compared to other reviews 3.Consider applicability  External generalizability  Who can you apply results to outside the studies  Treatment description (reporting)  If you wanted to apply the treatment is there enough a description of what was done (dose, delivery form etc)

6. When interpreting results 4.Consider benefit to harm ratio  Make some judgment as to the risk:benefit ratio 5.Consider economic evaluation  Affordable  Relative to other treatments 6.Consider implications for future research  Suggest what future research should be done to help answer the questions you have now come up with

7. Interpret Results Make recommendations  State plans on describing what this adds to current knowledge in the area  How (dose, frequency etc), in whom, when should treatment be used  Chance to improve the clinical practice

8. Methods of presenting results  Can provide a summary table  Outline of important details of the included studies  Also may plan on summarizing the results and quality of each study in the results section of your systematic review  People can refer to the table for more detailed information

9. Methods of presenting the results  Table of ROB assessment, details of the intervention etc  Forest plots, funnel plots, other relevant plots  Idea: You should be able to generally explain how you are going to present your findings (visually or verbally) within the manuscript of your review

10. PRISMA Statement  Developed to improve the quality of reporting of systematic reviews.  List of criteria for reporting your systematic review  It is available at http://www.prisma- statement.org/statement.htmhttp://www.prisma- statement.org/statement.htm

11. Work plan and Timeline  Define discrete activities to be done by certain dates  Timelines  Create a figure outlining activities to be done.

12. Systematic Review Critical Appraisal  Why?  If using reviews to guide decisions about health-care, misleading reviews can be deadly  Accurate reviews can change practice Avoid; harmful or ineffective therapies Include; efficacious prognostic marker, diagnostic technique or treatment

13. Two things to worry about in reviews  Systematic error  Bias Validity: extend to which design and conduct protect against bias  Unsystematic error  Chance variation Confidence intervals help us Measure of precision (dispersion)

14. Checklists for assessment of systematic reviews  A number have been created  Oxman and Guyatt criteria (J Clin Epi 1991, v44 p91-98; p1271-1278)  Validated and has been expanded upon  All focus on the same sources of bias

15. Bias in systematic reviews Problem formulation  Is the question clearly focused? Study identification  Is the search for relevant studies thorough? Study selection  Are the inclusion criteria appropriate? Appraisal of studies

16. Bias in systematic reviews Data collection  Is missing information obtained from investigators? Data synthesis  How sensitive are the results to changes in the way the review was done?

17. Bias in systematic reviews Interpretation of results  Do the conclusions flow from the evidence that is reviewed?  Are the recommendations linked to the strength of the evidence?  Are judgments about preferences (values) explicit?  If there is “no evidence of effect” is caution taken not to interpret this as “evidence of no effect”?  Are subgroup analyses interpreted cautiously?

18. Oxman/Guyatt Criteria (1) Were the search methods reported? (2) Was the search comprehensive? (3) Were the inclusion criteria reported? (4) Was selection bias avoided? (5) Were the validity criteria reported? (6) Was validity assessed appropriately? (7) Were the methods used to combine studies reported? (8) Were the findings combined appropriately? (9) Were the conclusions supported by the reported data? (10) What was the overall scientific quality of the overview?

19. Assessment of Validity of a Systematic Review  Users’ Guides to the Medical Literature: VI How to use and overview  Oxman, A., Cook, D., Guyatt, G., JAMA (1994) Validity: 1. Did the overview address a focused clinical question? 2. Were the criteria used to select articles for inclusion appropriate? 3. Is it unlikely that important, relevant studies were missed? 4. Was the validity of the included studies appraised? 5. Were assessments of studies reproducible? 6. Were the results similar from study to study?

20. Results What are the results? 1. What are the overall results of the review? 2. How precise were the results? Will the results help me in caring for my patients? 1. Can the results be applied to my patient care? 2. Were all clinically important outcomes considered? 3. Are the benefits worth the harms and costs?

21. Expanded Criteria: For clinical questions!! I. Are the results of the study valid?  Primary guides: 1. Did the overview address a focused clinical question? 2. Were the criteria used to select articles for inclusion appropriate?

22. 1. Did the overview address a focused clinical question?  Most clinical questions can be formulated in terms of a simple relationship between the patient, some exposure (to a treatment, a diagnostic test, a cause, etc), and one or more specific outcomes of interest.  If the main question is not clear from the title or abstract, a good idea to move on to the next article.

23. 2. Were the criteria used to select articles for inclusion appropriate?  Keep the question in mind  Types of trials, patients, exposures and outcomes of interest  If yes; less likely to be bias in article selection

24. Expanded/Modified Criteria I. Are the results of the study valid? B. Secondary guides: 3. Is it unlikely that important, relevant studies were missed? 4. Was the validity of the included studies appraised? 5. Were assessments of studies reproducible? 6. Were the results similar from study to study?

25. 3. Is it unlikely that important, relevant studies were missed?  Search strategy?  use of  bibliographic databases, such as MEDLINE and EMBASE  checking the reference lists of the articles that are retrieved  personal contact with experts in the area Missed or Unpublished trials  Controls for “publication bias”  Higher likelihood of published studies to have positive results  Tested with funnel plot

26. Source Undetermined

27. 4. Was the validity of the included studies appraised?  Trial quality assessment  Dual, independent and using explicit criteria  Important to trial quality Junk in = junk out Methodologically weak trials have inflated treatment effects Therefore: a review using methodologically weak trials has questionable validity

28. 5. Were assessments of studies reproducible?  Two or more people participate in each decision reduces bias or chance  Good agreement between the reviewers, increases confidence in results of the review  Cohen’s Kappa (kappa coefficient)  Raw % agreement

29. 6. Were the results similar from study to study?  Are the studies different enough to preclude combining results  Clinical heterogeneity= patient population, outcome measure, exposure (length, intensity etc.) are heterogeneous  Methodological heterogeneity  Statistical heterogeneity = when differences of effects (mean and SD) between trials is greater than that expected due to chance  If so; there is some factor causing a difference between the groups  Therefore it may not make sense to combine results Qualitative analysis

30. Expanded/Modified Criteria II. What are the results?  What are the overall results of the review?  How precise were the results?

31. What are the overall results of the review?  Vote counts are not good:  Number with +ve VS –ve Because large and small studies are treated equally This does not consider clinically significant effects of non-statistically significant results  Meta-analysts weight studies according to size  larger studies receive more weight  overall results represent a weighted average of the results of the individual studies (Weighted mean difference)  Studies may also be given more or less weight depending on their quality Sensitivity analyses can be carried-out (e.g., on strong VS weak studies)

32. What are the overall results of the review?  Look for a summary measure  Dichotomous outcomes (Odds ratio etc) Same outcome measures  Mean difference (SMD) Different outcome measures (standardized) Differences in outcomes which are weighted by standard deviation  Look for a presentation of the results that conveys their practical importance (for example by translating the summary effect size back into natural units)

33. How precise were the results?  95% confidence interval (CI)  Measure of dispersion  Precision increases with decreasing size of the CI  If Odds Ratio CI crosses 1 then no effect is evident

34. Expanded/Modified Criteria III. Will the results help me in caring for my patients?  Can the results be applied to my population of intent (e.g., patient)?  Were all clinically important outcomes considered?  Are the benefits worth the harms and costs?

35. Can the results be applied to my patient care?  Can use overall effect; but may be too general  Subgroup analyses can be used  Be weary of conclusions drawn from comparisons between studies It is quite easy for differences between groups within different studies to arise due to chance  Hypothesized differences between subgroups are likely credible if: Differences are large A priori hypothesis was tested for the subgroup analysis Consistency across studies Biologic plausibility  If not met use overall effect

36. Were all clinically important outcomes considered?  Focused reviews of the evidence for individual outcomes are more likely to provide valid results  Consider those outside of the review BUT  a clinical decision requires considering all of them

37. Are the benefits worth the harms and costs?  Explicitly or implicitly; when making a clinical decision the expected benefits must be weighed against the potential harms and costs  Implicit: discomfort/invasiveness VS outcome severity  Explicit: Risk:benefit ratio

38. What are the conclusions regarding this meta-analysis?  Good/bad?  Clinically relevant?  How might this change practice?

39. Work-on Protocols…..

40. Thank you!!