1. How to Use Systematic Reviews Primary Care Conference June 27, 2007 David Feldstein, MD

2. Objectives Systematically analyze validity of a meta-analysis Systematically analyze validity of a meta-analysis Discuss results of a meta-analysis Discuss results of a meta-analysis Apply evidence from a meta-analysis to patient care decisions Apply evidence from a meta-analysis to patient care decisions

3. Systematic Review vs. Meta-Analysis Systematic Review Systematic Review –Critical evaluation of research to answer a focused question where an attempt is made to evaluate all available research –Also known as an overview Meta-analysis Meta-analysis –Quantitative strategy to combine the results of multiple studies into a single pooled estimate

4. Case A 55 y.o. male with a history of an MI in 1995 presents to clinic for an annual physical. A 55 y.o. male with a history of an MI in 1995 presents to clinic for an annual physical. He has been feeling well, but is concerned about the risk of future MI’s and death. He has been feeling well, but is concerned about the risk of future MI’s and death. He is on ASA, statin, ACE and B-blocker He is on ASA, statin, ACE and B-blocker His brother who also has CAD told him that he should take vitamin E His brother who also has CAD told him that he should take vitamin E Should I recommend vitamin E? Should I recommend vitamin E?

5. Question In patients with CAD does vitamin E supplementation decrease the risk of death? In patients with CAD does vitamin E supplementation decrease the risk of death?

6. Search PubMed PubMed –Clinical Queries »Find Systematic Reviews »“Vitamin E and cardiovascular disease”

7. Articles Eidelman RS. Hollar D. Hebert PR. Lamas GA. Hennekens CH. Randomized trials of vitamin E in the treatment and prevention of cardiovascular disease. Archives of Internal Medicine. 164(14):1552-6, 2004 Jul 26. Eidelman RS. Hollar D. Hebert PR. Lamas GA. Hennekens CH. Randomized trials of vitamin E in the treatment and prevention of cardiovascular disease. Archives of Internal Medicine. 164(14):1552-6, 2004 Jul 26. Shekelle PG. Morton SC. Jungvig LK. Udani J. Spar M. Tu W. J Suttorp M. Coulter I. Newberry SJ. Hardy M. Effect of supplemental vitamin E for the prevention and treatment of cardiovascular disease. Journal of General Internal Medicine. 19(4):380-9, 2004 Apr. Shekelle PG. Morton SC. Jungvig LK. Udani J. Spar M. Tu W. J Suttorp M. Coulter I. Newberry SJ. Hardy M. Effect of supplemental vitamin E for the prevention and treatment of cardiovascular disease. Journal of General Internal Medicine. 19(4):380-9, 2004 Apr.

8. Steps in Critical Appraisal Is the study valid? Is the study valid? What are the results? What are the results? Can I apply the results to my patient? Can I apply the results to my patient?

9. Critical Appraisal Worksheet - Validity 1. Did the overview explicitly address a sensible clinical question? 2. Was the search for relevant studies detailed and exhaustive? 3. Were the primary studies of high methodologic quality? 4. Were assessments of studies reproducible? Guyatt G, Rennie D. Users' guides to the medical literature : a manual for evidence-based clinical practice. Chicago, IL: AMA Press; 2002.

10. Critical Appraisal First read the abstract First read the abstract Each article has sections that are labeled Each article has sections that are labeled Look at the labeled areas to answer the corresponding validity questions for your assigned article Look at the labeled areas to answer the corresponding validity questions for your assigned article Feel free to look at the other article if time permits Feel free to look at the other article if time permits

11. Validity Did the overview address a sensible clinical question? Did the overview address a sensible clinical question?

12. Validity Basic Principles - Sensible question Basic Principles - Sensible question –Is it reasonable to expect similar results across trials? –Consider: »Population »Intervention »Outcomes –Take into account inclusion and exclusion criteria

13. Validity Was the search for relevant studies detailed and exhaustive? Was the search for relevant studies detailed and exhaustive?

14. Validity Basic Principles – Search Basic Principles – Search –Sensitive search to find all published data »Multiple databases –Avoid language bias »Non-English speaking authors more likely to publish positive studies in English journals –Avoid publication bias »Larger studies or positive results more often published »Need to search for unpublished data FP

15. Validity Exhaustive search: Exhaustive search: –MEDLINE, CINAHL –Cochrane controlled trials registry (CENTRAL) –Foreign language literature (EMBASE and LILACS) –References cited in primary sources –Conference proceedings –Unpublished data from experts in field, authors, drug manufacturers

16. Validity Were the primary studies of high methodologic quality? Were the primary studies of high methodologic quality?

17. Jadad Score Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996

18. Validity Basic Principles - Quality of individual studies Basic Principles - Quality of individual studies –Should look at criteria similar to User’s Guides for individual studies –Often use predefined scoring systems, but is not necessary –Lower quality trials overestimate treatment benefit »Allocation Concealment »Blinding

19. Jadad Scores (Shekelle) Trial Jadad Score CHAOS3 GISSI3 Haeger0 HOPE3 SPACE4

20. Validity Were assessments of studies reproducible? Were assessments of studies reproducible?

21. Validity Basic Principles – Assessment reproducible Basic Principles – Assessment reproducible –2 people independently »Evaluate studies for inclusion »Abstract data »Assess validity –To prevent bias or random error –Check agreement between reviewers

22. Validity Conclusions Useless Perfect 0 1 2 3 4 5 6 7 8 9 10 X X = JGIM   = Archives

23. Results Where the results similar from study to study? Where the results similar from study to study? –Heterogeneity

24. Heterogeneity Are differences between studies due to chance alone Are differences between studies due to chance alone Check forest plot Check forest plot Often statistically checked using x 2 test Often statistically checked using x 2 test –not sensitive –p<0.1 implies heterogeneity I 2 test newer method for evaluating I 2 test newer method for evaluating Authors should provide reasons for heterogeneity a priori Authors should provide reasons for heterogeneity a priori

25. Favors TreatmentFavors Control 1

26. Favors TreatmentFavors Control 1

27. Forest Plots

32. Results (JGIM) What is the magnitude of the treatment effect? What is the magnitude of the treatment effect? –Mortality Vit E v. placebo (secondary prevention) –Look at the top of Figure 2 pg 384

33. Results (JGIM) Relative Risk for death with Vit E = 0.96

34. Results (JGIM) How precise is the treatment effect? How precise is the treatment effect? »RR death = 0.96 »95% confidence interval = 0.84-1.10

35. Applying Results Is our patient so different from those in the study that its results can not apply? Is our patient so different from those in the study that its results can not apply? –No reason to think that our patient would not have similar results Is the intervention feasible in our setting? Is the intervention feasible in our setting? –Certainly What are our patients values and preferences? What are our patients values and preferences? –Prevention of MI and Death

36. Patient Conclusion What would you do with this patient? What would you do with this patient? Well done meta-analysis of reasonable quality individual trials Well done meta-analysis of reasonable quality individual trials No evidence of decrease in death No evidence of decrease in death Fairly tight confidence intervals Fairly tight confidence intervals I would not recommend vitamin E I would not recommend vitamin E

37. Objectives/ Wrap-up Systematically analyze validity of a meta-analysis Systematically analyze validity of a meta-analysis Discuss results of a meta-analysis Discuss results of a meta-analysis Apply evidence from a meta-analysis to patient care decisions Apply evidence from a meta-analysis to patient care decisions

38. Egger M, Ebrahim S, Smith GD. Where now for meta-analysis? Int J Epidemiol. 2002;31(1):1-5.