1. 116 (27%) 185 (43%) 49 (11%) How to critically appraise a systematic review Igho J. Onakpoya MD MSc University of Oxford Centre for Evidence-Based Medicine Nuffield Department of Primary Care Health Sciences Oxford igho.onakpoya@phc.ox.ac.uk

5. Learning outcomes  Principles of critical appraisal of SRs  How to undertake critical appraisal of SRs  How to appraise published SRs  Assessing the relevance of published SRs  2 sessions  Part I - Teaching (90 mins)  Part II - Class activity (90 mins)  1 short break in session 1 (5 mins)  15 minute break between sessions 1 and 2

6. Opening scenario Ms X is a 35 year old banker with five children. She lost her dad who was hypertensive three years ago due to a heart attack. She is worried about her risk of having high blood pressure but does not want to take any conventional medicines because “they have lots of side effects”. Her close friend has told her to take green tea but she is not sure whether “it works”. She therefore wants your opinion on whether this a good idea.

7. Get on the internet

8. Let’s see PubMed Clinical Queries!

9. Decide on which review/reviews meet your need.

10. Actually, we have done ours too!

11. What is a systematic review? Systematic identification, appraisal and synthesis of the evidence from studies addressing the same research question.

12. Why systematic reviews? Increasing strength of evidence Traditional (narrative) reviews

13. Don’t accept the conclusions of every systematic review hook, line and sinker!

14. How to read a systematic review Start with the title!

15. Next step: Useful pneumonic  Q : Was there a clear research question?  F : Did they find all relevant studies?  A : Was study quality assessed?  S : Were the results adequately summarized?

16. Step 1: Is there a clear and focussed research Q uestion?

17. Be sure of what you are looking for.

18. Clear Question?  Participants  Intervention  Comparator  Outcome

20. Based on the paper Onakpoya et al. Q : What is the PICO?

21. P: Normotensive or hypertensive adults I: Green tea C: Placebo/identical controls O: Blood pressure

22. Step 2: How did they Find the studies? 1.Which databases were searched, how many? 2.Was the search period specified? 3.Was grey literature searched? 4.Did they contact study authors? 5.Did they search bibliography?

23. F: How were studies included?  Independent determination of study eligibility  How many were eligible?  How many were excluded?  Were the reasons for exclusion reported?  Is there a list of excluded studies?  Resolution of disagreements

24. Flow charts could be helpful

25. QF

26. Step 3: Did they Assess study quality? Many quality assessment checklists available:  Cochrane  Jadad  CONSORT  STROBE  GRACE

27. Quality rating can be subjective

28. Onakpoya et al. The effect of cactus pear (Opuntia ficus-indica) on body weight and cardiovascular risk factors: a systematic review and meta-analysis of randomized clinical trials. Nutrition. 2015 May;31(5):640-6 Cochrane Assessing the risk of bias

29. Step 4: Did they Summarize the evidence? 1.Did they extract results data from included studies? 2.Did they statistically combine the data? 3.Did it make sense to combine the data?

30. Statistical data combination http://www.cochrane.org/

31. Basics of meta-analysis  Statistically combines data from individual studies: forest plot Uses fixed or random effects model  Can test for variations in trial designs and participants: heterogeneity  Can test for publication bias: funnel plot

32. What is a meta-analysis? A way to calculate an average Estimates an ‘average’ or ‘common’ effect Improves the precision of an estimate by using all available data

33. What is a meta-analysis? Optional part of a systematic review Systematic reviews Meta-analyses

35. At the bottom there’s a horizontal line. This is the scale measuring the treatment effect.

36. The vertical line in the middle is where the treatment and control have the same effect – there is no difference between the two

37. For each study there is an id The data for each trial are here, divided into the experimental and control groups This is the % weight given to this study in the pooled analysis

38. The label above the graph tells you what statistic has been used The data shown in the graph are also given numerically

39. The pooled analysis is given a diamond shape where the widest bit in the middle is located at the calculated best guess (point estimate), and the horizontal width is the confidence interval Note on interpretation If the confidence interval crosses the line of no effect, this is equivalent to saying that we have found no statistically significant difference in the effects of the two interventions

40. trials Confidence interval Line of no effect Overall effect

41. Effect of green tea supplementation on systolic blood pressure (mmHg).

42. Did the authors interpret the results correctly?

44. Beware of missing data!

45. Appendix 1a: Funnel plot of mean differences plotted against sample size. The vertical line represents the mean difference for all trials.

46. Based on the review findings what advice would you give Ms X? And why?

47. Questions to ponder on  Do the results appear valid and reliable?  How widely applicable are the study findings?  Are the results relevant to my local population?  Do the results apply to this patient/individual?

48. Useful pneumonic  Q : Was there a clear research question?  F : Did they find all relevant studies?  A : Was study quality assessed?  S : Were the results adequately summarized?

50. Part II: Class activity  Critically appraise the systematic review: 45 minutes  2 groups will look at chlorogenic acids  2 groups will look at steviol glycosides  Nominate someone to present for your group  We re-group at 4:30 pm  5 minute presentation summary of your review

51. Class activity

52. Any Questions?

53. Useful resources  Cochrane Handbook of Systematic Reviews of Interventions [updated March 2011]. http://handbook.cochrane.org/ http://handbook.cochrane.org/  Gough D, Oliver S, Thomas J. An Introduction to Systematic Reviews. SAGE Publications Ltd. 2012  PRISMA. Transparent Reporting of Systematic Reviews and Meta-Analyses. http://www.prisma- statement.org/index.htmhttp://www.prisma- statement.org/index.htm