1. Summary of Findings & Assessment of Quality of Evidence: Grade Workshop Sunday, October 17, 2010 0900 to 1700 Introduction

2. Introduction to facilitators Michelle Kho Jan Brozek Nancy Santesso Holger Schunemann Ingvil von Mehren Sæterdal

3. Agenda Mix of presentations, interactive sessions, hands-on work and small group discussions

4. Systematic review process

7. Risk of Bias

9. Meta-analysis

10. Sensitivity analyses High versus lower protein diets (studies with <20% losses to follow-up) Change in Systolic blood pressure (mmHg)

11. Heterogeneity

12. Subgroup analysis

13. Funnel Plot Medline Search Strategy for RCTs and Reviews -------------------------------------------------------------------------------- 1 diet, protein-restricted/ 2 diet, carbohydrate-restricted/ 3 1 or 2 4 diet fads/ 5 (carbohydrate* or protein*).ti,ab. 6 4 and 5 7 exp dietary proteins/ 8 dietary carbohydrates/ 9 (diet* or intake*).ti,ab. 10 (high* or increas* or rich or low* or restrict* or decreas* or reduc*).ti,ab. 11 (7 or 8) and 9 and 10 12((carbohydrate* or protein*) adj3 (high* or increas* or rich or low* or restrict* or decreas* or reduc*)).ti,ab. 13 12 and 9 14 3 or 6 or 11 or 13 15 randomized controlled trial.pt. 16 controlled clinical trial.pt. 17 randomized.ab. 18 placebo.ab. 19 clinical trials as topic.sh. 20 randomly.ab. 21 trial.ti. 22 or/15-21 23 humans.sh. 24 22 and 23 25 14 and 24

14. Systematic review process

15. Chapter 11: Presenting results and Summary of Findings Tables Chapter 12: Interpreting results and drawing conclusions Cochrane Handbook

16. Overview: Interpreting results of a review and GRADE how does GRADE fit into the process of moving from results to conclusions in systematic reviews what are the basic principles behind GRADE

17. Consider the following examples of moving from results to conclusions How would you interpret the results of the meta-analyses and conclusions made by the authors?

18. Authors’ conclusions Short term beneficial effects were found for fasting for 7 to 10 days followed by a vegetarian diet when compared to ordinary diet.

19. The pooled SMD for pain reduction comparing glucosamine to placebo was 0.61, which represents a moderate clinically significant treatment benefit in favour of glucosamine

21. What information do you think would increase or decrease your confidence in these results? What information do you think would indicate that more research is or is not necessary? Work with your neighbor and discuss for 5 mins

24. To make conclusions consider.... likelihood of effect and confidence in that effect

25. To make conclusions consider.... Likelihood/magnitude of effect – Mood will drop by 5 points on a scale of 0 to 10 when on a high protein diet. quality of the evidence – Low: This estimate will likely change with more research.

26. Summary of Findings Table a summary of the key findings from the systematic review for users a transparent aid and record of the authors’ interpretation of the results to make conclusions

27. Weighing the criteria for overall quality of evidence In fact in this example, – Allocation concealment is unclear in one of the studies – Only three of five studies measured major bleeding - a primary outcome in anticoagulation studies – suggesting selective outcome reporting – The confidence intervals include potential for harm or no harm I might say that my confidence in the results is “low” and that more research is likely to change the results

28. The pooled SMD for pain reduction comparing glucosamine to placebo was 0.61, which represents a moderate clinically significant treatment benefit in favour of glucosamine

29. Likelihood of and confidence in an outcome

30. Quality of evidence across studies for an outcome  HighFurther research is very unlikely to change our confidence in the estimate of effect or accuracy.   ModerateFurther research is likely to have an important impact on our confidence in the estimate of effect or accuracy and may change the estimate.   LowFurther research is very likely to have an important impact on our confidence in the estimate of effect or accuracy and is likely to change the estimate.   Very lowAny estimate of effect or accuracy is very uncertain.

31. GRADE: recommendation – quality of evidence Clear separation: 1) 4 categories of quality of evidence:  (High),   (Moderate),   (Low),   (Very low) ? – methodological quality of evidence – likelihood of bias – by outcome and across outcomes 2) Recommendation: 2 grades – conditional (aka weak) or strong (for or against an intervention)? – Balance of benefits and downsides, values and preferences, resource use and quality of evidence *www.GradeWorking-Group.org

32. GRADE Quality of Evidence In the context of a systematic review The quality of evidence reflects the extent to which we are confident that an estimate of effect is correct. In the context of making recommendations The quality of evidence reflects the extent to which our confidence in an estimate of the effect is adequate to support a particular recommendation.

33. Determinants of quality RCTs  observational studies   5 factors that can lower quality 1.limitations in detailed design and execution (risk of bias criteria) 2.Inconsistency (or heterogeneity) 3.Indirectness (PICO and applicability) 4.Imprecision (number of events and confidence intervals) 5.Publication bias 3 factors can increase quality 1.large magnitude of effect 2.all plausible residual confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed 3.dose-response gradient

34. 1. Design and Execution/Risk of Bias Examples: Inappropriate selection of exposed and unexposed groups Failure to adequately measure/control for confounding Selective outcome reporting Failure to blind (e.g. outcome assessors) High loss to follow-up Lack of concealment in RCTs Intention to treat principle violated

35. Design and Execution/RoB From Cates, CDSR 2008

36. Design and Execution/RoB Overall judgment required

37. 2. Inconsistency of results (Heterogeneity) if inconsistency, look for explanation – patients, intervention, comparator, outcome if unexplained inconsistency lower quality

38. Reminders for immunization uptake

40. Judgment – variation in size of effect – overlap in confidence intervals – statistical significance of heterogeneity –I2–I2

41. Inconsistency when 1 study? Do not downgrade

42. 3. Directness of Evidence generalizability, transferability, applicability differences in – populations/patients (HIC – L/MIC, women in general – pregnant women) – interventions (all techniques, new - old) – comparator appropriate (newer technique – old or no technique) – outcomes (important – surrogate: CIN I – cancer) indirect comparisons – interested in A versus B – have A versus C and B versus C – Cryo + antibiotics versus no intervention versus Cryo - antibiotics

43. EVIDENCE PROFILE Question: Cyrotherapy with antibiotics vs no antibiotics for histologically confirmed CIN 1 All rates presented at 12 months with assumption that events would occur within this time frame. 2 Indirect analysis between single arm observational studies Quality assessmentNo of patientsEffect QualityImportance No of studies DesignLimitations Inconsisten cy IndirectnessImprecisionOther Cryotherap y with antibiotics No antibiotics Relative (95% CI) Absolute Major infection (follow-up 12 months 1 ; requiring hospitalisation or blood transfusion) 16observation al studies no serious limitations no serious inconsistenc y serious 2 no serious imprecision none 0/1600 (0%) 10/4573 (0.22%) RD 0 (0 to 0) 0 fewer per 1000  IMPORTANT Resource use - not measured All severe adverse events (follow-up 12 months; (major infections and bleeding, pelvic inflammatory disease, stenosis, etc ) 17observation al studies no serious limitations no serious inconsistenc y serious 2 no serious imprecision none 0/1705 (0%) 22/5142 (0.43%) RD 0 (0 to 0) 0 fewer per 1000  IMPORTANT

44. 4. Publication Bias Should always be suspected – Only small “positive” studies – For profit interest – Various methods to evaluate – none perfect, but clearly a problem

45. Egger M, Smith DS. BMJ 1995;310:752-54 45 I.V. Mg in acute myocardial infarction Publication bias Meta-analysis Yusuf S.Circulation 1993 ISIS-4 Lancet 1995

46. Egger M, Cochrane Colloquium Lyon 2001 46 Funnel plot Standard Error Odds ratio 0.10.313 3 2 1 0 100.6 Symmetrical: No publication bias

47. Egger M, Cochrane Colloquium Lyon 2001 47 Funnel plot Standard Error Odds ratio 0.10.313 3 2 1 0 100.6 Asymmetrical: Publication bias? 0.4

48. 5. Imprecision Small sample size – small number of events Wide confidence intervals – uncertainty about magnitude of effect Extent to which confidence in estimate of effect adequate to support decision

49. Example: Immunization in children

51. For systematic reviews If the 95% CI excludes a relative risk (RR) of 1.0 and the total number of events or patients exceeds the OIS criterion, precision is adequate. If the 95% CI includes appreciable benefit or harm (we suggest a RR of under 0.75 or over 1.25 as a rough guide) rating down for imprecision may be appropriate even if OIS criteria are met.

52. Optimal information size We suggest the following: if the total number of patients included in a systematic review is less than the number of patients generated by a conventional sample size calculation for a single adequately powered trial, consider rating down for imprecision. Authors have referred to this threshold as the “optimal information size” (OIS)

53. 025.0%

54. 0

55. 0

56. 0

57. 2.0 0.50 Ischemic stroke point estimate and confidence interval Figure 1, Rating down for imprecision in guidelines: Thresholds are key Favors Intervention Favors Control Risk difference (%) Threshold if side effects and toxicity appreciable, NNT = 100. Confidence interval crosses threshold, rate down for imprecision Threshold if side effects, toxicity and cost minimal, NNT = 200. Entire confidence interval to left of threshold, do not rate down for imprecision

58. Figure 2: Corticosteroids to reduce hospital mortality in septic shock

59. Figure 4: Optimal information size given alpha of 0.05 and beta of 0.2 for varying control event rates and relative risks For any chosen line, evidence meets optimal information size criterion if sample size above the line

60. Total Number of EventsRelative Risk ReductionImplications for meeting OIS threshold 100 or less< 30%Will almost never meet threshold whatever control event rate 20030%Will meet threshold for control event rates for ~ 25% or greater 20025%Will meet threshold for control event rates for ~ 50% or greater 20020% Will meet threshold only for control event rates for ~ 80% or greater 300> 30%Will meet threshold 30025%Will meet threshold for control event rates ~ 25% or greater 30020%Will meet threshold for control event rates ~ 60% or greater 400 or more> 25%Will meet threshold for any control event rate 400 or more20%Will meet threshold for control event rates of ~ 40% or greater Table 1: Optimal information size implications from Figure 5

61. What can raise quality? 1. large magnitude can upgrade (RRR 50%/RR 2) – very large two levels (RRR 80%/RR 5) – criteria everyone used to do badly almost everyone does well

62. BMJ 2003 BMJ, 2003

63. Reminders for immunization uptake

64. What can raise quality? 2. dose response relation – (higher INR – increased bleeding) – childhood lymphoblastic leukemia risk for CNS malignancies 15 years after cranial irradiation no radiation: 1% (95% CI 0% to 2.1%) 12 Gy: 1.6% (95% CI 0% to 3.4%) 18 Gy: 3.3% (95% CI 0.9% to 5.6%)

65. In terms of high altitude sickness, symptoms generally do not manifest below 1500 m. From about 1500 to 2500 m, symptoms are generally mild, if experienced at all. At 2500 m, symptoms of mild to moderate acute mountain sickness (AMS) become quite common among unacclimatized visitors after rapid ascent. At this altitude high altitude pulmonary edema (HAPE) may also occur, but it is more common above 3000 m. Above 3000 to 4000 m, AMS is common among people who have not properly acclimatized, and the risk of severe consequences, including life-threatening HAPE and cerebral edema, is substantial.

66. What can raise quality? 3. all plausible residual confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed

67. All plausible residual confounding would result in an overestimate of effect  Hypoglycaemic drug phenformin causes lactic acidosis  The related agent metformin is under suspicion for the same toxicity.  Large observational studies have failed to demonstrate an association – Clinicians would be more alert to lactic acidosis in the presence of the agent Vaccine – adverse effects

68. Quality assessment criteria

69. Pulling it all together and drawing conclusions

70. carefully consider and assess all the factors that may influence the quality of evidence bear in mind that down- and upgrading for specific quality factors should be done in the context of all of the factors that influence the quality of evidence downgrading for one quality criterion may influence how the next quality criterion is dealt with

71. Within and among Downgrade or upgrade on a continuum Downgrade or upgrade – WITHIN each category – AMONG the categories

72. Example: Meta- analysis of 5 studies uncertainty about three factors: study limitations/RoB, inconsistency, and imprecision Uncertainty not serious enough to downgrade each factor Option to pick one or two levels to downgrade Indicate in footnotes why and why did not downgrade for those factors (e.g. There was some uncertainty but already downgraded for...)

73. Survival HR 0.77 (0.65 to 0.91)

74. How confident are you that these results are true?

75. Study limitations

76. No, there are no serious limitations Yes, there are serious limitations Yes, there are very serious limitations Would you downgrade for risk of bias?

77. From risk of bias to limitations in design

80. Quality now? High

81. Inconsistency

82. Who believes there is important inconsistency (rather than random error)? No, there is no serious inconsistency Yes, there is serious inconsistency Yes, there is very serious inconsistency

83. Quality now? High

84. Indirectness Direct comparison? Population? Intervention? Outcome?

85. Quality now? High

86. Publication bias

87. Quality now? High

88. Imprecision

89. Quality now? High No upgrading

90. Major bleeding RR 1.50 (0.26 – 8.80)

91. Study limitations

92. No, there are no serious limitations – although there …. Yes, there are serious limitations – most people would agree that selective reporting is…. Yes, there are very serious limitations – there is a risk of bias but only for the one criteria of selective reporting Would you downgrade for risk of bias?

93. From risk of bias to limitations in design

96. Quality now? Moderate

97. Imprecision

98. Quality now? Low Observational studies could have provided higher quality evidence

99. Flavanoids for Hemorrhoids venotonic agents – mechanism unclear, increase venous return popularity – 90 venotonics commercialized in France – none in Sweden and Norway – France 70% of world market possibilities – French misguided – rest of world missing out

100. Systematic Review 14 trials, 1432 patients key outcome – risk not improving/persistent symptoms – 11 studies, 1002 patients, 375 events – RR 0.4, 95% CI 0.29 to 0.57 minimal side effects is France right? what is the quality of evidence?

101. What can lower quality? Study limitations/risk of bias – lack of detail re concealment – questionnaires not validated rate down quality for study limitations/RoB? indirectness – no problem inconsistency, need to look at the results

103. Publication bias? size of studies – 40 to 234 patients, most around 100 all industry sponsored

105. What can lower quality? risk of bias – lack of detail re concealment – questionnaires not validated Inconsistency – heterogeneity p < 0.001; I 2 65.1% indirectness imprecision – RR 0.4, 95% CI 0.29 to 0.57 Publication bias – 40 to 234 patients, most around 100

106. Conclusions  WHO guidelines should be based on the best available evidence to be evidence based  GRADE is the approach used by WHO and gaining acceptance internationally  combines what is known in health research methodology and provides a structured approach to improve communication  Does not avoid judgments but provides framework  Criteria for evidence assessment across questions and outcomes  Criteria for moving from evidence to recommendations  Transparent, systematic  four categories of quality of evidence  two grades for strength of recommendations  Transparency in decision making and judgments is key