1. Grading evidence and recommendations The GRADE approach Holger Schünemann, MD, PhD for the GRADE Working Group

2. Professional good intentions and plausible theories are insufficient for selecting policies and practices for protecting, promoting and restoring health. Iain Chalmers

3. How can we judge the extent of our confidence that adherence to a recommendation will do more good than harm?

4. GRADE G rades of R ecommendation A ssessment, D evelopment and E valuation

5. What do you know about GRADE? Have prepared a guideline Have prepared a guideline Read the BMJ paper Read the BMJ paper Have prepared a systematic review and a summary of findings table Have prepared a systematic review and a summary of findings table Have attended a GRADE meeting, workshop or talk Have attended a GRADE meeting, workshop or talk

6. About GRADE o Began as informal working group in 2000 o Researchers/guideline developers with interest in methodology o Aim: to develop a common system for grading the quality of evidence and the strength of recommendations that is sensible and to explore the range of interventions and contexts for which it might be useful* o 13 meetings (~10 – 35 attendants) o Evaluation of existing systems and reliability* o Workshops at Cochrane Colloquia, WHO and GIN since 2000 *Grade Working Group. CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005

7. GRADE Working Group David Atkins, chief medical officer a Dana Best, assistant professor b Martin Eccles, professor d Yngve Falck-Ytter, associate director e Signe Flottorp, researcher f Gordon H Guyatt, professor g Robin T Harbour, quality and information director h Margaret C Haugh, methodologist i David Henry, professor j Suzanne Hill, senior lecturer j Roman Jaeschke, clinical professor k Regina Kunx, Associate Professor Gillian Leng, guidelines programme director l Alessandro Liberati, professor m Nicola Magrini, director n James Mason, professor d Philippa Middleton, honorary research fellow o Jacek Mrukowicz, executive director p Dianne O’Connell, senior epidemiologist q Andrew D Oxman, director f Bob Phillips, associate fellow r Holger J Schünemann, associate professor g,s Tessa Tan-Torres Edejer, medical officer/scientist t Jane Thomas, Lecturer, UK Helena Varonen, associate editor u Gunn E Vist, researcher f John W Williams Jr, professor v Stephanie Zaza, project director w a) Agency for Healthcare Research and Quality, USA b) Children's National Medical Center, USA c) Centers for Disease Control and Prevention, USA d) University of Newcastle upon Tyne, UK e) German Cochrane Centre, Germany f) Norwegian Centre for Health Services, Norway g) McMaster University, Canada h) Scottish Intercollegiate Guidelines Network, UK i) Fédération Nationale des Centres de Lutte Contre le Cancer, France j) University of Newcastle, Australia k) McMaster University, Canada l) National Institute for Clinical Excellence, UK m) Università di Modena e Reggio Emilia, Italy n) Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Italy o) Australasian Cochrane Centre, Australia p) Polish Institute for Evidence Based Medicine, Poland q) The Cancer Council, Australia r) Centre for Evidence-based Medicine, UK s) National Cancer Institute, Italy t) World Health Organisation, Switzerland u) Finnish Medical Society Duodecim, Finland v) Duke University Medical Center, USA w) Centers for Disease Control and Prevention, USA

8. Why guidelines? users looking for different things just tell me what to do (recommendation) what to do, and on strong or weak grounds – –recommendation and grade recommend, grade, evidence summary, values – –systematic review, value statement evidence from individual studies

9. Grading System current profusion: can there be consensus? current profusion: can there be consensus? trade-off benefits and risks trade-off benefits and risks –do it (or don’t do it) –probably do it (or probably don’t do it) quality of underlying evidence quality of underlying evidence –high quality (well done RCT) –intermediate (quasi-RCT) –low (well done observational) –very low (anything else)

10. Moving down poor RCT design, implementation poor RCT design, implementation –inconsistency indirect indirect –A vs B, but have A to C, B to C –patients, interventions, outcomes reporting bias reporting bias

11. Moving up magnitude of effect magnitude of effect dose-response dose-response biases favor control biases favor control

12. When to make a recommendation? –never patient values differ patient values differ just lay out benefits and risks just lay out benefits and risks –when evidence strong enough when very weak, too uncertain when very weak, too uncertain –clinicians need guidance intense study demands decision intense study demands decision

13. Why bother about grading? People draw conclusions about the People draw conclusions about the –quality of evidence –strength of recommendations Systematic and explicit approaches can help Systematic and explicit approaches can help –protect against errors –resolve disagreements –facilitate critical appraisal –communicate information However, there is wide variation in currently used approaches However, there is wide variation in currently used approaches

14. Who is confused? Evidence Recommendation II-2B II-2B C+ 1 C+ 1 StrongStrongly recommended StrongStrongly recommendedOrganization  USPSTF  ACCP  GCPS

15. Still not confused? EvidenceRecommendation BClass I BClass I C+ 1 C+ 1 IVC IVCOrganization  AHA  ACCP  SIGN Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral valve disease

16. Guidelines development process

17. Example ACCP First ACCP guidelines in 1986 (J. Hirsh; J. Dalen)First ACCP guidelines in 1986 (J. Hirsh; J. Dalen) Initially aimed at consensusInitially aimed at consensus Methodologists involved since beginningMethodologists involved since beginning Now formally convening every 2 to 3 yearsNow formally convening every 2 to 3 years > 200.000 copies in 2001> 200.000 copies in 2001 Seventh conference held in 2003Seventh conference held in 2003 87 panel members, 22 chapters87 panel members, 22 chapters Across subspecialtiesAcross subspecialties 565 recommendations, 230 new565 recommendations, 230 new Evidence Based RecommendationsEvidence Based Recommendations

18.  Evidence – recommendation:transparent link  Explicit inclusion criteria  Comprehensive search  Standardized considerationof study quality  Conduct/use meta-analysis  Grade recommendations  Acknowledge values andpreferences underlyingrecommendations What makes guidelines evidence based (in 2005)? Schünemann et al. Chest 2004

20. Schünemann HJ et al. Chest 2004

21. Transparent link between evidence and recommendations & Explicit inclusion criteria Albers et al. Chest 2004

23. Quality of evidence The extent to which one can be confident that an estimate of effect or association is correct. It depends on the: –study design (e.g. RCT, cohort study) –study quality/limitations (protection against bias; e.g. concealment of allocation, blinding, follow-up) –consistency of results –directness of the evidence including the populations (those of interest versus similar; for example, older, sicker or more co-morbidity) populations (those of interest versus similar; for example, older, sicker or more co-morbidity) interventions (those of interest versus similar; for example, drugs within the same class) interventions (those of interest versus similar; for example, drugs within the same class) outcomes (important versus surrogate outcomes) outcomes (important versus surrogate outcomes) comparison (A - C versus A - B & C - B) comparison (A - C versus A - B & C - B)

24. Quality of evidence The quality of the evidence (i.e. our confidence) may also be REDUCED when there is:  Sparse or imprecise data  Reporting bias The quality of the evidence (i.e. our confidence) may be INCREASED when there is:  A strong association  A dose response relationship  All plausible confounders would have reduced the observed effect  All plausible biases would have increased the observed lack of effect

25. Quality assessment criteria

26. Categories of quality High: Further research is very unlikely to change our confidence in the estimate of effect. High: Further research is very unlikely to change our confidence in the estimate of effect. Moderate: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Moderate: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Low: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low: Any estimate of effect is very uncertain. Very low: Any estimate of effect is very uncertain.

27. Judgements about the overall quality of evidence Most systems not explicit Most systems not explicit Options: Options: –strongest outcome –primary outcome –benefits –weighted –separate grades for benefits and harms –no overall grade –weakest outcome Based on lowest of all the critical outcomes Based on lowest of all the critical outcomes Beyond the scope of a systematic review Beyond the scope of a systematic review

28. Strength of recommendation The extent to which one can be confident that adherence to a recommendation will do more good than harm. trade-offs (the relative value attached to the expected benefits, harms and costs) trade-offs (the relative value attached to the expected benefits, harms and costs) quality of the evidence quality of the evidence translation of the evidence into practice in a specific setting translation of the evidence into practice in a specific setting uncertainty about baseline risk uncertainty about baseline risk

29. Judgements about the balance between benefits and harms Before considering cost and making a recommendation Before considering cost and making a recommendation For a specified setting, taking into account issues of translation into practice For a specified setting, taking into account issues of translation into practice

30. Clarity of the trade-offs between benefits and the harms the estimated size of the effect for each main outcome the estimated size of the effect for each main outcome the precision of these estimates the precision of these estimates the relative value attached to the expected benefits and harms the relative value attached to the expected benefits and harms the variation in values between people the variation in values between people important factors that could be expected to modify the size of the expected effects in specific settings; e.g. proximity to a hospital important factors that could be expected to modify the size of the expected effects in specific settings; e.g. proximity to a hospital

31. Judgements about recommendations We recommend… “Do it” We recommend… “Do it” We suggest… “Probably do it” We suggest… “Probably do it” We recommend… “Probably don’t do it” We recommend… “Probably don’t do it” We suggest… “Don’t do it” We suggest… “Don’t do it” No recommendation This could include considerations of costs; i.e. “Is the net gain (benefits-harms) worth the costs?”

32. Should healthy asymptomatic postmenopausal women have been given oestrogen + progestin for prevention in 1992? Quality of evidence across studies for Quality of evidence across studies for –CHD –Hip fracture –Colorectal cancer –Breast cancer –Stroke –Thrombosis –Gall bladder disease Quality of evidence across critical outcomes Quality of evidence across critical outcomes Balance between benefits and harms Balance between benefits and harms Recommendations Recommendations Will GRADE lead to change?

33. Oestrogen + progestin for prevention after WHI and HERS

35. Further developments Diagnostic tests Diagnostic tests Costs Costs (Equity) (Equity) Empirical evaluations Empirical evaluations Free software application Free software application

36. GRADE Profiler

37. Empirical evaluations Critical appraisal of other systems Critical appraisal of other systems Pilot test + sensibility Pilot test + sensibility “Case law” + practical experience “Case law” + practical experience Guidance for judgements Guidance for judgements –Single studies –Sparse data or imprecise data Agreement Agreement Validity? Validity? Comparisons with other systems Comparisons with other systems Alternative presentations Alternative presentations

38. Comparison of GRADE and other systems Explicit definitions Explicit definitions Explicit, sequential judgements Explicit, sequential judgements Components of quality Components of quality Overall quality Overall quality Relative importance of outcomes Relative importance of outcomes Balance between health benefits and harms Balance between health benefits and harms Balance between incremental health benefits and costs Balance between incremental health benefits and costs Consideration of equity Consideration of equity Evidence profiles Evidence profiles International collaboration International collaboration Software Software Consistent judgements? Consistent judgements? Communication? Communication?

39. Who is interested in GRADE WHO WHO American Endocrine Society American Endocrine Society American College of Chest Physicians (ACCP) American College of Chest Physicians (ACCP) Italian National Cancer Institute Italian National Cancer Institute Clinical Evidence Clinical Evidence Norwegian Centre for Health Services Norwegian Centre for Health Services UpToDate UpToDate Close relationship with Cochrane Collaboration Close relationship with Cochrane Collaboration American Society of Clinical Oncology (ASCO) American Society of Clinical Oncology (ASCO) American Thoracic Society (ATS) American Thoracic Society (ATS)

40. Questions?

41. GRADE for diagnostic tests

42. Taking account of costs Include important (disaggregated) costs in evidence summaries and balance sheets when relevant Include important (disaggregated) costs in evidence summaries and balance sheets when relevant –May be useful to aggregate and value (in monetary terms) –Always include disaggregated resource utilisation –Note when important information is missing –Published cost-effectiveness analyses are rarely helpful Assess the quality of the evidence for important costs (consumption of resources) as for other effects (Were quantities measured reliably?) Assess the quality of the evidence for important costs (consumption of resources) as for other effects (Were quantities measured reliably?) If costs are critical to a decision, low quality evidence can lower the overall quality of evidence If costs are critical to a decision, low quality evidence can lower the overall quality of evidence Costs are negotiable (the value of resources) Costs are negotiable (the value of resources) There are many possible criteria for making a recommendation There are many possible criteria for making a recommendation

43. Should activated protein C be given to patients in severe sepsis? An example with costs

44. GRADE evidence profile: Activated Protein C for sepsis Name:Jaeschke and Schunemann Name:Jaeschke and Schunemann Date: September 2004 Date: September 2004 Question: Should APC be used for severe sepsis? Question: Should APC be used for severe sepsis? Setting:ICU in Paris Setting:ICU in Paris Baseline risk:Severe sepsis or septic shock > 24 h Baseline risk:Severe sepsis or septic shock > 24 h References: Effectiveness: Bernard 2001. Efficacy and safety of recombinant human activated protein C for severe sepsis. NEJM 2001; 344:699 and Manns 2002. An economic evaluation of activated protein C treatment for severe sepsis. NEJM 2002;347:993. References: Effectiveness: Bernard 2001. Efficacy and safety of recombinant human activated protein C for severe sepsis. NEJM 2001; 344:699 and Manns 2002. An economic evaluation of activated protein C treatment for severe sepsis. NEJM 2002;347:993. Cost-effectiveness: Manns 2002. An economic evaluation of activated protein C treatment for severe sepsis. NEJM 2002;347:993. Cost-effectiveness: Manns 2002. An economic evaluation of activated protein C treatment for severe sepsis. NEJM 2002;347:993.

45. Possible criteria for making a recommendation Treatment effect Treatment effect Adverse effects Adverse effects Cost Cost Cost-effectiveness Cost-effectiveness Equity Equity Seriousness of the problem Seriousness of the problem Administrative restrictions Administrative restrictions

46. Quality assessment

47. Summary of findings

48. Values and preferences

49. Where would you prefer to live?

50. ← Option 1 Option 2 →

51. ← Option 1 (pink card) Option 2 → (green card)

52. You are hiking. Which of the following animals would you prefer to encounter?

53. ← Option 1 (pink card) Option 2 → (green card)

54. You are buying an ice cream. Which flavor do you prefer?

55. ← Option 1 (pink card) Option 2 → (green card) Chocolate Strawberry

56. You are buying a new car. Which one would you buy?

57. ← Option 1 (pink card) Option 2 → (green card) Yellow fox Red Ferrari

58. Sparse data Should antibiotic impregnated catheters or standard catheters be used after radical prostatectomy? One RCT One RCT Lower rate of symptomatic bacteriuria in the antibiotic group (RR 0.36, 95% CI 0.18 to 0.73). Lower rate of symptomatic bacteriuria in the antibiotic group (RR 0.36, 95% CI 0.18 to 0.73). 1 of 56 men in the antibiotic impregnated group had a symptomatic UTI compared with 6 of 68 who had standard catheters (RR 0.20, 95% CI 0.03 to 1.63) 1 of 56 men in the antibiotic impregnated group had a symptomatic UTI compared with 6 of 68 who had standard catheters (RR 0.20, 95% CI 0.03 to 1.63)

59. Reporting bias

60. GRADE profiler (GRADEpro)