1. Holger Sch ü nemann Yngve Falck-Ytter NICE, London December 11, 2006 GRADE – An introduction and workshop

2. Why bother about grading? People draw conclusions about the – –quality of evidence – –strength of recommendations Systematic and explicit approaches can help – –protect against errors – –resolve disagreements – –facilitate critical appraisal – –communicate information However, there is wide variation in currently used approaches and grading can be misused or misunderstood

3. Which grading system? EvidenceRecommendation BClass I C+ 1 IVC Organization   AHA   ACCP   SIGN Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral valve disease

4. Grading Systems Current profusion: can there be consensus?

5. GRADE G rades of R ecommendation A ssessment, D evelopment and E valuation

6. About GRADE o o Working group since 2000 o o Researchers/guideline developers with interest in methodology o o Aim: to develop a common system for grading the quality of evidence and the strength of recommendations that is sensible and to explore the range of interventions and contexts for which it might be useful* o o Evaluation of existing systems and reliability* o o Adopted by ATS, ACCP, ACP, WHO, Cochrane *Grade Working Group. CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005

7. GRADE Working Group David Atkins, chief medical officer a Dana Best, assistant professor b Peter A Briss, chief c Martin Eccles, professor d Yngve Falck-Ytter, associate director e Signe Flottorp, researcher f Gordon H Guyatt, professor g Robin T Harbour, quality and information director h Margaret C Haugh, methodologist i David Henry, professor j Suzanne Hill, senior lecturer j Roman Jaeschke, clinical professor k Gillian Leng, guidelines programme director l Alessandro Liberati, professor m Nicola Magrini, director n James Mason, professor d Philippa Middleton, honorary research fellow o Jacek Mrukowicz, executive director p Dianne O ’ Connell, senior epidemiologist q Andrew D Oxman, director f Bob Phillips, associate fellow r Holger J Sch ü nemann, associate professor g,s Tessa Tan-Torres Edejer, medical officer/scientist t Helena Varonen, associate editor u Gunn E Vist, researcher f John W Williams Jr, associate professor v Stephanie Zaza, project director w a) Agency for Healthcare Research and Quality, USA b) Children's National Medical Center, USA c) Centers for Disease Control and Prevention, USA d) University of Newcastle upon Tyne, UK e) German Cochrane Centre, Germany f) Norwegian Centre for Health Services, Norway g) McMaster University, Canada h) Scottish Intercollegiate Guidelines Network, UK i) F é d é ration Nationale des Centres de Lutte Contre le Cancer, France j) University of Newcastle, Australia k) McMaster University, Canada l) National Institute for Clinical Excellence, UK m) Universit à di Modena e Reggio Emilia, Italy n) Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Italy o) Australasian Cochrane Centre, Australia p) Polish Institute for Evidence Based Medicine, Poland q) The Cancer Council, Australia r) Centre for Evidence-based Medicine, UK s) National Cancer Institute, Italy t) World Health Organisation, Switzerland u) Finnish Medical Society Duodecim, Finland v) Duke University Medical Center, USA w) Centers for Disease Control and Prevention, USA

8. Guideline development process Prioritise Problems, establish panel  Systematic Review  Evidence Profile  Relative importance of outcomes  Overall quality of evidence  Benefit – downside evaluation  Strength of recommendation  Implementation and evaluation of guidelines GRADE

9. Guideline development process Prioritise Problems, establish panel  Systematic Review  Evidence Profile  Relative importance of outcomes  Overall quality of evidence  Benefit – downside evaluation  Strength of recommendation  Implementation and evaluation of guidelines GRADE Summary of Findings

10. GRADE Quality of evidence The extent to which one can be confident that an estimate of effect or association is correct. Although the degree of confidence is a continuum, we suggest using four categories: – –High – –Moderate – –Low – –Very low

11. Judgements about the quality of evidence The quality of the evidence (i.e. our confidence) depends on: study design (e.g. RCT, case-control study) study quality/limitations (protection against bias; e.g. concealment of allocation, blinding, follow-up) consistency of results directness of the evidence including the – –populations (those of interest versus similar; for example, older, sicker or more co-morbidity) – –interventions (those of interest versus similar; for example, drugs within the same class) – –outcomes (important versus surrogate outcomes) – –comparison (A - C versus A - B & C - B)

12. Moving quality down poor (RCT) design, implementation → →randomization, blinding, concealment, follow-up, intention to treat principle, early stopping for benefit inconsistency Indirect evidence → →patients, interventions, outcomes → →A vs B, but have A to C, B to C sparse or imprecise data reporting bias

13. Moving quality up Observational studies – high or moderate quality? Strong association → →strong association: RR > 2 or RR < 0.5 → →very strong association: RR > 5 or RR < 0.2 Dose response relationship – –bleeding risk associated with increasing INR (blood thinning with warfarin) Plausible confounders would have reduced the effect For example, plausible explanatory factors that were not adjusted for in studies comparing mortality rates of for-profit and not-for-profit hospitals would have reduced the observed effect. Thus, the evidence showing that for-profit hospitals have a higher risk of mortality is more convincing

15. Categories of quality High: Further research is very unlikely to change our confidence in the estimate of effect. Moderate: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low: Any estimate of effect is very uncertain.

16. Judgements about the overall quality of evidence Most systems not explicit Options: – –Benefits – –Primary outcome – –Highest – –Lowest Based on lowest of all the critical outcomes Beyond the scope of a systematic review

17. Levels of evidence: SIGN

18. Grading of recommendations

19. Some problems Oversimplified hierarchy based on study design – –Inadequate consideration of other factors – –Distinction between study design, quality of evidence and strength of recommendation blurred Systematic reviews included in the hierarchy – –rather than viewed as the basis for making judgements Expert opinion included in the hierarchy – –rather than explicitly considering the evidence underlying expert opinions Balance between desirable and undesirable effects – –Not reflected in the grade – –Not considered transparently Inadequate consideration of other factors that affect confidence in a recommendation Grading misused when recommendation not separated from the quality of the evidence

20. Example WHO Avian Influenza guidelines - key clinical questions: Population: H5N1 infected individuals Intervention: Neuraminidase Inhibitors, M2 Inhibitors, other pharmacological agents Comparison: no therapy/alternative management Outcomes: ?

21. Example WHO Avian Influenza guidelines - key clinical questions: Population: H5N1 infected individuals Intervention: Neuraminidase Inhibitors, M2 Inhibitors, other pharmacological agents Comparison: no therapy/alternative management Outcomes: Mortality?, Hospitalizations? Resource use?, Adverse outcomes?

22. Clinical Question Refinement Survey of panel members Outcome definition: List of potential outcomes circulated Feedback from panel Concealed rating of importance Consultation with Cochrane Consumers network

23. Judgment about the relative importance for each endpoint (scale from 9 to 1): 7 – 9: the endpoint is critical for decision making. 4 – 6: the endpoint is important but not critical. 1 – 3: the endpoint is not important. Only critical (and important) outcomes included Treatment: mortality, duration of hospitalization, incidence of lower respiratory tract complications, antiviral drug resistance and serious adverse events. Outcomes/endpoints

24. Evidence Summary and Quality Ratings Draft summaries sent to panel members for review and identification of gaps Restricted additional evidence at meeting Evidence profiles using GRADE methodology and GRADEpro software (v1.12)

25. Evidence Profiles Oseltamivir for treatment of H5N1 infection: - -

26. Evidence Summary Summary of findings No clinical trial of oseltamivir for treatment of H5N1 patients. 4 systematic reviews and health technology assessments (HTA) reporting on 5 studies of oseltamivir in seasonal influenza. Healthy adults, high risk adults or children for treatment of seasonal influenza Duration of treatment up to 5 days Several countries in the northern and southern hemispheres (no resource poor countries) 3 published case series describing H5N1 patients treated with oseltamivir. Many in vitro and animal studies. 50% worldwide

27. Case scenario A 13 year old girl who lives in rural Indonesia presented with flu symptoms and developed severe respiratory distress over the course of the last 2 days. She required intubation. The history reveals that she shares her living quarters with her parents and her three siblings. At night the family’s chicken stock shares this room too and several chicken had died unexpectedly a few days before the girl fell sick.

28. Who would recommend oseltamivir for this patient or similar patients? YES (pink) No (green)

29. Recommendations Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (????? recommendation, very low quality evidence).

30. Recommendations Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence).

31. Comparison of GRADE and other systems Explicit definitions Explicit, sequential judgements Components of quality defined Quality by outcome and overall quality Relative importance of outcomes Balance between health benefits and harms Balance between incremental health benefits and costs Evidence profiles International collaboration Consistent judgements? Communication?

32. GRADE Profiler