1. Grading evidence and recommendations Workshop W-069 Congress Hall ABEF Oct 6 2004

2. Professional good intentions and plausible theories are insufficient for selecting policies and practices for protecting, promoting and restoring health. Iain Chalmers

3. How can we judge the extent of our confidence that adherence to a recommendation will do more good than harm?

4. GRADE G rades of R ecommendation A ssessment, D evelopment and E valuation

5. GRADE Working Group David Atkins, chief medical officer a Dana Best, assistant professor b Peter A Briss, chief c Martin Eccles, professor d Yngve Falck-Ytter, associate director e Signe Flottorp, researcher f Gordon H Guyatt, professor g Robin T Harbour, quality and information director h Margaret C Haugh, methodologist i David Henry, professor j Suzanne Hill, senior lecturer j Roman Jaeschke, clinical professor k Gillian Leng, guidelines programme director l Alessandro Liberati, professor m Nicola Magrini, director n James Mason, professor d Philippa Middleton, honorary research fellow o Jacek Mrukowicz, executive director p Dianne O’Connell, senior epidemiologist q Andrew D Oxman, director f Bob Phillips, associate fellow r Holger J Schünemann, associate professorg g,s Tessa Tan-Torres Edejer, medical officer/scientist t Helena Varonen, associate editor u Gunn E Vist, researcher f John W Williams Jr, associate professor v Stephanie Zaza, project director w a) Agency for Healthcare Research and Quality, USA b) Children's National Medical Center, USA c) Centers for Disease Control and Prevention, USA d) University of Newcastle upon Tyne, UK e) German Cochrane Centre, Germany f) Norwegian Centre for Health Services, Norway g) McMaster University, Canada h) Scottish Intercollegiate Guidelines Network, UK i) Fédération Nationale des Centres de Lutte Contre le Cancer, France j) University of Newcastle, Australia k) McMaster University, Canada l) National Institute for Clinical Excellence, UK m) Università di Modena e Reggio Emilia, Italy n) Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Italy o) Australasian Cochrane Centre, Australia p) Polish Institute for Evidence Based Medicine, Poland q) The Cancer Council, Australia r) Centre for Evidence-based Medicine, UK s) University of Buffalo, USA t) World Health Organisation, Switzerland u) Finnish Medical Society Duodecim, Finland v) Duke University Medical Center, USA w) Centers for Disease Control and Prevention, USA Opinions do not necessarily represent those of the institutions with which the members of the GRADE Working Group are affiliated.

6. What do you know about GRADE? Have prepared a guideline Have prepared a guideline Read the BMJ paper Read the BMJ paper Have prepared a systematic review and a summary of findings table Have prepared a systematic review and a summary of findings table Have attended a GRADE meeting, workshop or talk Have attended a GRADE meeting, workshop or talk

7. Why bother about grading? People draw conclusions about the People draw conclusions about the –quality of evidence –strength of recommendations Systematic and explicit approaches can help Systematic and explicit approaches can help –protect against errors –resolve disagreements –facilitate critical appraisal –communicate information However, there is wide variation in currently used approaches However, there is wide variation in currently used approaches

8. Who is confused? Evidence Recommendation II-2B II-2B C+ 1 C+ 1 StrongStrongly recommended StrongStrongly recommendedOrganization  USPSTF  ACCP  GCPS

9. Still not confused? EvidenceRecommendation BClass I BClass I C+ 1 C+ 1 IVC IVCOrganization  AHA  ACCP  SIGN Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral valve disease

10. Guidelines development process

11. Quality of evidence The extent to which one can be confident that an estimate of effect or association is correct. It depends on the: –study design (e.g. RCT, cohort study) –study quality/limitations (protection against bias; e.g. concealment of allocation, blinding, follow-up) –consistency of results –directness of the evidence including the populations (those of interest versus similar; for example, older, sicker or more co-morbidity) populations (those of interest versus similar; for example, older, sicker or more co-morbidity) interventions (those of interest versus similar; for example, drugs within the same class) interventions (those of interest versus similar; for example, drugs within the same class) outcomes (important versus surrogate outcomes) outcomes (important versus surrogate outcomes) comparison (A - C versus A - B & C - B) comparison (A - C versus A - B & C - B)

12. Quality of evidence The quality of the evidence (i.e. our confidence) may also be REDUCED when there is:  Sparse or imprecise data  Reporting bias The quality of the evidence (i.e. our confidence) may be INCREASED when there is:  A strong association  A dose response relationship  All plausible confounders would have reduced the observed effect  All plausible biases would have increased the observed lack of effect

13. Quality assessment criteria

14. Categories of quality High: Further research is very unlikely to change our confidence in the estimate of effect. High: Further research is very unlikely to change our confidence in the estimate of effect. Moderate: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Moderate: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Low: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low: Any estimate of effect is very uncertain. Very low: Any estimate of effect is very uncertain.

15. Judgements about the overall quality of evidence most systems just use evidence about primary benefit/outcome most systems just use evidence about primary benefit/outcome but what about other outcomes (downsides)? but what about other outcomes (downsides)? options: options: –ignore all but primary outcome –basing it on the evidence for benefits –some blended approach –having separate grades for benefits and harms –weakest of any outcome Based on lowest of all the critical outcomes Based on lowest of all the critical outcomes Beyond the scope of a systematic review Beyond the scope of a systematic review

16. Strength of recommendation The extent to which one can be confident that adherence to a recommendation will do more good than harm. trade-offs (the relative value attached to the expected benefits, harms and costs) trade-offs (the relative value attached to the expected benefits, harms and costs) quality of the evidence quality of the evidence translation of the evidence into practice in a specific setting translation of the evidence into practice in a specific setting uncertainty about baseline risk uncertainty about baseline risk

17. Judgements about the balance between benefits and harms Before considering cost and making a recommendation Before considering cost and making a recommendation For a specified setting, taking into account issues of translation into practice For a specified setting, taking into account issues of translation into practice

18. Clarity of the trade-offs between benefits and the harms the estimated size of the effect for each main outcome the estimated size of the effect for each main outcome the precision of these estimates the precision of these estimates the relative value attached to the expected benefits and harms the relative value attached to the expected benefits and harms important factors that could be expected to modify the size of the expected effects in specific settings; e.g. proximity to a hospital important factors that could be expected to modify the size of the expected effects in specific settings; e.g. proximity to a hospital

19. Balance between benefits and harm Net benefits: The intervention does more good than harm. Net benefits: The intervention does more good than harm. Trade-offs: There are important trade- offs between the benefits and harms. Trade-offs: There are important trade- offs between the benefits and harms. Uncertain net benefits: It is not clear whether the intervention does more good than harm. Uncertain net benefits: It is not clear whether the intervention does more good than harm. Not net benefits: The intervention does not do more good than harm. Not net benefits: The intervention does not do more good than harm.

20. Judgements about recommendations This should include considerations of costs; i.e. “Is the net gain (benefits-harms) worth the costs?” Do it Do it Probably do it Probably do it No recommendation Probably don’t do it Probably don’t do it Don’t do it Don’t do it

21. Should healthy asymptomatic postmenopausal women have been given oestrogen + progestin for prevention in 1992? Quality of evidence across studies for Quality of evidence across studies for –CHD –Hip fracture –Colorectal cancer –Breast cancer –Stroke –Thrombosis –Gall bladder disease Quality of evidence across critical outcomes Quality of evidence across critical outcomes Balance between benefits and harms Balance between benefits and harms Recommendations Recommendations Will GRADE lead to change

22. Evidence profile: Quality assessment Oestrogen + progestin for prevention before WHI and HERS Oestrogen + progestin versus usual care

23. Oestrogen + progestin for prevention after WHI and HERS

24. GRADE for diagnostic tests

25. Challenges for GRADE Operationalise all steps Operationalise all steps Dissemination/buy in Dissemination/buy in –simple to do –easy to understand and use Tool and manual Tool and manual

26. GRADE profiler (GRADEpro)

27. Separation by outcomes

28. Work in groups of two take a pencil (and paper) take a pencil (and paper) write down the most important issues/questions you have about GRADE write down the most important issues/questions you have about GRADE

29. The 10 burning questions/issues about GRADE 1. Different experts 2. Prospective studies 3. Valuing benefits and harms – decide about tradeoffs 4. Low quality evidence leading to strong rec’s 5. How can one introduce/disseminate one single/uniform system 6. Empirical evidence for GRADE – how should we obtain it 7. Mechanisms for balancing benefits and cost 8. Reliability? 9. Other than RCT evidence 10. Decisions about quality of evidence/limitations of study design, guidance about magnitude of effect

30. The 10 burning questions/issues about GRADE 1. Other than RCT evidence 2. What type of cost and resources 3. Who judges the importance of outcomes 4. How can one evaluate whether all outcomes are reported? 5. Decisions about quality of evidence/limitations of study design

31. Small group sessions find a group find a group select spokes person select spokes person take 30 minutes to complete the task take 30 minutes to complete the task be prepared to criticise be prepared to criticise

32. Summary What is good about GRADE? – – – What is most challenging? – Takes too long – relative importance is difficult to work out – Difficult - much more time needed – What do we need to do next? – more time – –

33. The 10 burning questions/issues about GRADE 1. 1. 2. 2. 3. 3. 4. 4. 5. 5. 6. 6. 7. 7. 8. 8. 9. 9. 10. 10.

34. Summary What is good about GRADE? – – – What is most challenging? – – – What do we need to do next? – – –