1. GRADE – An introduction and workshop
Holger Schünemann
CADTH, Ottawa March 27 & 28, 2007

2. Grading evidence and recommendations
Why grade?
Grading the quality of evidence
Grading the strength of recommendations
Judgements about importance
Comparison of GRADE & other systems

3. Professional good intentions and plausible theories are insufficient for selecting policies and practices for protecting, promoting and restoring health.
Iain Chalmers

4. How can we judge the extent of our confidence that adherence to a recommendation will do more good than harm?

5. Why Grade Recommendations?
strong recommendations
high quality methods
large precise effect
few down sides of therapy
weak recommendations
low quality methods
imprecise estimate
small effect
substantial down sides

6. Why grade recommendations?
People draw conclusions about the
quality of evidence
strength of recommendations
Systematic and explicit approaches can help
protect against errors
resolve disagreements
facilitate critical appraisal
communicate information
However, there is wide variation in currently used approaches and grading can be misused or misunderstood

7. Which grading system? Evidence Recommendation II-2 B C+ 1
Strong Strongly recommended
Organization
USPSTF
ACCP
GCPS

8. Which grading system?
Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral valve disease
Evidence Recommendation
B Class I
C+ 1
IV C
Organization
AHA
ACCP
SIGN

9. Current profusion: can there be consensus?
Grading Systems
Current profusion: can there be consensus?

10. Grades of Recommendation Assessment, Development and Evaluation

11. About GRADE Working group since 2000
Researchers/guideline developers with interest in methodology
Aim: to develop a common system for grading the quality of evidence and the strength of recommendations that is sensible and transparent and to explore the range of interventions and contexts for which it might be useful*
Evaluation of existing systems and reliability*
Adopted by ATS, ACCP, ACP, WHO, NICE, Cochrane, several other organizations.
*Grade Working Group. CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005

12. GRADE Working Group David Atkins, chief medical officera
Dana Best, assistant professorb
Peter A Briss, chiefc
Martin Eccles, professord
Yngve Falck-Ytter, associate directore
Signe Flottorp, researcherf
Gordon H Guyatt, professorg
Robin T Harbour, quality and information director h
Margaret C Haugh, methodologisti
David Henry, professorj
Suzanne Hill, senior lecturerj
Roman Jaeschke, clinical professork
Gillian Leng, guidelines programme directorl
Alessandro Liberati, professorm
Nicola Magrini, directorn
James Mason, professord
Philippa Middleton, honorary research fellowo
Jacek Mrukowicz, executive directorp
Dianne O’Connell, senior epidemiologistq
Andrew D Oxman, directorf
Bob Phillips, associate fellowr
Holger J Schünemann, associate professorg,s
Tessa Tan-Torres Edejer, medical officer/scientistt
Helena Varonen, associate editoru
Gunn E Vist, researcherf
John W Williams Jr, associate professorv
Stephanie Zaza, project directorw
a) Agency for Healthcare Research and Quality, USA
b) Children's National Medical Center, USA
c) Centers for Disease Control and Prevention, USA
d) University of Newcastle upon Tyne, UK
e) German Cochrane Centre, Germany
f) Norwegian Centre for Health Services, Norway
g) McMaster University, Canada
h) Scottish Intercollegiate Guidelines Network, UK
i) Fédération Nationale des Centres de Lutte Contre le Cancer, France
j) University of Newcastle, Australia
k) McMaster University, Canada
l) National Institute for Clinical Excellence, UK
m) Università di Modena e Reggio Emilia, Italy
n) Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Italy
o) Australasian Cochrane Centre, Australia
p) Polish Institute for Evidence Based Medicine, Poland
q) The Cancer Council, Australia
r) Centre for Evidence-based Medicine, UK
s) National Cancer Institute, Italy
t) World Health Organisation, Switzerland
u) Finnish Medical Society Duodecim, Finland
v) Duke University Medical Center, USA
w) Centers for Disease Control and Prevention, USA

13. a) Agency for Healthcare Research and Quality, USA
b) Children's National Medical Center, USA
c) Centers for Disease Control and Prevention, USA
d) University of Newcastle upon Tyne, UK
e) German Cochrane Centre, Germany
f) Norwegian Centre for Health Services, Norway
g) McMaster University, Canada
h) Scottish Intercollegiate Guidelines Network, UK
i) Fédération Nationale des Centres de Lutte Contre le Cancer, France
j) University of Newcastle, Australia
k) McMaster University, Canada
l) National Institute for Clinical Excellence, UK
m) Università di Modena e Reggio Emilia, Italy
n) Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Italy
o) Australasian Cochrane Centre, Australia
p) Polish Institute for Evidence Based Medicine, Poland
q) The Cancer Council, Australia
r) Centre for Evidence-based Medicine, UK
s) National Cancer Institute, Italy
t) World Health Organisation, Switzerland
u) Finnish Medical Society Duodecim, Finland
v) Duke University Medical Center, USA
w) Centers for Disease Control and Prevention, USA
x) University of London, UK
Y) BMJ Clinical Evidence, UK

14. Guideline development process
Prioritise Problems, establish panel 
Systematic Review
Evidence Profile
Relative importance of outcomes
Overall quality of evidence
Benefit – downside evaluation
Strength of recommendation
Implementation and evaluation of guidelines
GRADE

15. GRADE Quality of evidence
The extent to which one can be confident that an estimate of effect or association is correct.
Although the degree of confidence is a continuum, we suggest using four categories:
High
Moderate
Low
Very low

16. Judgements about the quality of evidence
The quality of the evidence (i.e. our confidence) depends on:
study design (e.g. RCT, case-control study)
study quality/limitations (protection against bias; e.g. concealment of allocation, blinding, follow-up)
consistency of results
directness of the evidence including the
populations (those of interest versus similar; for example, older, sicker or more co-morbidity)
interventions (those of interest versus similar; for example, drugs within the same class)
outcomes (important versus surrogate outcomes)
comparison (A - C versus A - B & C - B)

17. Moving quality down poor (RCT) design, implementation
randomization, blinding, concealment, follow-up, intention to treat principle, early stopping for benefit
inconsistency
Indirect evidence
patients, interventions, outcomes
A vs B, but have A to C, B to C
sparse or imprecise data
reporting bias

18. Moving quality up Observational studies – high or moderate quality?
Strong association
strong association: RR > 2 or RR < 0.5
very strong association: RR > 5 or RR < 0.2
Dose response relationship
bleeding risk associated with increasing INR (blood thinning with warfarin)
Plausible confounders would have reduced the effect

20. Categories of quality
High: Further research is very unlikely to change our confidence in the estimate of effect.
Moderate: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.
Low: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.
Very low: Any estimate of effect is very uncertain.

21. Judgements about the overall quality of evidence
Most systems not explicit
Options:
Benefits
Primary outcome
Highest
Lowest
Based on lowest of all the critical outcomes
Beyond the scope of a systematic review

22. Levels of evidence: SIGN

23. Grading of recommendations

24. Problems with other systems
Oversimplified hierarchy based on study design
Inadequate consideration of other factors
Distinction between study design, quality of evidence and strength of recommendation blurred
Systematic reviews included in the hierarchy
rather than viewed as the basis for making judgements
Expert opinion included in the hierarchy
rather than explicitly considering the evidence underlying expert opinions
Balance between desirable and undesirable effects
Not reflected in the grade
Not considered transparently
Inadequate consideration of other factors that affect confidence in a recommendation
Grading misused when recommendation not separated from the quality of the evidence

25. Example WHO Avian Influenza guidelines - key clinical questions:
Population: H5N1 infected individuals
Intervention: Neuraminidase Inhibitors, M2 Inhibitors, other pharmacological agents
Comparison: no therapy/alternative management
Outcomes: ?

26. Example WHO Avian Influenza guidelines - key clinical questions:
Population: H5N1 infected individuals
Intervention: Neuraminidase Inhibitors, M2 Inhibitors, other pharmacological agents
Comparison: no therapy/alternative management
Outcomes: Mortality?, Hospitalizations? Resource use?, Adverse outcomes?

27. Clinical Question Refinement
Survey of panel members
Outcome definition:
List of potential outcomes circulated
Feedback from panel
Concealed rating of importance
Consultation with Cochrane Consumers network
A list of potential outcomes for each question and intervention that the panel considered was initially developed by two reviewers for each question. The team preparing the evidence summaries independently scored the relative importance of each considered outcome from 1-9, where 7-9 indicated the outcome was critical for a decision or recommendation, 4-6 indicated it was important, and 1-3 indicated it was not important. Because the relative importance of some outcomes depended on whether a drug was being used for treatment or chemoprophylaxis, this was done for two scenarios. The individual scores were discussed and disagreements were resolved by consensus. Outcomes were included in the approximate order of their relative importance in evidence tables and outcomes that were considered not important (a score of 3 or less) were not included. Panel members were also asked before the panel meeting to provide additional outcomes that should be addressed and to rate the relative importance of the outcomes. The panel repeated a concealed rating exercise during the meeting for questions that dealt with chemoprophylaxis because the panel reconsidered whether mortality is a critical outcome under these circumstances. The Cochrane Consumers network was consulted through their electronic discussion list for identification of additional important outcomes not included in the list of potential outcomes developed by the reviewers and panel. This involvement did not reveal additional important outcomes.

28. Outcomes/endpoints
Judgment about the relative importance for each endpoint (scale from 9 to 1):
7 – 9: the endpoint is critical for decision making.
4 – 6: the endpoint is important but not critical.
1 – 3: the endpoint is not important.
Only critical (and important) outcomes included
Treatment: mortality, duration of hospitalization, incidence of lower respiratory tract complications, antiviral drug resistance and serious adverse events.

29. Evidence Summary and Quality Ratings
Draft summaries sent to panel members for review and identification of gaps
Restricted additional evidence at meeting
Evidence profiles using GRADE methodology and GRADEpro software (v1.12)

30. Oseltamivir for treatment of H5N1 infection:
Evidence Profiles
Oseltamivir for treatment of H5N1 infection: - -

31. Evidence Summary Summary of findings
No clinical trial of oseltamivir for treatment of H5N1 patients.
4 systematic reviews and health technology assessments (HTA) reporting on 5 studies of oseltamivir in seasonal influenza.
Healthy adults, high risk adults or children for treatment of seasonal influenza
Duration of treatment up to 5 days
Several countries in the northern and southern hemispheres (no resource poor countries)
3 published case series describing H5N1 patients treated with oseltamivir.
Many in vitro and animal studies.
< 400 cases; mortality > 50% worldwide

32. Strength of recommendation
“The extent to which we can be confident that desirable effects of an intervention outweigh undesirable effects.”
quality of the evidence
translation of the evidence into practice in a specific setting
uncertainty about baseline risk
trade-offs (the relative value attached to the expected benefits, harms and costs)

33. Strength of recommendation
Undesirable consequences
harms
more burden
costs
Desirable consequences
health benefits
less burden
savings

34. Strength of recommendation
Desirable consequences:
reduction in morbidity and mortality
improvement in quality of life
reduction in the burden of treatment (such as having to take medication or the inconvenience of blood tests)
reduced resource expenditures
Undesirable consequences:
adverse effects that have a deleterious impact on these broad outcome categories

35. Categories of recommendations
Although the degree of confidence is a continuum, we suggest using two categories: strong and weak.
Strong recommendation: the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects.
Weak recommendation: the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident.
Recommend
 
Suggest
? ?

36. Judgements about recommendations

37. Judgements about the strength of a recommendation
Reasons for not being confident can include:
absence of high quality evidence
imprecise estimates
uncertainty or variation in how different individuals value the outcomes
small net benefits
uncertainty whether the net benefits are worth the costs (including the costs of implementing the recommendation)

38. Judgements about the strength of a recommendation
No precise threshold for going from a strong to a weak recommendation
The presence of important concerns about one or more of the above factors make a weak recommendation more likely.
Panels should consider all of these factors and make the reasons for their judgements explicit.
Recommendations should specify the perspective that is taken (e.g. individual patient, health system) and which outcomes were considered (including which, if any costs).

39. Implications of a strong recommendation
Patients: Most people in your situation would want the recommended course of action and only a small proportion would not
Clinicians: Most patients should receive the recommended course of action
Policy makers: The recommendation can be adapted as a policy in most situations

40. Implications of a weak recommendation
Patients: The majority of people in your situation would want the recommended course of action, but many would not
Clinicians: Be prepared to help patients to make a decision that is consistent with their own values
Policy makers: There is a need for substantial debate and involvement of stakeholders

41. Recommendations
Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (????? recommendation, very low quality evidence).

42. Recommendations
Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence).

43. Comparison of GRADE and other systems
Explicit definitions
Explicit, sequential judgements
Components of quality defined
Quality by outcome and overall quality
Relative importance of outcomes
Balance between health benefits and harms
Balance between incremental health benefits and costs
Evidence profiles
International collaboration
Consistent judgements?
Communication?

44. Summary
GRADE system harmonizes grading of recommendations and quality of (overall) evidence
Two levels of strength (strong and weak)
Four levels of quality (high, moderate, low and very low)
Quality for each recommendation based on overall quality of evidence
Transparency is key