1. Holger Schünemann, MD, PhD Chair, Department of Clinical Epidemiology & Biostatistics Professor of Clinical Epidemiology, Biostatistics and Medicine McMaster University, Hamilton, Ontario, Canada

2. 1. Formulating questions Guidelines are a way of answering questions about clinical, communication, organisational or policy interventions, in the hope of improving health care or health policy. It is therefore helpful to structure a guideline in terms of answerable questions. WHO Guideline Handbook, 2008

3. Different types of questions Background Questions Definition: e.g. What is Human Papilloma Virus (HPV) infection? Mechanism: e.g. How does HPV cause cancer? Foreground Questions Efficacy: e.g. What is the efficacy of an HPV vaccine? Recommendations/decisions: e.g. e.g. Should we use HPV vaccine?

4. Different types of questions Background Questions Definition: e.g. What is Human Papilloma Virus (HPV) infection? Mechanism: e.g. How does HPV cause cancer? Foreground Questions Efficacy: e.g. What is the efficacy of an HPV vaccine? Recommendations/decisions: e.g. e.g. Should we use HPV vaccine? Actionable items

5. 2. Choosing outcomes  Every decision comes with desirable and undesirable consequences  Developing recommendations must include a consideration of desirable and undesirable consequences

6. Desirable and undesirable consequences  desirable effects  lower mortality  improvement in quality of life, fewer hospitalizations  reduction in the burden of treatment  reduced resource expenditure  undesirable consequences  deleterious impact on morbidity, mortality or quality of life, increased resource expenditure

7. Limitations of older systems & approaches  confuse quality of evidence with strength of recommendations  lack well-articulated conceptual framework  criteria not comprehensive or transparent  focus on single outcomes

8. G rades of R ecommendation A ssessment, D evelopment and E valuation CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005, AJRCCM 2006, Chest 2006, BMJ 2008, Lancet ID 2007, PLOS Medicine 2007

9. GRADE Working Group David Atkins, chief medical officer a Dana Best, assistant professor b Martin Eccles, professor d Francoise Cluzeau, lecturer x Yngve Falck-Ytter, associate director e Signe Flottorp, researcher f Gordon H Guyatt, professor g Robin T Harbour, quality and information director h Margaret C Haugh, methodologist i David Henry, professor j Suzanne Hill, senior lecturer j Roman Jaeschke, clinical professor k Regina Kunx, Associate Professor Gillian Leng, guidelines programme director l Alessandro Liberati, professor m Nicola Magrini, director n James Mason, professor d Philippa Middleton, honorary research fellow o Jacek Mrukowicz, executive director p Dianne O ’ Connell, senior epidemiologist q Andrew D Oxman, director f Bob Phillips, associate fellow r Holger J Sch ü nemann, professor g,s Tessa Tan-Torres Edejer, medical officer t David Tovey, Editor y Jane Thomas, Lecturer, UK Helena Varonen, associate editor u Gunn E Vist, researcher f John W Williams Jr, professor v Stephanie Zaza, project director w a) Agency for Healthcare Research and Quality, USA b) Children's National Medical Center, USA c) Centers for Disease Control and Prevention, USA d) University of Newcastle upon Tyne, UK e) German Cochrane Centre, Germany f) Norwegian Centre for Health Services, Norway g) McMaster University, Canada h) Scottish Intercollegiate Guidelines Network, UK i) F é d é ration Nationale des Centres de Lutte Contre le Cancer, France j) University of Newcastle, Australia k) McMaster University, Canada l) National Institute for Clinical Excellence, UK m) Universit à di Modena e Reggio Emilia, Italy n) Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Italy o) Australasian Cochrane Centre, Australia p) Polish Institute for Evidence Based Medicine, Poland q) The Cancer Council, Australia r) Centre for Evidence-based Medicine, UK s) National Cancer Institute, Italy t) World Health Organisation, Switzerland u) Finnish Medical Society Duodecim, Finland v) Duke University Medical Center, USA w) Centers for Disease Control and Prevention, USA x) University of London, UK Y) BMJ Clinical Evidence, UK

10. The GRADE approach Clear separation of 2 issues: 1) 4 categories of quality of evidence: very low, low, moderate, or high quality?  methodological quality of evidence  likelihood of systematic deviation from truth  by outcome 2) Recommendation: 2 grades – conditional or strong (for or against)?  Quality of evidence only one factor *www.GradeWorkingGroup.org

11. Determinants of quality  RCTs start high  observational studies start low  5 factors lower the quality of evidence  limitations in detailed design and execution  inconsistency  indirectness  reporting bias  imprecision  3 factors can increase the quality of evidence

12. Example: Limitations in Design and Execution  Limitations – observational studies  Failure to develop and apply appropriate eligibility criteria - under- or over-matching in case-control studies  Selection of exposed and unexposed in cohort studies from different populations  Flawed measurement of both exposure and outcome (e.g. recall bias in CC studies)  Differential surveillance for outcome in exposed and unexposed in cohort studies  Failure to adequately measure/control for confounding  Failure to match for prognostic factors and/or adjustment in statistical analysis

13. Quality assessment criteria

14. From Evidence to Recommendation 14

15. Categories of recommendations Although the degree of confidence is a continuum, we suggest using two categories: strong and weak/conditional.  Strong recommendation: the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects.  Conditional recommendation: the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident. Recommend   Suggest  

16. Determinants of the strength of recommendation

17. Judgements about the strength of a recommendation  No precise threshold for going from a strong to a weak recommendation  The presence of important concerns about one or more of these factors make a weak recommendation more likely.  Panels should consider all of these factors and make the reasons for their judgements explicit.  Recommendations should specify the perspective that is taken (e.g. individual patient, health system) and which outcomes were considered (including which, if any costs).

18. Finally: There are no RCTs!  We will do a consensus statement/guideline (and not use rigorous methods)  Do you think that those using the recommendations would like to be informed about the basis (explanation) for a recommendation if they were asked (by their patients)?  I suspect the answer is “yes” 18

19. There are no RCTs! Cont’d  Another reason for using structured approaches: any form of recommendation needs agreement/consensus – whether based on high or lower quality evidence (voting as a forced form of consensus)  Ergo: Consensus statement is a misnomer in regards to differentiating from guideline  The level of detail depends on other aspects:  Funds, time, greater interest, higher priority  Transparency is key 19

20. Conclusions  Clinical practice guidelines should be based on the best available evidence  GRADE provides a structure approach to improve communication – official WHO system  Criteria for evidence assessment across questions and outcomes  Criteria for moving from evidence to recommendations  Transparent, systematic  four categories of quality of evidence  two grades for strength of recommendations  Transparency in decision making and judgments is key

22. What can raise quality? 2. dose response relation  (higher INR – increased bleeding)  childhood lymphoblastic leukemia  risk for CNS malignancies 15 years after cranial irradiation  no radiation: 1% (95% CI 0% to 2.1%)  12 Gy: 1.6% (95% CI 0% to 3.4%)  18 Gy: 3.3% (95% CI 0.9% to 5.6%) 3. all plausible confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed

23. All plausible confounding would result in an underestimate of the treatment effect  Higher death rates in private for-profit versus private not-for-profit hospitals  patients in the not-for-profit hospitals likely sicker than those in the for-profit hospitals  for-profit hospitals are likely to admit a larger proportion of well-insured patients than not-for- profit hospitals (and thus have more resources with a spill over effect)

24. All plausible biases would result in an overestimate of effect  Hypoglycaemic drug phenformin causes lactic acidosis  The related agent metformin is under suspicion for the same toxicity.  Large observational studies have failed to demonstrate an association  Clinicians would be more alert to lactic acidosis in the presence of the agent

26. Implications of a strong recommendation  Patients: Most people in your situation would want the recommended course of action and only a small proportion would not  Clinicians: Most patients should receive the recommended course of action  Policy makers: The recommendation can be adapted as a policy in most situations

27. Implications of a weak/conditional recommendation  Patients: The majority of people in your situation would want the recommended course of action, but many would not  Clinicians: Be prepared to help patients to make a decision that is consistent with their own values  Policy makers: There is a need for substantial debate and involvement of stakeholders

28. Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)?

29. Summary of findings Transmission: No human to human transmission Patient or population: Hospitalised, clinical and serologically confirmed cases of avian influenza

30. Oseltamivir for Avian Flu Summary of findings: No clinical trial of oseltamivir for treatment of H5N1 patients. 4 systematic reviews and health technology assessments (HTA) reporting on 5 studies of oseltamivir in seasonal influenza. Hospitalization: OR 0.22 (0.02 – 2.16) Pneumonia: OR 0.15 (0.03 - 0.69) 3 published case series. Many in vitro and animal studies. No alternative that is more promising at present. Cost: ~ Euro 50 per treatment course

31. What would you recommend?  Strong recommendation: the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects.  Weak recommendation: the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident.

32. Judgments about the strength of a recommendation - oseltamivir for treatment of patients hospitalised with avian influenza (H5N1) FactorsComments Balance between desirable and undesirable effects “The benefits are uncertain, but potentially large.” Quality of the evidence“The quality of the evidence is very low.” Values and preferences“All patients and care providers would accept treatment for H5N1 disease.” No alternative Costs (resource use)“The cost is not high for treatment of sporadic cases.”

33. Who would recommend oseltamivir for these patients (no other alternative)?  YES (green card)  No (pink card)

34. Example: Oseltamivir for Avian Flu Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (????? recommendation, very low quality evidence). Schunemann et al., The Lancet ID, 2007

35. Example: Oseltamivir for Avian Flu Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence). Values and Preferences Remarks: This recommendation places a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment. Schunemann et al., The Lancet ID, 2007

36. Other explanations Remarks: Despite the lack of controlled treatment data for H5N1, this is a strong recommendation, in part, because there is a lack of known effective alternative pharmacological interventions at this time. The panel voted on whether this recommendation should be strong or weak and there was one abstention and one dissenting vote. Schunemann et al., The Lancet ID, 2007