1. This sessions will be interactive. So please sit with people with whom you will enjoy a conversation

2. GRADE: A splendid opportunity for teaching (and learning) evidence-based practice PlanPlan –justify the title –GRADE background –two steps quality of evidencequality of evidence strength of recommendationstrength of recommendation –profiles and summary of findings –an example in applying GRADE

3. Why a splendid opportunity your students are likely to be seeing a lot of ityour students are likely to be seeing a lot of it exemplifies three key principles of EBHCexemplifies three key principles of EBHC –need for systematic reviews of best evidence –hierarchy of evidence –need for values and preferences if you understand GRADE you understand how to use evidence to inform practiceif you understand GRADE you understand how to use evidence to inform practice

4. Menu fundamentals of GRADEfundamentals of GRADE –interactive lecture exercises in applying GRADEexercises in applying GRADE optional interactive lecturesoptional interactive lectures –GRADE and diagnosis –GRADE and resource use –GRADE and subgroup analysis –GRADE and stopping early for benefit –values and preferences in decision-making

5. Plan GRADE backgroundGRADE background two stepstwo steps –confidence in estimates (quality of evidence) –strength of recommendation evidence profilesevidence profiles

6. Plan GRADE backgroundGRADE background two stepstwo steps –confidence in estimates (quality of evidence) –strength of recommendation quality and strength can differquality and strength can differ profiles and summary of findingsprofiles and summary of findings importance of values/preferencesimportance of values/preferences an exercise in applying GRADEan exercise in applying GRADE

7. Plan GRADE backgroundGRADE background two stepstwo steps –confidence in estimates (quality of evidence) –strength of recommendation importance of values/preferencesimportance of values/preferences an exercise in applying GRADEan exercise in applying GRADE

8. Plan GRADE backgroundGRADE background two stepstwo steps –confidence in estimates (quality of evidence) –strength of recommendation application to breast cancer screeningapplication to breast cancer screening –contrast with USPSTF

9. Plan GRADE backgroundGRADE background two stepstwo steps –confidence in estimates (quality of evidence) –strength of recommendation profiles and summary of findingsprofiles and summary of findings importance of values/preferencesimportance of values/preferences an exercise in applying GRADEan exercise in applying GRADE GRADE and economic evaluationGRADE and economic evaluation

10. any experience participating in guideline panels?any experience participating in guideline panels? Is grading recommendations a good idea?Is grading recommendations a good idea? Why?Why? experience with gradingexperience with grading –systems used?

11. Why Grade Recommendations? strong recommendationsstrong recommendations –strong methods –large precise effect –few down sides of therapy weak recommendationsweak recommendations –weak methods – imprecise estimate – small effect – substantial down sides

12. Summarizing recommendations clinicians need succinct summariesclinicians need succinct summaries should includeshould include –confidence in estimates –summaries of best estimates of effect all patient-important outcomesall patient-important outcomes –strength of recommendations GRADE working groupGRADE working group –BMJ 2004 and 2008

13. Grading good idea, but which grading system to use? many availablemany available –Australian National and MRC –Oxford Center for Evidence-based Medicine –Scottish Intercollegiate Guidelines (SIGN) –US Preventative Services Task Force –American professional organizations AHA/ACC, ACCP, AAP, Endocrine society, etc....AHA/ACC, ACCP, AAP, Endocrine society, etc.... cause of confusion, dismaycause of confusion, dismay

16. Dilemma: proliferation of systems Solution: common international grading system? GRADE (Grades of recommendation, assessment, development and evaluation)GRADE (Grades of recommendation, assessment, development and evaluation) international groupinternational group –Australian NMRC, SIGN, USPSTF, WHO, NICE, Oxford CEBM, CDC, CC ~ 30 meetings over last eleven years~ 30 meetings over last eleven years (~10 – 50 attendants)(~10 – 50 attendants)

17. Common international grading system? GRADE (Grades of recommendation, assessment, development and evaluation)GRADE (Grades of recommendation, assessment, development and evaluation) international groupinternational group –Australian NMRC, SIGN, USPSTF, WHO, NICE, Oxford CEBM, CDC, CC ~ 30 meetings over last 11 years~ 30 meetings over last 11 years (~10 – 50 attendants)(~10 – 50 attendants)

18. Grading system – for what? interventionsinterventions –management strategy 1 versus 2 what grade is not aboutwhat grade is not about –individual studies (body of eidence) –prognostic questions in patients with heart failure is high uric acid associated with increased riskin patients with heart failure is high uric acid associated with increased risk –diagnostic accuracy questions in lung cancer, what is the accuracy of CT scanning of the mediastinumin lung cancer, what is the accuracy of CT scanning of the mediastinum

19. GRADE Uptake Agencia sanitaria regionale, Bologna, Italia Agency for Health Care Research and Quality (AHRQ) Allergic Rhinitis and Group - Independent Expert Panel American Association for the study of liver diseases American College of Cardiology Foundation American College of Chest Physicians American College of Emergency Physicians American College of Physicians American Endocrine Society American Gastroenterology Association American Society for Colposcopy and Cervical Pathology American Society of Gastrointestinal Endoscopy American Society of Interventional Pain Physicians American Thoracic Society (ATS) Ärztliches Zentrum für Qualität in der Medizin - Germany Austrian Ludwig Boltzmann Institute for HTA BMJ Clinical Evidence British Medical Journal Canadian Agency for Drugs and Technology in Health Canadian Cardiovascular Society Canadian Society of Nephrology Canadian Task Force on Preventive Health Care Centre for Disease Control Committee on Immunization Practices COMPUS at CADTH - Canada Centers for Disease Control Cochrane Collaboration Critical Ultrasound Journal Dutch Institute for Healthcare Improvement CBO EBM Guidelines Finland Emergency Medical Services for Children National Resource Center European Association for the Study of the Liver European Monitoring Centre for Drugs and Drug Addicaton European Respiratory Society European Society of Thoracic Surgeons Evidence-based Nursing Sudtirol, Alta Adiga, Italy Evidence-Based Tuberculosis Diagnosis (tbevidence.org) - Canada Finnish Office of Health Technology Assessment German Agency for Quality in Medicine German Center for Evidence-based Nursing "sapere aude" - Germany Heelth Inspectorate for Scotland Infectious Disease Society of America Institute for Clinical Systems Improvement Japanese Society of Oral and Maxillofacial Radiology Japanese Society for Temporomandibular Joint Joslin Diabetes Center Journal of Infection in Developing Countries Kaiser Permanente Kidney Disease International Guidelines Organization Ministry of Health and Long-term Care, Ontario National and Gulf Centre for Evidence-based Medicine National Board of Health and Welfare - Sweden National Institute for Clinical Excellence (NICE) National Kidney Foundation NHS Quality Improvement Scotland - UK Norwegian Knowledge Centre for the Health Services Ontario MOH Medical Advisory Secretariat Panama and Costa Rica National Clinical Guidelines Program Polish Institute for EBM Scottish Intercollegiate Guideline Network (SIGN) Society of Critical Care Medicine Society of Pediatric Endocrinology Society of Vascular Surgery Spanish Society of Family Practice (SEMFYC) Stop TB Diagnostic Working Group Surviving sepsis campaign Swedish Council on Technology Assessment in Health Care Swedish National Board of Health and Welfare University of Pennsylvania Health System for EB Practice UpToDate WINFOCUS World Allergy Organization World Health Organization (WHO)

20. GRADE uptake

22. What are we grading? two componentstwo components confidence in estimate of effect adequate to support decision (quality of body of evidence)confidence in estimate of effect adequate to support decision (quality of body of evidence) high, moderate, low, very lowhigh, moderate, low, very low strength of recommendationstrength of recommendation strong and weakstrong and weak

23. Confidence in estimate (quality of evidence) no confidence totally confident High Moderate Low Very Low

24. Health Care Question (PICO) Systematic reviews Studies Outcomes Important outcomes Rate the quality of evidence for each outcome, across studies RCTs start high, observational studies start low (-) Study limitations Imprecision Inconsistency of results Indirectness of evidence Publication bias likely Final rating of quality for each outcome: high, moderate, low, or very low (+) Large magnitude of effect Dose response Plausible confounders would ↓ effect when an effect is present or ↑ effect if effect is absent Decide on the direction (for/against) and grade strength (strong/weak*) of the recommendation considering: Quality of the evidence Balance of desirable/undesirable outcomes Values and preferences Decide if any revision of direction or strength is necessary considering: Resource use *also labeled “conditional” or “discretionary” Rate overall quality of evidence (lowest quality among critical outcomes) S1S2S3S4 OC1OC2OC3OC4 OC1OC3 Critical outcomes OC4 Generate an estimate of effect for each outcome OC2 S5

25. Structured question patients: lymphoma patients at risk of developing chemotherapy-induced febrile neutropenia granulocyte colony-stimulating (G-CSF) alternative not using G-CSF

26. Need to define all patient-important outcomes and evaluate their importance

27. Structured question patients: – –women considering breast cancer screening – –age 40-9; 50 to 74; > 75 – –no  risk genetic mutation chest radiation intervention – –film mammography alternative – –no screening

28. Need to define all patient-important outcomes and evaluate their importance desirable consequences – –reduction in breast cancer mortality undesirable consequences – –false positive screening results - anxiety – –invasive procedures from positive results – –complications of invasive procedures – –unnecessary diagnosis and treatment

29. Determinants of confidence RCTs start highRCTs start high observational studies start lowobservational studies start low what can lower confidence?what can lower confidence? –detailed design and execution –inconsistency –indirectness –imprecision –reporting bias

30. Determinants of confidence RCTs start highRCTs start high observational studies start lowobservational studies start low what can lower confidence?what can lower confidence? risk of biasrisk of bias inconsistencyinconsistency indirectnessindirectness imprecisionimprecision publication biaspublication bias

31. Determinants of confidence RCTs start highRCTs start high observational studies start lowobservational studies start low 5 limitations can lower confidence5 limitations can lower confidence detailed design and executiondetailed design and execution –concealment, blinding, loss to follow-up inconsistencyinconsistency –variability in results (heterogeneity) publication biaspublication bias

32. Determinants of confidence risk of biasrisk of bias –concealment – blinding – loss to follow-up imprecisionimprecision –wide confidence intervals publication biaspublication bias

33. Determinants of confidence RCTs start highRCTs start high observational studies start lowobservational studies start low 5 limitations can lower confidence5 limitations can lower confidence BiasBias –study design and implementation concealment, blinding, loss to follow-upconcealment, blinding, loss to follow-up –publication bias ImprecisionImprecision –wide confidence intervals

36. Determinants of confidence RCTs start highRCTs start high observational studies start lowobservational studies start low limitations can lower confidence?limitations can lower confidence? BiasBias –detailed design and execution concealment, blinding, loss to follow-upconcealment, blinding, loss to follow-up ImprecisionImprecision –wide confidence intervals

37. Determinants of confidence RCTs start highRCTs start high observational studies start lowobservational studies start low limitations can lower confidence?limitations can lower confidence?

38. Determinants of confidence 5 limitations can lower confidence5 limitations can lower confidence risk of biasrisk of bias –concealment, blinding, loss to follow-up imprecisionimprecision inconsistencyinconsistency –variability in results (heterogeneity) IndirectnessIndirectness publication biaspublication bias

39. Determinants of confidence 5 limitations can lower confidence5 limitations can lower confidence risk of biasrisk of bias –concealment, blinding, loss to follow-up imprecisionimprecision inconsistencyinconsistency –variability in results (heterogeneity) publication biaspublication bias

40. Risk of Bias well establishedwell established –concealment –intention to treat principle observed –blinding –completeness of follow-up more recentmore recent –selective outcome reporting bias

41. Consistency of results if inconsistency, look for explanationif inconsistency, look for explanation –patients, intervention, outcome, methods judgment of consistencyjudgment of consistency –variation in size of effect –overlap in confidence intervals –statistical significance of heterogeneity –I2–I2–I2–I2

42. Relative Risk with 95% CI for Vitamin D Non-vertebral Fractures Chapuy et al, (2002) 0.85 (0.64, 1.13) Pooled Random Effect Model 0.82 (0.69 to 0.98) p= 0.05 for heterogeneity, I 2 =53% Chapuy et al, (1994) 0.79 (0.69, 0.92) Lips et al, (1996) 1.10 (0.87, 1.39) Dawson-Hughes et al, (1997) 0.46 (0.24, 0.88) Pfeifer et al, (2000) 0.48 (0.13, 1.78) Meyer et al, (2002) 0.92 (0.68, 1.24) Trivedi et al, (2003) 0.67 (0.46, 0.99) Favours Vitamin D Favours Control Relative Risk 95% CI

43. Relative Risk with 95% CI for Vitamin D (Non-Vertebral Fractures, Dose >400) Chapuy et al, (1994) 0.70 (0.69, 0.92) Dawson-Hughes et al, (1997) 0.46 (0.24, 0.88) Pfeifer et al, (2000).48 (0.13, 1.78) Chapuy et al, (2002) 0.85 (0.64, 1.13) Trivedi et al, (2003) 0.67 (0.46, 0.99) Pooled Random Effect Mode 0.75 (0.63 to 0.89) p= 0.26 for heterogeneity, I 2 =24% Favours Vitamin D Favours Control Relative Risk 95% CI

44. Relative Risk with 95% CI for Vitamin D (Non-Vertebral Fractures, Dose = 400) Lips et al (1996) 1.10 (0.87, 1.39) Meyer et al (2002) 0.92 (0.68, 1.24) Pooled Random Effect Mode 1.03 (0.86 to 1.24) p = 0.35 heterogeneity, I 2 =0% Favours Vitamin D Favours Control Relative Risk 95% CI

45. Should we believe sub-group analysis? within-study comparison? Nowithin-study comparison? No large difference in effect Borderlinelarge difference in effect Borderline unlikely chance Yes, p = 0.006unlikely chance Yes, p = 0.006 consistent across studies Yesconsistent across studies Yes a priori hypothesis Yesa priori hypothesis Yes one of small number hypotheses Yesone of small number hypotheses Yes biologically compelling Yesbiologically compelling Yes shall we believe sub-group analysis?shall we believe sub-group analysis?

46. Confidence judgments: Directness populationspopulations –older, sicker or more co-morbidity interventionsinterventions –warfarin in trials vs clinical practice outcomesoutcomes –important versus surrogate outcomes –glucose control versus CV events

47. Flatulence Figure 6: Hierarchy of outcomes according to their patient-importance to assess the effect of phosphate lowering drugs in patients with renal failure and hyperphophatemia Importance of endpoints Critical for decision making Important, but not critical for decision making Of low patient- importance 2 5 Pain due to soft tissue Calcification / function 6 Fractures 7 Myocardial infarction 8 Mortality 9 3 4 1 Coronary calcification Ca 2+ /P- Product Surrogates of declining importance Bone density Ca 2+ /P- Product Soft tissue calcification Ca 2+ /P- Product Lower by one level for indirectness Lower by two levels for indirectness

48. Alendronate Risedronate Placebo Directness interested in A versus B available data A vs C, B vs C

49. Imprecision small sample sizesmall sample size –small number of events wide confidence intervalswide confidence intervals –uncertainty about magnitude of effect how do you decide what is too wide?how do you decide what is too wide? primary criterion:primary criterion: –would decisions differ at ends of CI

50. Imprecision small sample sizesmall sample size –small number of events wide confidence intervalswide confidence intervals –uncertainty about magnitude of effect

51. Precision atrial fib at risk of strokeatrial fib at risk of stroke warfarin increases serious gi bleedingwarfarin increases serious gi bleeding –3% per year 1,000 patients 1 less stroke1,000 patients 1 less stroke –30 more bleeds for each stroke prevented 1,000 patients 100 less strokes1,000 patients 100 less strokes –3 strokes prevented for each bleed where is your threshold?where is your threshold? –how many strokes in 100 with 3% bleeding?

52. 01.0%

53. 0

54. 0

55. 0

56. 00.5%1.0%

57. Example: clopidogrel or ASA? pts with threatened strokepts with threatened stroke RCT of clopidogrel vs ASARCT of clopidogrel vs ASA –19,185 patients ischaemic stroke, MI, or vascular death compared – –939 events (5·32%) clopidogrel – –1021 events (5·83%) with aspirin RR 0.91 (95% CI 0.83 – 0.99) (p=0·043) rate down for precision?rate down for precision?

58. 01.0% Clopidogrel or ASA for threatened vascular events RCT 19,185 patients 1.7% - 0.9 – 0.1% RR 0.91 (95% CI 0.83 – 0.99)

59. 0 Non-inferiority

60. 0

61. 0

62. Imprecision – additional problem small trials, large effectsmall trials, large effect –likely to be overestimate analogy to stopping earlyanalogy to stopping early lack of prognostic balancelack of prognostic balance solution: optimal information sizesolution: optimal information size –# of pts from conventional sample size calculation –specify control group risk, α, β, Δ

63. Fluoroquinolone prophylaxis in neutropenia: infection-related mortality Total number of events: 47

64. Fluoroquinolone prophylaxis in neutropenia: infection-related mortality sample size 1,002 α 0.05, β 0.20, Δ 0.25 RRR, CER 7% N = 6,000

65. Stroke – Fixed Effects

67. Rating down for precision if OIS not met rate down for imprecisionif OIS not met rate down for imprecision –unless very large ss (? > 1,000 per group) if OIS met and CI exclude RR = 1, don’t downgrade if OIS met and CI exclude RR = 1, don’t downgrade if OIS met and CI includes, RR = 1, downgrade only if RR 1.25if OIS met and CI includes, RR = 1, downgrade only if RR 1.25

68. Criteria for NOT rating down CI narrow enough to permit confident recommendation for or againstCI narrow enough to permit confident recommendation for or against if positive benefit outcome, safeguard against false positiveif positive benefit outcome, safeguard against false positive –OIS or -number threshold met (300)

69. Precision, bottom lines consider rating down for precisionconsider rating down for precision –sample size small, CI wide look at upper and lower boundaries of CIlook at upper and lower boundaries of CI –if decisions differ, likely rate down if effect large (RRR > 30%) look againif effect large (RRR > 30%) look again –OK if OIS met (check figure, or 300 events) –total sample >4,000

70. Publication bias high likelihood could lower qualityhigh likelihood could lower quality when to suspectwhen to suspect number of small studiesnumber of small studies industry sponsoredindustry sponsored

73. Funnel Plot Fish oil on mortality

74. Funnel plots what else can cause asymmetry?what else can cause asymmetry? small trials were positively biasedsmall trials were positively biased small trials enrolled a more responsive populationsmall trials enrolled a more responsive population small trials had superior implementation of interventionssmall trials had superior implementation of interventions

75. What can raise confidence? large magnitude can rate up one levellarge magnitude can rate up one level –very large two levels common criteriacommon criteria –everyone used to do badly –almost everyone does well –quick action hip replacement for hip osteoarthritiship replacement for hip osteoarthritis

76. Dose-response gradient childhood lymphoblastic leukemiachildhood lymphoblastic leukemia risk for CNS malignancies 15 years after cranial irradiationrisk for CNS malignancies 15 years after cranial irradiation no radiation: 1% (95% CI 0% to 2.1%) 12 Gy: 1.6% (95% CI 0% to 3.4%) 18 Gy: 3.3% (95% CI 0.9% to 5.6%).

77. Observational study starts low How else can they move plausible confounders strengthen association FP hospitals higher mortality than NFP hospitals – –NFP treat sicker patients – –FP treat wealthier patients can go to very low for all 5 limitations

78. Events vanishingly small in controls series of 10,000 patients undergoing colonoscopy incidence of associated colonic rupture high quality evidence

79. Confidence assessment criteria

80. Best estimates of effect binary outcomes – –relative risks similar across baseline risk need absolute risk to make tradeoffs risk difference from baseline risk and RR baseline risk – –ideally prediction rule observational studies – –observational studies without rule – –median or RCTs

81. Quality Assessment Summary of Findings Quality Relative Effect (95% CI) Absolute risk difference Outcome Number of participants (studies) Risk of Bias ConsistencyDirectnessPrecision Publication Bias Myocardial infarction 10,125 (9) No serious limitations No serious imitations No serious limitations Not detected High 0.71 (0.57 to 0.86) 1.5% fewer (0.7% fewer to 2.1% fewer) Mortality 10,205 (7) No serious limitations Possiblly inconsistent No serious limitations Imprecise Not detected Moderate or low 1.23 (0.98 – 1.55) 0.5% more (0.1% fewer to 1.3% more) Stroke 10,889 (5) No serious limitaions No serious limitations Not detected High 2.21 (1.37 – 3.55) 0.5% more (0.2% more to 1.3% more0 Beta blockers in non-cardiac surgery

85. Quality Assessment Summary of Findings Quality Relative Risk (95% CI) p-value Illustrative risks Outcome No. of patients (studies) Risk of BiasInconsistencyIndirectnessImprecision Publication Bias Example control rate Associated risk with PVL Hospital mortality 1,664 (9) Inability to blind. 2 trials stopped early with few events and large effects; were also confounded by ‘open lung’ strategies. p = 0.07 I 2 = 45.6% Varied populations, interventions. Not robust in sensitivity analyses DirectPreciseUndetected Moderate (due to inconsistency) 0.82 (0.68 – 0.99) p = 0.04 40% 32.8% (27.2 – 39.6) Barotrauma 1,497 (7) Inability to blind. p = 0.24 I 2 = 25.3% Varied populations, interventions DirectImpreciseUndetected Moderate (due to imprecision) 0.90 (0.66 – 1.24) p = 0.53 NS Paralysis 1,202 (5) Inability to blind. p = 0.004 I 2 = 59% Varied populations, interventions, measurements DirectPrecise Not assessed Moderate (due to inconsistency) 1.37 (1.04 – 1.82) p = 0.03 30% 41.1% (31.2 – 54.6) Dialysis 173 (2) Inability to blind. p = 0.26 I 2 = 22.8% Varied populations, interventions DirectImprecise Not assessed Moderate (due to imprecision) 1.76 (0.79 – 3.90) p = 0.16 NS Pressure limited ventilation

86. Quality Assessment Summary of Findings Quality Relative Risk (95% CI) p-value Illustrative risks Outcome No. of patients (studies) Risk of BiasInconsistencyIndirectnessImprecision Publication Bias control rate vaccinated rate Zoster episodes 38,546 (1) No serious riskonly one study DirectPreciseUndetected High not reported 11.12 per 1,000 patient- years 5.42 (difference 5.7 per 1,000 pt-years (p< 0.001) Post- herpetic neuralgia 38,546 (1) No serious riskonly one study DirectPreciseUndetected High not reported 1.38 per 1,000 patient- years 0.46 (difference 0.92 per 1,000 pt- years (p< 0.001) Serious adverse events 38,546 (1) No serious risk only one study DirectPreciseUndetected High Not reported 13 per 1,000 19 (difference 6 per 1,000) Zoster vaccine

87. PopulationNo. of participants (trials) † Higher PEEP Lower PEEP Adjusted Relative Risk (95% CI; P-value) ‡ Adjusted Absolute Risk Difference (95% CI) Quality Patients with ARDS 1892 (3)324/951 (34.1%) 368/941 (39.1%) 0.90 (0.81 to 1.00; 0.049) -3.9% (-7.4% to -0.04%)High Patients without ARDS 404 (3)50/184 (27.2%) 41/220 (18.6%) 1.37 (0.98 to 1.92; 0.065) 6.9% (-0.4% to 17.1%)Moderate (imprecision) High versus low PEEP in ALI and ARDS

88. Whipples procedure pancreatic cancer with or without duodenectomy

90. Patients or population: Anyone taking a long flight (lasting more than 6 hours) Settings: International air travel Intervention: Compression stockings 1 Comparison: Without stockings Outcomes Illustrative comparative risks * (RANGE OF UNCERTAINTY) Relative effect (95% CI) Number of participants (studies) Quality of the evidence (GRADE) Comments Assumed riskCorresponding risk Without stockingsWith stockings (95% CI) Symptomatic deep vein thrombosis (DVT) See comment Not estimable 2637 (9 studies) See comment0 participants developed symptomatic DVT in these studies. Symptomatic deep vein thrombosis – surrogate symptomless deep vein thrombosis Low risk population 2 RR 0.10 (0.04 to 0.25) 2637 (9 studies)   Moderate 3 Estimates of control group asymptomatic thrombosis from the primary studies range from 15 per 1,000 in low risk patients to 25 per 1,000 in high risk patients 5 per 10,0000.5 per 10,000 (0 to 1) High risk population 2 18 per 10,0001.8 per 10,000 (0.5 to 4) Superficial vein thrombosis 13 per 10006 per 1000 (2 to 15) RR 0.45 (0.18 to 1.13) 1804 (8 studies)   Moderate 4

91. Overall level of evidence most systems just use evidence about primary benefit outcomemost systems just use evidence about primary benefit outcome but what about others (risk)?but what about others (risk)? what to do?what to do? optionsoptions –ignore all but primary –weakest of any outcome –some blended approach –weakest of critical outcomes

93. Strength of recommendations degree of confidence that desirable effects of adhering to recommendation outweigh undesirable effects.degree of confidence that desirable effects of adhering to recommendation outweigh undesirable effects. strong recommendationstrong recommendation –benefits clearly outweigh risks/hassle/cost –risk/hassle/cost clearly outweighs benefit

94. Strength of recommendations degree of confidence that desirable effects of adhering to recommendation outweigh undesirable effects.degree of confidence that desirable effects of adhering to recommendation outweigh undesirable effects. strong recommendationstrong recommendation –benefits clearly outweigh risks/hassle/cost –risk/hassle/cost clearly outweighs benefit what can downgrade strength?what can downgrade strength?

95. Strength of Recommendation strong recommendationstrong recommendation –benefits clearly outweigh risks/hassle/cost –risk/hassle/cost clearly outweighs benefit what can downgrade strength?what can downgrade strength? low confidence in estimateslow confidence in estimates close balance between up and downsidesclose balance between up and downsides

96. Risk/Benefit tradeoff aspirin after myocardial infarctionaspirin after myocardial infarction –25% reduction in relative risk –side effects minimal, cost minimal –benefit obviously much greater than risk/cost warfarin in low risk atrial fibrillationwarfarin in low risk atrial fibrillation –warfarin reduces stroke vs ASA by 50% –but if risk only 1% per year, ARR 0.5% –increased bleeds by 1% per year

97. Strength of Recommendations Resuscitate fast in septic patient - do it! Prone ventilation in failing patient with ARDS – –Probably do it – –Probably do not do it

98. Strength of Recommendations Aspirin after MI – do it Warfarin rather than ASA in Afib -- probably do it -- probably don’t do it -- probably don’t do it

100. Significance of strong vs weak variability in patient preferencevariability in patient preference –strong, almost all same choice (> 90%) –weak, choice varies appreciably interaction with patientinteraction with patient –strong, just inform patient –weak, ensure choice reflects values use of decision aiduse of decision aid –strong, don’t bother –weak, use the aid quality of care criterionquality of care criterion –strong, consider –weak, don’t consider

101. Weak recommendation practice will varypractice will vary –according to what? interpretation of evidenceinterpretation of evidence –clopidogrel in stroke patients’ values and preferencespatients’ values and preferences –atrial fibrillation inclination to gamble (risk aversion)inclination to gamble (risk aversion) –HRT

102. When evidence is low confidence choice more preference dependentchoice more preference dependent risk aversionrisk aversion steroids for pulmonary fibrosissteroids for pulmonary fibrosis –low quality evidence in support of benefit –high quality evidence of toxicity

103. When confidence is low recommendation to the hopeful patientrecommendation to the hopeful patient –I’m likely to deteriorate –if something might work, let’s try it –damn the torpedoes recommendation to the fearful patientrecommendation to the fearful patient –doctor, you mean you know it’s toxic diabetes, skin changes, body habitus, infection, osteoporosisdiabetes, skin changes, body habitus, infection, osteoporosis –you don’t know for sure it works? –are you crazy? weak recommendation mandatedweak recommendation mandated

104. Strong recommendation when confidence is low? known benefit, strong recommendation for one of two alternativesknown benefit, strong recommendation for one of two alternatives –antipyretics in children with chickenpox –but which one: ASA or acetaminophen benefit: high quality evidence of equivalencebenefit: high quality evidence of equivalence harm: low quality evidence that harm differs appreciablyharm: low quality evidence that harm differs appreciably –Reye syndrome from ASA strong recommendation for acetaminophenstrong recommendation for acetaminophen

105. Strong recommendation when confidence is low? BlastomycosisBlastomycosis –low quality evidence amphotericin more effective than itraconazole –high quality evidence more toxic patients with life threatening blastopatients with life threatening blasto –life and death situation –strong recommendation for ampho

106. Strong recommendation when confidence is low? head to toe CT scanninghead to toe CT scanning –prevent cancer deaths very low quality evidence of benefitsvery low quality evidence of benefits moderate quality evidence re risks, high re costsmoderate quality evidence re risks, high re costs strong recommendation againststrong recommendation against

107. Presentation strong and weakstrong and weak –discomfort with “weak” –alternative wording: discretionary, conditional strongstrong –“we recommend”… discretionarydiscretionary –“we suggest…” nevernever –we recommend (or suggest) you consider… always: quality of evidence and gradealways: quality of evidence and grade

108. When (not to) GRADE “good to remind/alert” if no systematic review undertakenif no systematic review undertaken no sensible person would consider contraryno sensible person would consider contrary –We recommend that the patient, and the clinician responsible for the patient’s care, should be made aware of any change in a prescribed medication, including change to a generic drug very general (not sufficiently specific)very general (not sufficiently specific) –We suggest that long-term maintenance immunosuppression be tailored to individual patient’s adverse events or risk of adverse events

109. Explicit comparator we recommend hourly urine volume measurement for at least 24 hourswe recommend hourly urine volume measurement for at least 24 hours –in contrast to every 2 hours, every 3…? we suggest measuring serum creatinine in all KTRs at leastwe suggest measuring serum creatinine in all KTRs at least –daily for 7 days –2 to 3 X per week for weeks 2 to 4 –every 2 weeks for months 4 to 6

110. Value and preference statements underlying values and preferences always presentunderlying values and preferences always present sometimes crucialsometimes crucial important to make explicitimportant to make explicit

111. Values and preferences Stroke guideline: patients with TIA clopidogrel over aspirin (Grade 2B). Underlying values and preferences: This recommendation to use clopidogrel over aspirin places a relatively high value on a small absolute risk reduction in stroke rates, and a relatively low value on minimizing drug expenditures.

112. Values and preferences peripheral vascular disease: aspirin be used instead of clopidogrel (Grade 2A). Underlying values and preferences: This recommendation places a relatively high value on avoiding large expenditures to achieve small reductions in vascular events.

113. Flavanoids for Hemorrhoids venotonic agentsvenotonic agents –mechanism unclear, increase venous return popularitypopularity –90 venotonics commercialized in France –none in Sweden and Norway –France 70% of world market possibilitiespossibilities –French misguided –rest of world missing out

114. Systematic Review 14 trials, 1432 patients14 trials, 1432 patients key outcomekey outcome –risk not improving/persistent symptoms –11 studies, 1002 patients, 375 events –RR 0.4, 95% CI 0.29 to 0.57 minimal side effectsminimal side effects is France right?is France right? what is the quality of evidence?what is the quality of evidence?

115. What can lower confidence? risk of biasrisk of bias –lack of detail re concealment –questionnaires not validated indirectness – no problemindirectness – no problem inconsistency, need to look at the resultsinconsistency, need to look at the results

117. Publication bias? size of studiessize of studies –40 to 234 patients, most around 100 all industry sponsoredall industry sponsored

119. What can lower confidence? detailed design and executiondetailed design and execution –lack of detail re concealment –questionnaires not validated inconsistencyinconsistency –almost all show positive effect, trend –heterogeneity p < 0.001; I 2 65.1% indirectnessindirectness imprecisionimprecision –RR 0.4, 95% CI 0.29 to 0.57 publication biaspublication bias –40 to 234 patients, most around 100

120. Is France right? recommendationrecommendation –yes –no against use strengthstrength –strong – weak

121. Resource Use why not cost?why not cost? -may lead to focus on cost of intervention rather than downstream resource use -resource use emphasizes alternative uses of resources (opportunity cost)

122. Resource Use just another outcome? yes and noyes and no who benefits?who benefits? –other outcomes usually clear –costs borne by different payers across societies and within (age)across societies and within (age) some argue costs aren’t relevant to clinicians when third party payersome argue costs aren’t relevant to clinicians when third party payer

123. Why resource use different costs vary much more than other outcomescosts vary much more than other outcomes –across jurisdictions –within jurisdictions –over time even when resource use the same, implications may differeven when resource use the same, implications may differ –year’s supply of expensive drug –nurses’ salary in U.S., 6 in Poland, 30 in China

124. Why resource use different opportunity cost differs by perspectiveopportunity cost differs by perspective hospital pharmacy, fixed budgethospital pharmacy, fixed budget –new expensive drug, clear what give up envelope public spendingenvelope public spending –more on health, less on education, social services –will refraining from spending on drugs really mean more for other services? –should envelope include military spending?

125. Implications unbearable lightness of resource useunbearable lightness of resource use consider balance of desirable and undesirable before considering resource useconsider balance of desirable and undesirable before considering resource use may decide not to consider resource use at allmay decide not to consider resource use at all –intervention not useful –desirable consequences >>>> undesirable –relevant only when difference small

126. Similarities with other outcomes only consider important resource useonly consider important resource use need estimate of difference between trt and controlneed estimate of difference between trt and control explicit judgments about the quality of the evidence, special judgmentsexplicit judgments about the quality of the evidence, special judgments –perspective –how to judge quality of evidence –? use of economic model

127. Evidence summary includes quality of evidence, summary of findingsincludes quality of evidence, summary of findings –“balance sheet”, special form of grade profile resource use and not just costsresource use and not just costs –can judge whether resource use applicable to local setting –focus on coss relevant to them (pharmacy) –apply unit costs to local setting

128. Example question patientspatients –women with pre-eclampsia interventionintervention –intravenous magnesium RCT done in 33 countriesRCT done in 33 countries –over 9,000 patients for presentation of resource use evidence need to specify perspectivefor presentation of resource use evidence need to specify perspective –health system

129. Quality assessment StudiesDesignLimitationsInconsistencyIndirectnessImprecisionNo of patients Relative effect (95% CI) Quality Eclampsia Duley 2003RCTNoone trial onlyNo 9,992RR 0.41 (0.29-0,58) High Maternal death Duley 2003RCTNoone trial onlyNoImprecision9,992RR 0.54 (0.26-1,10) Moderate

130. Quality assessment Design Limita- tions Inconsis- tency Indirect- ness Impre- cision ResourcesCosts per patient Studies per patient(US $; year 2001) Place bo MgSO4PlaceboMgSO4 Magnesium sulphate High GNI 06020 Simon 2005Middle GNI RCTNo one trial only No 0603High Low GNI 0605 Administration of the drug High GNI 01066 Simon 2005Middle GNI RCTNo one trial only No 01014High Low GNI 0108 Other hospital resources a, b High GNI Large variationre sources c NA 12,83912,818 Simon 2005 Middle GNIRCTNo one trial only No NA 1,4121,416 Modera te Low GNI NA 155157

131. Outcomes Typical control group risk Typical absolute effect (95% CI) Relative effect (95% CI) Nr. of participants (studies) Quality of the evidence Comments Clinical outcomes Eclampsia Severe RR 0.41 (0.29 - 0.58) 11,444  High 27 per 1,00016 fewer per 1,000 (11 to 19) Not severe 15 per 1,0009 fewer per 1,000 (6 to 11) Maternal death Severe RR 0.54 (0.26 - 1.10) 10,795   Moderate 2 6 per 1,000 3 fewer per 1,000 (0.6 more to 4 fewer) Not severe 3 per 1,000 1 fewer per 1,000 (0.3 more to 2 fewer) Side effects 46 per 1,000 3 196 more per 1,000 (165 to 231) RR 5.26 (4.59 - 6.03) 9.992  High Mostly flushing. Other side effects include nausea, vomiting, slurred speech, muscle weakness, dizziness, drowsiness, confusion and headache.

132. Magnesium sulphate ampoules 06 10 ml. ampoules per woman 9.996  High Cost High GNI Middle GNI Low GNI $20 more per patient $ 3 more per patient $ 5 more per patient Administration of magnesium sulphate 01 per woman9.996  High Cost High GNI Middle GNI Low GNI $66 per patient $14 per patient $ 8 per patient Resources for administering magnesium sulphate included midwife time (main cost), intravenous cannula/needle, syringe, IV fluids, drug. Other hospital resources Varied widely9.996   Moderate 5 There was large variation in the use of other hospital resources in both intervention and control groups. Cost High GNI Middle GNI Low GNI $12,839 $ 1,416 $ 157 $20 less per woman (0 to 60) $ 4, less per woman (0 to 10) $ 2 less per woman (1 to 3) Other hospital costs have been adjusted based on the influence of eclampsia to control for the many other factors that influenced these costs. Resource use from the perspective of the health system control grop difference trt vs control

133. Issues in resource use broad perspective desirablebroad perspective desirable –narrow perspectives ignore much resource use –users can pick costs relevant to them –either health care system or societal indirect costs controversialindirect costs controversial indirect evidence of resources useindirect evidence of resources use –costs only reported –RCT but doesn’t reflect practice ulcer prevention everyone gets repeat endoscopyulcer prevention everyone gets repeat endoscopy

134. Confidence in estimaates for resource use rules basically the samerules basically the same –RCTs start high, observational low may need multiple sources of evidencemay need multiple sources of evidence –RCTs may not fully report resource use variation across settingsvariation across settings –RCT may not reflect real world –time frame may extend beyond trial different quality for different resourcesdifferent quality for different resources –mag sulphate versus hospital resources

135. Formal economic models limitationslimitations –supported by industry, biased –setting specific –reduces transparency –if evidence low quality, speculative –often many assumptions solution: develop own modelsolution: develop own model –OK if you are NICE with lots of resources even so, don’t include in profileeven so, don’t include in profile

136. Costs versus affordability intervention may be “cost-effective”intervention may be “cost-effective” –$10,000 per qaly gained but if applicable to huge proportion of population, may still be unaffordablebut if applicable to huge proportion of population, may still be unaffordable

137. healthy asymptomatic postmenopausal qomwn: HRT in 1992? Possible benefits –CHD, Hip fracture, Colorectal cancer Possible harms –Breast cancer –Stroke –Thrombosis –Gall bladder disease Can GRADE lead to change?

138. Evidence profile: Quality assessment Oestrogen + progestin for prevention in 1992 (before WHI and HERS) Oestrogen + progestin versus usual care

139. Oestrogen + progestin for prevention after WHI and HERS

140. Postulate major work in preparing guideline/HTA assessment is systematic review If already doing this, GRADE framework should add little history: Rolls-Royce and Volkswagen

141. VW and RR appraoches Rolls Royce (NICE)Rolls Royce (NICE) –systematic review for every outcome –production of evidence profiles –involvement of multiple constituencies including patientsincluding patients –inclusion of economic analysis cost $1 million per guidelinecost $1 million per guideline

142. MOPED GRADE UpToDateUpToDate –5,000 graded recommendations generate PICO (informal)generate PICO (informal) –no formal rating of outcome importance use of existing reviews, primary studiesuse of existing reviews, primary studies –no new evidence syntheses quality for key outcomesquality for key outcomes –5 reasons rating down, 3 up –no new evidence profiles, SoF tables recommendationsrecommendations –strong or weak, consider 3 factors –value and preference statements

143. ACCP formal structured questions no formal rating of outcome importance – –trying to change hit-and miss systematic reviews – –largely only available ones hit-and-miss individual study evidence summaries rare evidence profiles – –trying to change

144. VW approach take systematic reviews if availabletake systematic reviews if available if not, review key, accessible evidenceif not, review key, accessible evidence no meta-analysis if not doneno meta-analysis if not done no evidence profilesno evidence profiles small group make expert judgementsmall group make expert judgement

145. Conclusion clinicians, policy makers need summariesclinicians, policy makers need summaries –confidence in estimates –strength of recommendations explicit rulesexplicit rules –transparent, informative GRADEGRADE –transparent, systematic –increasing wide adoption