1. Why Grade the Evidence? target audience for Cochrane reviewstarget audience for Cochrane reviews –clinicians interested in the question –policy makers, consumers many in audience will have limitationsmany in audience will have limitations –time –methodological sophistication

2. Implications of Limitations limited time/sophisticationlimited time/sophistication –is evidence strong, inferences secure? –is evidence weak, inferences insecure? shorthand summaryshorthand summary –enhance usefulness of reviews Gray Elrodt in the BerkshiresGray Elrodt in the Berkshires –post-MI smoking cessation, aspirin, beta blocker, ACE inhibitor, statin

3. Grade for Specific Question too vague: alendronate in osteoporosistoo vague: alendronate in osteoporosis patients – post-menopausal womenpatients – post-menopausal women interventionintervention –daily alendronate, dose 10 to 20 mg. outcome – non-vertebral fracturesoutcome – non-vertebral fractures

4. What Influences Grade? study designstudy design –basic –detailed design and execution consistencyconsistency directnessdirectness reporting biasreporting bias

5. Summary Methodological Quality study designstudy design –randomization –observational study detailed design and executiondetailed design and execution –concealment –balance in known prognostic factors –intention to treat principle observed –blinding –completeness of follow-up

6. Summary Methodologic Quality consistency of resultsconsistency of results if inconsistency, look for explanationif inconsistency, look for explanation –patients, intervention, outcome, methods no clear thresholdno clear threshold –size of effect, confidence intervals, statistical significance

7. Risk Difference of Conversion to Sinus Rhythm Amiodarone vs Placebo or Digoxin or CCB Favours Control Favours Amiodarone ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' Bianconi 2000 0.02 (-0.06 to 0.11) Galperin 2000 0.33 (0.19 to 0.47) Hohnloser 2000 0.16 (0.06 to 0.26) Natale 2000 0.67 (0.51 to 0.83) Villani 2000 0.20 (0.06 to 0.33) Cowan 1986 0.08 (-0.19 to 0.36) Noc 1990 0.77 (0.52 to 1.00) Capucci 1992 -0.11 (-0.41 to 0.20) Cochrane 1994 0.13 (-0.06 to 0.33) Donovan 1995 0.03 (-0.21to 0.27) Hou 1995 0.21 (-0.01 to 0.43) Kondili 1995 0.14 (-0.16 to 0.44) Galve 1996 0.08 (-0.11 to 0.27) Kontoyannis 1998 0.31 (0.11 to 0.50) Bellandi 1999 0.27 (0.13 to 0.41) Cotter 1999 0.28 (0.13 to 0.43) Kochiadakis 1999 0.28 (0.15 to 0.41) Joseph 2000 0.19 (-0.02 to 0.39) Peukurinen 2000 0.52 (0.31 to 0.72) Vardas 2000 0.41 (0.28 to 0.53) Cybulski 2001 0.39 (0.24 to 0.54) Pooled Risk Difference 0.26 (0.18 to 0.34) 00.20.40.60.811.2-0.2-0.4-0.6 AF Duration > 48 hrs AF Duration </= 48 hrs

8. Relative Risk of Hospital Mortality: Adult Patients ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Shortell 653 144,159 1.43 Keeler 220 4,937 0.04 Hartz 2,368 3,107,616 11.38 Manheim MH 1,252 1,537,660 9.78 Manheim FS 1,617 2,228,593 2.59 Kuhn 2,580 3,353,676 12.34 Pitterle 3,482 4,529,206 14.11 Mukamel 1,653 5,298,812 17.21 Bond 3,224 4,210,468 12.66 Yuan Medical 3,316 7,386,000 11.90 Yuan Surgical -- 4,396,000 5.05 Lanska 799 16,983 0.00 McClellan 2,875 181,369 1.48 Sloan 2,360 7,079 0.03 Totals 26,399 36,402,558 100.00 0.70.80.911.11.21.3 Relative Risk and 95% CI Favours Private Not-For-Profit Favours Private For-Profit Study Number of Hospitals Number of Patients % Weight Random Effects Pooled Estimate

9. Directness of Evidence indirect treatment comparisonsindirect treatment comparisons –interested in A versus B –have A versus C and B versus C alendronate vs risedronatealendronate vs risedronate –both versus placebo, no head-to-head

10. Four Levels of “Directness” patients meet trials’ eligibility criteriapatients meet trials’ eligibility criteria not included, but no reason to questionnot included, but no reason to question –slight age difference, comorbidity, race some question, bottom line applicablesome question, bottom line applicable –valvular atrial fibrillation serious question about biologyserious question about biology –heart failure trials applicability to aortic stenosis

11. Levels of Directness interventionsinterventions –same drugs and doses –similar drugs and doses –same class and biology –questionable class and biology outcomesoutcomes –same outcomes –similar (duration, quality of life) –less breathlessness for role function –laboratory exercise capacity for q of life

12. Magnitude, Precision, Reporting Bias magnitude not generally part of qualitymagnitude not generally part of quality –but very large magnitude can upgrade precision not generally part of qualityprecision not generally part of quality –but sparse data can lower quality reporting biasreporting bias –high likelihood can lower quality

13. Grading System high qualitywell done RCThigh qualitywell done RCT intermediatequasi-RCTintermediatequasi-RCT lowwell done observationallowwell done observational insufficient anything elseinsufficient anything else

14. Moving Down study execution – –serious flaws can lower by one level – –fatal flaws can lower by two levels consistency – –important inconsistency can lower by one level directness of evidence – –some uncertainty re relevance lower by one level – –major uncertainty re relevance lower by two levels selection bias – –strong evidence lower by 1 level

15. Moving Up very strong association, up 2 levels – –insulin in diabetic ketoacidosis strong, consistent association with no plausible confounders, up 2 levels – –fluoride for preventing cavities strong association can move up 1 level – –? HRT ?

16. Conclusion challenges in gradingchallenges in grading –balancing simplicity and complexity –judgement always required consistent grading system requiredconsistent grading system required must consider study design, execution, consistency, directnessmust consider study design, execution, consistency, directness –magnitude, precision, publication bias only when extreme

17. Questions for Discussion should Cochrane reviews provide:should Cochrane reviews provide: –grades of evidence across studies for all important outcomes? –overall grade of evidence across outcomes? –same grading system across reviews? bonus; while Cochrane reviews shouldn’t provide recommendations should they include structured discussion includingbonus; while Cochrane reviews shouldn’t provide recommendations should they include structured discussion including –trade-offs –translation of evidence in to practice