1. April 2009 Netta Conyers-Haynes, Principal Consultant, Communications Kaiser Permanente National Guideline Program Implications of IOM SR Standards Wiley Chan, MD Physician, Internal Medicine Methodologist, KP National Guideline Program

2. Kaiser Permanente: Largest Non-Profit Health Care Program in the United States  Founded in 1945  8 regions in 9 states and District of Columbia  8.6 million members (as of 12/09)  15,129 physicians (as of 12/09)  164,098 employees (as of 12/09)  KP Care Management Institute (CMI)  KP National Guideline Program (NGP) Kaiser Foundation Hospitals Permanente Medical Groups Kaiser Foundation Health Plan Kaiser Permanente

3. Kaiser Permanente Evidence Network  15 KP National Evidence-Based Guidelines  Supported by full Systematic Reviews  3.5 Staff FTEs dedicated to KP NGP work  1 Principal Consultant, 2 Analysts, 2 Project Managers  1.5 Physician FTEs dedicated to KP NGP work  7 Physician EBM Methodologists  External vendor: Doctor Evidence  Search & Data Extraction  Technology platform Data repository, analysis & documentation

4. Kaiser Permanente National Guideline Program: Process & Methodology New Clinical Issue Scheduled Update Clinical Questions (CQ) Existing Guideline? CQ Match? Assess Guideline Acceptable? Existing SR by CQ? Assess SR Acceptable? New Relevant Studies? Evidence Search Abstract Article Review Inclusion/ Exclusion Data Extraction Critical Appraisal Qualitative/ Quantitative Synthesis Assess Evidence Quality Evidence Summary RationaleRecommendations Assess Implementability of Recommendations GDT Approval Internal Review GQ NGD Approval National Guideline Yes No Implementation  Overview Key to Abbreviations: CQClinical Question SRSystematic Review GDTGuideline Development Team GQGuideline Quality Committee NGDKP National Guideline Directors

5. Kaiser Permanente National Guideline Program: Process & Methodology New Clinical Issue Scheduled Update Clinical Questions (CQ) Existing Guideline? CQ Match? Assess Guideline Acceptable? GDT Approval Internal Review GQ NGD Approval National Guideline Yes Implementation Acceptable External Guideline  ADAPTE/AGREE

6. Kaiser Permanente National Guideline Program: Process & Methodology New Clinical Issue Scheduled Update Clinical Questions (CQ) Existing Guideline? CQ Match? Assess Guideline Acceptable? Existing SR by CQ? Assess SR Acceptable? New Relevant Studies? Evidence Summary RationaleRecommendations Assess Implementability of Recommendations GDT Approval Internal Review GQ NGD Approval National Guideline Yes No Implementation No Acceptable External Guideline Acceptable External Systematic Review No New Relevant Studies  AMSTAR  GRADE

7. Kaiser Permanente National Guideline Program: Process & Methodology New Clinical Issue Scheduled Update Clinical Questions (CQ) Existing Guideline? CQ Match? Assess Guideline Acceptable? Existing SR by CQ? Assess SR Acceptable? Evidence Search Abstract Article Review Inclusion/ Exclusion Data Extraction Critical Appraisal Qualitative/ Quantitative Synthesis Assess Evidence Quality Evidence Summary RationaleRecommendations Assess Implementability of Recommendations GDT Approval Internal Review GQ NGD Approval National Guideline Yes No Implementation No Acceptable External Guideline No Acceptable External Systematic Review  Internal Systematic Review  GRADE

8. Kaiser Permanente Systematic Review Processes, Responsibilities, & Tools  Clinical Questions  KP: Population, Intervention, Comparison, Outcome Timing, Setting  Evidence Search  Doctor Evidence with KP input Masters-trained Medical Librarians (MLIS)  Abstract/Article Review & Inclusion/Exclusion  Doctor Evidence Dual Inclusion/Exclusion with 3 rd party adjudication  KP: Evaluate exclusions

9. Kaiser Permanente Systematic Review Processes, Responsibilities, & Tools  Data Extraction  Doctor Evidence Dual extraction with adjudication & 3 rd party independent QA Masters-trained & graduate student data extractors Contact authors to clarify discrepancies in published data  KP: Quality Assurance  Critical Appraisal  KP: Cochrane Risk of Bias, Doctor Evidence platform  Qualitative/Quantitative Analysis  KP: Doctor Evidence platform

10. Doctor Evidence Meta-Analysis Page

11. Doctor Evidence Meta-Analysis Results

12. Doctor Evidence Meta-Analysis Study Selection Detailed study- and outcome-specific descriptions of Cochrane Risk of Bias are exposed by hovering Select and de-select studies and Recalculate

13. Doctor Evidence Study Summary Detailed study-specific descriptions of characteristics and outcomes are exposed by hovering

14. Kaiser Permanente Systematic Review Processes, Responsibilities, & Tools  Evidence Grading  KP: GRADE, GRADEPro Developing functionality in Doctor Evidence platform  Evidence Summary (SR)  Rationale  Recommendations & Guideline  KP: GRADE, GRADEPro, Word Developing functionality in Doctor Evidence platform

15. Kaiser Permanente National Guideline Program: Process & Methodology National Guideline Disease Management Accreditation (NCQA) Clinician & Staff Education Tools (Clinical Library) Patient Education Tools (KP.ORG, National Patient Instructions Regions (EHR, CDS, Operations)  Guideline Implementation Key to Abbreviations: EHRElectronic Health Record CDSClinical Decision Support KP.ORGKP publicly-accessible Internet site NCQANational Committee for Quality Assurance

16. IOM Systematic Review Standards Problem Areas for KP  2.6Develop a systematic review protocol  2.7Submit the protocol for peer review  2.7.1Provide a public comment period for the protocol  2.8Make the final protocol publicly available  5.2.1Use a 3 rd party to manage peer review  5.2.2Provide a public comment period for the report  5.3Publish the final report with free public access

17. IOM Systematic Review Standards Challenges  Very resource-intensive  Infinite needs - Finite resources  Balance between efficiency versus rigor  Investment in developing expertise & infrastructure  Migration of existing SRs & CPGs to new methods  Updating SRs & CPGs  Every 2 years?  Dynamic updating, based on evidence & impact?  External review  Patient & public involvement in SRs and CPGs

18. IOM Systematic Review Standards Opportunities  Improved SR & CPG rigor & transparency  Common evidence-grading & recommendation strength language  Collaborate with other SR & CPG developers  Prioritize & coordinate topics  Avoid duplication  Target funding  Share data extractions?  Devote more energy to CPG implementation  Collaborate with performance metric developers

19. Implications of IOM SR Standards Wiley.Chan@kp.org

20. Supplemental Slides

21. IOM Systematic Review Standards Congruent Areas for KP  2.1Establish a team with appropriate expertise  2.2 Manage bias & conflict of interest  2.3Ensure user & stakeholder input  2.4Manage bias & conflict of interest for individuals providing Input  2.5Formulate the topic  3.1Conduct a comprehensive systematic search

22. IOM Systematic Review Standards Congruent Areas for KP  3.2Address potentially biased reporting of research results  3.3Screen & select studies  3.4Document the search  3.5Manage data collection  3.6Critically appraise each study  5.1Prepare final report using a structured format  5.2Peer review the draft report

23. IOM Systematic Review Standards Congruent Areas for KP  4.1Use a prespecified method to evaluate the body of evidence  4.2Conduct a qualitative synthesis  4.3Decide if the SR will include a quantitative analysis  4.4.1Use expert methodologists  4.4.2Address heterogeneity  4.4.3Measures of statistical uncertainty  4.4.4Sensitivity analysis

24. Doctor Evidence Evidence Table Qizard