1. Development and Testing of the AHRQ QI Toolkit for Hospitals Donna O. Farley, PhD Peter Hussey, PhD RAND Corporation

2. Set of tools that hospitals can use to help improve performance in quality and patient safety Set of tools that hospitals can use to help improve performance in quality and patient safety The AHRQ Quality Indicators (QIs) The AHRQ Quality Indicators (QIs) – Inpatient Quality Indicators (IQIs) – Patient Safety Indicators (PSIs) Targeted to wide range of hospitals Targeted to wide range of hospitals – Independent or system-affiliated – Varying quality improvement experience What Is the Toolkit? 2

3. Toolkit Development Developed through a Task Order in the AHRQ ACTION program Developed through a Task Order in the AHRQ ACTION program RAND partnered with UHC to develop and test the toolkit RAND partnered with UHC to develop and test the toolkit AHRQ will continue toolkit support AHRQ will continue toolkit support 3

4. Applicable for hospitals with differing knowledge, skills, and needs Applicable for hospitals with differing knowledge, skills, and needs Serves as a “resource inventory” from which hospitals can select tools Serves as a “resource inventory” from which hospitals can select tools Different audiences for each tool (e.g., quality officer, finance officer, programmer) Different audiences for each tool (e.g., quality officer, finance officer, programmer) How Hospitals Can Use the Toolkit 4

5. What Are the Quality Indicators? Inpatient Quality Indicators – 28 indicators of quality in four sets Inpatient Quality Indicators – 28 indicators of quality in four sets – Volume, counts (6) – Mortality for conditions, rates (7) – Mortality for procedures, rates (8) – Utilization, rates (7) Patient Safety Indicators – Patient Safety Indicators – – 17 indicators and a composite indicator – Screen for adverse events for inpatients – Expressed as rates 5

6. The Development Process Develop Alpha Toolkit Identify tools to include Develop draft tools Field Test Alpha Toolkit Revise and Finalize Toolkit for Dissemination Perform Evaluation Improvement experiences Usability of toolkit Effects on QI values 6

7. Established principles to guide toolkit development Established principles to guide toolkit development Reviewed literature to guide design Reviewed literature to guide design Developed outline of toolkit based on steps of a quality improvement process Developed outline of toolkit based on steps of a quality improvement process Identified and developed specific tools for each step Identified and developed specific tools for each step Tool Development Steps 7

8. Technical Advisory Panel 8 Six-member panel Six-member panel Brought various skills and perspectives Brought various skills and perspectives – Hospital experience – Quality improvement – Relevant research skills Provided guidance throughout toolkit development Provided guidance throughout toolkit development – Toolkit design principles – Content of the tools

9. Parsimony in tool choice and design Parsimony in tool choice and design Target the most important factors for implementation Target the most important factors for implementation Provide tools that offer most value for a range of hospitals Provide tools that offer most value for a range of hospitals Readily accessible contents Readily accessible contents Enable hospitals to assess effectiveness of their actions Enable hospitals to assess effectiveness of their actions Principles Guiding Toolkit Development 9

10. Structure of the Toolkit Introduction and Roadmap A.Readiness to Change B.Applying QIs to the Hospital Data C.Identifying Priorities for Quality Improvement D.Implementation Methods E.Monitoring Progress and Sustainability of Improvements F.Return-on-Investment Analysis G.Existing Quality Improvement Resources 10

11. The Roadmap A navigational guide through the toolkit A navigational guide through the toolkit For each tool, it summarizes: For each tool, it summarizes: – Action step being taken – Brief description of the tool – Key audience(s) to use the tool – Position with lead role responsibility 11

12. Quality improvement collaborative Quality improvement collaborative Conducted by UHC Conducted by UHC 11 hospitals participated 11 hospitals participated Structured implementation process for improvements on the QIs Structured implementation process for improvements on the QIs Evaluation performed by RAND Evaluation performed by RAND Field Test Design 12

13. WHAT WE LEARNED

14. Evaluation Design Six hospitals participated in evaluation Six hospitals participated in evaluation Designed to learn: Designed to learn: – Hospital implementation strategies – Experiences in Improvement effort – Usefulness and usability of the tools Data collection Data collection – Pre/post interviews – Regular update calls during study period – Three post-interviews in site visits 14

15. Overall, Positive Feedback The tools were judged by the hospitals to be usable and useful The tools were judged by the hospitals to be usable and useful Hospitals varied widely in how many, and which, tools they chose to apply Hospitals varied widely in how many, and which, tools they chose to apply Toolkit was useful for achieving staff consensus on the extent of quality gaps and on evidence-based practices Toolkit was useful for achieving staff consensus on the extent of quality gaps and on evidence-based practices 15

16. PSI 4: Death among surgical inpatients w/ serious complications PSI 4: Death among surgical inpatients w/ serious complications PSI 7: Central venous catheter-related bloodstream infection PSI 7: Central venous catheter-related bloodstream infection PSI 12: Postoperative pulmonary embolism or deep vein thrombosis PSI 12: Postoperative pulmonary embolism or deep vein thrombosis PSI 13: Postoperative sepsis PSI 13: Postoperative sepsis PSI 15: Accidental puncture/laceration PSI 15: Accidental puncture/laceration PSI 19: Obstetric trauma-vaginal delivery w/o instrument PSI 19: Obstetric trauma-vaginal delivery w/o instrument All the Hospitals Chose to Address PSIs 16

17. Three Key Learnings Hospitals need to trust their data Hospitals need to trust their data Priority-setting is challenging Priority-setting is challenging Keep the tools short and simple Keep the tools short and simple 17

18. Need to Trust Your Data The IQI or PSI rates have to be credible: “If we’re running reports over coding information, we have to be mindful of coding issues before engaging medical staff. Need to be sure that we’re not wasting their time.” “If we’re running reports over coding information, we have to be mindful of coding issues before engaging medical staff. Need to be sure that we’re not wasting their time.” 18

19. Priority-Setting is Challenging Many hospitals commented on prioritization: “It’s a great benefit to look at data and explore it to see if it’s an issue… I don’t know if [hospitals] have the time to do that, unless it’s driven by corporate leadership or pay structure. [There are] so many other things that we’re mandated to report and improve, it’s hard to look for something else.” 19

20. Keep Tools Simple Users should be able to easily find the tools they need: “People have so much going on that it’s hard…” “I think we have to come up with simpler versions…” 20

21. Revised Toolkit To Address These Issues Added a documentation and coding tool to improve PSI validity Added a documentation and coding tool to improve PSI validity Made prioritization matrix tools flexible so a hospital can tailor it with factors it considers in priority-setting Made prioritization matrix tools flexible so a hospital can tailor it with factors it considers in priority-setting Simplified tools and instructions to increase usability Simplified tools and instructions to increase usability 21

22. QUESTIONS?