1. POSITIVE PREDICTIVE VALUE OF AHRQ PATIENT SAFETY INDICATORS IN A NATIONAL SAMPLE OF HOSPITALS AcademyHealth Annual Research Meeting June 9, 2008 Team presenter: Pat Zrelak PhD, CNRN, CNAA-BC 1 Team: Patrick Romano 1 ; Garth Utter 1 ; Richard White 1 ; Dan Tancredi 1 ; Ruth Baron 1 ; Banafsheh Sadeghi; Sheryl Davies; Jeff Geppert 2. Ruth Baron 1 ; Banafsheh Sadeghi; Sheryl Davies; Jeff Geppert 2. 1 University of California Davis, Sacramento, CA. 2 Battelle Memorial Institute, Elk Grove, CA.

2. Background of PSIs Set of quality indicators designed to identify potentially preventable problems that patients may experience as a result of contact with the health care system Set of quality indicators designed to identify potentially preventable problems that patients may experience as a result of contact with the health care system Based on severity adjusted inpatient hospital discharge data Based on severity adjusted inpatient hospital discharge data Initially developed through a contract with UC-Stanford Evidence-based Practice Center Initially developed through a contract with UC-Stanford Evidence-based Practice Center Little is known about the criterion validity of the PSIs across multiple hospitals Little is known about the criterion validity of the PSIs across multiple hospitals

3. AHRQ PSI Validation Pilot Goals Gather evidence on the criterion validity of the PSIs based on medical record review Gather evidence on the criterion validity of the PSIs based on medical record review Improve guidance about how to interpret & use the data Improve guidance about how to interpret & use the data Evaluate potential refinements to the specifications Evaluate potential refinements to the specifications Develop medical record abstraction tools Develop medical record abstraction tools Develop mechanisms for conducting validation studies on a routine basis Develop mechanisms for conducting validation studies on a routine basis

4. Positive Predictive Value The positive predictive value or post-test probability is the proportion of flagged cases who actually had the event. The positive predictive value or post-test probability is the proportion of flagged cases who actually had the event. The Positive Predictive Value (PPV) can be further defined as: The Positive Predictive Value (PPV) can be further defined as:

5. Methods Retrospective cross-sectional study design Retrospective cross-sectional study design Volunteer sample of collaborative partners Volunteer sample of collaborative partners – Facilitating organizations (e.g., Arizona) – Hospital systems – Individual hospitals Sampling based on administrative data Sampling based on administrative data Sampling probabilities assigned using AHRQ QI software to generate desired sample size nationally Sampling probabilities assigned using AHRQ QI software to generate desired sample size nationally

6. Data collection methods Each hospital identified chart abstractors Each hospital identified chart abstractors Training occurred via webinars Training occurred via webinars Medical record abstraction tools & guidelines Medical record abstraction tools & guidelines – Pretested in the Sacramento area – Targeted the ascertainment of the event, risk factors, evaluation & treatment, and related outcomes

7. Timeline ■ 10 indicators- divided into 2 phases of 5 each ■ Phase I review-  Training early 2007  Chart review 4 month process  4 th Qtr 2005, 2006, & 1 st Qtr 2007 ■ Phase II review –  Waiting OMB approval Phase III –sensitivity determination

8. Patient Safety Indicators Phase IPhase II Accidental puncture and laceration Foreign body left in during procedure Selected infection due to medical care Postoperative hemorrhage or hematoma Postoperative pulmonary embolism or deep vein thrombosis Postoperative physiologic and metabolic derangement Postoperative sepsisPostoperative respiratory failure Iatrogenic pneumothoraxPostoperative wound dehiscence

9. Medical record sample Phase IHospitalsSample Accidental puncture and laceration43249 Iatrogenic pneumothorax38205 Postoperative PE/DVT37155 Selected Infection due to Medical Care37194 Postoperative Sepsis33164 Overall47967

10. N=249 N=249 90% of events occurred during the hospitalization 90% of events occurred during the hospitalization 10% were false positives 10% were false positives – 8% identified by abstractor as miscoded – 2% present on admission Accidental Puncture or Laceration

11. Iatrogenic Pneumothorax N=205 N=205 89% of events occurred during the hospitalization 89% of events occurred during the hospitalization 11% were false positives 11% were false positives – 7% present or suspected at admission – 4% no documentation of event (miscoded)

12. Postoperative DVT or PE N=155 N=155 68% occurred during the index hospitalization 68% occurred during the index hospitalization 32% were false positives 32% were false positives – 16% had no surgical procedure performed in the OR – 16% did not have a new post-op PE or DVT

13. Selected Infection due to Medical Care N=194 N=194 61% occurred during the index hospitalization 61% occurred during the index hospitalization 39% were false positives 39% were false positives – 17% were present on admission – 22% had no documentation of an infection

14. Postoperative Sepsis N=164 N=164 40% of the events occurred during the hospitalization 40% of the events occurred during the hospitalization 60% were false positive 60% were false positive – 17% had no documentation of bacteremia, septicemia, sepsis or SIRS – 17% had infection (=14%) or sepsis (=3%) POA – 25% did not have elective surgery

15. Summary of PPVs Preliminary estimates PSIPPV% Accidental puncture or laceration 90% Iatrogenic pneumothorax 89% Postoperative DVT/PE 68% Selected infections due to medical care 63% Postoperative sepsis 40%

16. Recognizing limitations Data elements available via chart review Data elements available via chart review Time constraints (burden on collaborators) Time constraints (burden on collaborators) Inter-hospital variation Inter-hospital variation Volunteer sample Volunteer sample

17. Further analysis of: Further analysis of: – Potential preventability – Management/treatment & patient outcomes – Inter-hospital variation Evaluation of alternative ICD-9-CM specifications Evaluation of alternative ICD-9-CM specifications – Can we improve PPV through numerator or denominator changes? AHRQ QI Validation Pilot Next steps for analysis

18. Policy implications “Present at admission” data would substantially improve the PPV “Present at admission” data would substantially improve the PPV – Current PSI software has POA option – October 2008 release will require POA Coding changes are needed to enhance specificity and PPV in some areas Coding changes are needed to enhance specificity and PPV in some areas – AHRQ proposed codes for upper extremity and thoracic venous thromboses, and to distinguish acute from sub-acute thromboses – New codes for catheter-associated bloodstream infection Several PSIs have been endorsed by NQF Several PSIs have been endorsed by NQF – Accidental Puncture & Laceration (phase I) – Iatrogenic pneumothorax (phase I) – Foreign body (phase II) – Wound dehiscence (phase II)

19. AHRQ project team AHRQ project team – Mamatha Pancholi &Marybeth Farquhar Battelle training and support team Battelle training and support team – Laura Puzniak & Lynne Jones All of the validation pilot partners! All of the validation pilot partners!Acknowledgments