1. VALIDATING AHRQ PATIENT SAFETY INDICATORS with the VA NATIONAL SURGICAL QUALITY IMPROVEMENT PROGRAM POSTOPERATIVE ADVERSE EVENTS Dennis Tsilimingras, M.D., M.P.H. Center for Health Quality, Outcomes, & Economic Research (Bedford VAMC) Boston University School of Public Health

2. Co-Investigators Patrick Romano, M.D., M.P.H. 3, William Henderson, Ph.D. 4, Peter Rivard, M.H.S.A. 1,2, Susan Loveland, M.A.T. 1,2, Shibei Zhao, M.P.H. 1,2, Cindy Christiansen, Ph.D. 1,2, Anne Elixhauser, Ph.D. 5, Shukri Khuri, M.D. 6, Amy K. Rosen, Ph.D. 1,2 1 Center for Health Quality, Outcomes, & Economic Research (Bedford VAMC) 2 Boston University School of Public Health 3 University of California at Davis 4 NSQIP Denver Data Analysis Center 5 Agency for Healthcare Research and Quality 6 Surgical Service, VA Boston Healthcare System Funding Organization: Department of Veterans Affairs HSR&D Project IIR 02-144-1

3. Introduction Patient Safety Indicators (PSIs): - developed by the UC-Stanford and AHRQ collaboration - purpose to screen for potential safety problems using ICD-9-CM codes - limitation: PSIs have not been validated in the past

4. Research Objectives (1)To validate the AHRQ PSIs with postoperative adverse events in the VA National Surgical Quality Improvement Program (NSQIP). (2)To test alternative PSI definitions based on clinical information from NSQIP for the purpose of future refinement of the PSIs.

5. Methods AHRQ NSQIP Accepted PSIs: Postoperative Adverse Events: (1) Postoperative Sepsis (1) Systemic Sepsis (2) Postoperative Respiratory (2) Unplanned Intubation for Failure Respiratory Failure or Failure to Wean from Ventilator > 48hrs (3) Postoperative Pulmonary (3) Deep Vein Thrombosis Embolism or Deep Vein & Pulmonary Embolism Thrombosis (4) Postoperative Wound (4) Dehiscence Dehiscence (5) Postoperative Physiologic (5) Acute Renal Failure and Metabolic Derangements

6. Methods Study Sample VA administrative inpatient data from the Patient Treatment File (PTF): - from 130 VA hospitals - matched with NSQIP chart-abstracted records - FY 2001 (10/00 – 09/01) from both data sources

7. Methods NSQIP: - purpose is to provide reliable, valid, and comparative information regarding morbidity & mortality to all VA medical centers - employs trained nurse data collectors - collects preoperative patient characteristics and 30-day postoperative outcomes

8. Methods A hospitalization had a PSI if it met AHRQs administrative data definition that includes ICD-9-CM diagnostic and procedure codes. A hospitalization had a NSQIP event if it met NSQIPs clinical definition which was translated into ICD-9-CM diagnostic and procedure codes. Our alternative definition was based on a translation of NSQIP clinical definitions into ICD-9-CM diagnostic & procedure codes.

9. Methods Using NSQIP as the gold standard: - measured the sensitivity & positive predictive value of all 5 AHRQ PSIs - using the current AHRQ PSI definition - compared with the best alternative PSI definition These definitions were developed using PTF data and validated with NSQIP medical chart-abstracted data.

10. Postoperative Respiratory Failure ICD-9-CM Definitions Diagnostic codes Procedure Codes AHRQ 518.81 & 518.84 NSQIP 518.5 96.04 & 96.70-96.72 Alternative 518.5, 518.81, & 96.04 & 96.70-96.72 518.84

11. Results Sample 561,436 inpatient hospitalizations (PTF) 101,548 with surgical Diagnostic Related Groups 55,752 were matched with NSQIP data, which were used to calculate sensitivities and positive predictive values

12. Results Accepted PSIs: (1) Postoperative Sepsis (12,011 hospitalizations) (2) Postoperative Respiratory failure* (24,273 hospitalizations) *PSI includes diagnostic and procedure codes. Sensitivity(%) PPV(%) Current AHRQ PSI Definition 19 74 Alternative PSI Definition (NSQIP) 67 66 Sensitivity(%) PPV(%) Current AHRQ PSI Definition 32 44 Alternative PSI Definition (NSQIP) 37 45

13. Results (3) Postoperative Pulmonary Embolism or Deep Vein Thrombosis* (55,682 hospitalizations) (4) Postoperative Wound Dehiscence* (16,904 hospitalizations) *PSI includes diagnostic and procedure codes. Sensitivity(%) PPV(%) Current AHRQ PSI Definition 56 22 Alternative PSI Definition (NSQIP) 58 22 Sensitivity(%) PPV(%) Current AHRQ PSI Definition 29 72 Alternative PSI Definition (NSQIP) 61 57

14. Results (5) Postoperative Physiologic and Metabolic Derangements* (27,722 hospitalizations) *PSI includes diagnostic and procedure codes. Sensitivity(%) PPV(%) Current AHRQ PSI Definition 44 54 Alternative PSI Definition (NSQIP) 48 63

15. Results Comparing our results with Best and colleagues. Best et al. Identifying Patient Preoperative Risk Factors & Postoperative Adverse Events in Administrative Databases: Results from NSQIP. J Am Coll Surg. 2002;194:257-266. AHRQ PSI Our Results Best et al 2002 Results Sens(%)PPV(%) Sens(%)PPV(%) Sepsis 37 45 40 29 Respiratory Failure 67 66 9 52 PE or DVT 58 22 PE 49 48 DVT 12 18 Dehiscence 61 57 25 23 Physiologic & Metabolic Derangements 48 63 16 4

16. NSQIP clinical data are an excellent source for examining the validity of administrative-data-based indicators, such as the PSIs, as identifiers of potential inpatient surgical patient safety events. All 5 PSIs were improved by a simple modification to perform better in the VA. Conclusions

17. Significance PSIs may be a useful screening tool for identifying inpatient safety events that warrant further investigation. Such a tool can be constructed with the incorporation of clinical data definitions to administrative data definitions. The latest release of the AHRQ PSI software has incorporated certain findings from this study.