Refinement and Validation of the AHRQ Patient Safety Indicators Developed by UC-Stanford Evidence Based Practice Center Funded by the Agency for Healthcare Research and Quality EPC Team (PSI Development) PI: Kathryn McDonald, M.M., Stanford Patrick Romano, M.D., M.P.H, UC Davis Jeffrey Geppert, J.D., Ed.M., Stanford Sheryl Davies, M.A., Stanford Bradford Duncan, M.D., M.A., Stanford Kaveh G. Shojania, M.D., UCSF Support of Quality Indicators PI: Kathryn McDonald, M.M., Stanford Patrick Romano, M.D., M.P.H, UC Davis Jeffrey Geppert, J.D. Ed.M., Stanford Sheryl Davies, M.A., Stanford Mark Gritz, PhD, Battelle Greg Hubert, Battelle Denise Remus, Ph.D., RN, AHRQ

Acknowledgments Funded by AHRQ Contract No Support of Quality Indicators Contract No Data used for analyses: Nationwide Inpatient Sample (NIS), Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and Quality State Inpatient Databases (SID), 1997 (19 states). Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and Quality For more information:

Acknowledgments  We gratefully acknowledge the data organizations in participating states that contributed data to HCUP and that we used in this study: the Arizona Department of Health Services; California Office of Statewide Health and Development; Colorado Health and Hospital Association; CHIME, Inc. (Connecticut); Florida Agency for Health Care Administration; Georgia Hospital Association; Hawaii Health Information Corporation; Illinois Health Care Cost Containment Council; Iowa Hospital Association; Kansas Hospital Association; Maryland Health Services Cost Review Commission; Massachusetts Division of Health Care Finance and Policy; Missouri Hospital Industry Data Institute; New Jersey Department of Health and Senior Services; New York State Department of Health; Oregon Association of Hospitals and Health Systems; Pennsylvania Health Care Cost Containment Council; South Carolina State Budget and Control Board; Tennessee Hospital Association; Utah Department of Health; Washington State Department of Health; and Wisconsin Department of Health and Family Service.

Rationale for the PSIs Background: Perceived need for an inexpensive patient safety surveillance system based on readily available data UC-Stanford EPC charge: To review and improve the evidence base related to potential patient safety indicators (PSIs) that can be ascertained from data elements in a standardized, multi- state health data system, the Healthcare Cost and Utilization Project (HCUP).

Literature review to find candidate indicators  MEDLINE/EMBASE search guided by medical librarians at Stanford and NCPCRD (UK)  Few examples described in peer reviewed journals  Iezzoni et al.’s Complications Screening Program (CSP)  Miller et al.’s Patient Safety Indicators  Review of ICD-9-CM code book  Codes from above sources were grouped into clinically coherent indicators with appropriate denominators

Structure of indicators  All definitions were created using ICD-9-CM diagnosis and procedure codes (along with DRG, MDC, sex, age and procedure dates)  Numerator of each indicator is the number of cases with the complication of interest (e.g., Postop DVT/PE)  Denominator of each indicator is the number of hospitalizations (or patients) considered to be at risk (e.g. elective surgical patients)  Exclusions were defined to restrict the denominator to patients for whom the complication was less likely to have been present at admission, and more likely to have been preventable  The indicator “rate” is the numerator/denominator

PSI assessment methods  Literature review to gather data on coding and construct validity  ICD-9-CM coding consultant review (face validity)  Clinical panel review (face validity)  Empirical analyses of nationwide rates, hospital variation, impact of risk adjustment, and relationships among indicators

Clinical panel review  Intended to establish consensual validity  Modified RAND/UCLA Appropriateness Method  Physicians of various specialties/subspecialties, nurses, other specialized professionals (e.g., midwife, pharmacist)  Potential indicators were rated by 8 multispecialty panels; surgical indicators were also rated by 3 surgical panels  All panelists rated all assigned indicators (1-9) on:  Overall usefulness  Likelihood of identifying the occurrence of an adverse event or complication (i.e., not present at admission)  Likelihood of being preventable (i.e., not an expected result of underlying conditions)  Likelihood of being due to medical error or negligence (i.e., not just lack of ideal or perfect care)  Likelihood of being clearly charted in the medical record  Extent to which indicator is subject to bias due to case mix

Medical error and complications continuum Evaluation framework  Pre-conference ratings and comments/suggestions  Individual ratings returned to panelists with distribution of ratings and other panelists’ comments/suggestions  Telephone conference call moderated by PI and attended by note-taker, focusing on high-variability items and panelists’ suggestions ( mins)  Suggestions adopted only by consensus  Post-conference ratings and comments/ suggestions Medical error Nonpreventable Complications

Final selection of indicators  Retained indicators for which “overall usefulness” rating was “Acceptable” or “Acceptable-” :  Median score 7-9  Definite or indeterminate agreement  Excluded indicators rated “Unclear,” “Unclear-,” or “Unacceptable” :  Median score <7, OR  At least 2 panelists rated the indicator in each of the extreme 3-point ranges

PSIs reviewed  48 indicators reviewed in total  37 reviewed by multispecialty panel  15 of those reviewed by surgical panel  20 “accepted” based on face validity  2 dropped due to operational concerns  17 “experimental” or promising indicators  11 rejected

“Accepted” PSIs Selected postoperative complications   Postoperative thromboembolism   Postoperative respiratory failure   Postoperative sepsis   Postoperative physiologic and metabolic derangements   Postoperative abdominopelvic wound dehiscence   Postoperative hip fracture   Postoperative hemorrhage or hematoma Selected technical adverse events   Decubitus ulcer   Selected infections due to medical care Technical difficulty with procedures   Iatrogenic pneumothorax   Accidental puncture or laceration   Foreign body left in during procedure Other   Complications of anesthesia   Death in low mortality DRGs   Failure to rescue   Transfusion reaction Obstetric trauma and birth trauma   Birth trauma – injury to neonate   Obstetric trauma – vaginal delivery with instrument   Obstetric trauma – vaginal delivery without instrument   Obstetric trauma – cesarean section delivery

Romano et al., Health Affairs 2003; 22(2): National trends in PSI rates  Nationwide Inpatient Sample (NIS),  7.5 million discharges/1,000 hospitals/28 States  Approximates 20% sample of nonfederal acute care hospitals  Discharge level weights applied to generate national estimates for each year  Adjusted for age, gender, age-gender inter- actions, comorbidities, and DRG clusters  1,121,000 potential safety-related events affecting 1,070,000 hospitalizations

Estimated cases in 2000 Indicator Frequency ± 95% CI Rate per 100 Postoperative septicemia14,055 ± Postoperative thromboembolism75,811 ± 4, Postoperative respiratory failure12,842 ± Postoperative physiologic or metabolic derangement 4,003 ± Decubitus ulcer201,459 ± 10, Infection due to medical care54,490 ± 2, Postoperative hip fracture5,207 ± Accidental puncture or laceration89,348 ± 5, Iatrogenic pneumothorax19,397 ± 1, Postoperative hemorrhage/hematoma17,014 ±

Impact of patient safety events in 2000 (Zhan and Miller, JAMA 2003) Indicator Excess LOS (days) Excess charge ($) Postoperative septicemia 10.9$57,700 Postoperative thromboembolism 5.421,700 Postoperative respiratory failure 9.153,500 Postoperative physiologic or metabolic derangement 8.954,800 Decubitus ulcer 4.010,800 Selected infections due to medical care 9.638,700 Postoperative hip fracture 5.213,400 Accidental puncture or laceration 1.38,300 Iatrogenic pneumothorax 4.417,300 Postoperative hemorrhage/hematoma 3.921,400

Estimated cases in 2000 Indicator Frequency ± 95% CI Rate per 100 Birth trauma27,035 ± 5, Obstetric trauma –cesarean5,523 ± Obstetric trauma - vaginal without instrumentation 249,243 ± 12, Obstetric trauma - vaginal w instrumentation60,622 ± 3, Failure to rescue267,541 ± 5, Postoperative abdominopelvic wound dehiscence 3,858 ± Transfusion reaction138 ± Complications of anesthesia5,305 ± Death in low mortality DRGs5,912 ± Foreign body left during procedure2,710 ±

Impact of patient safety events in 2000 (Zhan and Miller, JAMA 2003) Indicator Excess LOS (days) Excess charge ($) Birth trauma -0.1 (NS) 300 (NS) Obstetric trauma –cesarean 0.42,700 Obstetric trauma - vaginal without instrumentation (NS) Obstetric trauma - vaginal w instrumentation Postoperative abdominopelvic wound dehiscence 9.440,300 Transfusion reaction 3.4 (NS) 18,900 (NS) Complications of anesthesia 0.2 (NS) 1,600 Foreign body left during procedure 2.113,300

National trends Romano, PS, Geppert, JJ, Davies, SM, Miller, M et al. A National Profile of Patient Safety in US Hospitals Based on Administrative Data, Health Affairs 2003;22(2):

National trends Romano, PS, Geppert, JJ, Davies, SM, Miller, M et al. A National Profile of Patient Safety in US Hospitals Based on Administrative Data, Health Affairs 2003;22(2):

National trends Romano, PS, Geppert, JJ, Davies, SM, Miller, M et al. A National Profile of Patient Safety in US Hospitals Based on Administrative Data, Health Affairs 2003;22(2):

National trends Romano, PS, Geppert, JJ, Davies, SM, Miller, M et al. A National Profile of Patient Safety in US Hospitals Based on Administrative Data, Health Affairs 2003;22(2):

National trends Romano, PS, Geppert, JJ, Davies, SM, Miller, M et al. A National Profile of Patient Safety in US Hospitals Based on Administrative Data, Health Affairs 2003;22(2):

Research/Policy Question Why are some PSIs increasing in incidence over time while others are decreasing?  Selective changes in coding practice  Changes in severity of illness or underlying risk of potential safety-related events  True changes in quality due to technical improvements in surgical or nursing technique, counterbalanced by inadequate staffing to prevent some complications

PSI Technical Review at Standard deviation of hospital effects: 1997 SID

PSI Technical Review at Ratio of hospital-level signal to total hospital variation: 1997 SID

PSI Technical Review at Year-to-year correlation of hospital effects: Florida SID

Risk adjustmentmethods  Must use only administrative data  APR-DRGs and other canned packages may adjust for complications  Final model  DRGs (complication DRGs aggregated)  Modified Comorbidity Index based on list developed by Elixhauser et al.  Age, Sex, Age-Sex interactions

Hospital level variation: Impact of bias, 1997 SID (summary) High BiasMedium BiasLow Bias Failure to rescue (44% change 2 deciles) Postop respiratory failure (11%) Postop abdominopelvic wound dehiscence (4%) Accidental puncture or laceration (24%) Postop hip fracture (8%)Obstetric trauma – cesarean birth (2%) Decubitus ulcer (26%)Iatrogenic pneumothorax (14%) Postop hemorrhage or hematoma (4%) Postop thromboembolism (14%) Postop physio/metabolic derangement (5%) Complications of anesthesia (<1%) Death in low mortality DRGs (13%) Obstetric trauma – vaginal birth with instrumentation (5%) Obstetric trauma – vaginal birth without Instrumentation (<1%) Postop sepsis (11%)Selected infections due to medical care (10%) Birth trauma (0%)

PSI Technical Review at PSIs loading on “catheter-related and technical complications” (factor 1)

PSI Technical Review at PSIs loading on “post/intraoperative complications” (factor 2)

PSI Technical Review at PSIs loading on neither factor (<1% variance explained)

Conclusions  Administrative data are appealing, but the development of indicators is time-consuming  Variations across hospitals and over time merit further exploration  Potentially useful screening tool for providers, provider associations, and health data agencies to identify possible safety problems  Ongoing support and validation work expected to offer many more insights into opportunities and obstacles in using administrative data for patient safety surveillance