Presentation is loading. Please wait.

Presentation is loading. Please wait.

Refinement and Validation of the AHRQ Patient Safety Indicators Developed by UC-Stanford Evidence Based Practice Center Funded by the Agency for Healthcare.

Published byLoraine Lewis Modified over 8 years ago

Similar presentations

Presentation on theme: "Refinement and Validation of the AHRQ Patient Safety Indicators Developed by UC-Stanford Evidence Based Practice Center Funded by the Agency for Healthcare."— Presentation transcript:

1 Refinement and Validation of the AHRQ Patient Safety Indicators Developed by UC-Stanford Evidence Based Practice Center Funded by the Agency for Healthcare Research and Quality EPC Team (PSI Development) PI: Kathryn McDonald, M.M., Stanford Patrick Romano, M.D., M.P.H, UC Davis Jeffrey Geppert, J.D., Ed.M., Stanford Sheryl Davies, M.A., Stanford Bradford Duncan, M.D., M.A., Stanford Kaveh G. Shojania, M.D., UCSF Support of Quality Indicators PI: Kathryn McDonald, M.M., Stanford Patrick Romano, M.D., M.P.H, UC Davis Jeffrey Geppert, J.D. Ed.M., Stanford Sheryl Davies, M.A., Stanford Mark Gritz, PhD, Battelle Greg Hubert, Battelle Denise Remus, Ph.D., RN, AHRQ

2 Acknowledgments Funded by AHRQ Contract No. 290-97-0013 Support of Quality Indicators Contract No. 290-02-0007 Data used for analyses: Nationwide Inpatient Sample (NIS), 1995-2000. Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and Quality State Inpatient Databases (SID), 1997 (19 states). Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and Quality For more information: http://www.qualityindicators.ahrq.gov

3 Acknowledgments  We gratefully acknowledge the data organizations in participating states that contributed data to HCUP and that we used in this study: the Arizona Department of Health Services; California Office of Statewide Health and Development; Colorado Health and Hospital Association; CHIME, Inc. (Connecticut); Florida Agency for Health Care Administration; Georgia Hospital Association; Hawaii Health Information Corporation; Illinois Health Care Cost Containment Council; Iowa Hospital Association; Kansas Hospital Association; Maryland Health Services Cost Review Commission; Massachusetts Division of Health Care Finance and Policy; Missouri Hospital Industry Data Institute; New Jersey Department of Health and Senior Services; New York State Department of Health; Oregon Association of Hospitals and Health Systems; Pennsylvania Health Care Cost Containment Council; South Carolina State Budget and Control Board; Tennessee Hospital Association; Utah Department of Health; Washington State Department of Health; and Wisconsin Department of Health and Family Service.

4 Rationale for the PSIs Background: Perceived need for an inexpensive patient safety surveillance system based on readily available data UC-Stanford EPC charge: To review and improve the evidence base related to potential patient safety indicators (PSIs) that can be ascertained from data elements in a standardized, multi- state health data system, the Healthcare Cost and Utilization Project (HCUP).

5 Literature review to find candidate indicators  MEDLINE/EMBASE search guided by medical librarians at Stanford and NCPCRD (UK)  Few examples described in peer reviewed journals  Iezzoni et al.’s Complications Screening Program (CSP)  Miller et al.’s Patient Safety Indicators  Review of ICD-9-CM code book  Codes from above sources were grouped into clinically coherent indicators with appropriate denominators

6 Structure of indicators  All definitions were created using ICD-9-CM diagnosis and procedure codes (along with DRG, MDC, sex, age and procedure dates)  Numerator of each indicator is the number of cases with the complication of interest (e.g., Postop DVT/PE)  Denominator of each indicator is the number of hospitalizations (or patients) considered to be at risk (e.g. elective surgical patients)  Exclusions were defined to restrict the denominator to patients for whom the complication was less likely to have been present at admission, and more likely to have been preventable  The indicator “rate” is the numerator/denominator

7 PSI assessment methods  Literature review to gather data on coding and construct validity  ICD-9-CM coding consultant review (face validity)  Clinical panel review (face validity)  Empirical analyses of nationwide rates, hospital variation, impact of risk adjustment, and relationships among indicators

8 Clinical panel review  Intended to establish consensual validity  Modified RAND/UCLA Appropriateness Method  Physicians of various specialties/subspecialties, nurses, other specialized professionals (e.g., midwife, pharmacist)  Potential indicators were rated by 8 multispecialty panels; surgical indicators were also rated by 3 surgical panels  All panelists rated all assigned indicators (1-9) on:  Overall usefulness  Likelihood of identifying the occurrence of an adverse event or complication (i.e., not present at admission)  Likelihood of being preventable (i.e., not an expected result of underlying conditions)  Likelihood of being due to medical error or negligence (i.e., not just lack of ideal or perfect care)  Likelihood of being clearly charted in the medical record  Extent to which indicator is subject to bias due to case mix

9 Medical error and complications continuum Evaluation framework  Pre-conference ratings and comments/suggestions  Individual ratings returned to panelists with distribution of ratings and other panelists’ comments/suggestions  Telephone conference call moderated by PI and attended by note-taker, focusing on high-variability items and panelists’ suggestions (90-120 mins)  Suggestions adopted only by consensus  Post-conference ratings and comments/ suggestions Medical error Nonpreventable Complications

10 Final selection of indicators  Retained indicators for which “overall usefulness” rating was “Acceptable” or “Acceptable-” :  Median score 7-9  Definite or indeterminate agreement  Excluded indicators rated “Unclear,” “Unclear-,” or “Unacceptable” :  Median score <7, OR  At least 2 panelists rated the indicator in each of the extreme 3-point ranges

11 PSIs reviewed  48 indicators reviewed in total  37 reviewed by multispecialty panel  15 of those reviewed by surgical panel  20 “accepted” based on face validity  2 dropped due to operational concerns  17 “experimental” or promising indicators  11 rejected

12 “Accepted” PSIs Selected postoperative complications   Postoperative thromboembolism   Postoperative respiratory failure   Postoperative sepsis   Postoperative physiologic and metabolic derangements   Postoperative abdominopelvic wound dehiscence   Postoperative hip fracture   Postoperative hemorrhage or hematoma Selected technical adverse events   Decubitus ulcer   Selected infections due to medical care Technical difficulty with procedures   Iatrogenic pneumothorax   Accidental puncture or laceration   Foreign body left in during procedure Other   Complications of anesthesia   Death in low mortality DRGs   Failure to rescue   Transfusion reaction Obstetric trauma and birth trauma   Birth trauma – injury to neonate   Obstetric trauma – vaginal delivery with instrument   Obstetric trauma – vaginal delivery without instrument   Obstetric trauma – cesarean section delivery

13 Romano et al., Health Affairs 2003; 22(2):154-166 National trends in PSI rates  Nationwide Inpatient Sample (NIS), 1995-2000  7.5 million discharges/1,000 hospitals/28 States  Approximates 20% sample of nonfederal acute care hospitals  Discharge level weights applied to generate national estimates for each year  Adjusted for age, gender, age-gender inter- actions, comorbidities, and DRG clusters  1,121,000 potential safety-related events affecting 1,070,000 hospitalizations

14 Estimated cases in 2000 Indicator Frequency ± 95% CI Rate per 100 Postoperative septicemia14,055 ± 10601.091 Postoperative thromboembolism75,811 ± 4,1560.919 Postoperative respiratory failure12,842 ± 9380.359 Postoperative physiologic or metabolic derangement 4,003 ± 4190.089 Decubitus ulcer201,459 ± 10,1042.130 Infection due to medical care54,490 ± 2,6580.193 Postoperative hip fracture5,207 ± 3270.080 Accidental puncture or laceration89,348 ± 5,6690.324 Iatrogenic pneumothorax19,397 ± 1,0250.067 Postoperative hemorrhage/hematoma17,014 ± 9680.206

15 Impact of patient safety events in 2000 (Zhan and Miller, JAMA 2003) Indicator Excess LOS (days) Excess charge ($) Postoperative septicemia 10.9$57,700 Postoperative thromboembolism 5.421,700 Postoperative respiratory failure 9.153,500 Postoperative physiologic or metabolic derangement 8.954,800 Decubitus ulcer 4.010,800 Selected infections due to medical care 9.638,700 Postoperative hip fracture 5.213,400 Accidental puncture or laceration 1.38,300 Iatrogenic pneumothorax 4.417,300 Postoperative hemorrhage/hematoma 3.921,400

16 Estimated cases in 2000 Indicator Frequency ± 95% CI Rate per 100 Birth trauma27,035 ± 5,6740.667 Obstetric trauma –cesarean5,523 ± 5970.593 Obstetric trauma - vaginal without instrumentation 249,243 ± 12,5708.659 Obstetric trauma - vaginal w instrumentation60,622 ± 3,10424.408 Failure to rescue267,541 ± 5,05617.424 Postoperative abdominopelvic wound dehiscence 3,858 ± 2890.193 Transfusion reaction138 ± 490.0004 Complications of anesthesia5,305 ± 4550.056 Death in low mortality DRGs5,912 ± 4330.043 Foreign body left during procedure2,710 ± 2040.008

17 Impact of patient safety events in 2000 (Zhan and Miller, JAMA 2003) Indicator Excess LOS (days) Excess charge ($) Birth trauma -0.1 (NS) 300 (NS) Obstetric trauma –cesarean 0.42,700 Obstetric trauma - vaginal without instrumentation 0.05 -100 (NS) Obstetric trauma - vaginal w instrumentation 0.07220 Postoperative abdominopelvic wound dehiscence 9.440,300 Transfusion reaction 3.4 (NS) 18,900 (NS) Complications of anesthesia 0.2 (NS) 1,600 Foreign body left during procedure 2.113,300

18 National trends 1995-2000 Romano, PS, Geppert, JJ, Davies, SM, Miller, M et al. A National Profile of Patient Safety in US Hospitals Based on Administrative Data, Health Affairs 2003;22(2):154-166.

19 National trends 1995-2000 Romano, PS, Geppert, JJ, Davies, SM, Miller, M et al. A National Profile of Patient Safety in US Hospitals Based on Administrative Data, Health Affairs 2003;22(2):154-166.

20 National trends 1995-2000 Romano, PS, Geppert, JJ, Davies, SM, Miller, M et al. A National Profile of Patient Safety in US Hospitals Based on Administrative Data, Health Affairs 2003;22(2):154-166.

21 National trends 1995-2000 Romano, PS, Geppert, JJ, Davies, SM, Miller, M et al. A National Profile of Patient Safety in US Hospitals Based on Administrative Data, Health Affairs 2003;22(2):154-166.

22 National trends 1995-2000 Romano, PS, Geppert, JJ, Davies, SM, Miller, M et al. A National Profile of Patient Safety in US Hospitals Based on Administrative Data, Health Affairs 2003;22(2):154-166.

23 Research/Policy Question Why are some PSIs increasing in incidence over time while others are decreasing?  Selective changes in coding practice  Changes in severity of illness or underlying risk of potential safety-related events  True changes in quality due to technical improvements in surgical or nursing technique, counterbalanced by inadequate staffing to prevent some complications

24 PSI Technical Review at http://www.qualityindicators.ahrq.gov/data/hcup/psi.htm Standard deviation of hospital effects: 1997 SID

25 PSI Technical Review at http://www.qualityindicators.ahrq.gov/data/hcup/psi.htm Ratio of hospital-level signal to total hospital variation: 1997 SID

26 PSI Technical Review at http://www.qualityindicators.ahrq.gov/data/hcup/psi.htm Year-to-year correlation of hospital effects: 1996-97 Florida SID

27 Risk adjustmentmethods  Must use only administrative data  APR-DRGs and other canned packages may adjust for complications  Final model  DRGs (complication DRGs aggregated)  Modified Comorbidity Index based on list developed by Elixhauser et al.  Age, Sex, Age-Sex interactions

28 Hospital level variation: Impact of bias, 1997 SID (summary) High BiasMedium BiasLow Bias Failure to rescue (44% change 2 deciles) Postop respiratory failure (11%) Postop abdominopelvic wound dehiscence (4%) Accidental puncture or laceration (24%) Postop hip fracture (8%)Obstetric trauma – cesarean birth (2%) Decubitus ulcer (26%)Iatrogenic pneumothorax (14%) Postop hemorrhage or hematoma (4%) Postop thromboembolism (14%) Postop physio/metabolic derangement (5%) Complications of anesthesia (<1%) Death in low mortality DRGs (13%) Obstetric trauma – vaginal birth with instrumentation (5%) Obstetric trauma – vaginal birth without Instrumentation (<1%) Postop sepsis (11%)Selected infections due to medical care (10%) Birth trauma (0%)

29 PSI Technical Review at http://www.qualityindicators.ahrq.gov/data/hcup/psi.htm PSIs loading on “catheter-related and technical complications” (factor 1)

30 PSI Technical Review at http://www.qualityindicators.ahrq.gov/data/hcup/psi.htm PSIs loading on “post/intraoperative complications” (factor 2)

31 PSI Technical Review at http://www.qualityindicators.ahrq.gov/data/hcup/psi.htm PSIs loading on neither factor (<1% variance explained)

32 Conclusions  Administrative data are appealing, but the development of indicators is time-consuming  Variations across hospitals and over time merit further exploration  Potentially useful screening tool for providers, provider associations, and health data agencies to identify possible safety problems  Ongoing support and validation work expected to offer many more insights into opportunities and obstacles in using administrative data for patient safety surveillance

Download ppt "Refinement and Validation of the AHRQ Patient Safety Indicators Developed by UC-Stanford Evidence Based Practice Center Funded by the Agency for Healthcare."

Similar presentations

Adverse Patient Safety Events: Costs of Readmissions and Patient Outcomes Following Discharge Didem M. Bernard, Ph.D. William E. Encinosa, Ph.D.

Adverse Patient Safety Events: Costs of Readmissions and Patient Outcomes Following Discharge Didem M. Bernard, Ph.D. William E. Encinosa, Ph.D.
VALIDATING AHRQ PATIENT SAFETY INDICATORS with the VA NATIONAL SURGICAL QUALITY IMPROVEMENT PROGRAM POSTOPERATIVE ADVERSE EVENTS Dennis Tsilimingras, M.D.,

VALIDATING AHRQ PATIENT SAFETY INDICATORS with the VA NATIONAL SURGICAL QUALITY IMPROVEMENT PROGRAM POSTOPERATIVE ADVERSE EVENTS Dennis Tsilimingras, M.D.,
Variations in Quality Outcomes Among Hospitals in Different Types of Health Systems, 1995 - 2000 Askar Chukmaitov, M.D., M.P.A. Askar Chukmaitov, M.D.,

Variations in Quality Outcomes Among Hospitals in Different Types of Health Systems, Askar Chukmaitov, M.D., M.P.A. Askar Chukmaitov, M.D.,
Preventable Hospitalizations: Assessing Access and the Performance of Local Safety Net Presented by Yu Fang (Frances) Lee Feb. 9 th, 2007.

Preventable Hospitalizations: Assessing Access and the Performance of Local Safety Net Presented by Yu Fang (Frances) Lee Feb. 9 th, 2007.
Differences in adverse events detected using different methods of identification? James M Naessens; Claudia R Campbell; Bjorn Berg; John J Lefante; Arthur.

Differences in adverse events detected using different methods of identification? James M Naessens; Claudia R Campbell; Bjorn Berg; John J Lefante; Arthur.
Session 3C: Overview of the AHRQ Quality Indicators Thursday, September 27, 2007 3:30 pm to 5:00 pm ET.

Session 3C: Overview of the AHRQ Quality Indicators Thursday, September 27, :30 pm to 5:00 pm ET.
Increasing Health Care Costs: the Price of Innovation? AcademyHealth Annual Research Meeting June 7, 2004 Claudia A. Steiner, MD, MPH Bernard Friedman,

Increasing Health Care Costs: the Price of Innovation? AcademyHealth Annual Research Meeting June 7, 2004 Claudia A. Steiner, MD, MPH Bernard Friedman,
Understanding the Influences on the Association between Nurse Staffing and Preventable Patient Complications Deborah Dang, PhD, RN 2007 Interdisciplinary.

Understanding the Influences on the Association between Nurse Staffing and Preventable Patient Complications Deborah Dang, PhD, RN 2007 Interdisciplinary.
Lessons from the AHRQ PSI Validation Pilot Project AHRQ QI Users Meeting September 10, 2008 Presenter: Patrick S. Romano, MD MPH; Professor, UC Davis Center.

Lessons from the AHRQ PSI Validation Pilot Project AHRQ QI Users Meeting September 10, 2008 Presenter: Patrick S. Romano, MD MPH; Professor, UC Davis Center.
Overview of the AHRQ QI Toolkit for Hospitals Courtney Gidengil, MD MPH Peter Hussey, PhD RAND Corporation.

Overview of the AHRQ QI Toolkit for Hospitals Courtney Gidengil, MD MPH Peter Hussey, PhD RAND Corporation.
Utilizing severity to interpret changing trends of hospitalized injury rates in the United States, 1988-2007 Claudia A. Steiner, MD, MPH 1 Li-Hui Chen,

Utilizing severity to interpret changing trends of hospitalized injury rates in the United States, Claudia A. Steiner, MD, MPH 1 Li-Hui Chen,
AHRQ Quality Indicators Toolkit Tool A.2 Instructions.

AHRQ Quality Indicators Toolkit Tool A.2 Instructions.
What Types of Websites and Reports Can MONAHRQ Generate? March 2015 Note: This is one of eight slide sets outlining MONAHRQ and its value, available at.

What Types of Websites and Reports Can MONAHRQ Generate? March 2015 Note: This is one of eight slide sets outlining MONAHRQ and its value, available at.
HCUPnet for State Policymakers - Utah’s Use Case Keely Cofrin Allen, Ph.D. Office of Health Care Statistics Utah Department of Health AHRQ State Quality.

HCUPnet for State Policymakers - Utah’s Use Case Keely Cofrin Allen, Ph.D. Office of Health Care Statistics Utah Department of Health AHRQ State Quality.
Day 1: Session III Implications of ICD-9-CM Coding Rules for Measuring QIs Presenters: Patrick Romano, UC Davis Rita Scichilone, American Health Information.

Day 1: Session III Implications of ICD-9-CM Coding Rules for Measuring QIs Presenters: Patrick Romano, UC Davis Rita Scichilone, American Health Information.
Impact of the MS-DRGs on the AHRQ QIs

Impact of the MS-DRGs on the AHRQ QIs
Overview of the Pediatric Indicator Module Presenters: Kathryn McDonald and Sheryl Davies, Stanford University AHRQ QI User Meeting September 26-27, 2005.

Overview of the Pediatric Indicator Module Presenters: Kathryn McDonald and Sheryl Davies, Stanford University AHRQ QI User Meeting September 26-27, 2005.
Gastroschisis and Mode of Delivery: National Trends and Evaluation of Outcomes, 1991 to 2005 T. Mac Bird University of Arkansas for Medical Sciences College.

Gastroschisis and Mode of Delivery: National Trends and Evaluation of Outcomes, 1991 to 2005 T. Mac Bird University of Arkansas for Medical Sciences College.
Agency For Healthcare Quality and Research Quality Indicators NH Health Care QA Commission AHRQ Subcommittee Report July 31, 2009.

Agency For Healthcare Quality and Research Quality Indicators NH Health Care QA Commission AHRQ Subcommittee Report July 31, 2009.
Assessing Patient Safety through Administrative Data: Adapting and Improving Existing Systems Patrick S. Romano, MD MPH Professor of Medicine and Pediatrics.

Assessing Patient Safety through Administrative Data: Adapting and Improving Existing Systems Patrick S. Romano, MD MPH Professor of Medicine and Pediatrics.

Similar presentations

Ads by Google