Presentation is loading. Please wait.

Presentation is loading. Please wait.

AHRQ IQI Clinical Validation Panels Sheryl Davies, MA Stanford University.

Published byMalcolm Usrey Modified over 10 years ago

Similar presentations

Presentation on theme: "AHRQ IQI Clinical Validation Panels Sheryl Davies, MA Stanford University."— Presentation transcript:

1 AHRQ IQI Clinical Validation Panels Sheryl Davies, MA Stanford University

2 Outline Purpose Purpose Composition and Recruitment Composition and Recruitment Rating Process Rating Process Overall themes Overall themes Indicator results Indicator results Take home lessons Take home lessons

3 Purpose of the Clinical Panels IQIs and PQIs IQIs and PQIs – Developed 1999-2001 – Based on established indicators – Did not undergo panel review Panel review establishes face validity of the indicators Panel review establishes face validity of the indicators Standardize available evidence for all AHRQ QIs Standardize available evidence for all AHRQ QIs – Establish face validity for one stakeholder group – Update evidence Panel review for IQIs considered for NQF endorsement Panel review for IQIs considered for NQF endorsement

4 Methods Modified RAND/UCLA Appropriateness Method (Nominal Group Technique) Modified RAND/UCLA Appropriateness Method (Nominal Group Technique) Physicians of various specialties/subspecialties and other health professionals were recruited with the assistance of relevant organizations Physicians of various specialties/subspecialties and other health professionals were recruited with the assistance of relevant organizations 22 contacted organizations nominated 22 contacted organizations nominated – 103 clinicians nominated 60 accepted 45 eligible Panelists selected in order to form diverse panels Panelists selected in order to form diverse panels

5 Panelists Female: 18% Female: 18% Academic: 71% Academic: 71% Geographic Geographic – East: 44% – West: 29% – Other: 27% Practice setting Practice setting – Urban: 67% – Suburban: 42% – Rural: 18% Funding of primary hospital: Funding of primary hospital: – Private: 44% – Public: 27% Underserved patient population: 56% Underserved patient population: 56%

6 Panel methods: Ratings Initial ratings Initial ratings – Packet of information summarizing evidence – Approx. 10 questions Tailored to the indicator type Tailored to the indicator type 9 point scale 9 point scale Overall usefulness for quality improvement, comparative reporting Overall usefulness for quality improvement, comparative reporting – Compiled ratings provided to panelists Conference call Conference call – Discuss differences – Consensus on definition changes Final ratings Final ratings – Empirical analyses provided – Using same questionnaire as initial ratings

7 Results: Overarching themes Case mix variability Case mix variability Reliability Reliability Volume measures as indirect measures of quality Volume measures as indirect measures of quality – Composite measures

8 Cardiac Panel Reviewed 5 indicators: Reviewed 5 indicators: – AAA Volume/Mortality – Pediatric Heart Surgery Volume/Mortality – Bilateral Catheterization 11 clinicians: vascular surgeons, pediatric cardiologists, pediatric cardiovascular surgeons, interventional cardiologists, pediatric ICU nurse, surgical nurse 11 clinicians: vascular surgeons, pediatric cardiologists, pediatric cardiovascular surgeons, interventional cardiologists, pediatric ICU nurse, surgical nurse

9 Cardiac Panel IndicatorOverall – QIOverall - Comparative AAA MortalityAcceptable (7)Unclear (6) AAA VolumeAcceptable (7) Bilateral catheterization Unclear (5)Unclear with disagreement (5) Pediatric heart surgery volume Acceptable (8) Pediatric heart surgery mortality Acceptable (8)

10 Cardiac Panel IndicatorOverall – QIOverall - Comparative AAA MortalityAcceptable (7)Unclear (6) AAA VolumeAcceptable (7) Case mix variability Ruptured vs. unruptured; endovascular vs. open Bias: Slight overadjustment for endovascular (12%) and underadjustment for ruptured (12%) Total volume (ruptured and unruptured) best predictor of outcomes Stratify by surgical approach (endovascular vs. open)

11 Cardiac Panel IndicatorOverall – QIOverall - Comparative AAA MortalityAcceptable (7)Unclear (6) AAA VolumeAcceptable (7) Case mix variability Ruptured vs. unruptured; endovascular vs. open Bias: Slight overadjustment for endovascular (12%) and underadjustment for ruptured (12%) Total volume best predictor of outcomes Stratify by surgical approach (endovascular vs. open)

12 Cardiac Panel IndicatorOverall – QIOverall - Comparative Pediatric heart surgery volume Acceptable (8) Pediatric heart surgery mortality Acceptable (8) Case mix variability Supported use of RACHS Correlations of hospital volume for each RACHS complexity are robust (r = 0.74 – 0.95) Best predictor of outcome is total volume, rather than by complexity

13 Cardiac Panel IndicatorOverall – QIOverall - Comparative Bilateral catheterization Unclear (5)Unclear w/ disagreement (5) Modification: Expand list of appropriate indications for bilateral catheterization Primarily a resource indicator Charting of indications may be poor May result in decrease of appropriate uses

14 Surgical Resection Panel Reviewed 4 indicators: Reviewed 4 indicators: – Esophageal Resection Volume/Mortality – Pancreatic Resection Volume/Mortality 13 clinicians: thoracic surgeons, general surgeons (including GI and oncology), oncologists, internist, gastroenterologists, surgical nurse 13 clinicians: thoracic surgeons, general surgeons (including GI and oncology), oncologists, internist, gastroenterologists, surgical nurse

15 Surgical Resection Panel IndicatorOverall – QIOverall - Comparative Esophageal resection mortalityAcceptable (7) Esophageal resection volumeAcceptable (7) Pancreatic resection mortalityAcceptable (7) Pancreatic resection volumeAcceptable (7)

16 Surgical Resection Panel Esophageal resection Case mix variability Risk adjustment performs well. Underestimates risk for patient with middle esophageal and unspecified site cancer (22%, 37%). Pancreatic resection Case mix variability Over-estimates risk for total pancreatectomy (68%), lesser extent underestimates risk for Whipple (15%). Issue raised with 3M. Low rates

17 Geriatric Panel Reviewed 4 indicators: Reviewed 4 indicators: – Acute Stroke Mortality – Hip Fracture Mortality – Hip Replacement Mortality – Incidental Appendectomy 14 clinicians: internists (including geriatrics and hospital medicine), neurologists, general surgeon, interventional radiologist, orthopedic surgeons, neurosurgeon, diagnostic radiologist, nurse, physical therapist 14 clinicians: internists (including geriatrics and hospital medicine), neurologists, general surgeon, interventional radiologist, orthopedic surgeons, neurosurgeon, diagnostic radiologist, nurse, physical therapist

18 Geriatric Panel IndicatorOverall – QIOverall - Comparative Acute stroke mortality Unclear (6.5)Unclear with disagreement (5) Incidental appendectomy Acceptable (7)Unclear (6) Hip fracture mortalityAcceptable (7) Hip replacement mortality Unclear due to disagreement (7) Unclear (6)

19 Geriatric Panel IndicatorOverall – QIOverall - Comparative Acute stroke mortality Unclear (6.5)Unclear with disagreement (5) Incidental appendectomy Acceptable (7)Unclear (6) Hip fracture mortalityAcceptable (7) Hip replacement mortality Unclear due to disagreement (7) Unclear (6) Case mix variability: Stroke type (hemorrhagic, ischemic, subarachnoid) Risk adjustment accounts for almost all difference in risk Patient factors such as delay in presenting for care

20 Geriatric Panel IndicatorOverall – QIOverall - Comparative Acute stroke mortality Unclear (6.5)Unclear with disagreement (5) Incidental appendectomy Acceptable (7)Unclear (6) Hip fracture mortalityAcceptable (7) Hip replacement mortality Unclear due to disagreement (7) Unclear (6) Exclude patients with hip fracture Case mix variability Risk adjustment somewhat overestimates risk for revision Rates very low, reliability concerns

21 Geriatric Panel IndicatorOverall – QIOverall - Comparative Acute stroke mortality Unclear (6.5)Unclear with disagreement (5) Incidental appendectomy Acceptable (7)Unclear (6) Hip fracture mortalityAcceptable (7) Hip replacement mortality Unclear due to disagreement (7) Unclear (6) Is it still being done? If it is still being done, it shouldnt be done. Then it is a good indicator I am having a hard time getting excited about this indicator

22 Geriatric Panel IndicatorOverall – QIOverall - Comparative Acute stroke mortality Unclear (6.5)Unclear with disagreement (5) Incidental appendectomy Acceptable (7)Unclear (6) Hip fracture mortalityAcceptable (7) Hip replacement mortality Unclear due to disagreement (7) Unclear (6) Limit to the elderly Case mix variability Risk adjustment accounts for both repair type and fracture location

23 Conclusion Overall good reception of indicators Overall good reception of indicators Recommendations considered in context of other validation efforts Recommendations considered in context of other validation efforts Indicator revisions implemented in February 2008 Indicator revisions implemented in February 2008 Further efforts to improve indicators or develop additional evidence Further efforts to improve indicators or develop additional evidence

24 Questions?

Download ppt "AHRQ IQI Clinical Validation Panels Sheryl Davies, MA Stanford University."

Similar presentations

Chapter 3: Clinical Decision-Making for Massage

Chapter 3: Clinical Decision-Making for Massage
Evidence-based Dental Practice Developing guidelines or clinical recommendations Slide #1 This lecture follows the previous online lecture on evidence.

Evidence-based Dental Practice Developing guidelines or clinical recommendations Slide #1 This lecture follows the previous online lecture on evidence.
Measure Criteria MD Measurement, P4P Phyllis Torda NCQA February 2008.

Measure Criteria MD Measurement, P4P Phyllis Torda NCQA February 2008.
WORKFORCE PLANNING June 2011 Amr Fouad Training & Research Sector Ministry of Health & Population.

WORKFORCE PLANNING June 2011 Amr Fouad Training & Research Sector Ministry of Health & Population.
The Application of Propensity Score Analysis to Non-randomized Medical Device Clinical Studies: A Regulatory Perspective Lilly Yue, Ph.D.* CDRH, FDA,

The Application of Propensity Score Analysis to Non-randomized Medical Device Clinical Studies: A Regulatory Perspective Lilly Yue, Ph.D.* CDRH, FDA,
Solving the Faculty Shortage in Allied Health 9 th Congress of Health Professions Educators 4 June 2002 Ronald H. Winters, Ph.D. Dean College of Health.

Solving the Faculty Shortage in Allied Health 9 th Congress of Health Professions Educators 4 June 2002 Ronald H. Winters, Ph.D. Dean College of Health.
Adverse Patient Safety Events: Costs of Readmissions and Patient Outcomes Following Discharge Didem M. Bernard, Ph.D. William E. Encinosa, Ph.D.

Adverse Patient Safety Events: Costs of Readmissions and Patient Outcomes Following Discharge Didem M. Bernard, Ph.D. William E. Encinosa, Ph.D.
VALIDATING AHRQ PATIENT SAFETY INDICATORS with the VA NATIONAL SURGICAL QUALITY IMPROVEMENT PROGRAM POSTOPERATIVE ADVERSE EVENTS Dennis Tsilimingras, M.D.,

VALIDATING AHRQ PATIENT SAFETY INDICATORS with the VA NATIONAL SURGICAL QUALITY IMPROVEMENT PROGRAM POSTOPERATIVE ADVERSE EVENTS Dennis Tsilimingras, M.D.,
Surgery volume and operative mortality: A re-examination using fixed-effects regression Amresh Hanchate, PhD Section of General Internal Medicine Boston.

Surgery volume and operative mortality: A re-examination using fixed-effects regression Amresh Hanchate, PhD Section of General Internal Medicine Boston.
Pay-for-Performance Programs: the U.S. Experience Eric Schneider, M.D., M.Sc. Harvard School of Public Health Brigham and Womens Hospital Boston, MA.

Pay-for-Performance Programs: the U.S. Experience Eric Schneider, M.D., M.Sc. Harvard School of Public Health Brigham and Womens Hospital Boston, MA.
Multinational Comparisons of Health Systems Data, 2008 Support for this research was provided by The Commonwealth Fund. The views presented here are those.

Multinational Comparisons of Health Systems Data, 2008 Support for this research was provided by The Commonwealth Fund. The views presented here are those.
1 Using Data to Drive Health System Performance Commissioned from Ovations by the National Primary and Care Trust Development Programme.

1 Using Data to Drive Health System Performance Commissioned from Ovations by the National Primary and Care Trust Development Programme.
Introduction Discussion Results Mobile Health Clinics are transportable health care units that deliver high-value community based health care to underserved.

Introduction Discussion Results Mobile Health Clinics are transportable health care units that deliver high-value community based health care to underserved.
Primary Care – Changing Future 1 PRIMIS 23 rd April 2002 Metropole Birmingham.

Primary Care – Changing Future 1 PRIMIS 23 rd April 2002 Metropole Birmingham.
2008 Johns Hopkins Bloomberg School of Public Health Setting Up a Smoking Cessation Clinic Sophia Chan PhD, MPH, RN, RSCN Department of Nursing Studies.

2008 Johns Hopkins Bloomberg School of Public Health Setting Up a Smoking Cessation Clinic Sophia Chan PhD, MPH, RN, RSCN Department of Nursing Studies.
2014 National Patient Safety Goals

2014 National Patient Safety Goals
Promoting Regulatory Excellence Self Assessment & Physiotherapy: the Ontario Model Jan Robinson, Registrar & CEO, College of Physiotherapists of Ontario.

Promoting Regulatory Excellence Self Assessment & Physiotherapy: the Ontario Model Jan Robinson, Registrar & CEO, College of Physiotherapists of Ontario.
Lecture 3 Validity of screening and diagnostic tests

Lecture 3 Validity of screening and diagnostic tests
THE COMMONWEALTH FUND Figure 1. More Than Two-Thirds of Opinion Leaders Say Current Payment System Is Not Effective at Encouraging High Quality of Care.

THE COMMONWEALTH FUND Figure 1. More Than Two-Thirds of Opinion Leaders Say Current Payment System Is Not Effective at Encouraging High Quality of Care.
7/16/08 1 New Mexico’s Indicator-based Information System for Public Health Data (NM-IBIS) Community Health Assessment Training July 16, 2008.

7/16/08 1 New Mexico’s Indicator-based Information System for Public Health Data (NM-IBIS) Community Health Assessment Training July 16, 2008.

Similar presentations

Ads by Google