Measuring Progress in Patient Safety Peter Pronovost, MD, PhD, FCCM Johns Hopkins University BNVBBVB

Exercise Please answer each question with a score of 1 to 5 Exercise Please answer each question with a score of 1 to 5. 1 is below average, 3 is average and 5 is above average How smart am I How hard do I work How kind am I How tall am I How good is the quality of care we provide

Improving Sepsis Care (n= 19 ICUs) 36% Reduction (NS) 69% Reduction (p < 0.001)

Improving Sepsis Care (n= 19 ICUs) 36% Reduction (NS) 69% Reduction (p < 0.001)

x Central Mandate Scientifically Sound Feasible Local Wisdom Safety efforts are where the x is. We need to migrate to be more scientifically sound and tap into local wisdom

Context Have we created a culture of safety? Conceptual model for measuring safety Process Outcome Structure How often do we harm? Have we reduced the likelihood of harm? How often do we do what we are supposed to? IT Context Have we created a culture of safety? Adapted from Donebedian

Keystone ICU Safety Dashboard 2004 2006 How often did we harm (BSI) 2.8/1000 How often do we do what we should 66% 95% How often did we learn from mistakes 100s % Needs improvement in Safety climate Teamwork climate 84% 82% 43% 42% Pronovost JAMA 2007

Pronovost BMJ 2008

Comprehensive Unit-based Safety Program (CUSP) Educate staff on science of safety http://www.jhsph.edu/ctlt/training/patient_safety.html Identify defects Assign executive to adopt unit Learn from one defect per quarter and implement teamwork tools The intervention we used to improve culture and learn from mistakes is the comprehensive unit based safety program. Your role in the csicu was as the executive who adopted that unit. You can tell some of the defects you surfaced. Pronovost J, Patient Safety, 2005

What can be measured as a valid rate? Rate requires Numerator- event Denominator- those at risk for event Surveillance for events and those at risk Minimal and Known Error Random error Systematic error

Sources Variation in Safety measures True variation in Safety V data quality/definition/methods of collection V case mix V historical rates Chance

Measuring Preventable Harm Measure rate or counts directly High sensitivity low specificity Estimate observed/expected (O/E) Low sensitivity and specificity Link process and outcome High specificity and moderate sensitivity

Process Measures Validity of the construct Validity of how we measure construct

It is Ok to have non-rate measures Self reported measures are generally not valid as rates A common mistake is interpreting a non-rate measure as a valid rate

Learning from Mistakes What happened? Why did it happen (system lenses) What could you do to reduce risk How to you know risk was reduced Create policy/process/procedure Ensure staff know policy Evaluate if policy is used correctly Pronovost 2005 JCJQI

Pronovost Health affairs in press Identify Hazards ( 3. Mitigate Risks 2. Analyze & Prioritize Hazards 4. Evaluate Effectiveness of Risk Reduction Patient Safety Learning Communities Patient safety learning communities relate to each other in a gear like fashion: as the identified hazards require stronger levels of intervention to achieve mitigation, the next learning community is engaged in action, eventually feeding back to the group that provided the initial thrust. Each group (unit, hospital, industry) follows the same four- step process, but they engage unique matrices of stakeholders to mitigate hazards that are within their locus of control. Pronovost Health affairs in press

GYN/OB JHOC Medicine Neurosciences Oncology Ophthalmology FAC: Fetal Assessment Center/OB Ultrasound  GSS - Shared Specialty Suite  Asthma & Allergy - Allergy & Clinical Immunology  BRU  GSS - Medical Oncology  GSS - Wilmer 110  GSS - GYN/OB 420  JHOPC - Express Testing  Asthma & Allergy - Pulmonary  EMU  IPOP Clinic - HIPOP Location  GSS - Wilmer Laser Center  GSS - GYN/REI  JHOPC - OR  Asthma & Allergy - Rheumatology  JHOPC Neurosciences  IPOP Clinic - IPOP Location  WECP & ER  HAL-2  JHOPC - PACU  Blalock 4 - Endoscopy  MEY 8 (12) Weinberg OPD - 1st Floor  Wilmer OR  JHOPC GYN/OB  WM - Shared Specialty Suite  Blalock 5 Echo Lab (2) MEY 9 (5) Weinberg OPD - 2nd Floor  Wilmer PACU  MCE    Cardiac CT  NCCU7  WGA 5 (5) Wilmer White Marsh  NEL-2 Nursery  CCP-5 (5) WGB 5  Wilmer: Other - E Balt Divisions  NEL-2 Obstetric OR  CCU-5 (7) WGC-5 (3) Wilmer: Other - Satellites  NEL-2 PACU  CVC  WGD 5  Nelson Harvey 2  CVIL- CardioVascular Interventional Lab  OSL-2  Dialysis Unit  OSL-3 Nursery  GSS - Internal Medicine  OSL-3  HAL-5 (5) WGB-4  HAL-8 (7) Hospitalist Unit (5) JHOPC - Exec Health & Travel Clinic  JHOPC - Medicine Clinics 

CAST Each contributing factor rate Each Intervention rate importance of the problem and contributing factors in causing the accident importance of the problem and contributing factors in future accidents Each Intervention rate How well the intervention solves the problem or mitigates the contributing factors for the accident Rates the team belief that the intervention will be implemented and executed as intended

“The way we do things around here” What is Culture*?: “The way we do things around here” here 1 attitude = opinion…everyone’s attitude = culture *aka Climate

Executive Perceptions vs. Frontline Perceptions: Executives overestimate: Teamwork Climate 4X Safety Climate 2.5X Executive Confidence vs. Executive Accuracy: -Often wrong but rarely in doubt… -Currently no incoming data-streams -Halo Effects -Frontline data fills the gap

* Statistically Significant 6 of 7 domains have shown statistically significant improvements since 2006. * Statistically Significant 28

71 Teamwork Climate 2008 67 Teamwork Climate 2007 64 Teamwork Climate 2006 62 Teamwork Climate 2005 This illustrates the teamwork domain scores (safety on the next slide) for each of the 150+ units across the organizaiton. It is important to notice the variability…even with an organizational domain score of 71. It also shows how many units are below the danger zone threshhold (60%). 29

60 Safety Climate 2006 65 Safety Climate 2007 70 Safety Climate 2008 59 Safety Climate 2005 30

#4. “I Would Feel Safe Being Treated Here As A Patient.”   % of respondents within an ICU that agree

#3. “Nurse Input Is Well Received In This ICU.”   % of respondents within an ICU that agree

#26. “In This ICU, It Is Difficult To Speak Up If I Perceive A Problem With Patient Care.”   % of respondents within an ICU that agree

#32. “Disagreements In This ICU Are Resolved Appropriately (i. e #32. “Disagreements In This ICU Are Resolved Appropriately (i.e. not who is right, but what is best for the patient).”   % of respondents within an ICU that agree

Questions for Reflection How do you know you are safer? How will you become more efficient in your measurement efforts? How will you better tap into local wisdom?

Focus and Execute