1. Risk Informed Evaluation of Patient Safety Training Anthony D. Slonim, MD, DrPH Vice President Medical Affairs Carilion Medical Center Senior Staff, Departments of Medicine and Pediatrics Carilion Clinic Professor, Medicine and Pediatrics Virginia Tech-Carilion School of Medicine

2. Objectives Decision-making science How do we normally make decisions? Picking up on Level 4…Can we train to improve our decision making results?

3. Decision Making Science

4. Bayes Theorem P (A/B)=P(B/A) * P(A) P(B)

5. Pattern Recognition How many squares do you see? Jumping to conclusions too quickly. Is there flashing in the squares? Your mind will play tricks on you!

6. Decision-Making Medical Decision-Making Process 1.Perception/Data gathering (training-H and P, Labs, Rads) “Amber light” is showing 2.Interpretation (training-pattern recognition and probability) “Amber light” means prepare to stop, maybe 3.Decision making (based on probability + experience) Stop or go 4.Action taking (reflex/“gut level response”/programming) Hit the brake or accelerator Marx, D Marx D and Slonim AD: Assessing patient safety risk before the injury occurs: An Introduction to Socio-Technical Probabilistic Risk Assessment. Quality and Safety in Healthcare 2003; 12 Suppl 2: 33-38.

7. Medical Decision-Making Perception/Data Gathering Interpretation Decision-Making Action Triage Nurse ED Nurse Physician X Get Help: Cardiology Consultation Make a Dx and Treat Do more testing-which test? (Pre-test probabilities)

8. Medical Decision-Making Perception/Data Gathering Interpretation Decision-Making Action Triage Nurse ED Nurse Physician X Get Help: Cardiology Consultation Make a Dx and Treat Do more testing-which test? (Pre-test probabilities)

9. Expert Decision Making: Practice, Practice, Practice * Expert – pattern matching against large mental library, quick, accurate if confirm correct answer * Novice – library is empty – slow, error prone process * Certain Diagnoses are Favored- Frequent, Recent, Serious * Heuristics – fixating on the wrong pattern

10. Pattern Recognition

11. Picking up on Level 4: Can we train for results?

12. Kirkpatrick’s Levels Level I Reactions How well trainees liked training Level II Learning The extent to which trainees understand and retain principles, facts, and techniques Level III Behavior The extent to which behavior changes as a result of training Level IV Results Impact of training on organizational criteria

13. Data Analytics ProcessImprovements Change Management Improved Outcomes Program Identification & Prioritization Elements of Quality Programs Quality Functions Research Education / Training

14. Why is there a safety problem ? Considerable variation in practice Based on opinion or consensus Evidence-based guidelines-unsupported Failure to create fail-safe processes Our providers may not know their work Policies and procedures We’re learning to work together We’re not sure of the results we’re looking for

15. Process Analysis Processes: A series of sequential steps governing interactions Between patients and providers Between providers and providers Examples of process analysis techniques: Root cause analysis-retrospective HAACP (hazard analysis and crit control points) FMEA (failure mode effects) PI methodology

16. Low-frequency, High Impact Events Low frequency, high-impact events Variable processes and practices Wrong site surgery The abduction of children from hospitals Deaths or major harm Process analysis helps to identify risk and prioritize interventions Decision support helps to guide decision making

17. Probabilistic Risk Assessment A hybrid between process analysis and decision support Identifies risk points and directs to interventions Is hierarchical and probabilistic Allows disentanglement of patient level risks, provider level risks, and system level risks Assigns probabilities for prioritization of risk reduction strategies Includes sociotechnical components into the models

18. Conceptual Framework Probabilistic Risk Assessment The Institution The Providers The Patient Quantitative Methods: Qualitative Methods:

19. The Prospective Risk Model The Top Three Risks

20. Training Evaluation Definition The systematic collection of descriptive and judgmental information necessary to make decisions related to instructional activities Ensures training Meets its stated objectives Changes trainee attitudes Increases trainee knowledge Develops trainee skills Transfers results to the job

21. Training Evaluation Important variables to consider: Organizational Factors Individual Factors Trainee Knowledge, Skills, and Attitudes Training Transfer Organizational Outcomes

22. Merging Kirkpatrick and ST PRA Socio-Technical Probabilistic Risk Assessment Good for examining low base rate events (Six Sigma) Models contributing causes Procedural tasks Team tasks Identifies the impact of an intervention Evidence base Empirically based Adjust and test the model Monte Carlo Changes in the likelihood of outcomes

23. Traditional Approaches Quick Wins “Fire-fighting” Burn-out / Fatigue Difficult-to-Sustain, Short-Term Results Typical ResultsImpact Time

24. Quality Fusion Approach Time Impact Quality Fusion Results Typical Results

25. Example

26. What is Escalation? Failure to rescue associated with Interpretation problems Throughput problems Put another way… When you do not realize the patient is in trouble OR you know the patient is in trouble, but you don’t respond as needed.

27. Common Course Ideally, we track the illness. As the patient gets worse (line goes up), we respond. As the patient improves we adjust. The patient condition The provider team response = =

28. Going Off Course The defect rate in our model is caused by failures to properly track the course of the illness. The patient condition The provider team response = =

29. Never On Course The provider team response The patient condition = = Sometimes, we’re off course right from the beginning and it’s difficult to get back on course.

30. The Prospective Risk Model The Top Three Risks

31. Conclusions A focus on results helps providers and patients Training on risk points can improve performance Leads to better results Requires alterations in decision making Enhances empiric data for better understanding training