1. Effectiveness of Simulation in Healthcare Joseph Barton, MD, MHMS September 18, 2014

2. Desired Benefits of Simulation Training Reduce patient risk Broadly applicable and widely available Consistent training and assessment Reduce costs Provide a clinical benefit

3. Kirkpatrick’s 4 Levels of Learning Evaluation – Reaction – Learning – Behavior – Results

4. Learning Evaluation LevelDescriptionEvaluation 1SurveysDid the trainee like it? 2AssessmentsDid the trainee learn? 3TransferIs a behavior changed? 4BenefitIs there a clinical outcome?

5. DOES SIMULATION-BASED EDUCATION WORK?

6. What is the Evidence? How effective is simulation as a modality? Number of recent meta analyses – Virtual patients – “Technology enhanced” simulation Virtual patients do not require special equipment Standardized patients aren’t well studied

7. HOW EFFECTIVE IS SIMULATION AS A TEACHING MODALITY?

8. Effectiveness of Simulation More than 600 studies comparing 36K plus subjects Three recent large meta analyses

9. Effectiveness of Simulation Cook DA, Erwin PJ, Triola MM. Computerized virtual patients in health professions education: a systematic review and meta- analysis. Acad Med. 2010;85:1589-1602. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011;306:978-988. Zendejas B, Brydges R, Wang AT, Cook DA. Patient outcomes in simulation-based medical education: a systematic review. J Gen Intern Med. 2013.

10. Effectiveness of Simulation Virtual patients and technology enhanced simulation show consistent, large and statistically significant benefits in – Knowledge – Instructor ratings – Computer scores – Patient care behaviors

11. Direct Patient Effects? Airway management, endoscopy, CVC insertion studied – Smaller benefits, still significant (major complications, mortality, length of stay) when compared to no intervention – Non simulation instruction did not reach statistical significance

12. Simulation based education is more effective than no instruction.

13. HOW DOES SIMULATION COMPARE TO OTHER INSTRUCTION?

14. Simulation vs other intervention Over 100 studies and 7K participants Challenging question due to different simulation interventions and due to variable comparison strategy

15. Simulation vs Other Intervention Cook DA, Brydges R, Hamstra SJ, et al. Comparative effectiveness of technology- enhanced simulation versus other instructional methods: a systematic review and meta-analysis. Simul Healthc. 2012;7:308-320. Cook DA, Hamstra SJ, Brydges R, et al. Comparative effectiveness of instructional design features in simulation-based education: systematic review and meta- analysis. Med Teach. 2013;35:e844-e875. Lorello GR, et al. Simulation-based training in anesthesiology: a systematic review and meta analysis. Br J Anaesth. 2014 Feb; 112(2):231-45. McGaghie, W. C., Issenberg, S. B., Cohen, E. R., Barsuk, J. H., & Wayne, D. B. (2011). Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Academic Medicine, 86, 706–711.

16. Simulation vs Other Intervention Simulation is non inferior to other approaches Technology enhanced simulation shows small, statistically significant benefit for knowledge and skills outcomes Approached statistical significance with direct patient benefit and provider behaviors

17. Simulation vs Other Intervention Anesthesia review showed moderate effects for satisfaction and skills, large effect for provider behavior, small effect for direct patient benefit compared with non simulation instruction Negligible effects comparing simulation to alternative simulation interventions Analysis showed inconsistencies in measurement of non technical skills

18. Pooled effect sizes for studies comparing training with no training. Effect sizes represent Cohen’s d or the nearly-equivalent Hedges’ g from random-effects meta-analysis; > 0.80 is large, 0.50–0.79 is moderate. Data derived from meta- analyses of Internet-based instruction (Cook et al. 2008b) virtual patients (Cook et al. 2010a), simulation-based instruction (McGaghie et al. 2011), and simulation-based instruction (Cook et al. 2011a)

19. Simulation based education is probably as good as (but is not necessarily better than) other types of instruction.

20. There is a high degree of variability between studies, suggesting that certain simulation interventions may be more effective in certain scenarios.

21. HOW CAN WE IMPROVE THE EFFECTIVENESS OF SIMULATION TRAINING?

22. Comparative Effectiveness Evaluates outcomes and processes leading to the outcome – Considers the costs, barriers, unforeseen consequences, and effective strategies associated with implementing therapies in practice. Studies comparing different simulation based approaches to explain what works, for what audience and in what context.

23. Comparative Effectiveness Sample sizes must be large Confounding can be a problem Effect size? – The difference between teaching and no teaching should be large, differences between teaching types may not be as apparent

24. Comparative Effectiveness Cook DA, Hamstra SJ, Brydges R, et al. Comparative effectiveness of instructional design features in simulation- based education: systematic review and meta-analysis. Med Teach. 2013;35:e844-e875.

25. Comparative Effectiveness 289 studies, 20K participants Feedback, repetition, range of difficulty, cognitive interactivity, clinical variation, distributed practice, individualized training, and longer training time significantly improve skill outcomes Patient outcome analysis revealed benefits of similar direction and magnitude that approached statistical significance

26. Future research should clarify how to choose between simulation and non- simulation approaches using rigorous qualitative studies that explore the strengths and appropriateness of each approach.

27. IS SIMULATION TRAINING COST EFFECTIVE?

28. What is the Value? Must consider costs – Simulator, faculty time, training expenses, facility fees, opportunity cost No study has offered a complete accounting of simulation costs

29. Cost Assessment Zendejas B, Wang AT, Brydges R, Hamstra SJ, Cook DA. Cost: the missing outcome in simulation-based medical education research: a systematic review. Surgery. 2013;153:160-176. Norman G, Dore K, Grierson L. The minimal relationship between simulation fidelity and transfer of learning. Med Educ. 2012;46:636-647.

30. What is the Value? Cost reporting is infrequent and incomplete Low-fidelity, low-cost training models can yield outcomes equal to much more expensive simulators

31. More expensive simulators are not necessarily better.

32. CONCEPTUALLY, SIMULATION TRAINING MAKES SENSE

33. Simulation Training = Patient Safety? Ziv A, Wolpe PR, Small SD, Glick S. Simulation-based medical education: an ethical imperative. Acad Med. 2003;78:783-788.

34. Simulation Education = Patient Safety? “Risk free” skill rehearsal – Repetitive, deliberate and structured practice “Risk free” assessment and feedback

35. Simulation based rehearsal is evidence supported when learning to work with “real patients.”

36. SIMULATION MAY NOT ALWAYS BE THE BEST LEARNING MODALITY

37. Is Simulation the Best Choice? Instructional design should be based on learning objectives, learner needs, safety concerns, resource utilization Virtual patients often used to teach clinical reasoning Technology enhanced simulation is most often for procedural training

38. How Do We Optimally Implement Simulation Education? More research to clarify how to choose between sim and non sim approaches Cost effectiveness research to determine true and comparison value of sim education

39. How Do We Optimally Implement Simulation Education? Comparative effectiveness research should focus on what works in simulation – for which audience – for what circumstances – and at what cost. Task analysis should focus on critical actions and can help determine level of fidelity necessary for type of training

40. How Do We Optimally Implement Simulation Education? Careful selection and sequencing of events surrounding the simulated task Appropriate faculty development and support Institutional commitment to simulation

41. QUESTIONS?