1. Creation of the Simulator Value Index Tool Adapted from workshop on 4.21.14 presented by American College of Surgeons Accreditation Education Institutes, Technologies & Simulation Committee) Deborah Rooney PhD James Cooke MD Yuri Millo MD David Hananel MEDICAL SCHOOL UNIVERSITY OF MICHIGAN

2. Disclosures o David Hananel, No Disclosures o Yuri Millo, No Disclosures o James Cooke, No Disclosures o Deborah Rooney, No Disclosures

3. Overview of Main Topics o Introduction of project o Overview of 2014 IMSH Survey results o Summary of 2014 ACS Consortium results o Working meeting to refine the AVI algorithm o Apply AVI algorithm in group exercise o Discuss next steps

4. Introduction: How it all started o ACS AEI, Technologies and Simulation Committee o Guidelines for Simulation Development (Millo, George, Seymour and Smith) o University of Michigan o Need to support faculty in sim purchase/decision-making process (Cooke) o Discourse o Definition of “value” o Differences across stakeholder role (institution, administration, clinician, educator, researcher...)

5. Introduction: How it all started o Reached consensus on factors used when considering a simulator purchase o Survey 1 o IMSH general membership, N=2800 o January, 2014 o Workshop 1, n=16 o IMSH, January, 2014 o Survey 2 o ACS AEI Consortium membership, N = 455 o March, 2014 o Workshop 2, n = ? o ACS AEI-March, 2014

6. Introduction: The Instrument o Began with 31-item survey accessed via www (Qualtrics) o 4-point rating scale o (1 = not considered/not important  4= critical to me when I consider a simulator purchase) o 6 Domains o Cost, Impact, Manufacturer, Utility, Assessment, Environment/Ergonomics) o Demographics o Country/Institution o Stakeholder role o Involvement o Follow-up

7. 44 5 2 1= Grenada 1= Chile 1= Peru 1=Czech Republic 2 2=Singapore 3 = New Zealand 1 1 95 total respondents, 72 individuals completed survey approximately 2+% of IMSH membership (2,800), 7 undesignated/16 incomplete IMSH Survey Sample: 67 institutions x 12 Countries

8. 6 1 1 3 4 4 1 1 1= Massachusetts 3 = Rhode Island 1 = New Jersey 1 3 1 2 1 3 1 1 1 1 1 50 participants from US IMSH Survey Sample: 44 institutions x 22 States/US

9. 46 58% 28 35% 26 33% 20 25% 6 8% 4 5% 1 1% n = 79 1 undesignated IMSH Survey Sample: Institution Affiliation

10. o Cost o Commercial Skills Centers (CSC) rated C1 (Purchase cost) lower than each of the other institutions, p =.001. o Manufacturer o CSCs rated M1 (Reputation of manufacturer) lower than each of the other institutions, p =.001. o Utility o CSCs rated U3 (Ease of data management) and o U11 (portability) lower than each of the other institutions, p =.001. o Ergonomics o Medical Schools rated item E2 (Ergonomic risk factor) much higher thank other institutions), p =.05.  CSCs rated E3 (Ease of ergonomic setup) lower than each of the other institutions, p =.001. IMSH Survey Results: Rating Differences by Institutional Affiliation

11. 31 39% 19 24% 7 9% 8 10% 14 18% n = 79 1=undesignated IMSH Survey Sample : Stakeholder Role

12. o Cost o Clinicians rated C2 (Cost of warranty) lower than the other stakeholders, p =.048. o Utility o Clinicians rated U11 (portability of simulator) higher than other stakeholders, p =.037. IMSH Survey Results: Rating Differences by Stakeholder Role

13. 37 46% 37 46% 4 5% 2 3% n = 80 IMSH Survey Sample : Involvement in Decision

14. o Although there are no differences across level of involvement, o There are different considerations during simulator purchasing process across; o Country o Institutional affiliation (commercial skills center may have unique needs) o Stakeholder role (Clinicians may have unique needs) o Keeping this in mind, let’s review the top factors considered IMSH Survey Results : Summary

15. AverageFactor (survey item number, item description)Domain 3.8 21- Technical stability/reliability of simulatorUtility 3.7 10- Customer service Manufacturer 3.4 16- Ease use for instructor/administrator Utility 3.4 19- Ease of use for learner Utility 3.36- Relevance of metrics to real life/clinical setting Impact 3.211- Ease of delivery and installation, orientation to simManufacturer 3.226- Reproducibility of task/scenario/curriculumAssmnt/Res 3.21- Purchase cost of simulatorCost 3.29- Reputation of manufacturerManufacturer 3.18- ScalabilityImpact 3.120- Quality of tutoring/feedback from sim to learnersUtility 3.17- Number of learners impactedImpact 3.02- Cost of warrantyCost 3.03- Cost of maintenanceCost 3.017- Ease of configuration/authoring sim's learning management systemUtility -Physical durabilityUtility The SVI Factors: Top 15+1 Factors Ranked

16. ACS Consortium Survey: Introduction o Identical Survey items, ratings o Added durability of simulator question o 31  32-item survey accessed www (Qualtrics) o 4-point rating scale o (1 = not considered/not important  4= critical to me when I consider a simulator purchase) o 6 Domains o Cost, Impact, Manufacturer, Utility, Assessment, Environment/Ergonomics) o Demographics o Country/Institution o Stakeholder role o Involvement o Follow-up

17. 49 1 2 1=UK 1=France 1=Italy 1 65 total respondents, 54 individuals completed survey approximately 12% of ACS membership (455), 2 undesignated ACS Survey Sample : 41 institutions x 7 Countries 1=Greece 1=Sweden

18. 8 1 1 3 5 4 1 1 8 = Massachusetts 1 = Rhode Island 1=Delaware 1 = Maryland 1 1 4 4 1 3 2 1 2 1 3 49 participants from US 47 indicated institution ACS Survey Sample: 36 institutions x 17 States/US 1 1 1

19. 37 67% 28 51% 24 44% 16 29% 2 4% 0 0% n = 55 ACS Survey Sample: Institution Affiliation 0 0%

20. 9 16% 27 48% 1 <2% 3 <6% 13 23% n = 56 ACS Survey Sample: Stakeholder Role 2 <4% 1 <2%

21. 25 45% 29 52% 2 3% n = 56 ACS Survey Sample: Involvement in Decision

22. o Although there are no differences across; o institution o stakeholder role o There are different considerations during simulator purchasing process across; o Level of involvement o (Self-reported “Responsible” folks are more concerned about number of learners impacted and Scalability) ACS Survey Results: Summary

23. But are there differences across IMSH and ACS membership? ACS Survey Results: Summary

24. Survey Results: IMSH v. ACS 4 (C2) 7 (I2)11 (M3)15 (U4) 22 (U11 )

25. o Cost o ACS members rated C2 (Cost of warranty) higher than the IMH members, bias =.40, p =.04. o Impact o ACS members rated I2 (Number of learners) higher than other stakeholders, bias =.53, p =.01. o Utility o ACS members rated U4 (Ease of report generation) higher than the IMH members, bias =.43, p =.02. o ACS members rated U11 (Portability of simulator) higher than other stakeholders, bias =.48, p =.01. Survey Results: Rating Differences by Conference

26. The SVI Factors: Top 15+1 Factors Ranked

27. Applying the SVI Tool o General impressions? What stood out? o What worked well? o What could have gone better? o Any surprises? o Usefulness? How might you use the SVI Tool at your institution? o Please complete the questions on “Feedback” Tab on the SVI Worksheet

28. Thank you: Our Contact Information o Deb Rooney University of Michigan dmrooney@med.umich.edu dmrooney@med.umich.edu o Jim Cooke University of Michigan o cookej@med.umich.edu cookej@med.umich.edu o David Hananel SimPORTAL & CREST University of Minnesota Medical School o dhananel@umn.edu dhananel@umn.edu o Yuri Millo Millo Group yuri.millo@millo-group.com yuri.millo@millo-group.com o Olivier Petinaux ACS American College of Surgeon, Division of Education opetinaux@facs.orgopetinaux@facs.org