Creation of the Simulator Value Index Tool Adapted from workshop on presented by American College of Surgeons Accreditation Education Institutes, Technologies & Simulation Committee) Deborah Rooney PhD James Cooke MD Yuri Millo MD David Hananel MEDICAL SCHOOL UNIVERSITY OF MICHIGAN

Disclosures o David Hananel, No Disclosures o Yuri Millo, No Disclosures o James Cooke, No Disclosures o Deborah Rooney, No Disclosures

Overview of Main Topics o Introduction of project o Overview of 2014 IMSH Survey results o Summary of 2014 ACS Consortium results o Working meeting to refine the AVI algorithm o Apply AVI algorithm in group exercise o Discuss next steps

Introduction: How it all started o ACS AEI, Technologies and Simulation Committee o Guidelines for Simulation Development (Millo, George, Seymour and Smith) o University of Michigan o Need to support faculty in sim purchase/decision-making process (Cooke) o Discourse o Definition of “value” o Differences across stakeholder role (institution, administration, clinician, educator, researcher...)

Introduction: How it all started o Reached consensus on factors used when considering a simulator purchase o Survey 1 o IMSH general membership, N=2800 o January, 2014 o Workshop 1, n=16 o IMSH, January, 2014 o Survey 2 o ACS AEI Consortium membership, N = 455 o March, 2014 o Workshop 2, n = ? o ACS AEI-March, 2014

Introduction: The Instrument o Began with 31-item survey accessed via www (Qualtrics) o 4-point rating scale o (1 = not considered/not important  4= critical to me when I consider a simulator purchase) o 6 Domains o Cost, Impact, Manufacturer, Utility, Assessment, Environment/Ergonomics) o Demographics o Country/Institution o Stakeholder role o Involvement o Follow-up

= Grenada 1= Chile 1= Peru 1=Czech Republic 2 2=Singapore 3 = New Zealand total respondents, 72 individuals completed survey approximately 2+% of IMSH membership (2,800), 7 undesignated/16 incomplete IMSH Survey Sample: 67 institutions x 12 Countries

= Massachusetts 3 = Rhode Island 1 = New Jersey participants from US IMSH Survey Sample: 44 institutions x 22 States/US

46 58% 28 35% 26 33% 20 25% 6 8% 4 5% 1 1% n = 79 1 undesignated IMSH Survey Sample: Institution Affiliation

o Cost o Commercial Skills Centers (CSC) rated C1 (Purchase cost) lower than each of the other institutions, p =.001. o Manufacturer o CSCs rated M1 (Reputation of manufacturer) lower than each of the other institutions, p =.001. o Utility o CSCs rated U3 (Ease of data management) and o U11 (portability) lower than each of the other institutions, p =.001. o Ergonomics o Medical Schools rated item E2 (Ergonomic risk factor) much higher thank other institutions), p =.05.  CSCs rated E3 (Ease of ergonomic setup) lower than each of the other institutions, p =.001. IMSH Survey Results: Rating Differences by Institutional Affiliation

31 39% 19 24% 7 9% 8 10% 14 18% n = 79 1=undesignated IMSH Survey Sample : Stakeholder Role

o Cost o Clinicians rated C2 (Cost of warranty) lower than the other stakeholders, p =.048. o Utility o Clinicians rated U11 (portability of simulator) higher than other stakeholders, p =.037. IMSH Survey Results: Rating Differences by Stakeholder Role

37 46% 37 46% 4 5% 2 3% n = 80 IMSH Survey Sample : Involvement in Decision

o Although there are no differences across level of involvement, o There are different considerations during simulator purchasing process across; o Country o Institutional affiliation (commercial skills center may have unique needs) o Stakeholder role (Clinicians may have unique needs) o Keeping this in mind, let’s review the top factors considered IMSH Survey Results : Summary

AverageFactor (survey item number, item description)Domain Technical stability/reliability of simulatorUtility Customer service Manufacturer Ease use for instructor/administrator Utility Ease of use for learner Utility Relevance of metrics to real life/clinical setting Impact Ease of delivery and installation, orientation to simManufacturer Reproducibility of task/scenario/curriculumAssmnt/Res Purchase cost of simulatorCost Reputation of manufacturerManufacturer ScalabilityImpact Quality of tutoring/feedback from sim to learnersUtility Number of learners impactedImpact Cost of warrantyCost Cost of maintenanceCost Ease of configuration/authoring sim's learning management systemUtility -Physical durabilityUtility The SVI Factors: Top 15+1 Factors Ranked

ACS Consortium Survey: Introduction o Identical Survey items, ratings o Added durability of simulator question o 31  32-item survey accessed www (Qualtrics) o 4-point rating scale o (1 = not considered/not important  4= critical to me when I consider a simulator purchase) o 6 Domains o Cost, Impact, Manufacturer, Utility, Assessment, Environment/Ergonomics) o Demographics o Country/Institution o Stakeholder role o Involvement o Follow-up

=UK 1=France 1=Italy 1 65 total respondents, 54 individuals completed survey approximately 12% of ACS membership (455), 2 undesignated ACS Survey Sample : 41 institutions x 7 Countries 1=Greece 1=Sweden

= Massachusetts 1 = Rhode Island 1=Delaware 1 = Maryland participants from US 47 indicated institution ACS Survey Sample: 36 institutions x 17 States/US 1 1 1

37 67% 28 51% 24 44% 16 29% 2 4% 0 0% n = 55 ACS Survey Sample: Institution Affiliation 0 0%

9 16% 27 48% 1 <2% 3 <6% 13 23% n = 56 ACS Survey Sample: Stakeholder Role 2 <4% 1 <2%

25 45% 29 52% 2 3% n = 56 ACS Survey Sample: Involvement in Decision

o Although there are no differences across; o institution o stakeholder role o There are different considerations during simulator purchasing process across; o Level of involvement o (Self-reported “Responsible” folks are more concerned about number of learners impacted and Scalability) ACS Survey Results: Summary

But are there differences across IMSH and ACS membership? ACS Survey Results: Summary

Survey Results: IMSH v. ACS 4 (C2) 7 (I2)11 (M3)15 (U4) 22 (U11 )

o Cost o ACS members rated C2 (Cost of warranty) higher than the IMH members, bias =.40, p =.04. o Impact o ACS members rated I2 (Number of learners) higher than other stakeholders, bias =.53, p =.01. o Utility o ACS members rated U4 (Ease of report generation) higher than the IMH members, bias =.43, p =.02. o ACS members rated U11 (Portability of simulator) higher than other stakeholders, bias =.48, p =.01. Survey Results: Rating Differences by Conference

The SVI Factors: Top 15+1 Factors Ranked

Applying the SVI Tool o General impressions? What stood out? o What worked well? o What could have gone better? o Any surprises? o Usefulness? How might you use the SVI Tool at your institution? o Please complete the questions on “Feedback” Tab on the SVI Worksheet

Thank you: Our Contact Information o Deb Rooney University of Michigan o Jim Cooke University of Michigan o o David Hananel SimPORTAL & CREST University of Minnesota Medical School o o Yuri Millo Millo Group o Olivier Petinaux ACS American College of Surgeon, Division of Education