1. Assessing operative autonomy Combining theory and software to make evaluation easy Jonathan Fryer MD, Professor of Surgery, Feinberg School of Medicine, Northwestern University

2. Disclosures I have made no financial gains from this project I may in the future I intend to continue work on this project regardless 2

3. What is the most essential goal of surgical training? 3

4. Operative Autonomy The ability to independently perform operations safely and effectively. 4

5. The Problem There is growing concern that graduating surgical residents are not achieving operative autonomy with essential procedures. 5 1.Bell RH. Why Johnny cannot operate. Surgery 146, 533–542 (2009). 2.Mattar SG et al. General Surgery Residency Inadequately Prepares Trainees for Fellowship: Results of a Survey of Fellowship Program Directors. Annals of Surgery September 2013 258, 440–449 (2013). 3.Coleman JJ et al. Early Subspecialization and Perceived Competence in Surgical Training: Are Residents Ready? Journal of the American College of Surgeons 216, 764–771 (2013). 4.Chen P. Are Today’s New Surgeons Unprepared? Well (2013). at http://well.blogs.nytimes.com/2013/12/12/are-todays-new-surgeons-unprepared

6. The Problem To be able to fix it…… You have to be able to measure it. 6

7. The Problem We don’t do a very good job of assessing residents in the OR. 7

8. The Problem – Currently, summative assessment of OR performance is based on: # of cases logged by resident –Role of resident in each case? Semi-annual global evaluations –Memory decay? 8

9. The Problem 9 …asking busy surgical faculty to fill out complex assessment forms in a timely manner, doesn’t work.

10. The Solution A simple assessment tool that: Assesses operative autonomy Doesn’t impede surgical workflow Facilitates high compliance and prompt completion

11. Theoretical Framework Inter-related constructs: – Supervision, Guidance, Autonomy, Performance Faculty Supervision (oversight) ≠ Faculty guidance (physical or verbal help) 1 Faculty Guidance = Resident Autonomy Resident Autonomy = ƒ (Resident performance) 11

12. The Solution – With every case faculty: Provide resident supervision. Assess and document the level of operative autonomy achieved by the resident. Progressively reduce the level of operative guidance they provide to resident.

13. The “Zwisch” Scale 4 levels of operative guidance – Show & Tell – Active Help – Passive Help – Supervision Only DaRosa, D. A. et al. A Theory-Based Model for Teaching and Assessing Residents in the Operating Room. Journal of Surgical Education 70, 24– 30 (2013).

14. Our method: PASS (Procedural Autonomy and Supervision System)

15. Today

16. Coming soon…

17. Study Design: Participants and Setting Department of general surgery at a large academic hospital All teaching faculty underwent formal frame-of-reference training per published protocol 1 All general surgery residents and trained faculty raters eligible for inclusion IRB-approved 17 1 George et al, J. Surg. Educ. 2013; 70

18. Results: Feasibility A 1 hour rater training session is sufficient to achieve reliable and accurate ratings 1 92% response rate using PASS 1 George, B. C. et al. Duration of Faculty Training Needed to Ensure Reliable OR Performance Ratings. J. Surg. Educ. 70, 703–708 (2013).

19. Results: PASS Sample (7 mos) Number of Residents31 By Year of Residency Year 1 Year 2 Year 3 Year 4 Year 5 9 6 5 5 6 Number of Attendings27 Number of Procedures1490 Number of Types of Procedures 127 19

20. Results: PASS Sample Relative Case Complexity Easiest 1/3Middle 1/3Hardest 1/3 193 (13.0%)895 (60.1%)402 (27.0%) 20

21. Results: Validity: Zwisch Levels by PGY 21 p=<.001 p=0.21 23.2%

22. Results: Validity: Zwisch Levels by Complexity 22 p=<.001

23. Results: Validity: Zwisch Level by Prior Experience 23 p=<.001

24. Study Design: Data Collection Sample 2: Video Sample – 8 procedures video recorded for additional review (subset of PASS sample) – Rated by operating faculty, in-person OR observer, and video reviewer using Zwisch scale (blinded to other scores) – Rated by 2 additional video reviewers using other OR assessment instruments (modified OPRS and O- SCORE) 24

25. Results: Video Sample 25 Number of Residents4 (PGY 2 to 5) Number of Attendings2 Number of Procedures8 Number of Types of Procedures 5 2 Laparoscopic cholecystectomy 2 Open inguinal hernia repair 2 Parathyroidectomy 1 Total thyroidectomy 1 Laparoscopic ventral hernia repair

26. Results: Reliability Inter-rater reliability – Zwisch ratings – Operating attending, OR observer, and video rater – ICC =.90, 95% CI =.72 -.98, p <.001. 26

27. Itemρp-value Operative Performance Rating System (OPRS) Degree of prompting or direction-.92.001 Instrument handling.94.005 Respect for tissue.94.005 Time and motion.94<.001 Operation flow.95<.001 Overall performance.95<.001 Ottawa Surgical Competency OR Eval. (O-SCORE) Knowledge of procedural steps.94<.001 Technical performance.93.001 Visuospatial skills.92.001 Efficiency and flow.86.007 Communication.92.001 Results: Validity: Zwisch Level correlation with other OR assessment tools 27

28. Benefits Faculty and residents constantly reminded of ultimate goal …. i.e. operative autonomy. Establishes a conceptual framework for teaching and learning in the OR. Data can be used to: – Help faculty and residents to set learning goals. – Help programs monitor operative progress and identify those who may need additional attention. – Address regulatory requirements for OR supervision and operative performance assessment. – Establish national norms

29. Limitations So far, studied only at a single institution Validity analysis based on small convenience sample Raters not blinded to resident PGY level Comparison with only selected items of OPRS and O- SCORE Unmeasured confounders (time of day, supervising surgeon experience, etc) 29

30. Conclusion The Zwisch rating scale is a reliable and valid measure of faculty guidance and resident autonomy Deployed on PASS the Zwisch scale can be used to feasibly record evaluations for the vast majority of operations performed by residents 30

31. Vision All surgical subspecialties. Other procedural specialties. Other medical professionals who need to learn to perform complex clinical tasks. Other trades or professions where trainees need to learn to independently perform complex tasks safely and effectively. 31

32. Acknowledgements Surgical Education Research & Development Team Jay Zwischenberger Eric Hungness Shari Meyerson Debra DaRosa Jonathan Fryer Ezra Teitelbaum Brian GeorgeMary Schuller Research supported by: Excellence in Academic Medicine Program from the State of Illinois Augusta Webster Educational Innovation Grant from the Northwestern University Center for Education in Medicine

33. Theoretical basis Global assessment of performance is simpler, more accurate, and more reliable than checklists 1 Faculty guidance is related to resident performance 2 Faculty can accurately and reliably rate the amount of guidance provided to residents 3 1.Regehr, G., MacRae, H., Reznick, R. K. & Szalay, D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 73, 993–997 (1998). 2.Chen, X. (Phoenix), Williams, R. G., Sanfey, H. A. & Dunnington, G. L. How do supervising surgeons evaluate guidance provided in the operating room? The American Journal of Surgery 203, 44–48 (2012). 3.George, B., Teitelbaum, E., DaRosa, D., Hungness, E., Meyerson, S., Fryer, J., Schuller, M., Zwischenberger, J. Duration of Faculty Training Needed to Ensure Reliable O.R. Performance Ratings. Journal of Surgical Education 70(6), 703-708 (2013).

34. Study Over 7 months 1490 evaluations 27 faculty 31 residents

35. Study Design: Rating Scales Zwisch Procedural Complexity Operative Performance Rating System (OPRS) 1 – 6 general items only--excludes items that pertain only to specific procedures Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) 2 – 5 intra-operative items only--excludes items that did not pertain to intra-operative performance. 35 1 Chen et al, The American Journal of Surgery 2012; 203 2 Gofton et al, Acad. Med. 2012; 87

36. Results: Validity Convergent Validity for Guidance/Autonomy and Resident Performance – Zwisch level vs. PGY – Zwisch level vs. Complexity – Zwisch level vs. Resident Experience Construct Validity for Guidance/Autonomy – Zwisch level vs. OPRS guidance item Construct Validity for Resident Performance – Zwisch level vs. OPRS performance items – Zwisch level vs. O-SCORE performance items 36

37. Learning Vector 37 Independence Dependence Autonomy ImmatureMature Learners Adapted from Stritter FT, Baker RM, Shahady EJ. Handbook for the academic physician. 1986.

38. The Team Dr. Debra DaRosa Dr. Brian George Dr. Shari Meyerson Dr. Ezra Teitelbaum Mary Schuller Dr. Nathaniel Soper Dr. Joseph Zwischenberger 38

39. Impact so far Over 1000 evaluations collected in 6 months Response rate > 90% Changes in teaching They love to use it!

40. Next steps Dictation of feedback Reports

41. Results: Validity Convergent Validity for Guidance/Autonomy and Resident Performance – Zwisch level vs. PGY – Zwisch level vs. Complexity – Zwisch level vs. Resident Experience Construct Validity for Guidance/Autonomy – Zwisch level vs. OPRS guidance item Construct Validity for Resident Performance – Zwisch level vs. OPRS performance items – Zwisch level vs. O-SCORE performance items 41

42. Theoretical Framework 42 HelpingWatching

43. Next Steps I am actively trying to bring this to MGH It needs additional development before it can be launched here Multiple other departments have already committed to supporting this project 43

44. Questions? 44

45. Results

46. 50% = 60 procedures

47. Benefits all stakeholders Provides useful data to all stakeholders – Faculty – Program – Regulators – Researchers – Residents

48. Current Status Milestone AchievedCost / time Development of v1.0 mobile app$200,000 / 8 months Development of v0.9 administrative interface (beta) $75,000 / 3 months Integration with Northwestern EMR $45,000 / 2 months Development of v2.0 iOS app$160,000 / 7 months (ongoing) Total$480,000 + operational expenses

49. Road Map Planned Technical MilestonesTarget launch date v2.0 for iOS at NorthwesternFebruary 2014 v1.0 Administrative interface at Northwestern April 2014 v2.0 for Android at NorthwesternJune 2014 System integration at MGHJune 2014 v2.0 iOS at MGHJuly 2014 V2.0 Android at MGHOctober 2014

50. 12 month budget Expense ItemCost Design and specification$30,000 Software Development$225,000-$300,000 QA testing$30,000 Server hosting and maintenance$25,000 User training$5,000 Administrative$30,000 Total$345,000 - $420,000

51. The “Zwisch” Scale 4 levels of guidance – Show & Tell – Active Help – Passive Help – Supervision Only DaRosa, D. A. et al. A Theory-Based Model for Teaching and Assessing Residents in the Operating Room. Journal of Surgical Education 70, 24–30 (2013).

52. Theoretical Framework 52 Stritter FT et al., Handbook for the academic physician. 1986. Chen et al., The American Journal of Surgery 2012; 203 Gofton et al., Acad. Med. 2012; 87