1. The Objective Structured Assessment of Technical Skills (OSATS) Helen M. MacRae, M.D., FRCSC, D.H. Gales Director Technical Skills Centre, University of Toronto

2. Certifying competence of new trainees Certifying Competence in New Technology (for example laparoscopic surgery or robotic surgery) Assessing the impact of interventions The Need for a Reliable and Valid Measure

3. The OSATS l A multi-station “bell-ringer” examination l Segments of 6 to 8 procedures l Performed on “bench models” simulating relevant anatomical structures l Knowledgeable OR nurses assisting

4. The OSATS (continued) l 15 minutes per procedure l Candidates move from station to station l Each station marked by qualified surgeon l Examiner uses two marking systems l Task-specific checklists l Global rating scales

5. Vascular Anastamosis Station Sample of Checklist Scoring (5 of 24) 1.Selects appropriate suture 2.Avoids excessive trauma / handling of artery with instruments 3.Guides suture down to desired position following bites 4.Uses correct needle angle >80% time. 5.Follows curve of needle on entrance / exit > 80% time 0 0 0 0 0 1 1 1 1 1

6. Vascular Anastamosis Station Sample of Global Rating (3 of 5) Respect for tissue 12345 Consistently handled tissue appropriately with minimal damage to tissue Frequently caused damage by inappropriate use of instruments Careful handling of tissue but occasionally caused inadvertent damage Time and motion 12345 Many unnecessary moves Efficient time/motion but some unnecessary moves Clear economy of movement and maximum efficiency Flow of Operation 12345 Frequently stopped operating and seemed unsure of next move Demonstrated some forward planning with reasonable progression of procedure Obviously planned course of operation with effortless flow from one move to the next

7. OSATS - Reliability l Internal consistency (across stations) l Inter-rater reliability (within stations)

8. Validity Measures l Clinical PGY level –are more senior trainees scoring higher? l Live animal version of exam –are better “live model” operators scoring higher? l Evaluation of final product –are those with better outcomes scoring higher? l Faculty rankings of clinical performance –are better ranked trainees scoring higher?

9. OSATS - Validity l Correlation with PGY level l Correlation with other measures

10. Summary of OSATS l A non-patient lab-based platform for testing technical skills which shows: l Good Reliability l Good Validity l Widespread Feasibility l Widespread Applicability

11. Conclusions We can reliably and validly test some of the procedural skills of medicine in a formal, structured manner outside of the clinical setting