The Objective Structured Assessment of Technical Skills (OSATS) Helen M. MacRae, M.D., FRCSC, D.H. Gales Director Technical Skills Centre, University of.

Slides:

Advertisements

Similar presentations

Assessment of Initial Clinical Competence for NP Regulation: The Development of an Objective Structured Clinical Examination Marie Napolitano RN, PhD,

Advertisements

OSCEs Kieran Walsh OSCE Means “objective structured clinical examination” Assesses competence Developed in light of traditional assessment methods.

Randomized prospective evaluation of surgical trainees in 'warm up' simulations of laparoscopic ventral hernia repair Richardson, C MD, St. Hill, C MD,

Implementing Evidence- Based Practice Training in a Scientist-Practitioner Program David DiLillo Director of Clinical Training University of Nebraska-Lincoln.

GCSE Statistics Coursework Sets 1 & 2 February 2013.

S tructured O bjective C linical E xamination SOC E.

Self-Assessment of Surgical Skill Acquisition with Computer-Based Video Training and the Impact on Self-Directed Training Nathan Jowett Vicki LeBlanc George.

Using Rubrics for Evaluating Student Learning. Purpose To review the development of rubrics for the purpose of assessment To share an example of how a.

Erica Schurter and Molly Mead Department of Information Access.

Measuring Student Learning March 10, 2015 Cathy Sanders Director of Assessment.

Educational Outcomes: The Role of Competencies and The Importance of Assessment.

Workshop Curricula review Ghent University Workshop Tempus Curricula Review October 2005.

Using Rubrics for Evaluating Student Learning Office of Assessment and Accreditation Indiana State University.

Updates for Medicine & Surgery Exams in Year 5 Curriculum Retreat 2007 Comparing Outcomes from New and Old Curricula 8 September 2007 Paul B. S. Lai HBP.

EVALUATION OF DR.MOHAMMED AL NAAMI, FRCSC, FACS, M Ed. Using O bjective S tructured C linical E xamination (OSCE)

Training the OSCE Examiners

Purpose of the Standards

Improving Communication in the Emergency Department: The 5 Cs Model of Consultation Educational Soundbites CORD Academic Assembly 2011 San Diego, CA Chad.

Assessment of Communication Skills in Medical Education

End of Shift Milestone Evaluation Forms

Classroom Assessments Checklists, Rating Scales, and Rubrics

Unit 1 – Preparation for Assessment LO 1.1&1.2&1.3.

Assessment tool OSCE AH Mehrparvar,MD Occupational Medicine department Yazd University of Medical Sciences.

Assessing the practical/clinical skills of students: Presented at the RITAL Conference 2010 By: Florence Davidson, Radiography Lecturer, CPUT, GSH campus.

Matthew Roberts, Vanessa Bacal, Mohammed Mahdi, Ethan D. Grober Mount Sinai & Women’s College Hospital, Division of Urology, Department of Surgery, University.

Learner Assessment Win May. What is Assessment? Process of gathering and discussing information from multiple sources to gain deep understanding of what.

Understanding Meaning and Importance of Competency Based Assessment

Student assessment AH Mehrparvar,MD Occupational Medicine department Yazd University of Medical Sciences.

Can a Brief On-line Education Tool Improve Surgical Resident Operative Dictations? A Prospective Evaluation Alicia Kieninger, MD, Yi Wei Zhang, MD, Anna.

Meaningful Evaluation: Framework, Process, Impact Inis Jane Bardella, M.D., FAAFP Associate Dean for Faculty Development and Global Health Initiatives.

Dr. Paul Townsend, D.C. Director of Practical Testing, Research & Development NBCE Part IV Practical Examination Duties of a Part IV Examiner Dr. LeRoy.

Training, Supervision and Competency in Gynaecological Surgery Dr Dina Bisson Consultant Obstetrician and Gynaecologist and Training Programme Director.

Developing an Assessment System B. Joyce, PhD 2006.

Programme Objectives Analyze the main components of a competency-based qualification system (e.g., Singapore Workforce Skills) Analyze the process and.

International Diabetes Federation (IDF) East Mediterranean and Middle East Region (EMME) Workshop on Professional Educational Methodology in Diabetes

University Teaching Symposium January 9,  Individual Development and Educational Assessment  Kansas State University  A diagnostic to.

Performance-Based Assessment HPHE 3150 Dr. Ayers.

Feasibility and Value of a Procedural Workshop for Surgery Residents Based on Phase-II of the APDS/ ACS National Skills Curriculum Dimitrios Stefanidis.

BOOT CAMP: Educational Outcomes After Four Successive Years of Preparatory Simulation-based Training at Onset of Internship Gladys L Fernandez MD, Richard.

Does the Rate of Achieving Competency on simulated Laparoscopy Training Programs Predict Actual Skill in Performing Laparoscopic Cholecystectomy? Warren.

Matthew Schill, BS, Debbie Tiemann, RN, Mary Klingensmith, MD, L. Michael Brunt, MD Department of Surgery and Institute for Minimally Invasive Surgery.

Measuring Non-technical Aspects of Surgical Clinician Development in an Otolaryngology Residency Training Program Jennifer J. Shin, M.D. S.M. Michael J.

Great Debate: Simulation A Resident’s Perspective Ernest (Ted) Gomez, MD, MTR PGY-3 Resident Department of Otorhinolaryngology – Head and Neck Surgery.

Training the Trainers Assessing the Learner Progress By Dr Malik Zaben IMET

Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.

Department of Surgery Development of PGY-1 Surgery Preparatory Course Curricula: Identification of Key Curricular Components Mara B. Antonoff MD Jonathan.

30/10/2006 University Leaders Meeting 1 Student Assessment: A Mandatory Requirement For Accreditation Dr. Salwa El-Magoli Chair-Person National Quality.

Investigate Plan Design Create Evaluate (Test it to objective evaluation at each stage of the design cycle) state – describe - explain the problem some.

Monitor and Revise Teaching. ObjectivesObjectives Describe how to monitor teaching List ways to contribute to broader evaluations Explain how to review.

NEW DIRECTIONS IN SURGICAL EDUCATION AND TRAINING Ajit K. Sachdeva, MD, FRCSC, FACS Director, Division of Education American College of Surgeons Challenges.

S tructured O bjective C linical E xamination P ractical.

1 IT/Cybersecurity - ICRDCE Conference Day Aligning Program, Course, and Class Objectives / Outcomes.

Objective Methods for Assessment of Technical Skills in Otolaryngology–Head & Neck Surgery Residents: A Systematic Review Érika MERCIER1, Ségolène CHAGNON-MONARQUE1,

A community of learners improving our world

ST1 Induction Day The MAC Thursday

Matthew S. Holden, Zsuzsanna Keri, Tamas Ungi, Gabor Fichtinger

CHAPTER 3: Practical Measurement Concepts

Objective vs Subjective

Clinical Assessment Dr. H

Improving Surgical Technical Quality

Father Muller Medical College & Hospital, Mangalore, Karnataka.

Competency, Milestones, and Credentialing: Evaluations and Simulation

Using an ‘Oral Board’ exam to assess for EPA 10 in

PART B MFPH RESULTS CHECKLIST

TESTS AND EXAMS TCOP 2014 FACULTY OF ECONOMICS & FINANCE

Revised Higher Course Event

CHAPTER 5: Selecting and Using Assessment Instruments

Laboratory-based vascular anastomosis training: A randomized controlled trial evaluating the effects of bench model fidelity and level of training on.

Deborah Anne Banker Committee Chair

Purpose of this presentation

Presentation transcript:

The Objective Structured Assessment of Technical Skills (OSATS) Helen M. MacRae, M.D., FRCSC, D.H. Gales Director Technical Skills Centre, University of Toronto

Certifying competence of new trainees Certifying Competence in New Technology (for example laparoscopic surgery or robotic surgery) Assessing the impact of interventions The Need for a Reliable and Valid Measure

The OSATS l A multi-station “bell-ringer” examination l Segments of 6 to 8 procedures l Performed on “bench models” simulating relevant anatomical structures l Knowledgeable OR nurses assisting

The OSATS (continued) l 15 minutes per procedure l Candidates move from station to station l Each station marked by qualified surgeon l Examiner uses two marking systems l Task-specific checklists l Global rating scales

Vascular Anastamosis Station Sample of Checklist Scoring (5 of 24) 1.Selects appropriate suture 2.Avoids excessive trauma / handling of artery with instruments 3.Guides suture down to desired position following bites 4.Uses correct needle angle >80% time. 5.Follows curve of needle on entrance / exit > 80% time

Vascular Anastamosis Station Sample of Global Rating (3 of 5) Respect for tissue Consistently handled tissue appropriately with minimal damage to tissue Frequently caused damage by inappropriate use of instruments Careful handling of tissue but occasionally caused inadvertent damage Time and motion Many unnecessary moves Efficient time/motion but some unnecessary moves Clear economy of movement and maximum efficiency Flow of Operation Frequently stopped operating and seemed unsure of next move Demonstrated some forward planning with reasonable progression of procedure Obviously planned course of operation with effortless flow from one move to the next

OSATS - Reliability l Internal consistency (across stations) l Inter-rater reliability (within stations)

Validity Measures l Clinical PGY level –are more senior trainees scoring higher? l Live animal version of exam –are better “live model” operators scoring higher? l Evaluation of final product –are those with better outcomes scoring higher? l Faculty rankings of clinical performance –are better ranked trainees scoring higher?

OSATS - Validity l Correlation with PGY level l Correlation with other measures

Summary of OSATS l A non-patient lab-based platform for testing technical skills which shows: l Good Reliability l Good Validity l Widespread Feasibility l Widespread Applicability

Conclusions We can reliably and validly test some of the procedural skills of medicine in a formal, structured manner outside of the clinical setting