It’s more than just competency-based education: It’s about radical curricular change Richard K. Reznick, MD, MEd, FRCSC, FACS Dean, Faculty of Health.

It’s more than just competency-based education: It’s about radical curricular change
Richard K. Reznick, MD, MEd, FRCSC, FACS Dean, Faculty of Health Sciences, Queen’s University CEO Southeastern Ontario Academic Medical Organization

Fellowship in Surgical Education

The story really starts here: And it’s a young and short story
Brent Laura Jeramie Sebastian Kevin Matt Mollon Kember Larouche Tomescu Koo Tsuji

With a long prologue!

And the prologue has been brewing for three decades
Libby Zion Revisited

Training is too long! Undergraduate 4 years Master’s Degree 2 years
Medical School Residency 4-5 years Fellowship 2-3 years Research Training 2-5 years

It’s a Problem: The House Officer is Dead
United States 80 hours Canada 72 hours Ireland 49 hours Australia 68-75 hours New Zealand Taiwan 70 hours Brazil 60 hours Great Britain Switzerland 50 hours The Netherlands 48 hours Denmark 42 hours Germany 40 hours Belgium Sweden France 35 hours Norway

Transatlantic comparison of the competence of surgeons at the start of their professional career
Domain T (40) P CIP Cognitive .52 .53 0.18 .856 PAME Judgment .85 .79 3.90 <.01 OSATS Technical .78 .75 1.33 0.19 Schiven, Reznick, Ten Cate, Grantcharov, Regehr, Satterthwaite, Thijssen, MacRae British Journal of Surgery 2010; 97: 443–449

OSATS Scores Examiner Bias

18/121 A procedures > 10 83/121 A procedures < 5 31/121 A procedures < 1
63/121 mode = 0 Bell RH et al Annals of Surgery: 249: : 2009

Is there such a thing as a surgical principle: Or if you take out a right colon, can you take out a left colon? Transfer, the process of using knowledge or skills acquired in one context in a new or varied context, has long been the topic of spirited debate in the research community

Detterman “educators must assume that transfer is as rare as volcanic eruptions and should operate as though it is virtually unachievable as an instructional goal”

Average Number of Passes Per Year

The message from Strauss, Otto and Starkes
“ we argue that the difference between expert performers and normal adults reflects a life-long period of deliberate effort to improve performance in a specific domain.” K. Anders Ericsson et al

There is a Tension

What is this beast: Competency-based Medical Education
The solution to all problems in PGME The genesis of problems in PGME “the competencies” Competence versus Time Curricular tweaking Radical reform

Competency-based Surgical Training not so new!
Over 500 years old No matter of person shall employ are said crafts unless he be worthy and expert in all subjects belonging to the said crafts, and he be diligently and advisedly examined Charter of the Royal College of Surgeons of Edinburgh, 1512

Medical educators now begin with the end in mind and focus on the competencies needed by graduates of medical education needed to meet those they serve, and effect the outcomes of healthcare Frank J and Danoff D: Medical Teacher, 2007: 29: 642

CBME Competency-based medical education has been criticized for being reductionistic, that is focusing on atomistic skills and failing to capture the essence of professional activities as manifested by complex integrated capabilities Swing SR Medical Teacher, 2010: 32: 663

Competency-based Education May not the Holy Grail (necessary, but not sufficient)
“Understanding, I have been urging, is active, is an engagement, is a form of agency, and is a form of self-expression. It is an expression of individuality; and it strikes against the conventionalism of competence, insofar as competency is a fulfillment of standards determined in advance”. Ronald Barnett

Competency-based Education: May not be the Holy Grail
“…tasks are broken down into their sub-units and the assessee is graded according to successful, serial completion of each and these stages. Such a competency construct is a learning paradigm, it is not the same as competence, which is a step on the road to professional excellence”. Talbot M, Med Educ 2004: 38:587

Meet Joe Hofstee Angoff Nedelsky Contrasting Groups Ebel The marginally competent, acceptably capable, bare pass, but a pass

Raising the Bar An explicitly stated mission of competency-based surgical residency is not only that the bar will continue to be set very high but that deliberate efforts (must) be made to raise it even further Bhatti and Cummings 2007: Academic Medicine: 82:569

And it begs the question. Are we striving for competence, or virtuosity?

We are at a precipice

A Six Point Plan to Restructure and Dramatically Shorten Training
1. Tackle politics and finances head on 2. Achieving the competencies through modular-based training 3. Dramatically accelerate the pace of procedural skill acquisition 4. Diminish wasted time 5. Incorporate meaningful assessment into day-to- day activities 6. Develop and promote a culture of collegiality

MOHLTC funding signed September 10, 2009

2. Modular-based training linked to specific learning objectives
1. Specific modular objectives form the basis of learning contracts 2. Objectives must be comprehensive 3. Objectives are learner-based and streamlined to ultimate career 4. Deploy web-based curricular materials, access to point of care (wireless) information, optimize VR training, and a focus on providing data for evidence based decision making

21 Modules to teach the competencies
Perception and Judgment Communication and Professionalism Compassion and Determination Dexterity

Modular-based training linked to specific learning objectives They finish when they finish!
1. Specific modular objectives form the basis of learning contracts 2. Objectives must be comprehensive 3. Objectives are learner-based and streamlined to ultimate career 4. Deploy modern technology, such as web-based curricular materials, access to point of care (wireless) information, optimize VR training, and a focus on providing data for evidence based decision making

This is not about turning professional training into vocational training

How surgeons disclose errors (we’re not doing so well)
Used the word error or mistake 57% Took responsibility for error 65% Offered verbal apology 47% Acknowledged patient’s emotions 55% Discussed future prevention 8% Offered second opinion 20% Chan D, Gallagher T, Reznick R, Levinson W. Surgery: 138: :2006

3. Dramatically accelerate the pace of procedural skill acquisition
Developing a “pre-program” of basic skills focusing on fundamental technical skills Dramatically ramping up skills laboratory (ex vivo) practice, using virtual reality, cadavers, surrogate tissue and inanimate training models Developing programs of structured and deliberate practice Maximizing each “real” patient experience by the use of pre-operative technical sessions, by videotape review of self and experts, and by debriefing sessions

Starting to operate on the decline

So we need to find ways to train specific skills benchmarked to proficiency criteria: AKA You can’t enter the O.R. until you are prepared

Surgeon in Africa My Simulator FLS Box Simulator in Africa

Skill Transfer: Laparoscopic Surgery Grantcharov et al
Skill Transfer: Laparoscopic Surgery Grantcharov et al., Randomized clinical trial of VR simulation for laparoscopic skills training, BJS 2004 Randomized, controlled, blinded design Pre- and post-tests in OR Gallbladder dissection of from liver Assessment using modified global ratings scales

Better economy of motion
Less errors Better economy of motion Faster

Basic Lap Chole Lecture Anatomy Technique Videos 20 Surgery Residents
Randomization Simulation training to proficiency criteria How were residents assessed intraoperatively? Intra-operative assessment Video-recorded Reviewed by blinded observer 42

Second generation haptic VR Simulators
Improvements in computer and graphics technology have lead to more advanced simulators More realistic graphics Haptic or force feedback Ability to perform full procedures Computer chip picture Grapshics picture Simulation picture 43

So the group at McGill drew on the important principles of OSATS, and developed the GOALS assessment
Which stands for … It tests the 5 domains that you see here Each is rated on a 5 point scale with descriptive anchors at points 1,3, and 5 You should also notice from this slide, that although I’m currently living in Montreal, this picture tells you where my true allegiance lies Based on the global rating scale developed by Dr. Reznik for open surgery 44

Mean GOALS Score Blinded Reviewer
*p=.001 45

4. Diminish Wasted Time Eliminating or minimizing time wasted secondary to a hierarchical model Eliminating or minimizing time wasted doing non-educational activities 3. Increasing support services, increasing nurse autonomy, rationalizing calls, and optimizing technological solutions to service problems 4. Critically assessing the need for and the context of night call 5. Seriously addressing the issue of sleep deprivation

Clinical Effects of Sleep Deprivation
STUDY TASK Friedman et al., 1973 EKG interpretation Christensen et al, 1977 X-ray Interpretation Poulton et al., 1978 Review lab results Engel et al., 1978 Simulated patients Reznick and Folse, 1987 Suturing task Storer et al., 1989 ETT, IV insertion Lingenfelser et al., 1994 Haynes et al, 2003 Surgery outcomes Howard et al., 2003 Anesthesia simulator Ellman et al., 2004 Stone et al., 2004 AMSITE examination Uchal, 2005 Laparoscopic simulator Jakubowicz, 2005 Simulated endoscopic sinus surgery STUDY TASK Goldman et al., 1972 Videotape:performance Friedman et al., 1973 EKG interpretation Beatty et al., 1977 Patient monitoring Storer et al., 1989 arterial catheterization Denisco at al., 1987 Bertram, 1988 Review resident notes Jaques at al., 1990 In-training examination Gottleib et al., 1991 Medication errors, LOS Mann and Danz, 1993 X-ray Interpretation Lingenfelser et al., 1994 Taffinder et al., 1998 VR lap. simulator Grantcharov et al., 2001 MIST-VR Landrigen et al., 2004 Medical errors Jensen et al., 2004 laparoscopic simulator Eastridge, 2004 This a list of all papers that I can find assessing the effects of sleep deprivation on clinically-relevant tasks (as opposed to standardized neuropsychological tests) across all specialties (note: the Ellman study is the only staff outcomes study). Studies in chronological order and reported with chart on left being negative studies and the one on tight are positive studies. Surgical studies are highlighted in red. This is the same info as the last 2 slides but crammed onto one slide. 48

Epworth Sleepiness Scale
How likely would you be to doze off in the following situations? Surgery (SD) Non - Surgery (SD) Sitting and Reading 2.22 (.84)** 1.53 (.97) Watching TV 2.13 (.93)** 1.39 (.98) Sitting inactive in a public place 1.74 (.89)** 0.91 (.92) Passenger in a car for 1 hour 2.03 (.93)* 1.72 (1.06) Lying down to rest 2.41 (.85)* 2.14 (.92) Sitting and talking to someone .52 (.70)** 0.16 (.42) Sitting quietly after lunch without alcohol 1.21 (.94)* 0.92 (.92) In a car stopped for a few minutes in traffic .56 (.85) 0.39 (.76) TOTAL EPWORTH SCORE 12.84 (4.7)** 9.17 (4.8) Included in the questionnaire was the Eppworth Sleepiness Scale. The ESS is a well-validated scale that requires individuals to rate their likelihood of falling asleep in a number of common scenarios where 0 = no chance and 3 = high chance. It has been shown to correlate with objective measures of sleepiness. In our survey, the ESS was used as a reflection of subjective sleepiness at the time participants were rotating through their own specialty. The results show that surgery residents report being significantly more sleepy on their own specialty rotations compared to non-surgical residents. ** p < 0.01 * p < 0.05 Woodrow SI, Park J, Murray BJ, Wang C, Bernstein M, Reznick RK, Hamstra SJ: Differences in the Perceived Impact of Sleep Deprivation among Surgical and Non-Surgical Residents. Medical Education 2008: 42 (5): pp 49

Sleep Deprivation Impact (SDI) Scale
Surgery (SD) Non-Surgery (SD) Less concentration during patient rounds 3.76 (1.11)** 4.54 (0.78) Unable to retain new patient information 3.67 (1.03)** 4.32 (0.80) Less effective clinical judgment 3.70 (1.12)** 4.18 (0.91) Inappropriate discharge orders 2.93 (1.28)** 3.75 (1.09) More irritable 4.20 (0.92)** 4.50 (0.79) Decreased empathy towards patients 3.55 (1.34)** 4.08 (1.04) Poor interaction with colleagues 3.76 (1.04)* 4.09 (1.00) Poor communication with patients 3.62 (1.20)** 4.16 (0.92) Decreased motivation to learn 4.07 (0.99)** 4.56 (0.75) Diminished desire to teach 4.16 (0.89)* 4.51 (0.84) Slower 4.14 (0.80)** Unsafe Driving 3.59 (1.21)* 3.95 (1.14) TOTAL SDI SCORE 45.2 (8.54)** 51.5 (5.97) Patient Care Professionalism Next residents were asked how much they agreed with a series of statements suggesting a detrimental effect of sleep deprivation on different aspects of performance in which a higher score reflects greater concern. clinical care tasks such as writing appropriate discharge orders and professional behaviours such as communicating with patients. There was also a general category that for example asked residents whether they felt unsafe driving home in this condition. Of the 18 questions we asked them we ultimately identified 12 that applied to both groups of residents and whose score we totalled into what we call the sleep deprivation impact scale. Responses for both individual items and total score is reported in this figure. Overall the internal consistency of the scale using chronbach’s alpha was 0.89. Although both groups of residents generally agreed that these different aspects of performance were affected by sleep deprivation, surgical residents consistently agreed less strongly across all 12 items as well as the summed score suggesting that they perceived their clinical performance as less impacted by sleep deprivation then did non-surgical residents. We also found that 15.4% of the variance in this total score was accounted for by group suggesting a moderate effect size of specialty as a factor in SDI score determination. 1 = strongly disagree, 2 = somewhat disagree, 3 = no opinion, 4 = somewhat agree, 5 = strongly agree alpha = 0.89, eta2 = .154 ** p < 0.01 * p < 0.05 50

The surgical “machismo”
Surgery residents work longer hours Surgery residents perceive less impact of sleep deprivation on their own clinical performance Surgery residents report increased sleepiness ESS Surgery: ESS Non-surgery: 9.17 Surgery residents at the U of T work longer hours than other groups of residents. This is likely not a surprising finding to anyone in this room. Associated with these increased hours is a higher level of sleepiness. But how sleepy are they? Research from the field of sleep medicine suggest that the average Epworth score for a normal population 8 with a score of 11 being considered pathologic. Patients with sleep apnea and narcolepsy score and 17.5 respectively. These patients have been found to be at increased risk for events threatening their personal safety such a motor vehicle collisions as a result of their sleepiness. Although recent studies of resident physician have suggested that they too might be at higher risk for both medical errors and MVCs when sleep deprived, our results suggest on a day-to-day basis surgery residents may be more at risk than other residents Epworth Score Normal 8 (<11) Obstructive Sleep Apnea 11.7 Narcolepsy 17.5 51

5. Benchmarking towards Excellence: Daily Assessment is the Key
Rigorous, reliable and regular assessment of the multitude of competencies with a special focus on technical achievement Liberal use of formative assessment Linking evaluation instruments to goals, objectives and desired competencies Training the evaluators Focusing effort on performance-based evaluation systems, such as PAME, OSATS, Using a diverse array of assessors, including self, other health professionals, patients, peers, and faculty

Assessment Toolbox PAME OSATS ICSAD VR Metrics OSCES Orals MCQ 360
ITER

OSATS our first small step
Reznick RK, Regehr G, MacRae H, Martin J, McCulloch W: Am J Surg 1997: 173: pp

Summary of OSATS Experiments: Nine Years of Research
Inter-rater and Inter-station reliability Construct valid (PGY6>5>4>3>2>1) Concurrent validity (r=.81 with faculty) Not just Toronto (across Ontario) Central preparation and peripheral delivery (Chicago and Los Angeles) Used now in RACS, RCS and GMC

Name: Dr. Demitrios Athenas DOB Oct 10, 1960 Medical School University of Patras Residency G.S. 5 years, Athens University Practice 2 years community teaching hospital, Athens, Greece ECFMG PASSED TOFEL/TSE REFERENCES Positive letter from an elective rotation in Houston ABSITE 30th percentile for PGY-5 DECISION: ????

It’s about knowing where they stand (OSATS scores across years of training)
Candidate 1 > Candidate 2 > Candidate 3 > Candidate 4 > Candidate 5 >

6. Promote a Culture of Collegiality: Improve Communication through a Culture Change
Embrace a multi-professional and patient centered approach to patient care 2. Focus on patient safety Focus on professionalism as a distinct enabler of learning

The WHO Checklist was piloted in 8 cities
EURO EMRO PAHO I London, UK Amman, Jordan Toronto, Canada WPRO I Manila, Philippines PAHO II Seattle, USA WPRO II Auckland, NZ AFRO Ifakara, Tanzania SEARO New Delhi, India

Results

Don Berwick , IHI "I cannot recall a clinical care innovation
in the past 30 years that has shown results of the magnitude demonstrated by the surgical checklist. This is a change ready right now for adoption by every hospital that performs surgery." Don Berwick , IHI

If we want to avoid everyone’s favorite sport

New Training Model We haven’t proved anything yet
Peter Ferguson Bill Kraemer New Training Model We haven’t proved anything yet Ben Alman July 2009 and July supra-numery residents Division of Orthpaedic Surgery Estimate $200,000 per year, per program Competency based, not time based Thank you Royal College

CBC Module 2: Hip fracture Cases logged for regular 3m and 6m junior rotations vs CBC Module

CBC Residents outperform regular Residents on 9/9 technical skills

CBC residents report more self-confidence

OLD WAYS WON’T WORK It’s more than just competency-based
Challenge curricular models Ramp up procedural skill acquisition Rigorously assess educational outcomes Promote team care

Orthopaedic Competency Based Curriculum (CBC) Progress Report
July 27 – August – Module 1 – Technical Skills (Module Leader – Dr Oleg Safir) Half days in SSC, half days in clinic/OR Core elements Gowning Prepping & draping Soft tissue handling Power tools Suturing Casting

The Dutch Experiment Based on CanMEDS Modular Structure
Assessment strategy to be formative and portfolio-based QA Measures (faculty development and educational quality control) Scheele and colleagues : Medical Teacher; 01/01/08

The Dutch Experiment STEP I Structuring the Content into Themes
Anatomic Region (Orthopaedics) Context (Psychiatry) Symptoms (General Surgery) Context and Symptoms (Ob Scheele and colleagues : Medical Teacher; 01/01/08

The Dutch Experiment-STEP II Choosing Information-Rich Activities (Entrustable Professional Activities) A task of high importance for daily practice A high risk or error prone task A task that is exemplary of (a number) of Can MEDS competencies) TenCate O.2005: Med Educ: 39:1176

The Dutch Experiment-STEP III Using Assessment toolbox to match EPA’s to CanMEDS Roles
Forceps Obstetric Drills C Section Expert DOP, MCQ, Vol Oral OSCE, MCQ DOP Communicator MCQ, OSCE Collaborator Manager Advocate Scholar Critical appraisal Professional

“There is nothing more useless than a merely well informed man”
Albert North Whitehead

S KNOWS HOW (Competence) DOES SHOWS HOW KNOWS

The handshake of the world's oldest surgeon St
The handshake of the world's oldest surgeon St. Petersburg's Fyodor Uglov, 99, who was still doing operations last spring, is as firm as that of a 30-year-old athlete.

Paraphrasing Piaget True learning occurs when we are deeply perturbed; when our world view is so challenged that we must reject our current conceptualizations, and we then must (painfully) rebuild our knowledge structures

Let me be so bold as to propose to you a new model of training
An aspiring iconoclast AKA surgical educator gone mad

Harvard sweetens reward for doctors who teach
Tackle the Finances and Politics What percent of your GNP is spent on health care? What percent of your GNP is spent on health training? Harvard sweetens reward for doctors who teach 3 hospitals agree to double budget By Liz Kowalczyk, Globe Staff | March 9, 2007 Harvard Medical School will increase by millions of dollars a year its payments to doctors for teaching students, a recognition of how difficult it has become to persuade busy physicians to devote time to educating the next generation of care givers. $100 per hour

Manual and video of fiberoptic intubation
Familiarization with bronchoscope Model training (n=12) Didactic training (n=12) Pre-test of fiberoptic manipulation ability Fiberoptic intubation in OR on healthy, anesthetized and paralyzed female patients with “easy” airways

Skills Curricula Some skills can be taught with low tech simulation
And so a natural corollary has been to include a curriculum within the training program of residents going regularly to the skills lab for training. At the University of Toronto, this involves weekly sessions of 2 hours for PGY1’s. Despite this however, some doubt remains as to the true effectiveness that this curriculum has.

Successful Tracheal Intubation
* % of Subjects * p < 0.01

And we need to properly teach new technologies

Schema Distributed Test Massed Transfer (4 weeks) & (One day)
Moulton CA, Dubrowski A, MacRae H, Graham B, Grober E, Reznick RK: Annals of Surgery 2006: 244: pp

Distributed practice Massed practice
CME courses: 2 days, 25 hours! Does it makes sense? Distributed practice 1 2 3 4 R T 1 week rest 1 month rest 1 hour block Massed practice 1 2 3 4 R 1 month rest 1 hour block

* * *p <.05 Group Inter-rater reliability = 0.89
Microsurgical drill Rat Group 30 Distributed Massed * * *p <.05 Inter-rater reliability = 0.89 20 Global Ratings (max. 25) 10 Pre-test Post-test Retention Transfer Test

4. Diminish Wasted Time Eliminating or minimizing time wasted secondary to a hierarchical model Eliminating or minimizing time wasted doing non-educational activities 3. Increasing support services, increasing nurse autonomy, rationalizing calls, and optimizing technological solutions to service problems 4. Critically assessing the need for and the context of night call 5. Seriously addressing the issue of sleep deprivation

Clinical Effects of Sleep Deprivation
STUDY TASK Friedman et al., 1973 EKG interpretation Christensen et al, 1977 X-ray Interpretation Poulton et al., 1978 Review lab results Engel et al., 1978 Simulated patients Reznick and Folse, 1987 Suturing task Storer et al., 1989 ETT, IV insertion Lingenfelser et al., 1994 Haynes et al, 2003 Surgery outcomes Howard et al., 2003 Anesthesia simulator Ellman et al., 2004 Stone et al., 2004 AMSITE examination Uchal, 2005 Laparoscopic simulator Jakubowicz, 2005 Simulated endoscopic sinus surgery STUDY TASK Goldman et al., 1972 Videotape:performance Friedman et al., 1973 EKG interpretation Beatty et al., 1977 Patient monitoring Storer et al., 1989 arterial catheterization Denisco at al., 1987 Bertram, 1988 Review resident notes Jaques at al., 1990 In-training examination Gottleib et al., 1991 Medication errors, LOS Mann and Danz, 1993 X-ray Interpretation Lingenfelser et al., 1994 Taffinder et al., 1998 VR lap. simulator Grantcharov et al., 2001 MIST-VR Landrigen et al., 2004 Medical errors Jensen et al., 2004 laparoscopic simulator Eastridge, 2004 This a list of all papers that I can find assessing the effects of sleep deprivation on clinically-relevant tasks (as opposed to standardized neuropsychological tests) across all specialties (note: the Ellman study is the only staff outcomes study). Studies in chronological order and reported with chart on left being negative studies and the one on tight are positive studies. Surgical studies are highlighted in red. This is the same info as the last 2 slides but crammed onto one slide. 94

Epworth Sleepiness Scale
How likely would you be to doze off in the following situations? Surgery (SD) Non - Surgery (SD) Sitting and Reading 2.22 (.84)** 1.53 (.97) Watching TV 2.13 (.93)** 1.39 (.98) Sitting inactive in a public place 1.74 (.89)** 0.91 (.92) Passenger in a car for 1 hour 2.03 (.93)* 1.72 (1.06) Lying down to rest 2.41 (.85)* 2.14 (.92) Sitting and talking to someone .52 (.70)** 0.16 (.42) Sitting quietly after lunch without alcohol 1.21 (.94)* 0.92 (.92) In a car stopped for a few minutes in traffic .56 (.85) 0.39 (.76) TOTAL EPWORTH SCORE 12.84 (4.7)** 9.17 (4.8) Included in the questionnaire was the Eppworth Sleepiness Scale. The ESS is a well-validated scale that requires individuals to rate their likelihood of falling asleep in a number of common scenarios where 0 = no chance and 3 = high chance. It has been shown to correlate with objective measures of sleepiness. In our survey, the ESS was used as a reflection of subjective sleepiness at the time participants were rotating through their own specialty. The results show that surgery residents report being significantly more sleepy on their own specialty rotations compared to non-surgical residents. ** p < 0.01 * p < 0.05 Woodrow SI, Park J, Murray BJ, Wang C, Bernstein M, Reznick RK, Hamstra SJ: Differences in the Perceived Impact of Sleep Deprivation among Surgical and Non-Surgical Residents. Medical Education 2008: 42 (5): pp 95

Sleep Deprivation Impact (SDI) Scale
Surgery (SD) Non-Surgery (SD) Less concentration during patient rounds 3.76 (1.11)** 4.54 (0.78) Unable to retain new patient information 3.67 (1.03)** 4.32 (0.80) Less effective clinical judgment 3.70 (1.12)** 4.18 (0.91) Inappropriate discharge orders 2.93 (1.28)** 3.75 (1.09) More irritable 4.20 (0.92)** 4.50 (0.79) Decreased empathy towards patients 3.55 (1.34)** 4.08 (1.04) Poor interaction with colleagues 3.76 (1.04)* 4.09 (1.00) Poor communication with patients 3.62 (1.20)** 4.16 (0.92) Decreased motivation to learn 4.07 (0.99)** 4.56 (0.75) Diminished desire to teach 4.16 (0.89)* 4.51 (0.84) Slower 4.14 (0.80)** Unsafe Driving 3.59 (1.21)* 3.95 (1.14) TOTAL SDI SCORE 45.2 (8.54)** 51.5 (5.97) Patient Care Professionalism Next residents were asked how much they agreed with a series of statements suggesting a detrimental effect of sleep deprivation on different aspects of performance in which a higher score reflects greater concern. clinical care tasks such as writing appropriate discharge orders and professional behaviours such as communicating with patients. There was also a general category that for example asked residents whether they felt unsafe driving home in this condition. Of the 18 questions we asked them we ultimately identified 12 that applied to both groups of residents and whose score we totalled into what we call the sleep deprivation impact scale. Responses for both individual items and total score is reported in this figure. Overall the internal consistency of the scale using chronbach’s alpha was 0.89. Although both groups of residents generally agreed that these different aspects of performance were affected by sleep deprivation, surgical residents consistently agreed less strongly across all 12 items as well as the summed score suggesting that they perceived their clinical performance as less impacted by sleep deprivation then did non-surgical residents. We also found that 15.4% of the variance in this total score was accounted for by group suggesting a moderate effect size of specialty as a factor in SDI score determination. 1 = strongly disagree, 2 = somewhat disagree, 3 = no opinion, 4 = somewhat agree, 5 = strongly agree alpha = 0.89, eta2 = .154 ** p < 0.01 * p < 0.05 96

The surgical “machismo”
Surgery residents work longer hours Surgery residents perceive less impact of sleep deprivation on their own clinical performance Surgery residents report increased sleepiness ESS Surgery: ESS Non-surgery: 9.17 Surgery residents at the U of T work longer hours than other groups of residents. This is likely not a surprising finding to anyone in this room. Associated with these increased hours is a higher level of sleepiness. But how sleepy are they? Research from the field of sleep medicine suggest that the average Epworth score for a normal population 8 with a score of 11 being considered pathologic. Patients with sleep apnea and narcolepsy score and 17.5 respectively. These patients have been found to be at increased risk for events threatening their personal safety such a motor vehicle collisions as a result of their sleepiness. Although recent studies of resident physician have suggested that they too might be at higher risk for both medical errors and MVCs when sleep deprived, our results suggest on a day-to-day basis surgery residents may be more at risk than other residents Epworth Score Normal 8 (<11) Obstructive Sleep Apnea 11.7 Narcolepsy 17.5 97

It’s not necessarily axiomatic that all things majestic have an strong underlying network, that all things meager lack an extensive foundation

5. Benchmarking towards Excellence: Daily Assessment is the Key
Rigorous, reliable and regular assessment of the multitude of competencies with a special focus on technical achievement Liberal use of formative assessment Linking evaluation instruments to goals, objectives and desired competencies Training the evaluators Focusing effort on performance-based evaluation systems, such as PAME, OSATS, Using a diverse array of assessors, including self, other health professionals, patients, peers, and faculty

Assessment Toolbox PAME OSATS ICSAD VR Metrics OSCES Orals MCQ 360
ITER

6. Develop and Promote a Culture of Collegiality
Embrace a multi-professional and patient centered approach to patient care 2. Focuses on patient safety Focus on professionalism as a distinct enabler of learning

Sharp Contrasts F-16 Fighter Pilot Training
Pre-simulator sessions Simulator sessions Debriefing of simulator sessions Pre-flight review 1 hour flight 6 hours of one on one debriefing

VR Training- A Part of our Future?
Estimates suggest 4B industry by 2013* Recent FDA recommendation that VR training precede carotid stenting Frost & Sullivan study suggest 6 mo ROI** 1:0.2 versus 1:0.8 * US Market for VR in Surgery and Imaging MedMarket Dilligence LLC (2005) **Return of Investment Study for Medical Simulation Training (Frost & Sullivan 2004)

Lessons for England Boy, are we presumptuous
The pyramid being abandoned, now shorten training Time based training is antiquated An ST1 can and should take out a colon See one, simulate 100, do one Break down the hierarchy

Shorten Training Medical school 4-6 years (could be 3)
Foundation 2 years (eliminate) ST Core 2 -3 years (eliminate) ST Higher 6 years CBC ~ 4 years Research 2-3 years only for academics Fellowship 1 year leave as is

You are accepting the best and the brightest
GPA>3.6 MCAT>11 Awards Sports Music Volunteerism Parted the Red Sea

Skills Labs: An Emerging Reality for Every Training Program

Attentional capacity threshold
Technical skills Depth perception Environment Operative Judgment Additional Learning Technical skills Depth perception Environment Operative Judgment Additional Learning Attentional capacity threshold Trained Novice Novice

Technical skills Depth perception Environment Operative Judgment Additional Learning Attentional capacity threshold Technical skills Depth perception Environment Operative Judgment Additional Learning Trained Novice Novice

50 PGY 1-3 surgical residents
Training Phase 50 PGY 1-3 surgical residents n = 21 n = 10 n= 19 Control High-Fidelity Low-Fidelity Didactic Silicone Tube Live Rat-Vas Deferens

Results -Post-Test Chi-square: p=0.048

Playing well in the sandbox

A new concept of graded responsibility
“sign-off” on cognitive elements Review of 1000’s of intra-operative situations and controversies Accepted algorithms Obligatory web-cast during first year of “independence” Mandatory second opinions for defined conditions

We are accepting he best and the brightest

Essentials of Surgical Training
Clearly defined selection criteria An efficient, fair and transparent selection process A “fit-for-purpose learning environment Appropriate access for trainees to clinical practice Trained motivated trainers An integrated progression of learning Effective and objective assessment of competency progression Creating an attitude that encourages life-long learning Thomas WEG: ANZ J Surg: 79: (2009)

What do team members hear?
Surgeons: “Surgeons are the only ones who really care if all the patients get done – the nurse is a shift worker.” Anesthetists: “Surgeons have a distorted sense of operative time – it’s all about their time.” Nurses: “Surgeons are just worried about money – if a case gets cancelled it’s less money for them.”

6. Develop and promote a culture of collegiality 7. If it ain’t broke, don’t fix it 8. Address potential deleterious byproducts

This is an example of a bench model for burr hole insertion.
125

Do sub-specialists need generalist training?
MSKCC Ground Rounds Do sub-specialists need generalist training? Richard K. Reznick, MD, MEd, FRCSC, FACS R.S. McLaughlin Professor and Chair Department of Surgery University of Toronto

There is a Tension Call for Generalism Need for Focused Training
Reduced Work Week and Sleep Deprivation Training is Already Too Long Patient Safety and Public Expectation Need for Graduated Independence I Did It! Marital Disharmony, Substance Abuse, Depression

We’ve got a big problem! Current Forces General Effect
Effect on Training Fiscal Restraint Pressure for speed Fewer opportunities Growth of Knowledge Increasingly complex cases Fewer opportunities Societal Influences Medical error Patient safety The O.R. Itself Increasingly stressful

SCORE Competency-based education Promoting simulation and skills labs
Common curricula Assessment of competence

Assessment toolbox for technical skill is rich
OSATS MISTELS Error assessment Product analysis ICSAD VR Metrics Video-tape review Intra-op review ITER Patient outcomes

Models of Surgical Education
Vs.

Is there such a thing as a surgical principle: Or if you can take out a left colon can you take out a right colon? Transfer, the process of using knowledge or skills acquired in one context in a new or varied context, has long been the topic of spirited debate in the research community

How much gain? Plate Fixation - Global Score
PGY PGY 1 (course) PGY 1 (control) Pretest Posttest Delay Test

No gain for similar task! K-Wire Fixation - Global Score
PGY 5 PGY 1 (course) PGY 1 (control) Experiment replicated with upper and lower limb casting Pretest Posttest Delay Test

Detterman “educators must assume that transfer is as rare as volcanic eruptions and should operate as though it is virtually unachievable as an instructional goal”

Generalizations about transfer
Transfer of knowledge and procedures occurs far less often than educators and educational researchers would hope (The literature is replete with testimonials to the relative ineffectiveness of “training for transfer” studies) Transfer is an essential aspect of competent performance in any complex domain (There is ample evidence in the literature that highly competent learners, experts regularly perform tasks for which they have never been directly trained)

Generalizations about transfer
Transfer is more apt to occur when the learning environment is intentionally orchestrated to encourage cross-situation and cross-domain transfer and rewards such efforts (explicitness of instructional content, attention to basic problem-solving procedures or strategies, multiple and varied opportunities for transfer, and mindful reflection and abstraction of the materials to be taught)

Identifying the seeds of transfer
Stages of domain learning (Novice learners have limited and fragmented knowledge of the domain and their interests are relegated more by the aspects of the immediate situation or context) Domain-specific analogical reasoning (willingness to engage in effortful processing) Profiles of academic achievement (domain knowledge, motivation, and general strategic ability are three needed ingredients)

Recognizing potential instructional barriers to transfer
Mentioning versus teaching (teachers invested far too little time exploring important knowledge and strategies richly and deeply) Instructional enabling (given high marks if they perform well on relatively undemanding tasks, display compliance, and behave as good citizens in the classroom) Decontextualized and disconnected instruction (Learners need a well articulated map for what could or should transpire)

Nurturing transfer Stage a three-pronged attack
focus on the learner, the task and the context Strive for principled understanding central concepts within this domain can function as the source or grounding for analogical reasoning Model and reward analogical thinking

Anatomy of a Surgeon is Complex
Skill transfer Communication and Professionalism Compassion and Determination Dexterity

Concepts of Professionalism
Two Departments of Surgery 43 semi-structured interviews 25 residents, 18 faculty 700 pages of transcript

Skill transfer Communication and Professionalism Compassion and Determination Dexterity

What can we learn from others? Axel Strauss, Jim Otto, Janet Starkes
Began playing the violin at 4 and won his first competition at age 8 First German artist to win the Naumburg Violin Award in 1998 Described by an opponent as “meaner than a bear and tougher than an old boot Professor and Chair, Department of Kinesiology, McMaster University,

Message from Jim Otto Determination Grit Playing with adversity
Desire to be the best

Skill transfer Communication and Professionalism Compassion and Determination Dexterity and Technical Skill Development

Skills Labs: An Emerging Reality for Every Training Program

50 PGY 1-3 surgical residents
Training Phase 50 PGY 1-3 surgical residents n = 21 n = 10 n= 19 Control High-Fidelity Low-Fidelity Didactic Silicone Tube Live Rat-Vas Deferens

Manual and video of fiberoptic intubation
Familiarization with bronchoscope Model training (n=12) Didactic training (n=12) Pre-test of fiberoptic manipulation ability Fiberoptic intubation in OR on healthy, anesthetized and paralyzed female patients with “easy” airways

Skills Curricula Procedural skills can be taught in the laboratory
And so a natural corollary has been to include a curriculum within the training program of residents going regularly to the skills lab for training. At the University of Toronto, this involves weekly sessions of 2 hours for PGY1’s. Despite this however, some doubt remains as to the true effectiveness that this curriculum has.

Successful Tracheal Intubation
* % of Subjects * p < 0.01

VR Training- A Part of our Future?
Estimates suggest 4B industry by 2013* Recent FDA recommendation that VR training precede carotid stenting Frost & Sullivan study suggest 6 mo ROI** 1:0.2 versus 1:0.8 * US Market for VR in Surgery and Imaging MedMarket Dilligence LLC (2005) **Return of Investment Study for Medical Simulation Training (Frost & Sullivan 2004)

Simulators can make a difference

Post-test on Simulator
Randomization of Residents Video Instruction & Equipment Familiarization Pre-test on Simulator (Case 1) Simulator Practice (Cases 1, 2, 4, 5, 6) No Intervention Post-test on Simulator (Case 3) Clinical Post-test Treatment Group Control Familiarization Phase Simulator Pre-test Phase Treatment Phase Simulator Post-test Phase Prospective randomized design 2 groups Control group No intervention—ie, no specific training outside of the endoscopy suite Treatment group 2-3 hour practice session on simulator

Effects of Simulator Training on Clinical Colonoscopic Performance Park, Reznick et al., Unpublished Data Mean +/- 1.0 SE * * p = 0.04

Sharp Contrasts F-16 Fighter Pilot Training
Pre-simulator sessions Simulator sessions Debriefing of simulator sessions Pre-flight review 1 hour flight 15 hours of one on one debriefing

What is Telesimulation?
Uses the camera in the simulator as a webcam Skype is used for video conferencing software – Free!

A new concept of graded responsibility
“sign-off” on cognitive elements Review of 1000’s of intra-operative situations and controversies Accepted algorithms Obligatory web-cast during first year of “independence” Mandatory second opinions for defined conditions

Focused Training We have been indoctrinated with the philosophy that the route to expertise is the creation of a totipotential trainee, one with a broad base of experiences. And that this broad base is essential to the development of a surgical specialist

It’s not necessarily axiomatic that all things majestic have an strong underlying network, that all things meager lack an extensive foundation

Anatomy of a surgeon is complex
Transfer of skills a challenge Emerging concepts of professionalism New reality means new models Simulation a part of our future Halsted is dead

Mike Gartner (before a game)
Eat well, good sleep review videotapes of last game anticipate game situations preview potential mistakes prayer

The pendulum needs equilibrium
Sleep deprivation, restricted hours vs. volume outcome relationship There is no “I” in team vs. the intimacy of the patient-doctor relationship The expectation to operate is a right or a privilege?

Value of Haptic Feedback
Virtual-reality-based laparoscopic surgical training: The role of simulation fidelity in haptic feedback 1 Authors: Hyun K. Kim a; David W. Rattner b; Mandayam A. Srinivasan a

time path length error score
An Evidence-Based Virtual Reality Training Program for Novice Laparoscopic Surgeons Rajesh Aggarwal, MRCS,* Teodor P. Grantcharov, PhD,† Jens R. Eriksen, MD, et al,

Virtual reality simulation objectively differentiates level of carotid stent experience in experienced interventionalists Van Herzeele, Aggarwal, Choong, et al: JOURNAL OF VASCULAR SURGERY November 2007

Royal College of Physicians and Surgeons of Canada
World Leader in Physician Competency Approval of a new model of Residency Education Supportive of Hybrid Programs

A Six Point Plan to Restructure and Dramatically Shorten Training
1. Tackle politics and finances head on 2. Achieving the competencies through modular-based training 3. Dramatically accelerate the pace of procedural skill acquisition 4. Diminish wasted time 5. Incorporate meaningful assessment into day-to- day activities 6. Develop and promote a culture of collegiality

It’s a Problem: No Time for Practice
“ It’s not practice that makes perfect, (stupid) It’s perfect practice that makes perfect.” Vince Lombardi

Principles of Good Practice
Morning practice most effective Strong correlation between rest and effective practice Maximum benefit from a combination of individual practice, sessions with a coach, and performances

Technical skills Depth perception Environment Operative Judgment Additional Learning Technical skills Depth perception Environment Operative Judgment Additional Learning Attentional capacity threshold Trained Novice Novice

Technical skills Depth perception Environment Operative Judgment Additional Learning Attentional capacity threshold Technical skills Depth perception Environment Operative Judgment Additional Learning Trained Novice Novice

"If I don't practice one day, I know it; two days, the critics know it; three days, the public knows it." - Heifetz

2. Modular-based training linked to specific learning objectives
1. Specific modular objectives form the basis of learning contracts 2. Objectives must be comprehensive 3. Objectives are learner-based and streamlined to ultimate career 4. Deploy modern technology, such as web-based curricular materials, access to point of care (wireless) information, optimize VR training, and a focus on providing data for evidence based decision making

You have to know when to slow down!

Are we creating the shift-work doctor?
-If that was your Sr. Resident, has he acted appropriately? Why or why not? -In total, we presented 4 of these types of scenarios portraying equally challenging dilemmas

Concepts of Professionalism
Two Departments of Surgery 43 semi-structured interviews 25 residents, 18 faculty 700 pages of transcript Park J, Woodrow SI, Reznick RK, Beales J, MacRae HM: Patient Care is a Collective Responsibility: Perceptions of Professional Responsibility in Surgery Surgery :142:107: 2007

Responsibilities Patient Care Self Education Colleagues Patient
Relationships Safety Continuity Principle Family & Affect Health Lifestyle Value Gatekeepers Inculcation Obligation Obedience -Our analysis yielded a large model describing 4 major responsibilities: patient care, self, education, colleagues, all within the context of duty hour limitation -We will present exemplars of the 4 major themes and won’t go into detail on each of the subthemes because of time constraints

Five Criteria to Define Competency
Focus on performance of the end product Reflects expectations that are applications of what is learned in the programme Expressible in terms of measurable behaviour Should use a standard that is NOT reliant on performance of other learners Inform learners, as well as other stakeholders, about what is expected of them Albanese et al: 2008, Med Educ 42:248

The explicit relationship of simulation-based procedural education to established clinical practice benchmarks leads to an outcomes based competency benchmarked assessment strategy for simulation. In such a setting, the use of non outcomes based standards-setting methods (ex Angoff, Hofstee etc.) would only serve to weaken the link between competency assessment and clinical performance. Michelson JD and Manning L: AmJSurg: 2008,196:609

Brief Overview of “the Competency Movement”
1990 Society for Teaching of Family Medicine proposes “Essentials for Family Practice 1993 CanMEDS project inaugurated focusing on doctors competencies to meet societal needs 1996 Medical school objectives project of the AAMC

Brief Overview of “the Competency Movement” (cont.)
ACGME/ABMS agreed to 6 competencies for certification 1999 Harden and collegeus published a 5 part guide to “Outcome-Based Education”

Focused Modular Training Turn it upside down

Are we creating the shift-work doctor?
-If that was your Sr. Resident, has he acted appropriately? Why or why not? -In total, we presented 4 of these types of scenarios portraying equally challenging dilemmas

This is not about turning professional training into vocational training

How surgeons disclose errors (we’re not doing so well)
Used the word error or mistake 57% Took responsibility for error 65% Offered verbal apology 47% Acknowledged patient’s emotions 55% Discussed future prevention 8% Offered second opinion 20% Chan D, Gallagher T, Reznick R, Levinson W. Surgery: 138: :2006

Lessons for England Boy, are we presumptuous
35 is too old! Consider a competency-based paradigm Participator learning>observational learning Quickly transform a rank novice to trained novice We are living in a world of team care

Skill Transfer: Laparoscopic Surgery
Randomized, controlled, blinded design Pre- and post-tests in OR Gallbladder dissection of from liver Assessment using modified global ratings scales Grantcharov et al., Randomized clinical trial of VR simulation for laparoscopic skills training, BJS 2004

Better economy of motion
Less errors Better economy of motion Faster

Warm-up in a Virtual Reality Environment Improves Performance in the Operating Room
Sonal Arora, Rajesh Aggarwal, Dan Catalyud, Irina Kruglikova, Svend Schulze,Peter Funch-Jensen, Teodor Grantcharov Department of Surgery D, Glostrup University Hospital, Glostrup, Denmark Department of Biosurgery and Surgical Technology, Imperial College, London Department of Surgery L, Aarhus University Hospital, Aarhus, Denmark Division of General Surgery, St. Michael’s Hospital, Toronto, Canada

Introduction VR simulators validated for assessment of surgical skill
Evidence of transferability of skills from VR to the OR Implementation in clinical practice is still limited Warm up used by other high performance industries (athletes, musicians etc) Should surgeons undergo warm up before operating on patients?

Aims To evaluate the impact of a short VR warm-up curriculum on laparoscopic performance in the Operating Room

Methods Participants 10 General Surgeons (5 experienced) at two University Hospitals All patients had Grade 1 biliary colic with no complicating factors. BMI< 35

Methods Study Design

Methods Simulator curriculum
warm-up program performed immediately prior to the operative procedure three exercises: lifting and grasping, clip applying and fine dissection one run of each exercise on medium level

Methods Outcomes OSATS-based global rating scale
Procedures videotaped and assessed by two blinded independent observers Statistical Analysis Comparison of performance scores with or without warm-up curriculum (Wilcoxon’s test) Reliability of assessment scores provided by 2 observers (Cronbach’s alpha)

Results Inter-rater reliability of OSATS based global rating (Cronbach’s alpha =0.92)

Results

Results Surgical performance better for the warm up compared to control (p = 0.042)

Conclusions Warm-up improves laparoscopic performance in the operating room. This warm-up program is short, easy to perform, and realistic to implement in a busy surgical department. Warm up may enhance procedural outcome and contribute to improved patient safety and better resource utilization Further studies should determine the optimal composition of a surgical VR warm-up program

Training Impact and Cost efficiency of a Comprehensive Curriculum for Advanced Laparoscopic Skills: A Randomized Controlled Trial

Background Laparoscopic Suturing:
Most technically demanding laparoscopic skill to acquire Requires extensive practice prior to proficiency Requisite skill for any MIS procedure

Purpose of the Study To design and validate two proficiency-based training curricula involving tools with proven validity for advanced skills training in Minimally Invasive Surgery (MIS)

Purpose of the Study Demonstrate transfer of skills to the OR
Perform a cost analysis of the efficiency of pre-OR training

Study Design Randomized Controlled Trial
Department of General Surgery, Univ. of Toronto Tertiary-level teaching hospitals

Study Design Subjects 26 General Surgery residents PGY 3 & 4
Intermediate MIS experience Exclusion Criteria: < 10 MIS cases as the “primary” surgeon

LAPARPSCOPIC SUTURING STUDY DESIGN LapSim VR Suturing Task x2
26 Surgical Residents VR Group (N=10) TB Group (N=10) Baseline Assessment VR Curriculum Trainer Box Curriculum LapSim VR Suturing Task x2

Virtual Reality Curriculum
LapSim® (Surgical Science, Gothenburg Sweden) VR Surgical Simulator w/ haptic interface (Immersion Medical, USA). Proficiency-based progression Summative and formative feedback from LapSim

LAPARPSCOPIC SUTURING STUDY DESIGN
26 Surgical Residents VR Group (N=10) TB Group (N=10) Baseline Assessment Trainer Box Curriculum

Trainer Box Curriculum
FLS trainer box / “Intracorporeal Suturing Task” Expert instructor / examiner FLS normalized scoring criteria Feedback: From expert instructor

Trainer Box Curriculum
Courtesy of 3-DMed

LAPARPSCOPIC SUTURING STUDY DESIGN Lap.Fundo Model / FLS Box
26 Surgical Residents VR Group (N=10) TB Group (N=10) Baseline Assessment VR Curriculum Trainer Box Curriculum Post Test 1 Lap.Fundo Model / FLS Box

Post-Test 1 Laparoscopic Nissen fundoplication model
Synthetic esophagus and gastric fundus Identical setup as for FLS grp Courtesy of Mount Sinai Hospital Surgical Skills Center, University of Toronto

Post-Test 1 Both groups FLS trainer box Expert rater / FLS certified

Post-Test 1

Post-Test 1 3 interrupted, three-throw, squared knots to perform the “wrap” Instruments and sutures same as in OR Familiarization Live scoring

Post-Test 1 Metrics: Laparoscopic suturing checklist
OSATS global rating scale

Endoscopic Suturing Check List (Moorthy et al. 2004)
(A-1) Needle Position (entry to incision) Y N 1 Held at 1/2 to 2/3 from the tip 2 Angle of 90 degrees +/- 20 degrees 3 Uses tissue or other instrument for stability 4 Attempts at positioning (3 or <3) (B-1) Needle Driving Through Tissue (entry to incision) 5 Entry at 60 to 90 degrees to the tissue plane 6 Driving with one movement 7 Single point of entry through tissue 8 Removing the needle along its curve (A-2) Needle Position (incision to exit) 9 10 Angle of 90 degrees +/- 20 11 12 (B-2) Needle Driving (incision to exit) 13 14 (C) Pulling the Suture Through 15 Needle held on needle holder in view at all times 16 Using pulley concept or walking along the suture (D) Technique of Knotting Y N 17 Two-handed Overwrap/underwrap followed by same 1 OR if one handed, one followed by the other 18 Correct "C" loop (no "S" or "O" loops) 19 Smoothly executed throw, no fumbles 20 Correct inverse "C" loop (no "S" or "O" loops) 21 22 Knot squared (capsized reef/surgical) 23 Correct third "C" loop (no "S" or "O" loops) 24 (E) Knot Slippage After the First Two Throws 25 Knot left loose to slip 26 Knot slippage attempts (3 or <3) (F) Knot Quality 27 All throws squared 28 Secure knot 29 All knots laid on the side (not over the incision) TOTAL /29

Global Rating scale – OSATS (Martin et al. Br J Surg, 1997). 1997
2 3 4 5 Respect For Tissue Frequently used unnecessary force on tissue or caused damage Careful handling of tissue but occasionally caused inadvertent damage Consistently handled tissues appropriately with minimal damage Time & Motion Many unnecessary moves Efficient time/motion but some unnecessary moves Economy of movement and maximum efficiency Instrument Handling Repeatedly makes tentative or awkward moves with instruments Competent use of instruments although occasionally appeared stiff or awkward Fluid moves with instruments and no awkwardness Knowledge of Instruments Frequently asked for the wrong instrument or used an inappropriate instrument Knew the names of most instruments and used appropriate instrument or the task Obviously familiar with the instruments required and their names Use of Assistants Consistently placed assistants poorly or failed to use them Good use of assistants most of the time Strategically used assistant to the best advantage at all times Flow of Operation & Forward Planning Frequently stopped operating or needed to discuss next move Demonstrated ability for forward planning with steady progression Obviously planned course of operation with effortless flow Knowledge of Specific Procedure Deficient knowledge. Needed specific instruction Knew all important aspects of the operation Demonstrated familiarity with all aspects of the operation

26 Surgical Residents VR Group (N=10) TB Group (N=10) Baseline Assessment VR Curriculum Trainer Box Curriculum In the OR Post Test 1 Post Test 2

Post-Test 2 Patient consent
Intra-operative performance during Laparoscopic Nissen Fundoplication One of two blinded MIS Surgeons

Post-Test 2 Video capture for analysis Post-operative scoring Metrics:
Laparoscopic suturing checklist OSATS global rating scale

26 Surgical Residents 3 MIS Surgeons VR Group (N=10) TB Group (N=10) Ctrl Group (N=6) Expert Group (N=3) Baseline Assessment VR Curriculum Trainer Box Curriculum Post Test 1 Post Test 2

No significant differences (Experience, demographics)
RESULTS 26 Surgical Residents 3 MIS Surgeons VR Group (N=10) TB Group (N=10) Ctrl Group (N=6) Expert Group (N=3) No significant differences between Groups (Experience, demographics)

No significant difference (LapSim Suturing Task)
RESULTS 26 Surgical Residents 3 MIS Surgeons VR Group (N=10) TB Group (N=10) Ctrl Group (N=6) Expert Group (N=3) Baseline Assessment No significant difference (LapSim Suturing Task)

Training time to proficiency - VR Grp significantly faster
RESULTS 26 Surgical Residents VR Group (N=10) TB Group (N=10) Baseline Assessment VR Curriculum Trainer Box Curriculum Training time to proficiency - VR Grp significantly faster (p<.002)

No significant difference Lap.Fundo Model / FLS Box
RESULTS 26 Surgical Residents VR Group (N=10) TB Group (N=10) Baseline Assessment VR Curriculum Trainer Box Curriculum Post Test 1 No significant difference (Time, CL, OSATS) Lap.Fundo Model / FLS Box

Post Test 1 (Trainer Box) Post Test 2 (Intra-operative)
RESULTS 26 Surgical Residents 3 MIS Surgeons VR Group (N=10) TB Group (N=10) Ctrl Group (N=6) Expert Group (N=3) Baseline Assessment VR Curriculum Trainer Box Curriculum Post Test 1 (Trainer Box) Post Test 2 (Intra-operative)

Results Post-Test 2: 29 operative cases recorded
Compared subjects performance: Time to suture completion Laparoscopic suturing checklist scores OSATS scores

Time (Sec)

Checklist Scores

OSAT Scores

Time to complete an intra-corporeal knot
control group

Cost Analysis: Comparison
Based on 5 Residents 1 Year Cost 5 Year Cost 10 Year Cost TRAINER BOX $ ,500.00 $ ,790.00 $ ,580.00 CTRL Curriculum $ ,690.00 $ ,450.00 $ ,900.00 VR Curriculum $ ,500.00 $ ,000.00 $ ,500.00

Cost Analysis: Comparison
Based on 10 Residents 1 Year Cost 5 Year Cost 10 Year Cost TRAINER BOX $ ,975.00 $ ,165.00 $ ,330.00 CTRL Curriculum $ ,380.00 $ ,900.00 $ ,800.00 VR Curriculum $ ,500.00 $ ,000.00 $ ,500.00

Impact on the MIS Curriculum
Residency programs can choose either/both curricula depending on their needs/resources Shifting the initial learning curve to the skills laboratory Safety / Resources / Time Simulation-based curricula are more cost effective than conventional OR training

Learning curve patterns in laparoscopic surgery. Can everyone learn it?
Teodor Grantcharov MD, PhD, Peter Funch-Jensen MD, DSc 1Department of Surgery, The Western Pennsylvania Hospital, Temple University School of Medicine, Pittsburgh, USA 2Department of Surgery, Aarhus University Hospital, Aarhus, Denmark

Laparoscopic surgery loss of 3-D degraded tactile feed-back
hand-eye coordination ”fulcrum effect”

Learning curves: the concept
1 2 proficiency zone performance 3 learning zone performance number of cases

Design 37 surgical trainees (< 10 lap. procedures)
well-validated assessment tool (Procedicus MIST) 10 repetitions of 6 VR tasks in 1 month standardized distribution of practice proficiency levels established by experienced lap. surgeons (>100 lap. procedures) Friedman’s non-parametric test

Procedicus MIST™ PC based system basic surgical skills (CoreSkills)
intracorporal suturing module progressive complexity training and assessment of key psychomotor skills

Procedicus MIST™: performance score
TIME to complete the tasks number of ERRORS ECONOMY of movements

MIST –valid tool for assessment of laparoscopic psychomotor skills
MIST-VR can differentiate between surgeons of different experience levels MIST-VR performance score correlates with performance score demonstrated during in-vivo procedures learning curves proportional to experience MIST-VR perfomence score demonstrates good test-retest reliability

VR as assessment tools: EAES consensus
VR simulators are valid tools for training and assessment of lapsroscopic technical skills VR offers unique advantages and systems are considered suitable for implementation in an integrated surgical training and assessment programme EAES Working Group Recommendations, 2005

Results: time 10 20 30 40 50 60 70 80 90 1 2 3 4 5 6 7 8 9 attempt 8%

Results: errors 5 10 15 20 25 30 1 2 3 4 6 7 8 9 attempt 8%

Results: economy of movements
5 10 15 20 25 30 35 40 45 50 1 2 3 4 6 7 8 9 attempt 8%

Conclusions 4 types of learning curves identified
8% of surgical trainees failed to demonstrate improvement with practice not everyone can achieve proficiency with laparoscopic technical skills challange for the training programmes of the future selection: when, how, by whom?

Time (Sec)

Reznick and Folse Am.J.Surg. 154:520-25, 1987
12 surgery resident (PGY1-6) Rested ~7 hr. Fatigued <3hr. Factual recall test, identify abnormal laboratory values, purdue pegboard, simulated suturing No difference observed in any task Small sample size Heterogeneous sample population Sleep Measures Non-validated performance measures Relevant tasks??? Many authors have argued that one of the reasons why the sleep deprivation literature examining residents’ performance demonstrates inconsistent results is the nature of the study design – especially in studies performed before the 1990’s. In particular: Most have small sample sizes of about 10 or 12 individuals Many use a heterogeneous study population, increasing the variance seen when measuring performance Measures used to quantify sleep may not have been reliable e.g. self-reporting (I am using EEG which is the gold standard, but even actigraphy – motion detector wrist watches – are much improved methods). Plus definitions of rested and sleep deprived are often variable – in some studies rested subjects only got 6 hours of sleep. In the sleep medicine world that is sleep deprived. Performance measures used are always different. The use of standard neuropsychological tests questions the translation of results into the clinical world. Some studies – like this one –developed their own more clinically relevant tasks but until recently, no one validated their measures……. We now know better………….

Years of study in the O.R

Years of Study in the O.R. Focus on communication failures
Qualitative methodologies Ethnographic approach 100’s of pages of transcript Emerging themes

Preliminary Checklist Results 252 cases: Frequency of Communication Failure

Effect of the Checklist 172 cases randomized Frequency of Failures
p<.001

It’s more than just competency-based education: It’s about radical curricular change Richard K. Reznick, MD, MEd, FRCSC, FACS Dean, Faculty of Health.

Similar presentations

Presentation on theme: "It’s more than just competency-based education: It’s about radical curricular change Richard K. Reznick, MD, MEd, FRCSC, FACS Dean, Faculty of Health."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

It’s more than just competency-based education: It’s about radical curricular change Richard K. Reznick, MD, MEd, FRCSC, FACS Dean, Faculty of Health.

Similar presentations

Presentation on theme: "It’s more than just competency-based education: It’s about radical curricular change Richard K. Reznick, MD, MEd, FRCSC, FACS Dean, Faculty of Health."— Presentation transcript:

Similar presentations

About project

Feedback