Preparing for cbd wORKPLACE based assessment Nancy Dudek MD MEd Anesthesia Grand Rounds September 23, 2015

Objectives Describe the concept of entrustment as it relates to resident assessment Properly use a different style of rating anchors Write comments that are specific, behavior-based and supported by examples

Competency Based Medical Education

Assessment Different Much more frequent assessments that are documented “Assessment for learning” Coaching Standard is competent for independent practice

Clinical Case Assessment Tool Based on the O-SCORE (Ottawa Surgical Competency Operating Room Evaluation) Intended to primarily be used to assess one case Looks at three areas of performance Pre-operative Intra-operative Post-operative Rating scale uses competent for independent practice as the standard Area for qualitative comments

What concerns do you have?

Focus on a couple of areas today Rating scales Narrative assessment

Part One – Rating Scale Anchors

Traditional rating scale anchors 1 – Consistently below expectations 2 – Sometimes below expectations 3 – Meets expectations 4 – Sometimes above expectations 5 – Consistently above expectations What works? What does not work?

Rating scale anchors 1 – “Staff had to do” 2 – “Staff had to talk resident through” 3 – “Staff had to prompt resident from time to time” 4 – “Staff needed to be in room just in case” 5 – “Staff did not need to be there” What do you think?

Other tools using these anchors O-SCORE – Ottawa Surgical Competency Operating Room Evaluation Tool to assess the performance of a trainee on one surgical procedure Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa surgical competency operating room evaluation (O-SCORE): a tool to assess surgical competence. Academic Medicine 2012;87(10):1401-1407. OCAT – Ottawa Clinic Assessment Tool Tool to assess the performance of a trainee during one clinic Rekman J, Hamstra SJ, Dudek NL, Seabrook C, Wood TJ, Balaa F, Brandys T, Gofton W. Development of the Ottawa Clinic Assessment Tool (OCAT). Association for Surgical Education Annual Meeting, Seattle, WA, April 23. OBAT – Ontario Bronchoscopy Assessment Tool Tool to assess the performance of a trainee on one bronchoscopy Voduc N, Dudek NL, Wood TJ. Validation of a new bronchoscopy assessment tool in a clinical setting. Ottawa Conference, Ottawa, ON, 2014.

Do they work? Highly reliable & excellent evidence for validity A large improvement on most other assessment tools Do not need rater training beyond reading the instructions Residents accept “low marks” Staff uses whole scale

Other work Study with three commonly used workplace-based assessment tools Changed the traditional ratings scale anchors to align with increasing independence Compared the reliability of the tools with the two different rating scale anchors Significantly increased reliability with new anchors Crossley J, Johnson G, Booth J, Wade W. Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales. Medical Education. 2011; 45: 560-569.

Why do these scales seem to work?

Entrustment More meaningful to staff Change culture of giving and receiving feedback Each resident’s progress can been seen Removes the moving target as independence is now the goal

Part Two – Narrative Assessment

Narrative assessment CBME has an increased focus on narrative assessment Documentation of the verbal feedback (which is an assessment of how they did that day)

Narrative component of CCAT Describe aspects of performance Done well Should be done differently Recommendations for next steps in learning

What do you think of this comment? Great case – confident, thorough, at expected level of training or beyond.

How could you improve it?

High quality comments Specific Focus on behaviour/performance and not attitude Use examples

Okay comments Responds well to feedback Communication skills need work Read more Great case presentations

Better comments Responds positively to feedback. Ex) Noted that you missed a quads lag by not first checking passive ROM of the knee. Reviewed proper technique for quads testing. On observation at a later point during the clinic you had altered your physical exam appropriately. Tendency to use too much medical jargon when explaining issues to patients. Ex) In the patient with an abnormal lesion on the chest x-ray you said, “It could be an infiltrate, a granuloma, a malignancy…” Focus anatomy reading on the brachial plexus…. need to be able to draw the plexus out so that neurological lesions can be mapped on the plexus Case presentations in clinic are succinct and include all relevant info Ex) Patient with depression and back pain… you were able to focus on the issues relevant to the question asked by the referring doctor in presenting the case

Summary Do not be afraid to assign the rating that describes the performance i.e. Do NOT worry about level of training Prepare residents for the different approach Focus on writing useful comments on your CCAT Specific, behaviour-based and supported by examples

Questions