1. Preparing for cbd wORKPLACE based assessment
Nancy Dudek MD MEd
Anesthesia Grand Rounds
September 23, 2015

2. Objectives
Describe the concept of entrustment as it relates to resident assessment
Properly use a different style of rating anchors
Write comments that are specific, behavior-based and supported by examples

3. Competency Based Medical Education

4. Assessment
Different
Much more frequent assessments that are documented
“Assessment for learning”
Coaching
Standard is competent for independent practice

5. Clinical Case Assessment Tool
Based on the O-SCORE (Ottawa Surgical Competency Operating Room Evaluation)
Intended to primarily be used to assess one case
Looks at three areas of performance
Pre-operative
Intra-operative
Post-operative
Rating scale uses competent for independent practice as the standard
Area for qualitative comments

6. What concerns do you have?

7. Focus on a couple of areas today
Rating scales
Narrative assessment

8. Part One – Rating Scale Anchors

9. Traditional rating scale anchors
1 – Consistently below expectations
2 – Sometimes below expectations
3 – Meets expectations
4 – Sometimes above expectations
5 – Consistently above expectations
What works?
What does not work?

10. Rating scale anchors 1 – “Staff had to do”
2 – “Staff had to talk resident through”
3 – “Staff had to prompt resident from time to time”
4 – “Staff needed to be in room just in case”
5 – “Staff did not need to be there”
What do you think?

11. Other tools using these anchors
O-SCORE – Ottawa Surgical Competency Operating Room Evaluation
Tool to assess the performance of a trainee on one surgical procedure
Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa surgical competency operating room evaluation (O-SCORE): a tool to assess surgical competence. Academic Medicine 2012;87(10):
OCAT – Ottawa Clinic Assessment Tool
Tool to assess the performance of a trainee during one clinic
Rekman J, Hamstra SJ, Dudek NL, Seabrook C, Wood TJ, Balaa F, Brandys T, Gofton W. Development of the Ottawa Clinic Assessment Tool (OCAT). Association for Surgical Education Annual Meeting, Seattle, WA, April 23.
OBAT – Ontario Bronchoscopy Assessment Tool
Tool to assess the performance of a trainee on one bronchoscopy
Voduc N, Dudek NL, Wood TJ. Validation of a new bronchoscopy assessment tool in a clinical setting. Ottawa Conference, Ottawa, ON, 2014.

12. Do they work? Highly reliable & excellent evidence for validity
A large improvement on most other assessment tools
Do not need rater training beyond reading the instructions
Residents accept “low marks”
Staff uses whole scale

13. Other work
Study with three commonly used workplace-based assessment tools
Changed the traditional ratings scale anchors to align with increasing independence
Compared the reliability of the tools with the two different rating scale anchors
Significantly increased reliability with new anchors
Crossley J, Johnson G, Booth J, Wade W. Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales. Medical Education ; 45:

14. Why do these scales seem to work?

15. Entrustment More meaningful to staff
Change culture of giving and receiving feedback
Each resident’s progress can been seen
Removes the moving target as independence is now the goal

16. Part Two – Narrative Assessment

17. Narrative assessment
CBME has an increased focus on narrative assessment
Documentation of the verbal feedback (which is an assessment of how they did that day)

18. Narrative component of CCAT
Describe aspects of performance
Done well
Should be done differently
Recommendations for next steps in learning

19. What do you think of this comment?
Great case – confident, thorough, at expected level of training or beyond.

20. How could you improve it?

21. High quality comments Specific
Focus on behaviour/performance and not attitude
Use examples

22. Okay comments Responds well to feedback Communication skills need work
Read more
Great case presentations

23. Better comments
Responds positively to feedback. Ex) Noted that you missed a quads lag by not first checking passive ROM of the knee. Reviewed proper technique for quads testing. On observation at a later point during the clinic you had altered your physical exam appropriately.
Tendency to use too much medical jargon when explaining issues to patients. Ex) In the patient with an abnormal lesion on the chest x-ray you said, “It could be an infiltrate, a granuloma, a malignancy…”
Focus anatomy reading on the brachial plexus…. need to be able to draw the plexus out so that neurological lesions can be mapped on the plexus
Case presentations in clinic are succinct and include all relevant info Ex) Patient with depression and back pain… you were able to focus on the issues relevant to the question asked by the referring doctor in presenting the case

24. Summary
Do not be afraid to assign the rating that describes the performance
i.e. Do NOT worry about level of training
Prepare residents for the different approach
Focus on writing useful comments on your CCAT
Specific, behaviour-based and supported by examples

25. Questions