1. Clinical Skills Verification Rater Training MODULE 3 Setting the Clinical Skills Verification Examination Climate and Giving Feedback Karen Broquet, M.D. Eugene Beresin, M.D. Nyapati Rau, M.D.

2. Module 3 Pre-test 1.Which of the following is assessed in the Clinical Skills Verification Evaluation? a. Patient Doctor Relationship b.Clinical Formulation c.Differential Diagnosis d.Treatment Planning

3. Module 3 Pre-test 2.In assessment of clinical skills evaluations, which of the following methods is most likely to demonstrate what a resident actually “does” according to Miller’s pyramid? a.Arrange Mock Oral Board-style evaluations b.Use Standardized Patients c.Perform the exercise during routine work d.Make use of the Hawthorne Effect e. Add a multiple choice test following the interview

4. Module 3 Pre-test 3.Which of the following rating methods is recommended for a documented Clinical Skills Verification Evaluation? a. Any rating form can be used b.Process notes provided by the resident c.Faculty should use a rating form he/she most prefers d.Faculty should all use the same rating form

5. Module 3 Pre-test 4.Which of the following is considered formative assessment? a. Year-end reviews of performance b.Pass/fail examinations c.Specific feedback of behavior in real time d.Anonymous 360 degree evaluations sent to the resident e.Periodic summative evaluations

6. CSV Exam MUST Assess… Ability to establish doctor-patient relationship Conduct a clinical interview Oral presentation of patient history and clinical findings

7. But can (and should) ALSO be…. Valuable educational experience Opportunity for Direct observation of clinical skills by faculty Formal evaluation of clinical skills Focused feedback

8. Feedback is Formative Process by which the teacher provides the learner with information about his/her performance for the purpose of improving performance (Ende 1983) Without specific and focused feedback, learners: Rely on self-assessment (unreliable!) May establish undesired behaviors and/or extinguish desired behaviors

9. Evaluation is Summative “Did I Pass the exam!?!”

10. Feedback = Coach Timely, immediate, ongoing Direct Verbal Face to face Guiding future performance

11. Evaluation = Judge Summative Not ongoing, occurs at the end of time together Assesses whether or not learner met performance standards

12. Quality of both feedback and evaluation depends on the observer’s detection, perception and recall of the resident’s performance How to maximize this: Incorporate direct observation into clinical teaching routine Brief (daily rounds) and not so brief (CSV exam) Normalizes observation/assessment experience Hawthorne effect: the more we know we are assessed, the better we perform Write it down! Specific, accurate feedback is better feedback Also helps with accurate evaluation Prepare for the encounter

13. Setting Expectations How long will the encounter be? Who will be observing? What are the goals of the encounter? i.e., is this a formal CSV evaluation? Assessment parameters? What rating instrument/form will be used? When will “results” be given? What are the consequences of “passing” or “failing” a CSV exam?

14. What Rating Form/Instrument to Use? Multiple forms accepted by the ABPN Forms vary by level of detail All faculty/CSV raters in a program should be trained on and use the same form

15. How Long Should the CSV Exam Be? ABPN: “about” 30 minutes for interview and 15 minutes for presentation AADPRT CS Assessment Task Force: 30-45 minutes for interview, 60-75 minutes total Programs should choose one format and use consistently

16. Guidelines for giving effective feedback – whatever the setting/encounter

17. Effective/Meaningful Feedback is… approached with teacher and learner working as allies with common goals given immediately in a private environment well timed and expected based on first-hand (observed) data focused on specific behaviors, not generalizations limited to behaviors that are remediable

18. Effective/Meaningful Feedback is… regulated in quantity (no more than 2-3 items for improvement) limited to actions and decisions, not assumed intentions or interpretations phrased in descriptive, nonjudgmental language closed with suggestions for improvement and a mutually agreed upon action plan

19. Barriers to Effective Feedback Faculty Time Lack of feedback skills Feels like being critical Want learners to like us Gestalt vs specific observations Impact of CSV ‘failure” Resident Time Lack of feedback experience Feels like being criticized Don’t want to expose weaknesses

20. Residents CRAVE feedback

21. To minimize barriers, faculty can… Understand the profound and beneficial educational value of feedback Provide the proper climate and preparation for the experience Elicit reciprocal feedback from trainees Gain the skills for giving both constructive and reinforcing feedback Use a consistent, common rating instrument

22. Cultural Phenomena – International Medical Grads Lack of feedback Experience Diverse cultural backgrounds Power dynamics between teacher and learner Accepted communication patterns Attitudes towards critique Vulnerability in evaluation Stress and difficult social situations Experience with discriminat ion

23. Important for all learners – especially IMGs Learner centered approach Explicitly prepare the resident for the expectations of the exam and the provision of feedback The importance of the teacher and the IMG working together cannot be over emphasized. Create a safe learning environment and clear expectations and educational objectives

24. Important for all learners – especially IMGs Invite learner to indicate his/her learning needs and provide a self-assessment. Normalize feedback as regular educational activity that includes areas to be reinforced and areas requiring improvement. Be clear when feedback is formative or summative (i.e evaluation)

25. Module 3 Post-test 1.Which of the following is assessed in the Clinical Skills Verification Evaluation? a. Patient Doctor Relationship b.Clinical Formulation c.Differential Diagnosis d.Treatment Planning

26. Module 5 Post-test 1.Which of the following is assessed in the Clinical Skills Verification Evaluation? a.Patient Doctor Relationship Diagnosis, biopsychosocial formulation and treatment planning are not assessed

27. Module 5 Post-test 2.In assessment of clinical skills evaluations, which of the following methods is most likely to demonstrate what a resident actually “does” according to Miller’s pyramid? a.Arrange Mock Oral Board-style evaluations b.Use Standardized Patients c.Perform the exercise during routine work d.Make use of the Hawthorne Effect e. Add a multiple choice test following the interview

28. Module 5 Post-test 2.In assessment of clinical skills evaluations, which of the following methods is most likely to demonstrate what a resident actually “does” according to Miller’s pyramid? c.Perform the exercise during routine work

29. Module 5 Post-test 3.Which of the following rating methods is recommended for a documented Clinical Skills Verification Evaluation? a. Any rating form can be used b.Process notes provided by the resident c.Faculty should use a rating form he/she most prefers d.Faculty should all use the same rating form

30. Module 5 Post-test 3.Which of the following rating methods is recommended for a documented Clinical Skills Verification Evaluation? d.Faculty should all use the same rating form Although the ABPN has approved a variety of different rating forms for the CSV exam, consistent use of and feedback regarding the same form by all raters in a program is associated with improved inter- rater reliability

31. Module 5 Post-test 4.Which of the following is considered formative assessment? a. Year-end reviews of performance b.Pass/fail examinations c.Specific feedback of behavior in real time d.Anonymous 360 degree evaluations sent to the resident e.Periodic summative evaluations

32. Module 5 Post-test 4.Which of the following is considered formative assessment? c.Specific feedback of behavior in real time Annual performance reviews, 360 degree evaluations and pass-fail examinations are all types of summative assessment.