Assessing Learners The Teaching Center Department of Pediatrics UNC School of Medicine The Teaching Center

Goals and Objectives Goal: »Overview the process of assessment of medical learners for the purpose of evaluation Objectives: »Outline differences between feedback and evaluation »Demonstrate that assessment should be matched with objectives »Describe various assessment methods »Review the “RIME” method of assessment The Teaching Center

Evaluation vs. Feedback Feedback: »Formative »Ongoing during a learning experience »Focused toward improvement Evaluation: »Summative »Completed at the end of a learning experience »Descriptive of performance »Often, for medical students, “the grade” The Teaching Center

Underlying Tenet of Assessment Assessment of the learner must match learning goals and objectives. »Plan for learner assessment after objectives are written and before the start of the learning encounter »Multiple assessment tools are typically needed to measure and document accomplishment of every learning objective in a learning encounter The Teaching Center

Six Competencies to Consider in Developing Learning Objectives and Assessing Learners Medical Knowledge Patient Care Communication Skills Practice-based learning and improvement Systems-based practice Professionalism The Teaching Center

Example Methods of Assessment 1.Observation 2.Multiple choice and other written testing 3.Oral examination 4.Chart review, Procedure logs, Case logs 5.Portfolio Development 6.Standardized patient encounters and Objective Structured Clinical Exams (OSCEs) The Teaching Center

1. Observation of the Learner Strengths: »Observation forms can be used to assess any competency if the learner has the opportunity to demonstrate it in the observed encounters »Commonly used and accepted in medical education Weaknesses: »Subjectivity »Lack of reliability because of different observer standards »Only as good as the observer makes them The Teaching Center

The Observation Form It is important to use observation forms that provide behavioral anchors. »Do not simply use ratings: Example: Professionalism that is “outstanding”, “average”, or “below average” makes for very subjective and unreliable assessments »Instead outline expected behaviors for each level of achievement Example: Regarding professionalism document specifics: maintains confidentiality, demonstrates honesty, maintains a professional appearance, etc. The more clearly the expected performance is outlined, the less subjective the observation evaluation form. The Teaching Center

2. Multiple Choice/Written Tests Strengths: »Efficient, inexpensive »Assesses medical knowledge well »Can assess some aspects of patient care Weaknesses: »Poor assessment of communication skills, professionalism, systems-based practice and practice-based learning »Sometimes reflects test-taking skill more than content knowledge The Teaching Center

3. Oral Examination Strengths: »Can assess thought processes »Depending on structure, can assess medical knowledge, patient care, practice-based learning, systems-based practice, patient care and professionalism. Weaknesses: »Does not assess communication skills with patients »Often not “real world” test of professionalism »Can be subjectively evaluated if there are not specifically defined expectations for responses The Teaching Center

4. Chart Review, Procedure Logs and Case Logs Strengths: »Documents patient care and therefore can be used for assessment of that. »Can also be used to assess practiced-based learning and improvement and systems-based practice. Weaknesses: »Does not assess communication skills with patients or professionalism. »Often does not assess the quality of care delivered. »Dependent on the documented findings, so cannot be used to assess medical accuracy. The Teaching Center

5. Portfolio Development Strengths: »Allows for documentation of self-assessment in any competency area »Can allow for documentation of evaluation from others as well Weaknesses: »Only as good as what is put into it – very dependent on learner effort The Teaching Center

6. Standardized patient encounters and OSCEs Strengths: »Can be very effective at assessing all competencies »Particularly useful in assessment of communication skills Weaknesses: »Expensive and time-consuming »Depend on learner buy-in and how realistic the encounter is made »Subjectivity is a possible concern so, again, behavioral anchors and clear expectations are important The Teaching Center

Matching Objectives to Assessment Examples Objectives: Communicate effectively Demonstrate knowledge Perform an accurate physical exam Utilize current evidence Assessment Method:  Standardized patient encounter  Multiple choice test  Observation  Portfolio containing literature reviews The Teaching Center

The RIME Method of Evaluation Reporter Interpreter Manager Educator The medical learner must go through these steps sequentially. These steps can be used to assess overall progress. The Teaching Center

Tips for Grading Medical Students Tips for Grading Medical Students Consider the RIME method, with expectation that the Honors medical student can Manage data Combine opinions to come to group consensus on the subjective pieces Clarify expectations and mark behaviors for various levels of performance Use a variety of assessment strategies The Teaching Center

Summary and Conclusion Learners should receive both feedback and evaluation Match assessment strategies to learning objectives »consider each competency Use multiple assessment strategies Know the strengths and weaknesses of each assessment strategy chosen The Teaching Center