1. Walk a Milestone in My Shoes for UME: Developing a Milestone Based Assessment Tool to Enhance the Grading Process in Undergraduate Medical Education Kathleen Brite, MD Christine Savi, PhD

2. Realistic Implementation of Milestone Assessment Building a reliable, intuitive, self-explanatory assessment Achieving stakeholder buy-in Implementation pitfalls, solutions Guiding resources It can be done!

3. Session Objectives 1.Explain how physician competencies drive desired student behavior outcomes and identify issues encountered in the UME environment relating to competency assessment, grade inflation, distribution, and rater reliability. 2.Discuss possible strengths/obstacles one may incur in developing a rubric outlining student performance and outline best practices for developing and implementing a milestone-based assessment. 3.Summarize the importance of stakeholder input and identify opportunities for collaboration among UME and GME programs.

4. Assessment of Resident Performance During the length of residency, multiple faculty and attendings teach and observe residents Each resident is assessed by their demonstration of observed performance in the ACGME behavioral areas: Interpersonal and Communication Skills Medical Knowledge Patient Care Practice-based Learning Professionalism Systems-based Practice Within each behavioral area are observable criterion on which the resident is observed at certain educational milestones in their medical development Narrative comments highlight/support behavioral observations

5. Transition from GME to UME GME Implemented Next Accreditation System (NAS) and milestone movement 2013: 7 specialty areas 2014: Remaining specialties Define Entrustable Professional Activities (EPAs), or the context the sub- competency can be observed Rate the student on the appropriate developmental level (milestone) UME Most programs: Traditional competency framework AAMC 13 pilot schools building a UME milestone-based framework Few programs: Implementing developmental competency progress U of Arizona, College of Medicine – Phoenix: Since 2013

6. Family, Community, Preventive Medicine Clerkship Distributed Model – 30 Sites across the state Diverse Sites – Rural, Urban Underserved, Academic Inconsistency in student assessment

7. Overview of Past Assessment tool Inconsistent Grade inflation – Clinical honors: 1 (being highest)

9. Grade Inflation Data Academic Year 2013-2014 – 37 out of 54 students earned Clinical Honors =68% – 44 % earned overall grade of Honors

10. Challenges in Student Assessment Faculty Development Inconsistency – Various degrees of exposure to students – Variety of personalities Curriculum overlap All leading to the quest for a more reliable tool…...

11. Assessment of Medical Student Performance – Old Version (1 competency area – 6 six sections, 5 pages) Highlights Anchor criteria in a separate document Contained only 3 levels of criterion Not specific to environment Subjective 5 pages Comments optional Holistic scoring No quality review Annual training

12. Current Challenges Discuss with a partner: Competency assessment alignment: UME/GME Grade inflation, distribution Reliability of faculty raters Other concerns?

13. Prevalent Trends

14. Assessment Development Identified need to bridge gap from GME milestone implementation to UME clerkship faculty Introduced paradigm already in use from pre-clerkship curriculum Collaborative, voluntary process with stakeholders Clerkship Directors Students Current, Alumni Subcommittee review (AES, CC), focus group feedback Administration: Academic and Student Affairs Assessment & Evaluation Director Residents Bi-monthly meetings after Clinical Curriculum Subcommittee meetings (faculty on campus) o Agenda-facilitated o Consensus-driven o Snacks o Informal approach

15. Assessment Development, cont’d Committee approval Assessment & Evaluation Subcommittee Clinical Curriculum Subcommittee School Curriculum Committee Timeframe – Development: 8 months – including approval process – Piloted: 4 weeks, revisions made – Faculty input – Student feedback – Utilized: 6 months, revisions recommended for subsequent year

16. Targeted Improvements Developmentally based on milestones of achievement Increasing levels of performance from preclinical to clinical Clerkship-contextual environment Progression over time: 4 – 12 weeks Specific, constructive feedback Why Change? Bridge the gap between Residency (GME) and Medical School (UME) Provide individualized, student-specific feedback based on clear expectations Guide developmental progression of behavior

17. Assessment of Medical Student Performance - New (6 competency areas – 1 pg.) Highlights Anchor criteria for all 3 levels Specific to environment Less Subjective 1 page Comments REQUIRED Analytic scoring Quality review Anytime training: online, email materials

18. Medical Student Assessment – New Tool (back)

19. Quality Control All assessments contain: – Comments: Specific comments about criteria and student progress – Rating levels = comments 3.0 = Exceptional performance 2.0 – 2.5 does not equal Exceptional performance Assessments returned if: – Comments are vague; state ‘nice job’, ‘overall, good student’ or not applicable to assessment criteria – Rating levels do not sync with comments – Returned assessments need to be re-submitted to meet mid/end deadline *Assessment Types – Formative Assessments Mid-Clerkship – Site or Clerkship Director End of Rotation – Faculty, Site Director – Summative Assessment Final Grade Form – Clerkship Director

20. Suggestions, Ideas, & Progress Learn from your group: What you are doing/will do to address competency assessment transition from UME/GME? What resources do you need?

21. Pearls of Progress

22. Grade Inflation Data Take 2 Old Assessment Tool: 44% Honors New Assessment Tool 2015-2016 – 49 out of 52 students earned Clinical Honors ranging from 2.6-3 = 94% – 21% earned overall grade of Honors  At university goal

23. Rater Agreement (ICC) by Competency Domain: 6 month implementation in FCPM End of Rotation Form OverallPatient Care CommunicationProfessionalismCritical Awareness & Quality Improvemen t Medical Knowledge Societal Awareness & Responsiveness Milestone- based New Tool (15 items) ICC (2,23) =.763 ICC (2,23) =.746 ICC (2,23) =.768 ICC (2,23) =.547 ICC (2,23) =.* ICC (2,23) =.* ICC (2,23) =.819 Global Benchmark - Previous Tool (31 items) ICC (2,13) =.665 ICC (2,13) =.672 ICC (2,13) =.779 ICC (2,13) =.332 ICC (2,13) =.867 ICC (2,13) = -.608 ICC (2,13) =.685

24. Implications for Residency Placement and Performance Address problematic behavior early Framework to identify and correct behavior Better clinical performance Implications for MSPE Crucial information for Program Directors

25. Cumulative Competency Graph MS2 Student A - MS2 Student A - MS2 Cohort

26. Survey says…

27. References 1.Wilkinson TJ, Frampton CM. Assessing performance in final year medical students. Can a postgraduate measure be used in an undergraduate setting? Med Educ. 2003; 37:233–240. 2.ACGME Milestones. Accreditation Council for Graduate Medical Education (ACGME) website. https://www.acgme.org/acgmeweb/tabid/430/ProgramandInstitutionalA ccreditation/NextAccreditationSystem/Milestones.aspx. Accessed October 17, 2014. https://www.acgme.org/acgmeweb/tabid/430/ProgramandInstitutionalA ccreditation/NextAccreditationSystem/Milestones.aspx 3.Clinical Learning Environment Review (CLER) Program. Accreditation Council for Graduate Medical Education (ACGME) website. https://www.acgme.org/acgmeweb/tabid/83/'/acgmeweb/tabid/436/Pro gramandInstitutionalAccreditation/NextAccreditationSystem/ClinicalLearn ingEnvironmentReviewProgram.aspx. Accessed March 10, 2015. https://www.acgme.org/acgmeweb/tabid/83/'/acgmeweb/tabid/436/Pro gramandInstitutionalAccreditation/NextAccreditationSystem/ClinicalLearn ingEnvironmentReviewProgram.aspx 4.Nasca TJ, Philibert I, Brigham T, Flynn TC. The Next GME Accreditation System – Rational and benefits. N Engl J Med, 2012; 366(11): 1051-1056.