1. Competency Tools to Measure Resident Progress
Carol Vandenakker-Albanese, MD
Residency Program Director
Department of PM&R
UC Davis Health System
November 15, 2014

2. ACGME Outcome Project
Competency-based education focused on learner performance in reaching specific objectives
Long-term initiative by ACGME to emphasize educational outcomes in the accreditation of residency training programs
Six core competencies for resident education
Medical Knowledge
Patient Care
Practice Based Learning and Improvement
Systems Based Practice
Professionalism
Interpersonal and Communication Skills

3. The ‘Next Accreditation System’ in a Nutshell
Continuous Accreditation Model – annually updated
Based on annual data submitted, other data requested, and program trends
RRCs will measure compliance through the evaluation of annual program data elements including:
Milestones (submitted twice a year)
ACGME Resident Survey
ACGME Faculty Survey
Procedural /Case Log Data

4. Procedural Logs: Required (Observed and Performed)
EMG/NCS
Axial epidural injection
Axial facet
Peripheral joint/intraarticular injection
Tendon sheath/bursa injection
Trigger points
Peripheral nerve injection
Botulinum Toxin Injection
Phenol injection
Programming baclofen pump
Refilling intrathecal baclofen pump
Ultrasound extremity
Ultrasound guidance

5. Case Logs: Optional (admission, consult, outpatient)
Acute neuromuscular pain
Amputation
Brain injury
Chronic pain management
Geriatric rehab
Neurologic disorders
Oncologic rehab
Pediatric rehab
Post fracture and post-operative joint arthroplasty
Pulmonary and cardiac
Rheumatologic
Soft Tissue disorders
Spinal Cord Injury
Stroke

6. Milestones are based on concept of Entrustable Professional Activity (EPA’s)
Real life patient care episodes, usually composed of elements of most if not all “competencies”
Benchmark of performance is the ability to be entrusted to perform care with “indirect supervision with direct supervision available”
Progression is then the achievement of EPA’s of increasing difficulty, risk, or sophistication
Proficiency is then the achievement of the most sophisticated EPA’s required of the resident

7. Uses and Implications of Milestones
ACGME
Accreditation –continuous monitoring of programs; lengthening of site visit cycles
Public Accountability –report at a national level on competency outcomes
Community of practice for evaluation and research, with focus on continuous improvement
Residency Programs
Guide curriculum development
More explicit expectations of residents
Support better assessment
Enhanced opportunities for early identification of under-performers

8. Challenges/Opportunities
Culture Change and Faculty Development
Program Directors, Designated Institutional Officials
Faculty
Selecting Review Committee Members
Improving quality of faculty observations/documentation
The “Community of Educators” in each specialty has come together and agree on:
core elements of the competencies (Milestones)
levels of performance
core methods of assessment

9. Program’s Assessment of Residents
Evaluation tools
Competency Committee
Semi-Annual Reporting to ACGME on each milestone for all residents
Annual reporting to ABPMR on resident status and evaluations

10. Toolbox: Competancy Evaluation Tools
Begin by using existing tools and faculty observations
Exams: Written, Oral, SAE
Clinical Observation: Simulation, ROCA
Multi-Source Feedback: 360 evaluations, chart reviews
Procedural Skill Evaluation
Workshops
Clinical assessments
Case Logs

11. Milestone Assessment
Determine which tools you will use to assess each milestone
Determine which rotations/ faculty are best to evaluate individual milestones
Determine which milestones may require additional tools
(Yearly updated Resident CV/ self-evaluation)

12. 27 PM&R Milestones Patient Care: 7 Systems Based Practice: 3
Practice-Based Learning and Improvement: 3
Professionalism: 3
Interpersonal and Communication Skills: 2
Medical Knowledge: 9

13. PC 2. Physiatric Physical Exam (including medical, neurological, and gait adapted for age and impairment)
Rotations:
Inpatient: General rehab, SCI, TBI
Outpatient: MSK, neurorehab
Peds
Tools:
ROCA
PE workshop

14. SBP 2. Team approach to enhance patient care coordination
SBP 2. Team approach to enhance patient care coordination. Rehabilitation team members may include OT, PT, SLP, etc.
Rotations:
Inpatient adult and pediatric rehab
Consults
Outpatient multidisciplinary clinics
Tools:
360 evaluations
Faculty evaluations

15. PROF 3. Professional behaviors and accountability to self, patients, society and the profession
Rotations:
Any (not all)
May want to pick most stressful
Tools
Faculty evaluations
Program director/ coordinator evaluations
Review of CV – involvement in service activities

16. MK 6. Musculoskeletal Disorders.
Rotations:
Outpatient clinics: sports and spine
Tools:
Faculty evaluations
ROCA
Chart reviews
SAE/ Written test
Mock Oral Exam

17. Using Milestones as Evaluation Tool
Self Assessment on milestones annually
Schedule milestone assessments throughout year
Incorporate milestones into rotation objectives/ evaluations
Incorporate selected milestones into electronic evaluations

18. Faculty Education Faculty must be familiar with milestones
Reminder: Milestone levels NOT equivalent to PGY level
Distribute evaluation tools at start of rotation
Identify which faculty are appropriate to evaluate which milestones
Review/ re-educate faculty based on quality of evaluations

19. Key Elements of Quality Evaluation
Trained Observers- Core Faculty
Common understanding of the expectations
Sensitive “eye” to key elements
Consistent evaluation of a given level of performance
Establish Minimum Numbers of Quality Observations (CCC)
Interpreter/Synthesizer Experts
Clinical Competency Committee (Resident Evaluation
Committee)