1. Core Competency Assessment in Emergency Medicine from Design to Implementation Christian Arbelaez, MD, MPH Assistant Residency Director Harvard Affiliated Emergency Medicine Residency Program Brigham and Women’s Hospital CORD Educational Soundbite San Diego, CA 2011

2. Core Competency Assessment in EM Goals To describe a comprehensive core competencies assessment program designed, implemented, and reviewed after one year of use in our residency program To share lessons learned from our experience

3. Core Competency Assessment in EM Outline Design (Oct 2008) Implementation (Jul 2009 - June 2010) 1 year Review (Summer 2010) Lessons Learned

4. Core Competency Assessment in EM Design: Goals The goals of the program for the residents –Robust feedback loop –Comprehensive yet practical –Multiple and repeated assessments –Assess a resident’s developmental progression –Identify promotion or remediation criteria The goal of the program for the residency –Identify need for programmatic change

5. Core Competency Assessment in EM Design: Process Review the ACGME guidelines Needs assessment and current areas of strengths and weakness Stakeholder meetings and focus groups Resource prioritization Develop action plan with timelines –Competency based curriculum and assessment program –Faculty development program –Concomitant research agenda and groups

6. Core Competency Assessment in EM Design: ACGME Domains Annual Competency Assessment –Competencies by PG year - expected, measured, remediation Chief Complaint Competency –3 chief complaints Procedural Competency –Pre-determined # of logged procedures –3 key index procedures ED bedside ultrasound Resuscitation Competency –Adult and Pediatric medical and trauma Off-service rotations

7. Core Competency Assessment in EM Design: 6 Core Competencies PGY 1 and PGY 2 –Interpersonal and Communication Skills –Professionalism –Medical Knowledge –Patient Care PGY 3 –Practice-Based Learning and Improvement PGY 4 –Systems-Based Practice

8. Core Competency Assessment in EM Design: HAEMR Annual Competency Assessment –Resident portfolio: all below + QI project, Research, PPT –360-degree eval - Faculty by PGY, Peer, Nursing, Self w/ P-Mex Chief Complaint Competency –PGY 1) CP 2) Abdominal pain 3) Weakness 4) Oral boards –Mentorship program Procedural Competency (A and B components) –Logs –Procedure cards: Endotracheal intubation and CVL –U/S program Resuscitation Competency (A and B components) –Logs –STRATUS Simulation center/Mock codes Off-service rotations

9. Core Competency Assessment in EM Implementation Implementation and roll-out plan –Phase 1 and phase 2 –Reasonable set deadlines –Phase 1 start date with new interns Key component – Clear and transparent communication plan for the residents, faculty, and all staff involved in assessment of the residents

10. Core Competency Assessment in EM Implementation: Communication Communication Plan –Resident group meetings Program objectives and expectations Changes in the electronic data management system Distribution of semi-annual assessment forms Examples of portfolio entries and self-reflection pieces –Resident individual meetings Program manager review portfolios in prep for semi-annual –Faculty meetings Faculty Primer: NI evals, 3 Procedures, Mentorship program Procedure cards - direct observation and constructive feedback

11. Core Competency Assessment in EM One-Year Review Critical review of program –Program evaluation: survey, meeting, retreat –PGY data report –Identified areas for improvement Action plan for Phase 2

12. Core Competency Assessment in EM One-Year Review: Phase 2 Annual Competency Assessment –Resident portfolio Resident learning contract and Professionalism checklist Promotion criteria –360-degree eval - Peer and Nursing, Self Chief Complaint Competency –Mentorship program Procedural Competency (A and B components) –Logs: Variability but on avg reaching targets by PGY3 –Intubation and CVL part B Resuscitation Competency (A and B components) –Logs: Variability but on avg reaching targets by PGY3 Off-service rotations –Agreements w Anesthesia and ICUs

13. Core Competency Assessment in EM Self-Assessment A. P-MEX form in NI B. Core Competency Self Assessment –PGY 1: Interpersonal Communication/Professionalism Who are you as a clinician? Define your ideal of professionalism in a physician –PGY2: Patient Care Describe a challenging patient encounter –PGY3: Practice Based Learning and Improvement Describe a patient encounter where there was a "medical error/Near miss" and what lessons you learned –PGY4: Systems Based Practice How do you successfully handle patient care and teaching while paying attention to the flow within the department C. Personal Reflection: –Goals for this year, Academic aspirations/thoughts for scholarly activity –Action plan/Next steps

14. Core Competency Assessment in EM One-Year Review: Promotion Criteria In order for promotion, at the PGY 3 end of year evaluation, the following are the basic requirements needed in the resident’s portfolio: –Signed resident learning contract –Completed Professionalism checklist –Completed semi-annual evaluations –one Chief Complaint CC assessment per year (including one oral boards) –Procedural/Resuscitation Competency ACGME targets reached by the end of PGY 3 –360 degree evaluations by the Clinical Nursing Group and Resident Peers –Self-assessment If need remediation, during PGY 4 –PGY 4 will be promoted with a remediation clause and a signed contract of expectations prior to granting of graduation the following year –Remediation with appropriate mentorship and resources

15. Core Competency Assessment in EM Lessons Learned Programmatic change –Time –Transparency –Engagement –Expectations and resistance –Action Plans with reasonable deadlines –Communication –Importance of review

16. Core Competency Assessment in EM Lessons Learned Highlights –Resident Portfolios Program manager meetings Adult learner Contract Professionalism Checklist PGY 3 Promotion Criteria –360 degree evaluations Peer and Clinical Nursing Leadership Group –Self-Assessment 360 degree evaluation Self-reflection questions Goals for year

17. Core Competency Assessment in EM 1.Residency programs are expected to be fully compliant and integrated with the ACGME core competency assessment guidelines by 2011 2.Residency programs should be proactive in designing and implementing an evaluation and assessment program that best fits their needs based on institutional strengths and available resources 3.We offer a comprehensive assessment program designed and implemented at our institution that is sensitive to the residents, faculty, and residency program needs for accountability, growth, remediation, and improvement Conclusions

18. Questions? Contact Information: Christian Arbelaez, MD, MPH 617-732-8947 carbelaez@partners.org