1. Educational Outcomes Service Group: Overview of Year One Lynne Tomasa, PhD May 15, 2003

2. Education of Program Directors and Faculty  Program directors were familiar with requirement and toolbox  Programs were mainly utilizing global assessment tools  Programs rarely utilized direct observation and evaluation  Evaluation tools were used mainly for summative feedback

3. Faculty Buy-In  Program directors did all or most of curriculum development, so group buy-in was minimal  Junior faculty members had limited experience in curriculum design/teaching  Programs with faculty at multiple sites had additional challenges

4. Curriculum Design  Program coordinators were helpful but had various amounts of expertise in this area  Many program directors were new and junior and had minimal experience in curriculum development

5. Six ACGME Domains  Overlapped with each other  Forced one to define it according to specialty  Patient care and medical knowledge were easiest to define and understand  Professionalism: “know it when you see it” but behaviors were not clearly documented

6. Six Domains  Interpersonal and communication skills made sense but often did not incorporate colleagues, staff, and other health professionals  Practice-based learning and improvement and systems-based practice were “fuzzy” and forced one to think outside the “box”  Systems-based practice was least intuitive  Health care professionals were now viewed as part of a team

7. Paradigm Shift  To Educational Outcomes  From Process and Structure Compares learner’s performance with that of a peer group Fails to provide clear understanding of what resident can or cannot do Cannot determine if benchmarks or performance indicators have been met

8. Educational Outcomes  Clearly defines curricular objectives or benchmarks to describe competencies  Faculty must be more accountable and possess the skills/competencies themselves  Requires a structure to record, collect, and analyze data  More costly to implement

9. “Competency” Defined A complex set of behaviors built on the components of knowledge, skills, attitudes, and “competence” as personal ability. A complex but demonstrable integration of numerous related objectives, the latter being discrete measurable behaviors. Carraccio, Wolfsthal, Englander, Ferentz, Martin, Academic Medicine, May 2002

10. Curricular Design Stepwise Approach 1. Competency identification 2. Determination of competency components and performance levels 3. Competency evaluation 4. Overall assessment of the process Carraccio, et al. 2002

11. Step 1: Competency Identification  Consensus of individual experts  Group consensus  Task analysis (document all activities over a period of time)  Critical-incident survey (describe observed incidents that reflect good or bad practice)  Behavioral-event interview (star performers describe critical clinical situations and characteristics of a good doctor)  Practitioner surveys

12. Step 2: Competency Components  Identification of “tasks” that sequentially or in sum, make up the competency  Benchmarks or performance indicators  Must be measurable  In aggregate, determine achievement of the specific competency

13. Step 2: Performance Levels  Performance levels set threshold for demonstrating competence  Each benchmark must be clearly defined to determine whether competence has been achieved  Must determine the methods by which the competency might be attained

14. Step 3: Competency Evaluation  Criterion-referenced measures compare performance against a set standard or threshold are the preferred methods  Adult learners  Incorporates a variety of methods and strategies for adult learners

15. Program Directors’ Concerns  Where’s the evidence? There is little Studies have utilized small numbers  PD time commitment  Faculty buy-in and education

16. Responses to ACGME Requirements  I wish this goes away  I’ll wait until our specialty board tells us exactly what to do  I/We don’t have the time  I have a large faculty in a variety of sites  This is one more thing we have to do  How do we know that this shift will produce better, more skillful physicians?

17. Common Themes  Residents were not an active participant in identifying competencies  Residents have either none or minimal knowledge about the goals of the ACGME Outcomes Project  Program directors feeling “overwhelmed” at the enormity of task

18. Common Themes  Want clearer guidelines: no consensus on what outcomes are desired  On one hand, Program Directors want autonomy and on the other they want more direction  Question of what outcomes are desired

19. Examples of Desired Outcomes  Improved career satisfaction  Decreased medication errors  Decreased hospital stays  Better balance between personal and professional life  Longer life for patients  Better compliance with preventive health measures

20. Motivating Factors  RRC visit in the near future  Enthusiastic faculty with interest in competency-based curriculum

21. Non-Motivating Factors  Implementation of duty-hours takes time away from teaching and curriculum development  RRC visit occurred just prior to July 2002  Lack of support from Director of Medical Education