1. Preparing the Future Primary Care Workforce Together
Primary Care Faculty Development Initiative (PCFDI)
CBME in the Ambulatory Setting
Nov

2. Outline CBME background Frameworks and outcomes
Key concepts and definitions
Frameworks and outcomes
Where we are/where we need to be
The role of milestones and entrustment in the assessment and evaluation of competence

3. Competency versus Competent
Competency: an observable ability of a health professional, integrating multiple components such as knowledge, skills, values and attitudes.
Competent: demonstrating the required abilities in all domains in a certain context at a defined stage of medical education or practice.
Adapted from: The International CBME Collaborators, 2009 3

4. What does competency-based medical education means to you?

5. Competency-Based Medical Education
is an outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using an organizing framework of competencies
Bottom line:
CBME = Outcomes-based Medical Education (OBME)
So what is the outcome, and what is the framework?
The International CMBE Collaborators 2009

6. The Framework: ACGME Competencies
Medical knowledge
Patient care and procedural skills
Interpersonal and communication skills
Practice-based learning and improvement
Systems-based practice
Professionalism

7. Outcome?

8. The Outcome
Frenk J, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010

9. What is the outcome?
A competent (at a minimum) practitioner aligned with:
IOM Six Aims for Quality
CMS Triple Aim
National Priorities Partnership

10. Individual Physician Readiness: The Gaps
Office-based Practice Competencies
Inter-Professional team skills
Clinical IT Meaningful Use skills
Population management skills
Reflective practice and CQI skills
Care Coordination
Continuity of Care
Leadership and management skills
Systems thinking
Procedural Skills
The health care system is complex. The comment is frequently made that it is not due to the physician. It’s the system, the payers, the incentives etc. However, you can look at physician performance. Jay Crossen did just that in recent graduates entering the Kiaser –Perm Health care system in California. This is the data on all trainees and is not specialty specific.
Crosson Health Affairs 2011

11. Is CBME/OBME Just a “Fad”?
Pet rocks
Leisure suits
Streaking
Disco music
Yugos
Pokemon
Tickle me Elmo
…probably not…

12. Is CBME/OBME a Paradigm Shift?
Thomas Kuhn (1962): “Normal science, the activity in which most scientists inevitably spend almost all of their time, is predicated on the assumption that the scientific community knows what the world is like. Much of the success of the enterprise derives from the community’s willingness to defend that assumption, if necessary at considerable cost”
Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press. Chicago Pg. 5.

13. Could the Same be True of UME and GME?
“Normal medical education, the activity in which most faculty inevitably spend almost all of their time, is predicated on the assumption that the medical educational community knows what the world is like. Much of the success of the enterprise derives from the community’s willingness to defend that assumption, if necessary at considerable cost”
Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press. Chicago Pg. 5.

14. Is CBME/OBME a Paradigm Shift?
Maybe…but perhaps that is not the main point:
CBME is yet another stage on what should be the ongoing evolution and improvement of medical education
The focus on outcomes is worthy of our attention

15. The Transition to Competency
Fixed length, variable outcome
Structure/Process
Knowledge acquisition
Single subjective measure
Norm referenced evaluation
Evaluation setting removed
Emphasis on summative
Competency Based
Education
Competency Based
Knowledge application
Multiple objective measures
Criterion referenced
Evaluation setting: DO
Emphasis on formative
Variable length, defined outcome
Caraccio et al 2002

16. The definition of expected outcomes or competencies
Milestones
The definition of expected outcomes or competencies

17. Milestones
A significant point in development that identifies the discrete knowledge, skills, and attitudes expected of learners as they progress through training.
Milestones should enable the trainee, program and the certification board to know an individuals trajectory of competency acquisition.

18. Dreyfus & Dreyfus Development Model
PGY3
Expert/
Master
PGY1
Proficient
MS4
Competent
MS3
Advanced Beginner
Reality is skills develop over time and training. We expect different levels of competence for different skills. For example history taking
Novice
Time, Practice, Experience
Dreyfus SE and Dreyfus HL. A 1980
Carraccio CL et al. Acad Med 2008;83:761-7

19. Reporting Milestones-IM

20. “Curricular” Milestone
ACGME
Competency
Developmental Milestones
Informing
ACGME Competencies
Approximate
Time Frame
Trainee to
Achieve
Stage
Assessment
Methods/Tools
Clinical skills and reasoning
Manages patients using clinical skills of interviewing and physical examination
Historical Data Gathering
Acquire accurate and relevant history from the patient in an efficiently customized, prioritized, and hypothesis driven fashion
Seek and obtain appropriate, verified, and prioritized data from secondary sources (e.g. family, records, pharmacy)
Obtain relevant historical subtleties that inform and prioritize both differential diagnoses and diagnostic plans, including sensitive, complicated, and detailed information that may not often be volunteered by the patient
6 months
9 months
18 months
Standardized patient
Direct Observation
Simulation
I specifically have a box around the 18 month milestone in preparation for the next slide…
Sub-bullet
“Curricular” Milestone

21. Milestones Benefits
Provide the learner with a clear path of progression
There are no surprises
Allow for rich formative feedback. Learners know where they are and where they need to go
Define specific behaviors that can focus assessment

22. Milestones Criticisms
Milestones are too reductionist
Checklist = competence
Checking off a milestones list does not equal competent practice in a highly complex health care environment
Operationalize the milestones to develop and apply meaningful assessment and evaluation.

23. Entrustment/Entrustable Professional Activities (EPAs)
A framework for work-based assessment?

24. Entrustable Professional Activities
EPAs represent the routine professional-life activities of physicians based on their specialty and subspecialty
The concept of “entrustable” means:
‘‘a practitioner has demonstrated the necessary knowledge, skills and attitudes to be trusted to independently perform this activity.’’1
1Ten Cate O. Acad Med. 2007;82(6):542–547.

25. An Entrustable Professional Activity
Part of essential work for a qualified professional
Requires specific knowledge, skill, attitude
Acquired through training
Leads to recognized output
Observable and measureable, leading to a conclusion
Reflects the competencies expected…
EPA’s together constitute the core of the profession
Here is a table that was produced by Olle ten Cate who developed the framework for EPA’s and has been instrumental to the work of our community. I like this example because it provides some concrete description of what an EPA might look like –
Part of the essential work for qualified professional – similar to what I just read
Requires specific knowledge, skill, attitude – makes sense if it is
Acquired though training - not something that you are born with or comes pre-packaged
Leads to a recognized output – you know it when you see it
Observable and measurable –
Reflects the competencies expected (competencies – the things that they are able to do)
The Milestones fit into this picture here with the last two bullets – where Milestones link to EPA’s – more on this to come.
Also important to note that the EPA’s together constitute the core of the profession – emphasize together
Consider these criteria when reviewing some examples of what I think are EPA’s.
ten Cate et al.
Acad Med 2007

26. “Entrustment in Medical Education”
Focused assessments around what
faculty and training programs already
“entrust” trainees to do?
Reflects the most important outcome of training: a trainee’s readiness to bear professional responsibility”
Enables work-based assessment focusing on demonstrating competence in desired outcomes of training.

27. Baystate Ambulatory LMT Model
Learners: have Direct supervision
Faculty member sees every patient
Managers: have Indirect supervision
Faculty member discretion to see patient
Teachers: Oversight from faculty
Resident discretion to allow patients to leave before precepting
Adapted from Sudeep K. Aulakh & Michael J. Rosenblum. Presented at ICRE 2012, Ottawa.

28. Ambulatory Milestone: Demonstrates patient-centered interviewing using the Invite, Listen, Summarize format
Failure -Frequently does not use these skills
Needs work -Inconsistently uses these skills
Competent -Consistently uses Invite & Listen; Summarizes in a reporter fashion
Proficient -Consistently uses all three skills, Summarizes interpreted information
Expert -Consistently uses all three skills, Summarizes interpreted information in complex cases
Competencies: Interpersonal communication & Patient Care
Adapted from Sudeep K. Aulakh & Michael J. Rosenblum. Presented at ICRE 2012, Ottawa.

29. Competencies, Milestones and EPAs
Characteristic
Competencies
Milestones
EPAs
Granularity
Low
Moderate to High
Low to Moderate
Synthetic/Integrated
Moderate
High
Practicality (application)
Conceptual

30. The Synthesis – Analytic Tension
Physicians do not apply each competency independently in caring for patients
As a result, judging overall performance is a synthetic/integrative assessment activity
However,
You will often have to pull things apart (analysis) to create shared mental models and to provide meaningful and actionable feedback to the resident

31. What has this resident been entrusted to do?
Lets watch a video.
What has this resident been entrusted to do?
If this were your institution, could you attest that the resident had the required competence to provide this care?

32. With Your Neighbor -
Pick a goal of training that would meet a national priority for ambulatory-based care.
Identify two to three entrustments in training that could serve as a focus for assessment in that competency?
What assessment methods would you use?
What process will you use to make an entrustment decision?
What facilitators and barriers would you face?

33. Judgment and Synthesis: as Guiding Framework and Blueprint
The “System”
Accreditation:
ACGME/RRC
Institution and Program
Residents
Program Aggregation
Assessments within Program:
Direct observations
Audit and performance data
Multi-source FB
Patient experience
Simulation
ITExam
Judgment and Synthesis:
Committee
NAS Milestones
Board Reporting
No Aggregation
Faculty, PDs and others
Certification:
Board
Milestone and EPAs
as Guiding Framework and Blueprint

34. Break