1. Eric Holmboe, MD Employment: ACGME
56th ASH Annual Meeting Disclosure Statement
Eric Holmboe, MD
Employment: ACGME
Discussion of off-label drug use: not applicable

2. Realizing the Promise of Competency-based Medical Education: From Theory to Practice

3. Disclosures Employed by the ACGME
I receive royalties from Mosby-Elsevier for a textbook on assessment
I am a member of the board of NBME and Medbiquitous

4. Outline Why CBME and why now? CBME and educational theories
External forces driving change
Nostalgialitis Imperfecta
CBME and educational theories
CBME, NAS and professional self-regulation

5. CBME: Start with System Needs
Frenk J, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010 5

6. Early Principles: CBME
World Health Organization (1978):
“The intended output of a competency-based programme is a health professional who can practise medicine at a defined level of proficiency, in accord with local conditions, to meet local needs.”
McGaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based Curriculum Development in Medical Education. World Health Organization, Switzerland, 1978.

7. A competent (at a minimum) practitioner aligned with: CMS Triple Aim
What Are The Outcomes?
A competent (at a minimum) practitioner aligned with:
CMS Triple Aim

8. Current Realities: Health System to Training System Performance

10. AHRQ Quality Report 2013 Measure Focus Measure Name/Description
Baseline Rate
Most Recent Rate
Aspirational Target
Aspirin Use
Outpatient visits at which adults with cardiovascular disease are prescribed/maintained on aspirin
47%13
53%14
Increase to 65% by 2017
Blood Pressure Control
Adults with hypertension who have adequately controlled blood pressure
46%15
53%16
Cholesterol Management
Adults with high cholesterol who have adequate control
33%17
32%18
Smoking Cessation
Outpatient visits at which current tobacco users received tobacco cessation counseling or cessation medications
23%19
22%20

11. Arnie Milstein 2010
Since physician graduates of American medical education organizations typically lead or heavily influence US health care delivery, one source of indirect, broad, outcome-based evidence [of the effectiveness of the medical education enterprise] is the overall performance of the US health care system. The width of the performance gaps on the aims of effectiveness, safety and efficiency understandably reduces society’s confidence that physicians are adequately honoring their Hippocratic promises.
Milstein A. Trailing Winds and Personal Risk Tolerance: An External Perspective on the Opportunity for Medical Educators to Fulfill Their Social Contract Permanently. Presented at ABIMF Summer Forum, August 2010

12. Policymakers Raising Concern
Institute of Medicine (2008)
Resident Duty Hours: Enhancing Sleep, Supervision, and Safety
Retooling for an Aging America
Congress ( )
Reductions in GME funding
Request to IOM to review GME regulation
MedPAC ( )
June 2010 Report influenced reform legislation
Institute of Medicine (2014)
More accountability for GME funding
Innovation fund

13. Evaluating Residency Programs Using Patient Outcomes JAMA 2009;302(12):1277-1283. Asch, DA, et.al.
Difference remains
after correction for
USMLE performance
Excess Risk ∆ 32%
Q1 vs Q5 ∆

14. The Medical Students “Strike Back”
Average # of physician visits in last six months of life (teaching hospitals in red)
From:
What Kind of Physician Will You Be?
Variation in Health Care and Its Importance for Residency Training
Dartmouth Institute for Health Policy & Clinical Practice 2012

15. Environment and Conservative Practice
Sirovich BE, Lipner RS, Johnston M, Holmboe ES. The Association Between Residency Training and Internists’ Ability to Practice Conservatively. JAMA IM

16. Nostalgialitis Imperfecta
Syndrome characterized by the following signs and symptoms:
“When I was an intern…<insert superlative>”
“Medicine was so much better 25 years ago”
Reality: Not really…
“Younger physicians today are less professional, skilled, etc. because of <insert favorite complaint>”

17. Harvard Medical Practice Study
Methods:
Investigated prevalence of adverse events due to medical management
Review of 30,121 medical records from 51 randomly selected acute care hospitals
Results:
Adverse events occurred in 3.7% of hospitalizations
27.6% due to medical negligence
13.6% resulted in death

18. Harvard Medical Practice Study
Study conducted in 1984 in the state of New York
My senior year ( ) as a medical student at the University of Rochester

19. Past, Present and Future
“Those who forget the past are condemned to repeat it” George Santayana “The blind spot of contemporary [education] is experience” Francisco Varela

20. The “Miracle” of Medical Education

21. Part of the Solution (Camelot), or Just a Fad (Trojan Horse)?
Competency-Based Medical Education
Part of the Solution (Camelot), or Just a Fad (Trojan Horse)?

22. Is CBME Just a “Fad”? …probably not… Pet rocks Leisure suits Streaking
Disco music
Yugos
Pokemon
Tickle me Elmo
…probably not…

23. Is CBME a Paradigm Shift?
Maybe…but perhaps that is not the main point:
CBME is yet another stage on what should be the ongoing evolution and improvement of medical education
The focus on outcomes is worthy of our attention

24. What Exactly is CBME?
An outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using an organizing framework of competencies1
1Frank, JR, Snell LS, ten Cate O, et. al. Competency-based medical education: theory to practice. Med Teach. 2010; 32: 638–645

25. Origins of CBET CBET Behaviorism Thorndike
Scientific Management Taylor
Progressive Education Dewey
Operant conditioning
Objective-based instruction
Minimum competency tests
Mastery-based learning
Criterion-referenced tests
Instructional design
CBET
McCowan; CDHS, 1998

26. Experiential Learning: David Kolb

27. Socio-cultural Theory: Key Principles
Subject matter and learning processes not uniform: diverse as the people
Learning highly influenced by social milleau
Learning mediated by artefacts and “sign” systems (e.g. language)
Learning situated within context where it occurs
Subject matter, content and process inseparable
Adversarial interactions (people or institutions) produces different learning
Yardley S, Teunissen PM, Dornan T. Experiential learning: AMEE guide 63. Med Teach. 2012; 34:e

28. Experiential Learning
Predominant Curriculum in GME
Identity
key outcome
Yardley S, Teunissen PM, Dornan T. Experiential learning: AMEE guide 63. Med Teach. 2012; 34:e

29. Deliberate Practice Ericsson & Lehmann, 1996:
“Individualized training activities
especially designed by a coach or teacher
to improve specific aspects of an individual's performance through repetition and successive refinement.
To receive maximal benefit from feedback,
individuals have to monitor their training
with full concentration, which is effortful
and limits the duration of daily training”.

30. Deliberate Practice and Expertise
From Anders Ericsson: Used by Permission

31. Design and Sequencing of Training Activities
* Monitor students’ development
* Design and select training tasks for individual students
Professional teachers and coaches
From Anders Ericsson: Used by Permission

32. Expert Performance vs. Everyday Skills
What this picture illustrates is the qualitative difference between the course of improvement of expert performance and of everyday activities.
In the initial phase of learning, or the cognitive phase, novices try to understand the activity and concentrate on avoiding mistakes (example). This is where you are concentrating on what you are doing.
In the middle phase of learning, the ‘‘associative’’ phase, performance is smoother. You don’t need to concentrate as hard to perform at an acceptable level, and salient mistakes increasingly rare.
Finally, after a period of training and experience, individuals are able to perform the tasks automatically and with a minimal amount of effort. However there is a consequence to this automation.
When automaticity occurs, people lose conscious control over execution of the skill, and can no longer make specific intentional adjustments or modifications. When this automated phase of learning as been attained, performance reaches a stable plateau with no further improvements.
Expert performers are able to counteract this automaticity and they remain in the cognitive or associate phase of learning. The way that they accomplish this is by deliberately constructing and seek out training situations in which the desired goal exceeds their current level of performance. This means they continue to practice. That is part of our job- to help trainees construct and seek out training situations that can further their skills set. But it is not any type of practice- and people have studied the way an expert practices, vs someone who is just very good.
Ericsson KA. Acad Med. 2004

33. Other Key Theories Self-determination Theory Self-efficacy Theory
(Ryan and Deci)
Intrinsic motivators much better than extrinsic motivators
Related to sense of autonomy and independence
Self-efficacy Theory
(Bandura)
Effectiveness at task leads to further learning and development
We don’t like being “bad” at things

34. Effective Learning Strategies
Practice testing
Concept of retrieval practice
Can be very simple – write down what you just learned
Multiple low-stakes practice tests quite effective for retention
Interleaved practice
Mix different concepts and competencies into the learning
Elaborative Interrogation and Self-Explanation
Distributed practice
Willingham DT.

35. Getting to Expertise…

36. Translating CBME into Action: The Milestones, NAS and Professional Self-Regulation

37. The Milestones and NAS in a Nutshell
A Continuous Accreditation Model based on assessment of annual data – this list is not all encompassing and is subject to change
Annual program data (resident/faculty information, major program changes, citation responses, program characteristics, scholarly activity, curriculum)
Aggregate board pass rate
Resident clinical experience
Resident survey and faculty survey (latter is new)
Semi-annual resident Milestone evaluations
10 year Self-Study and Self-Study Visit
Clinical Learning Environment Review (CLER) Visits
Most of the annual program data will be the same information currently used to accredit programs. The difference is that each program will be considered annually by the RRC rather than every 5 years (or less) under the current system. The program data parameters considered integral to program performance will necessarily vary between specialties. Expect them to include most of the factors currently considered by the Residency Review Committees. Programs that seem to be doing well will remain accredited while programs performing below expectations will get a closer review by their respective RRC. For the first time, this system will also help to identify strong performers and (hopefully) foster recognition and adoption of “best practices” by all programs.
Milestones will be one element of this evaluation, but they are by no means the sole, or even primary, determinant of program accreditation. There is no minimum or average score required in order to remain accredited, but residents are expected to show progression over the course of residency training, and training programs are expected to use the Milestones data internally to refine and guide the curriculum and assessment approaches. The ACGME recognizes that reviewing Milestone attainment in the wrong way could be counterproductive if it were to cause programs to falsely inflate resident assessments out of fear of losing accreditation, and it is the ACGME’s intent that Milestones will be used honestly and thoughtfully in resident evaluation.
Programs will be expected to report the results of a self assessment to the ACGME every ten years, and this self-assessment will be associated with a 10 year self study ACGME site visit.
Institutions with ACGME-accredited programs will participate in the Clinical Learning Environment Review program to assess their clinical learning environment. Because the CLER program is meant to be a quality improvement tool, institutional transparency will be encouraged in order to maximize continuous improvement. In the spirit of quality improvement, the data collected during a CLER visit will not be used for program accreditation.

38. Dreyfus & Dreyfus Development Model
Expert/
Master
Proficient
Competent
Advanced Beginner
Reality is skills develop over time and training. We expect different levels of competence for different skills. For example history taking
Novice
Time, Practice, Experience
Dreyfus SE and Dreyfus HL. 1980
Carraccio CL et al. Acad Med 2008;83:761-7

39. Milestones
By definition a milestone is a significant point in development.
Milestones should enable the trainee and the program to know an individual’s trajectory of competency development.

40. Defining Competency Based Education
Frank did a systematic review of the literature to come up with a definition of CBME. Most descriptions of CBME focus on a defined outcome and learners advance along a series of defined milestones on their way to competence. The idea is that each of these milestones are demonstrable abilities that can be observed and assessed.
Assessment of competencies is criterion referenced, in that learners are measured against a set of standards and not to other learners. Keep this in mind- we will come back to this later.
So the key in CBME is authentic assessment of what trainees are doing
Observable
&
Assessed
Defined outcome
Frank JR et al. Med Teach. 2010;32:631-7

41. Milestones as Roadmap Observations: Journey not a straight line
More than one path (but not infinite paths)
“If you don’t know where you are going, any road will get you there”

42. Dreyfus & Dreyfus Development Model
MILESTONES
Curriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Expert/
Master
Proficient
Competent
Advanced Beginner
Reality is skills develop over time and training. We expect different levels of competence for different skills. For example history taking
Novice
Time, Practice, Experience
Dreyfus SE and Dreyfus HL. 1980
Carraccio CL et al. Acad Med 2008;83:761-7

43. What Milestones Are Not:
A complete description of:
Clinical Competence of any individual
The elements of competence in a specialty/subspecialty
Promotion Criteria
Graduation Criteria
The totality of a discipline
The sole determinants to be used in Competency Based Medical Education
“Tools” to Close Programs

44. Entrustable Professional Activities
EPAs represent the routine professional-life activities of physicians based on their specialty and subspecialty
The concept of “entrustable” means:
‘‘a practitioner has demonstrated the necessary knowledge, skills and attitudes to be trusted to perform this activity [unsupervised].’’1
1Ten Cate O, Scheele F. Competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Acad Med. 2007; 82(6):542–547.

45. Entrustment in GME
As faculty, we “entrust” trainees to do many things without direct supervision
Admit patients to hospital from the ED
Night float
Clinic preceptor sign-out (without seeing the patient)
What justifies these “entrustments”?
How do we know when and if to make such entrustments?

46. Dyad Conversation
What do you entrust your fellows to do with only reactive (indirect) supervision?
How do you decide?

47. CBME relies heavily on the judgments of humans.
There is No Holy Grail…
Holmboe’s Cousin
CBME relies heavily on the judgments of humans.
The goal is to enhance the probability of making better judgments for the benefit of both patients and learners

48. Professional Self-Regulatory System
Unit of Analysis:
Program
Residents FB
Assessments within Program:
Direct observations
Audit and performance data
Multi-source FB
Simulation
ITExam J U D G E M N T
Accreditation
Qual/Quant “Data” Synthesis:
Committee P U B L I C D D D FB
Certification and Credentialing FB
Faculty, PDs and others
Unit of Analysis:
Individual
Milestones and EPAs
as Guiding Framework and Blueprint

49. Revised Conceptual Model of Rapid Cycle Change
Milestone Journey:
Revised Conceptual Model of Rapid Cycle Change
Revised conceptual model of rapid cycle change.
Tomolo A M et al. Qual Saf Health Care 2009;18:

50. Thank You and Questions eholmboe@acgme.org