1. Milestones: Are You Ready for July 1st ?
Kirk Lalwani, MD, FRCA, MCR
Associate Professor,
Chair,
Resident Evaluation and Clinical Competence Committee,
Anesthesiology and Perioperative Medicine.
On behalf of the AMIGOS
(Anesthesiology Milestones Implementation Group Operatives)

2. Disclosure No conflict of interest
Some of my slides are borrowed from the faculty educational resources on the ACGME website

3. Implementing Milestones
Objectives:
Outline the Next Accreditation System (NAS)
Define a Milestone
Review our Milestones strategy
Discuss assessment of Anesthesiology Milestones
Three key points will be addressed in this presentation.

4. Abbreviations NAS- Next Accreditation System
CLER- Clinical Learning Environment
ACGME- Accreditation Council for Graduate Medical Education
RRC- Residency Review Committee

5. The Next Accreditation System
This slide presentation provides general information regarding the Next Accreditation System (NAS), including the reasons for its development, the components of NAS, and the differences between NAS and the “old” Accreditation System.
This slide deck was created in December 2013 and care should be taken to ensure that updates and changes that have occurred since then are incorporated to provide an up to date and accurate presentation.

6. Why ‘Next Accreditation System’ (NAS)?
“Self-regulation is a fundamental professional responsibility, and the system for educating physicians answers to the public for the graduates it produces.”
“The Next GME Accreditation System – Rationale and Benefits”
Nasca T.J., Philibert I., Brigham T., Flynn T.C.
N Engl J Med 2012; 366:

7. Goals of NAS Strengthen resident development in
Professionalism
Communication Skills
Systems Based Practice
Practice Based Learning
Enhance public accountability
More explicit definition of a good physician (Milestones)
Patient safety (Clinical Learning Environment Review Program)
Reduce burden required for accreditation

8. ‘Next Accreditation System’ …in a Nutshell
Accreditation on the basis of educational outcomes
NAS : Advance from an episodic ` biopsy ’ model to annual data collection or ` continuous oversight ’.
RRCs will measure compliance through the evaluation of annual program data elements.

9. The Next Accreditation System
Continuous
Observations
Promote
Innovation
Assess
Program
Improvement(s)
Identify Opportunities for Improvement
The NAS seeks to provide continuous observations and identify opportunities for programs to improve and ensure that programs in good standing are free to innovate with regard to Detail requirements.
Program
Makes
Improvement(s)

10. The Next Accreditation System
We will now have annual data collection
Trends in annual data
Milestones
Resident and Faculty Surveys
Resident and Faculty Scholarly activities
Case Log Data
ITE / ABA Examination pass rates
Programs will be evaluated annually and data reviewed include trends, Milestone levels, results of Resident and Faculty surveys, and scholarly activity for faculty members, residents and fellows. For clinical experience, Case Logs will be reviewed for those specialties that require Case Log reporting. For specialties that do not have Case Log reporting, questions that address the clinical environment have been added to the Resident Survey. In addition, programs that receive Continued Accreditation will be free to innovate on Detail requirements.

11. Clinical Learning Environment Review Visits
Oversight of
Transitions of Care
Duty Hours Policy
Fatigue Management and Mitigation
Education and Monitoring of Professionalism
Involvement in Institutional Quality Improvement and Safety initiatives
Supervision policies
Components of the site visit – encourage people to review the slides and presentations on the ACGME website concerning the CLER visits.
Additional information available at:

12. Milestones

13. What are Milestones?
Specific behaviors, attributes, or outcomes in the six general competency domains to be demonstrated by residents during residency.
(ACGME and Specialty Boards)
Skill and knowledge-based development that commonly occurs by a specific time.

14. Why ????
ACGME requires it
Program Accreditation

15. Who, When, and How???? All residents in all ACGME-accredited programs
Subspecialty fellows from July 2015
New assessment methodology as well as existing tools
6-monthly reporting of Milestones to ACGME in January and July
Reporting starts July 2014

16. Expected Benefits of Milestone Assessments
For Residents
Clarify expectations
Feedback should identify specific areas to work on
Earlier identification of under-performers
Aspirational goals for residents who exceed expectations
In early experience with Milestone assessment, some observations have been that residents appreciate the more detailed and specific evaluations that milestones allow, compared to previous evaluations merely stating they had done a “good job.” Milestone evaluations provide a process to set expectations on what more there is to learn or what a resident has to accomplish to move to the next level. One of the goals in looking at each resident more closely is to identify residents who have gotten by because they are nice and easy to work with, but who actually had significant deficiencies in different skills and areas in the subcompetencies. The focus on specific skills and knowledge in the Milestones help to identify these residents and determine which in competency or subcompetency areas in which they need to improve.

17. Expected Benefits of Milestone Assessment
For the Program
Curriculum development
Accreditation requirements
Earlier identification of under-performers
For the Public
Better definition of what a physician can do at the completion of training
Program is accountable to a common standard
Possible use for board certification
Some program directors have also suggested other benefits. In looking at the Milestones, a program may discover that the current curriculum may not be teaching a competency to the expectations set for Level 4. The benefits to the public may be less clear to individual programs, but allow policy makers to understand some of the specific skills that future physicians in practice are learning. Currently the ACGME only expects each program to collect and use the Milestones in assessing their residents. The ACGME does not expect that every resident meets a specific level at a particular point in residency. The goal is to use Milestone assessment to thoughtfully and honestly evaluate each resident. The Review Committee will not look at an individual resident’s Milestone attainment. Each specialty board will determine on its own whether and how to use Milestone data in board certification.

18. Pragmatic Milestones Development Strategy Minimalist Comprehensive
Program implements new milestone aligned tools, which help residents understand expectations
Comprehensive
Program changes assessment system, invests time in faculty development and education to improve rater expertise and CCC validity
Minimalist
CCC assigns milestone levels using existing tools

19. Process: June 2013
Assembled AMIGOS Group from RECCC, faculty, and residents to represent all clinical areas
Survey to rank importance of each Milestone, following which 17 / 25 chosen
Small groups assigned 2-3 Milestones each to develop instruments or methodology
Large group review and feedback
Education office worked with Evalue to incorporate tools into system

20. General Strategy “Good now is better than perfect later….”
Priorities
Ease of use
Should not be burdensome for residents, faculty, RECCC or education office staff
No PAPER !
App for ‘on-the-go’ assessments - procedures, H&P
Implementation feasibility
Ready by July 2014
Integrate with existing system (Evalue)
Ensuring adequate data to assess Milestones comprehensively
A Milestone for every competency
17/25 Milestones chosen
Work in progress, as refinements will be made and new tools added over time

21. Which Milestones did the AMIGOS select?

22. Medical Knowledge (MK): 1
MK1- Knowledge of biomedical, clinical, epidemiological, and social sciences as outlined in the American Board of Anesthesiology Content Outline
ITE
AKT
ABA Basic Examination
ACLS certification
Direct Clinical Observation

23. Patient Care (PC): 8 of 10
PC1-Preanesthetic Evaluation, Assessment, and Preparation
PC2- Anesthetic choice and conduct
PC3- Periprocedural pain management
PC4- Management of perianesthetic complications
PC5- Crisis management
PC6- Triage and management of critically ill patient in a non- operative setting
PC7- Acute, chronic, and cancer related pain consultation and management
PC8- Technical skills: Airway management
PC9- Technical skills: Monitoring and Equipment
PC10-Technical skills: Regional anesthesia

24. Example of PC-8 Assessment
PC-8 (Technical skills: Airway management)
Direct Clinical Observation (DCO)
Daily evaluation form
Airway management checklist
Objective Structured Clinical Examination (OSCE) / Objective Structured Assessment Test (OSAT)
Simulation / Standardized Patient (SP)

25. Professionalism (Prof): 3 of 5
Prof1- Responsibility to patients, families, and society
Prof2- Honesty, integrity, and ethical behavior
Prof 3- Commitment to institution, department and colleagues
Prof4- Receiving and giving feedback
Prof 5- Responsibility to maintain personal, emotional, physical and mental health

26. Example of Prof-1 Assessment
Prof 1 (Responsibility to Patient, Family, and Society)
360 degree evaluations
Patients
Healthcare team members
Peer evaluations
Administrative staff
Direct Clinical Observation (DCO)
Daily evaluation form
Written comments

27. Interpersonal and Communication Skills (ICS): 2 of 3
ICS1- Communication with patients and families
ICS2- Communication with other professionals
ICS3- Team and Leadership skills

28. Practice Based Learning (PBL): 2 of 4
PBL1 Incorporates quality improvement and patient safety initiatives into personal practice
PBL2- Analysis of practice to identify areas in need of improvement
PBL3- Self-directed learning
PBL4- Education of patient, family, students, residents and other health professionals

29. Systems Based Practice (SBP) 1 of 2
SBP1- Systems-based approaches to patient care
SBP2- Coordination of patient care within the healthcare system

30. How will we Assess Milestone Levels?
Milestones are a summary of how a resident is progressing based on information from other sources
Milestones are not assessment tools to be used for gathering detailed information
They do not replace end-of-rotation forms, simulation, 360’s, Daily evaluations etc.
More pieces of data allow for more precision
There will be a minimum number of evaluations required for procedural assessments
The Milestones will ultimately be a summary of the information gathered by several evaluations. Some subcompetencies may not be measured on every rotation or every experience. For example, if residents must complete one quality improvement project or one scholarly activity during residency, there may be just a single assessment during the program for that project.
Some milestones may be difficult to measure with existing assessment tools or forms, and new methods may have to be developed. For example, a resident might need to serve on a quality committee with an assessment by the committee chair to address certain questions. An individual program may not have collected this information before, so a new assessment tool to address the milestones for this activity might be necessary.

31. Clinical Competency Committee (RECCC)
Self
Evaluations
Case Logs
Mock Orals
Professionalism
Indicators like
pharmacy, billing, evals etc
End-of-Rotation Evaluations
Unsolicited
Comments
ITE
Sim
Lab
Student Evaluations
Nursing and Ancillary Personnel Evaluations
Clinical Workplace Evaluations
Clinical Competency Committee
Overall the goal is to have the CCC review input from different sources. Assessment methods might include the evaluations that are currently being used by programs. Not all of the evaluations of one resident will be consistent, so the role of the CCC will be to consider them all and make an overall decision on how the resident is progressing, which milestones have been met, and which to continue to work toward.
OSCE
Peer Evaluations
Patient/
Family Evaluations
Assessment of Milestones

33. Anesthesiology Milestones
Level 4 is the Graduation Target
Not every resident will achieve Level 4 in every Milestone
Residents required to substantially meet most Milestones at Level 4
Residents are not expected to achieve Level 5 during residency
Residents may achieve a level of competency in specific Milestones sooner than expected
Milestones permit the identification of outstanding performance exceeding what is expected and usually observed in a resident

34. Milestone Template
This is what the five levels look like in the tabular form that will be used for Milestone reporting. Note that the form offers check-off boxes between Levels, recognizing that a resident might be best described at Level 2.5, for example, if he or she has met most of the Level 2 and some but not all of the Level 3 milestones. Level 5 milestones are aspirational goals that only a few exceptional residents will reach, but that not every resident would be expected to attain. Level 4 describes the graduation target, but every resident does not need to reach Level 4 in every competency or subcompetency in order to graduate. It is still the purview of the program director to determine if and when a resident is ready for independent practice.

35. Developmental Progression or Set of Milestones
General Competency
Developmental Progression or Set of Milestones
Subcompetency
Milestone
PC1. Preanesthetic Evaluation, Preparation
Level 1
Level 2
Level 3
Level 4
Level 5
Performs comprehensive H&P
Identifies medical issues that may affect anesthesia care
Identifies elements and process of informed consent
Identifies medical issues relevant to anesthesia care
Optimizes preparation of non-complex patients
Obtains informed consent but recognizes when assistance is needed
Identifies medical issues relevant to subspecialty anesthesia care
Optimizes preparation of complex or subspecialty patients
Obtains informed consent of patients with complex problems or requiring subspecialty anesthesia care
Performs assessment of complex or critically ill patients with conditional independence
Obtains informed consent in complicated clinical situations with conditional independence 
Independently performs assessment for all patients
Independently serves as a consultant to other members of the healthcare team
Consistently ensures that informed consent is obtained by using all available resources
It will help to agree on some terminology. The “General Competency” refers to one of the six Core Competencies. In this example, “PC” is for “Patient Care”, one of six Core Competencies. “PC1. History (Appropriate for age and impairment)” is a subcompetency under the first of several subsections of “Patient Care”. A Milestone is actually the description of how a resident is doing and the entire set of milestones for one subcompetency can be called a Set of Milestones. Notice that the Milestone narratives do not indicate level of education or the years in the program but by levels merely to signify that the next Milestone is more advanced than the level below it. Residents are expected to progress at different rates.

37. Instruments to Measure Milestones
Daily Evaluation Form for OR (Evalue)
Daily Evaluation Form for ICU (Evalue)
Daily Evaluation Form for Pain (Evalue)
Technical Skills forms (Evalue) (‘App’?)
Arterial line
CVP Line
Nerve Block / Epidural / Caudal
Airway Management
Peer evaluations
Assessment of Patient Interaction
Regional assessments
Simulation assessments

38. ‘QUASAR’ Project:
Quality in Anesthesiology,
Systems Assessment and Research

39. Resident ‘QUASAR’ Project: Quality in Anesthesiology, Systems Assessment and Research
Quality Improvement
Systems Based Practice
Practice-Based Learning
CA1- Background research, identify area of study
CA2- Identify quality measure to be studied and change to be implemented, study effects, refine
CA3- Write summary, submit abstract to national conference, or write a manuscript for publication

40. Summary Next logical extension of competency-based evaluation
Mostly familiar, not burdensome
Levels 1 and 2 do NOT signify a ‘Fail’ grade
Evaluations will be more specific and focused
Direct clinical observation to evaluate procedures and patient interactions

41. We Need Your Data !
APOM NEEDS YOU

42. Acknowledgements AMIGOS Nicole Conrad Oumou Diallo Judy Freeman
Julio Gonzalez
Karen Hand
Izumi Harukuni
Amy Miller Juve
Diana Kim
Ed Kahl
Dean Lao
David Larsen
Kim Mauer
Michele Noles
Annie Riley
Peter Schulman
Katie Seligman
Chris Swide
Linda Wylie
David Wilson
Glenn Woodworth