1. The ACGME Self-Study and 10-Year Site Visit
2015 Fall Program Director Retreat
September 24, 2015
Alan J. Smith, PhD, MEd
Assistant Dean and Director for GME
ACGME Designated Institutional Official

2. Disclosure
Adapted from original ACGME Webinar presentation by Susan Guralick, MD, DIO, Winthrop University Hospital, and Ingrid Philibert, PhD, MBA, ACGME Senior Vice President for Field Activities, June 18, 2015.

3. Objectives
Discuss elements of the self-study and ongoing program evaluation and improvement on which it is based
Explain the sequence and components of the self-study and the 10-year site visit
Describe program and institutional practices to assist with self-study preparation and execution
Explain the components of the 10-year accreditation site visit and Review Committee review

4. Self-Study Scope
Assess program performance and ongoing improvement efforts
Facilitate improvement in areas already in substantial compliance with the accreditation standards
Initial period: since last accreditation review
Ultimately a 10-year interval
Review improvement activities, successes achieved, and areas in need of improvement
Based on successive Annual Program Evaluations, ACGME Resident and Faculty Survey data, and other relevant information, stakeholder input

5. Strengths and Areas for Improvement
Important to acknowledge and celebrate
What should definitely be continued, especially in an environment of limited resources
Areas for Improvement identified by:
Citations, areas for improvement, and other information from ACGME
Annual Program Evaluation
Other program/institutional data sources
Expected: Longitudinal tracking of strengths and areas for improvement through successive Annual Program Evaluations

6. Improvement Viewed in Context: Program Aims, Opportunities and Threats
Aims as a way to differentiate programs
Set and reassessed as part of the Annual Program Evaluation
Context: Opportunities and Threats
External attractive factors that, if acted upon, will contribute to the program flourishing or may have a negative effect
While the program cannot fully control them, it is beneficial to have plans to mitigate their effect

7. Defining Program Aims
Aims are set and revisited/revised as part of the Annual Program Evaluation
Relevant considerations
Who are our residents/fellows?
What do we prepare them for?
Fellowship
Academic practice
Leadership and other roles
Who are the patients/populations we care for?

8. Benefits of a Focus on Program Aims
Suggests a relevant dimension of the program:
Types of graduates produced for specific community needs, practice settings and roles
Allows for a more “tailored” approach to creating a learning environment
Focus on specific aims can produce highly desirable graduates that match patient and healthcare system needs 1
Enhances the focus on functional capabilities of graduating residents
Fits with a milestones-based approach to assessment
1Hodges BD.”A Tea-Steeping or i-Doc Model for Medical Education?.” Acad Med85(9) Sept 2010, pp. S34-S44.

9. Defining “Opportunities”
Opportunities are external attractive factors that, if acted upon, will contribute to the program flourishing
What are capabilities for further evolving the program? How can the program capitalize on them?
Has there been recent change in the program’s context that creates an opportunity?
Are these opportunities ongoing, or is there a narrow window for them? How critical is the timing?

10. Defining “Threats”
Threats include external factors that affect the program
While the program cannot fully control them, it is beneficial to have plans to mitigate their effect
What factors beyond the program’s control may place it at risk?
What are the changes in residents’ specialty choice, regulation, or other factors that may affect the future success of the program?
Are there challenges or unfavorable trends in immediate context that may affect the program?
E.g., faculty burdened with heavy clinical load that prevents effective teaching and mentorship

11. Coordinated Self-Study for Core and Subspecialty Programs
Coordination of curriculum and program resources
Needs of core and subspecialty programs considered collectively
Subspecialties can access core resources
Core oversight of fellowships
Access common strengths, areas for improvement
Coordinated action plans for areas for improvement
Increase efficiency
Less time and resources spent, coordinated collection and review of data

12. Self-Study Process Who should conduct the self study?
Not defined by ACGME
Members of the Program Evaluation Committee (PEC) are the logical choice
PEC membership may be expanded (beyond required two faculty members and one resident)
Data gathering
Many potential sources, including ACGME Resident and Faculty Surveys, program and institutional data
Important: stakeholder (residents, faculty and relevant others) participation, input and perspective
Data as evidence to support conclusions

13. Self-Study Data Gathering
Annual Program Evaluation data, ACGME Resident and Faculty Survey data, other program and institutional data
Focus on data gathering as a learning experience
Evaluate strengths and areas for improvement
Explore opportunities and threats
Reflect stakeholder (residents, faculty, and relevant others) participation, input, and perspective
Data as evidence to support conclusions

14. Tracking Improvements
Design and implement solutions
Identify individual or group that will be responsible
Identify and secure resources
Timeline
Follow-up is key: endure all issues addressed
Documentation to facilitate ongoing tracking
A simple spreadsheet recording improvements achieved and ongoing (See handout 3)
Record improvements over multiple years

15. Resident Participation
Resident participation critical:
They are the beneficiaries of the educational program
They have first hand knowledge of areas that need improvement
Double benefit:
Residents help improve their own education
Resident participation in “educational QI effort” can be used to meet the requirement for resident involvement in quality and safety improvement

16. Self-Study and the Self-Study Summary
The program conducts its self-study
After the self-study, the program uploads the Self-Study Summary in ADS
Information on areas for improvement identified in the self-study are not included in the Summary
Timing consideration:
Programs expected to upload the self-study summary in the month the Review Committee indicated for their first site visit in NAS
Self-Study Site Visit scheduled months after uploading the Self-Study Summary in ADS

17. The Self-Study Summary
Brief document prepared by the program, uploaded in ADS (See handout 1)
ACGME Template: 2300 words (~4-5 pages) for core program, less for small subspecialty programs
Sections: Key Self-Study dimensions
Aims
Opportunities and Threats
Self-Study process
Who was involved, how were data collected and interpreted
How is ongoing improvement achieved through sequential Annual Program Evaluations
Omitted by design: information on strengths and areas for improvement

18. The 10-Year Site Visit A full accreditation site visit
Review of all applicable requirements
12-18-month period is by design, to allow programs to implement improvements
“Summary of Achievements”
ACGME template uploaded through ADS
~ 1200 words, describing key improvements accomplished from the self-study
Program provides information ONLY on the improvements that were realized from their self-study, no information on areas that have not been resolved
Program may provide an update to its Self-Study Summary

19. The 10-Year Site Visit Site visit by a team
Site visit team provides verbal feedback
Key strengths and suggestions for improvement
Site visit team submits a written report through ADS for the Review Committee (RC)

20. Review Committee Review of the 10-Year Site Visit
Available to the Review Committee:
ADS data
Program’s self-study summary and any updates
Program’s summary of improvements achieved as a result of the self-study (which does not include data on areas still in need of improvement)
The site visit report from the 10-year (accreditation) site visit
Review of program aims and context from the self-study summary allows the RC to assess self-study effectiveness
Improvements the program reports are paired with program aims and context from self-study summary
Data on improvements achieved are one measure of effectiveness

21. Review Committee Review of the 10-Year Site Visit
RC provides a Letter of Notification from the 10-year (Full Accreditation) Site Visit
Citations and areas for improvement
RC feedback on the self-study taken into consideration:
Program aims and context
Improvements reported and verified during the 10-year visit
Effectiveness of the self-study, based on the improvements reported by the program
Formative feedback (no accreditation impact) for the initial RC assessment of self-study effectiveness

22. University of Utah Program Self-Study Dates
Anesthesiology + Pain Management
October 2021
Dermatology
July 2021
Emergency Medicine
September 2016
Family Medicine
May 2020
Internal Medicine + subspecialty fellowships
May 2019
Medicine/Pediatrics
Medical Genetics
Neurology + subspecialty fellowships
November 2021
Neurosurgery
January 2018
OB/GYN
December 2021
Occupational Medicine
October 2019

23. University of Utah Program Self-Study Dates
Ophthalmology
May 2020
Orthopedic Surgery + subspecialty fellowships
June 2019
Otolaryngology + subspecialty fellowships
February 2020
Pathology + subspecialty fellowships
April 2019
Pediatrics + subspecialty fellowships
October 2022
PM&R
Psychiatry + subspecialty fellowships
Radiology + subspecialty fellowships
November 2016
Radiation Oncology
March 2018
Surgery + subspecialty fellowships
June 2020
Urology + subspecialty fellowships
December 2020
Plastic Surgery
May 2021

24. Handouts Self Study Summary Template
Suggested Annual Program Evaluation Template
Suggested Annual Program Evaluation Action Plan and Follow-up Template
JGME article: Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ ;7(3):

25. Sample Timeline for IM Core and Subspecialty Programs*
Day 1. ACGME Notification of Self-Study Submission Due Date
Week 1. Recruit Self-Study Planning Committee
Program Director and Associate PDs
Program Coordinator
Fellowship Program Directors and Associate PDs
Faculty member
Resident
Fellow
GME staff?
DIO?
* Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):

26. Sample Timeline for IM Core and Subspecialty Programs*
Weeks 2-5. Weekly meetings of the planning committee
Independent work done by committee members between meetings
Week 2. View ACGME self-study webinar (acgme.org)
Review self-study process
Develop self-study timeline for the program
Create a self-study planning committee worksheet
Week 3. Create/review Self-Study SWOT Template
Add/edit items as relevant to local/individual programs
* Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):

27. Sample Timeline for IM Core and Subspecialty Programs*
Week 4. Create or adapt self-study survey draft documents
Simple surveys: faculty, residents, fellows, coordinators, nurses & other relevant staff, others
Week 5. Define the focus group process
Week 6. Present Self-Study Planning work to GMEC/Discussion
* Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):

28. Sample Timeline for IM Core and Subspecialty Programs*
Weeks 6-8. Formation of Program-Specific Self-Study Committees
Should include PEC members plus others as appropriate (e.g., CCC members, Program Coordinator, Chief Residents, Residents/Fellows, Medical Educator
Week 7. Surveys Distributed
Weeks 7-8. Survey Data Collected
Weeks 8-9. Program-specific committees meet
Program-specific self-study planning
Define program aims and perform SWOT analysis
Weeks Survey groups participate in focus groups to delve into/clarify information gathered in surveys
* Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):

29. Sample Timeline for IM Core and Subspecialty Programs*
Weeks Self-Study Program-Specific Committees Meet Weekly for:
Data analysis
SWOT analysis review with additional data collected
Perform full self-self study
Create self-study document
Weeks Write Self-Study document for submission to ACGME
Weeks Finalize Self-Study documents for submission.
* Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):

30. THANKS!

32. Resident/Fellow Schedules in E*Value

33. Truth or Consequences . . .
Providing an accurate resident / fellow training schedule to the institution is a foundational program responsibility
Accurate, timely training schedules in E*Value are essential for:
Affiliate billing (over $20M yearly)
CMS/Medicare Reimbursement (over $30M yearly)
Information/Data Integrity with Affiliates & Government

34. The Problem
GME Office must routinely resolve missing FTEs at month-end
Discrepancies and late reporting of Leave of Absences, vacation days, or changes in training location are rampant
GME dedicating an employee 3-4 full days each month to audit and resolve schedule omissions, conflicts and mistakes in E*Value
Unprofessional, embarrassing after-the-fact corrections to affiliate bills are commonplace due to scheduling errors

35. Program Director Support Needed
Ensure that your Program Coordinator / Manager knows where your people will be training before the rotation begins
Ensure that training schedules are confirmed and input in E*Value by the 15th of each month
Promptly report any changes that occur after the 15th to GME
Encourage your Program Coordinator / Manager to utilize the monthly schedule verification checklist provided by GME and to double-check schedules before the 15th

36. We’d really rather not go there, but …
The University currently assesses a charge to departments or individuals for payroll corrections or missing scheduled training classes. Please help us avoid implementing this type of ‘penalty’ system.
Thank You!