1. ACGME 2016 Annual Education Conference BR 11 How to Create a Successful Game Plan for APES and Self-Studies February 25, 2016 Nancy Piro, PhD, Program Manager/Education Specialist Ann Dohn, MA, DIO APEs

2. Conflict of Interest Nancy Piro, PhD - No Conflicts of Interest to Declare Ann Dohn, MA - No Conflicts of Interest to Declare

3. Agenda – Everything you always wanted to know about Your APE … and How to Make it Simple! ACGME requirements Available data Matching data to requirements Making it easy – SWOT / Guidebook /Electronic toolbox

4. Creating a Game Plan for APEs

5. Overall Questions Why do we have to do an Annual Program Evaluation (APE)? Who does it? What is required? What must we review and evaluate? What has to be documented? Do we have to save it? Can you help make this easier?

6. Managing it all: How will I pull this off? Deep breath….don’t panic!!

7. APE Data and YOU ! The Program Coordinator plays a crucial role in determining what data, in what format needs to be collected and consolidated for your APEs? Your coordinator is your ally in this!

8. Where do I begin?

9. Program Evaluation Committee (PEC) 9

10. Program Evaluation Committee – ACGME Requirements The program director must appoint the Program Evaluation Committee (PEC) (Core) The Program Evaluation Committee – must be composed of at least two program faculty members and should include at least one resident; (Core) – must have a written description of its responsibilities; and, (Core) ACGME Common Program Requirements - effective: July 1, 2013 Template to make it easy to do…..

11. Program Evaluation Committee – TEMPLATE http://med.stanford.edu/gme/GME_Community.html

12. Program Evaluation Committee (PEC) – should actively participate in: planning, developing, implementing, and evaluating educational activities of the program; (Detail) reviewing and making recommendations for revision of competency-based curriculum goals and objectives; (Detail) addressing areas of non-compliance with ACGME standards; and, (Detail) reviewing the program annually using evaluations of faculty, residents, and others, as specified below. (Detail) ACGME Common Program Requirements - effective: July 1, 2013

13. PEC Function There are no requirements on how the PEC should carry out its duties The PEC or the program director may carry out the improvement plans The work of the PEC can go beyond meeting minimum standards

14. Annual Program Evaluation (APE) 14

15. RESIDENT PERFORMANCE The most recent aggregated written evaluations of the residents submitted by faculty and other evaluators In-training/In-service exam scores Procedure logs (if applicable) Scholarly activity (publications, presentations, grant awards, etc.) Learning portfolios: documented quality improvement activities

16. FACULTY DEVELOPMENT ABMS certification status for all faculty Updated faculty CVs Documentation (faculty survey; attendance logs) of faculty participation in: – CME-type activities directed toward acquisition of clinical knowledge and skills and also activities directed toward developing teaching abilities, professionalism, and abilities for incorporating the core competencies into practice – Teaching (conferences, grand rounds, journal clubs, lecture- based CME events, workshops, directed QI projects, practice-improvement self study). Faculty actively involved in mentor relationships with residents/fellows.

17. GRADUATE PERFORMANCE Aggregated board exam pass rates Aggregated alumni survey results (typically, such surveys target physicians one year and five years after graduation – survey questions may inquire about such items as current professional activities of graduates and perceptions on how well prepared they are as a result of the program) Other outcome measures – Practice location (underserved areas) – Academic Affiliations – Scholarly Activity

18. ALUMNI SURVEYS

19. PROGRAM QUALITY The most recent aggregated written evaluations of the program submitted by faculty The most recent aggregated written evaluations of the program (and/or specific rotations) submitted by residents The most recent aggregated written evaluations of the faculty submitted by residents Faculty’s recent scholarly activity (publications, presentations, grant awards, etc.)

20. PROGRAM QUALITY - Continued The most recent ACGME survey results The most recent GME House Staff survey results (if you have them) The most recent GMEC Internal/Special Review Report All communications from the ACGME or RRC Program Report Card/Scorecard Results (if you have them) – Trend Analyses

21. PROGRAM QUALITY - Continued Curriculum Overall and rotation-specific goals and objectives – Are they appropriate? – Do they align with the core competencies? Didactic curriculum – Is there at least one regular conference targeted to the residents’ level? Opportunities for scholarly activity Compliance with any new standards established by the ACGME, RRC, ABMS, etc. – Assessment Methods » Are evaluation tools appropriate? Do they align with the core competencies? – Resources: » Personnel (PD, PC, faculty), Affiliated Training Sites, Patient/Procedure Volume, Learning Environment (space, call rooms, books, computers, etc.)

22. PROGRESS ON THE PREVIOUS YEAR’S ACTION PLAN Review progress / (attempts to resolve problems) with respect to last year’s Annual Review delineating identified areas of weakness.

23. Ten Year Self-Study and Self-Study Visit (…in 2024?) Self- Study VISIT Ongoing Improvement AE Self- Study Yr 0 Yr 1 Yr 3Yr 4 Yr 5 Yr 6 Yr 7 Yr 8 Yr 9Yr 10 Yr 2 AE Annual Program Evaluation (PR V.C.) Resident performance Faculty development Graduate performance Program quality Documented improvement plan AE: Annual Program Evaluation From ACGME: Program Requirements 2013

24. So how do we make this easy? Complete APE before end AY ends 24

25. Annual Program Evaluation (APE) – CHECKLIST 25

26. Annual Program Evaluation (APE) – SWOT SWOT Analysis 26

27. APE GUIDEBOOK

28. ANNUAL PROGRAM EVALUATION AP TOOL Action Plan 28

29. APE “SWOT” Analysis Tool Fishbone – Ishikawa Diagram

30. Program Aims AIM setting is part of the annual program evaluation Who are our residents/fellows? What do we prepare them for? – Academic / practice … – Leadership and other roles … Who are the patients/populations we care for? AIMS are a way to differentiate programs Self-study ultimately evaluates program effectiveness in meeting these aims Moves beyond improvement solely based on compliance with minimum standards Assessment of relevant initiatives and their outcomes

31. SWOT ANALYSES – Definitions Strengths and Weaknesses – Internal Factors Strengths Program factors that are likely to have a positive effect on (or be an enabler to) achieving your program’s aims are strengths. Important to acknowledge and celebrate What should definitely be continued (important question in an environment of limited resources)

32. SWOT ANALYSES – Definitions Strengths and Weaknesses – Internal Factors Weaknesses Program factors that are likely to have a negative effect on (or be a barrier to) achieving your program’s objectives are weaknesses. Citations, areas for improvement and other information from ACGME The Annual Program Evaluation and other program/institutional data sources

33. SWOT ANALYSES – Definitions Threats and Opportunities Threats - Factors that pose risks. External Factors and conditions that are likely to have a negative effect on achieving the program’s objectives, or making the objective redundant or un-achievable are called threats. While the program cannot fully control them, beneficial to have plans to mitigate their effect What external factors may place the program at risk? What are changes in residents’ specialty choice, regulation, financing, or other factors that may affect the future success of the program? Are there challenges or unfavorable trends in immediate context that may affect the program? e.g., faculty burdened with heavy clinical load that prevents effective teaching and mentorship

34. SWOT ANALYSES – Definitions Threats and Opportunities Opportunities are: Factors and contexts external to the program (institutional, local, regional and national) that can affect the program Opportunities - Factors that favor the program, that the program may take advantage of / leverage External Factors that are likely to have a positive effect on achieving or exceeding your program’s objectives not previously considered are called opportunities. What are capabilities for further evolving the program; how can the program capitalize on them? Has there been recent change in the program’s context that that creates an opportunity? Are these opportunities ongoing, or is there a narrow window for them? How critical is the timing?

35. Fishbone – Ishikawa Diagram SWOT Analysis Completed Example

36. Electronic Toolbox for You! http://med.stanford.edu/gme/GME_Community.html Program Evaluation Committee Policy Template Program Evaluation Checklist & Agenda Annual Program Evaluation Guidebook: Diagrams & Action Plans Annual Program Evaluation Guidebook: Diagrams & Action Plans – Annual Program Evaluation Checklist Annual Program Evaluation Checklist – Annual Program Evaluation Agenda (PDF / DOC)PDFDOC – A Quick Method to Analyze Program Evaluations A Quick Method to Analyze Program Evaluations Program Improvement Action Plan

37. Questions & Contact Information Nancy Piro: npiro@stanford.edunpiro@stanford.edu Ann Dohn: adohn1@stanford.eduadohn1@stanford.edu