Presentation is loading. Please wait.

Presentation is loading. Please wait.

Department of Graduate Medical Education (GME) Now That We Have an Institutional Report Card---How Do We Use It? Conference Session: SES010 2012 ACGME.

Similar presentations


Presentation on theme: "Department of Graduate Medical Education (GME) Now That We Have an Institutional Report Card---How Do We Use It? Conference Session: SES010 2012 ACGME."— Presentation transcript:

1 Department of Graduate Medical Education (GME) Now That We Have an Institutional Report Card---How Do We Use It? Conference Session: SES010 2012 ACGME Annual Education Conference Ann Dohn, MA, DIO, and Nancy Piro, PhD, Program Manager/Education Specialist Department of Graduate Medical Education Stanford Hospital & Clinics

2 Department of Graduate Medical Education (GME) Disclosure No conflicts of interest to report

3 Department of Graduate Medical Education (GME) Session Objectives At the end of this session, participants will be able to: 1) Effectively use objective data in Balanced Report Cards to make Institutional and Programmatic decisions. 2) Understand strategies for gaining Institutional Administration (C-Suite) support for DIO/GMEC Decisions based on balanced report cards 3) Predict future ACGME review outcomes based on empirical data from balanced report cards

4 Department of Graduate Medical Education (GME) Setting the Stage How does GME administration function in 2012? What are the barriers? What are the challenges? How can we be heard by the C-Suite? What language should we speak? How do we optimize success?

5 Department of Graduate Medical Education (GME) How does GME Administration Function? How do we know? What do we measure? What management strategies do we use?

6 Department of Graduate Medical Education (GME) Historically….. GME has not used objective data in making many decisions Instead, decisions were often made based on: –Politics –Financial considerations –Special deals –Hunches –Ability to scream loudly and frequently –Fear

7 Department of Graduate Medical Education (GME) But what is going on in the environment? Evidenced based –Balanced score cards –Increased accountability for resources –Transparency

8 Department of Graduate Medical Education (GME) Fitting GME within the overall business culture Remember…your C-Suite colleagues often have that magical MBA ….. They think differently They use a different language

9 Department of Graduate Medical Education (GME) Fitting GME within the overall business culture We need to understand how the C-Suite thinks AND learn to think the same way… AND speak their language… To gain their support!!!

10 Department of Graduate Medical Education (GME) What Considerations does the C-Suite have with Respect to Data? Data must have: –An Organizational focus –Regular Means of Data Collection –Include External Assessments as well as Internal Assessments Data used must be –Standardized (comparable) / lean –Capable of transformation –Aligned with mission statement

11 Department of Graduate Medical Education (GME) DIOs need to be able to make decisions based on comparative data Data must: –Be Organizationally Focused –Have Regular Means of Data Collection –Include External Assessments as well as Internal Assessments –Standardized –Capable of transformation –Aligned with mission statement

12 Department of Graduate Medical Education (GME) Conclusion CEOs and DIOs have the same requirements with respect to their data needs

13 Department of Graduate Medical Education (GME) But what models for data do we have? Few data models exist for GME The concept of Institutional Accountability is relatively new. –Until the ACGME Outcome Project, there was no centralized curriculum oversight in GME, unlike medical schools / UME.

14 Department of Graduate Medical Education (GME) The Report Card Vision In 2005, Stanford hired its first PhD in GME –The vision was to develop tools to construct evidence based decision-making for Graduate Medical Education consistent with our mission –We needed a Report Card…

15 Department of Graduate Medical Education (GME) SIGH…. It wasnt as easy as first thought!

16 Department of Graduate Medical Education (GME) Our First Attempt …

17 Department of Graduate Medical Education (GME) Background on Institutional Report Cards Government and Industry Models –Multiple models exist and can be used as per specific purpose: GRPA (Government Performance and Results Act) Organizational Report Cards Balanced Scorecard Benchmarking Program Evaluations Social Indicators No Child Left Behind C BEST College Performance Testing

18 Department of Graduate Medical Education (GME) Which Model Should We Use? We needed a model that: – was organizationally focused and managed – had a track record of effective use – fit our existing structure with multiple programs and organizations – was flexible enough to be adapted for use on an annual basis – not an accreditation cycle-Regular Data Collection – was Easily Digestible – had internal and external measurement dimensions

19 Department of Graduate Medical Education (GME) Our Choice Balanced Scorecard Framework in an Organizational and/or Report Card Tool Best of Both Worlds

20 Department of Graduate Medical Education (GME) The Balanced Scorecard Approach The Balanced Scorecard is a performance measurement and performance management system developed by Robert Kaplan and David Norton (1992, 1996) – adopted by a wide range of leading edge organizations, both public and private. (The Balanced Scorecard--Measures That Drive Performance, Harvard Business Review, Jan- Feb 1992; and The Balanced Scorecard-Translating Strategy into Action, Harvard Business School Press, 1996)

21 Department of Graduate Medical Education (GME) Stanfords Report Card – Three Programs

22 Department of Graduate Medical Education (GME) Key to the Report Card

23 Department of Graduate Medical Education (GME) Balanced Scorecard Strategic Perspectives Institutional / Financial Growth Mission Vision Strategy Learning Program Processes Resident Do we continue to improve (outcomes)? Are our programs excelling? How do our residents see us? Are we putting our resources in the right places?

24 Department of Graduate Medical Education (GME) Selection of Balanced Report Card Measures PREPERIPOST INTERNALEXTERNAL RESIDENT PERCEPTIONS Voice of the Resident PROGRAM PROCESSES Faculty Eval of program Duty Hr Violations # Internal Citations LEARNING OUTCOMES FINANCIAL / GROWTH Resources GME Internal HS Survey Recommendation of Program GME Internal HS Survey Overall Satisfaction Resident Evaluation of Program Match Depth ACGME Survey Responses Teaching Supervision Scholarship Non Hostile/Intimidating Environment Address Concerns Confidentially Training Exams Board Pass Rates Career Placement Grants ACGME Cycle Length # ACGME Citations

25 Department of Graduate Medical Education (GME) Balanced ScoreCards Report Cards Had –Organizational focus –Regular Means of Data Collection –Include External Assessments as well as Internal Assessments Were –Standardized –Capable of transformation –Aligned with the Mission of Stanford Graduate Medical Education (GME)

26 Department of Graduate Medical Education (GME) Stanford Hospital & Clinics Use of the Balanced Report Card The Stanford Report Card is built on the Balanced Scorecard conceptual framework for translating an organizations vision into a set of performance indicators distributed among four perspectives adapted for GME: 1.Resident Perception Measurements 2.Program Processes 3.Learning Outcomes 4.Financial/Growth

27 Department of Graduate Medical Education (GME) Stanford Background Stanford University Medical Center currently sponsors 85 ACGME-accredited training programs with over 1100 enrolled residents and fellows.

28 Department of Graduate Medical Education (GME) Stanford University Medical Center Mission Dedication to pursuing the highest quality of patient care and graduate medical education, recognizing that one of its major responsibilities is the provision of organized educational programs. – Support of quality graduate medical education programs and excellence in residency training and research. – Guidance and supervision of the resident while facilitating the residents professional and personal development and ensuring safe and appropriate care for patients. – Provision of adequate funding of graduate medical education to ensure support of its faculty, residents, ancillary staff, facilities, and educational resources to achieve this important mission. – Ensuring that all of its graduate medical education programs meet or exceed the Institutional and Program Requirements promulgated by the Accreditation Council for Graduate Medical Education.

29 Department of Graduate Medical Education (GME) Translating the Stanford Hospital & Clinics GME Mission to the Balanced Report Card 1.Resident Perception Measurements Guidance and supervision of the resident while facilitating the residents professional and personal development and ensuring safe and appropriate care for patients. 2. Program Processes Ensuring that all of its graduate medical education programs meet or exceed the Institutional and Program Requirements promulgated by the Accreditation Council for Graduate Medical Education.

30 Department of Graduate Medical Education (GME) Stanford Hospital & Clinics Report Card 3. Learning Outcomes Support of quality graduate medical education programs and excellence in residency training and research. 4. Financial/Growth Provision of adequate funding of graduate medical education to ensure support of its faculty, residents, ancillary staff, facilities, and educational resources to achieve this important mission.

31 Department of Graduate Medical Education (GME) Stanford Hospital & Clinics Balanced Report Card Indicators are designed to measure SHCs progress toward achieving its vision; other indicators are designed to measure the long term drivers of success. Through the balanced report card SHC : –Monitors its current performance (finances, resident satisfaction, learning outcomes and program process results) –Monitors its efforts to improve processes, educate residents –Enhances its ability to grow, learn and improve the quality of its fellowship and residency educational programs.

32 Department of Graduate Medical Education (GME) Why GME Thinks We Need This ACGME and Institutions are increasingly holding DIOs and GME Committees accountable for their utilization of institutional resources. Actions / decisions must be based on documented real-time analyses of needs.

33 Department of Graduate Medical Education (GME) Selection of Balanced Report Card Measures PREPERIPOST INTERNALEXTERNAL RESIDENT PERCEPTIONS Voice of the Resident PROGRAM PROCESSES Faculty Eval of program Duty Hr Violations # Internal Citations LEARNING OUTCOMES FINANCIAL / GROWTH Resources GME Internal HS Survey Recommendation of Program GME Internal HS Survey Overall Satisfaction Resident Evaluation of Program Match Depth ACGME Survey Responses Teaching Supervision Scholarship Non Hostile/Intimidating Environment Address Concerns Confidentially Training Exams Board Pass Rates Career Placement Grants ACGME Cycle Length # ACGME Citations

34 Department of Graduate Medical Education (GME) Findings After several years of data collection…. –Some data is more valuable –Some data drives change

35 Department of Graduate Medical Education (GME) VOICE OF THE RESIDENT Internal Measures Resident Overall Evaluation of the Program Duty Hour Violations GME/HS Survey - Overall Satisfaction ? GME/HS Survey - Recommend Program ? External Measures Match Depth % Top Medical Schools ACGME Survey - % Compliant Responses Teaching Supervision Scholarship Non Hostile/Intimidating Environment Address Concerns Confidentially

36 Department of Graduate Medical Education (GME) PROGRAM PROCESSES Internal Faculty Overall Evaluation of the Program # Internal Citations External ACGME Cycle Length # ACGME Citations

37 Department of Graduate Medical Education (GME) Case Study - Stanford How the DIO uses the Balanced Report Card: –A Tale of Three Programs… –Starting from the bottom…. –Assuming that being at the bottom may mean needs more help

38 Department of Graduate Medical Education (GME) How Do We Use this Data? Look at Indicators that are Resident Driven – Voice of the Resident –Is there a discrepancy between the voice of the resident and the other indicators? Would the majority of the residents not choose the program again yet the program receives a five year accreditation cycle with one or no citations?

39 Department of Graduate Medical Education (GME) How Do We Use this Data? How Do the Programs Compare Against Each Other? How do they compare against their ACGME Cycles?

40 Department of Graduate Medical Education (GME) Stanfords Report Card – Three Programs

41 Department of Graduate Medical Education (GME) What does this tell the DIO? (Program A) Setting the stage: –2008-ACGME gave a 5 year cycle with 1 citation –2009-ACGME resident survey shows tanking program –2009-GME House Staff survey shows program in trouble –Duty Hour violations are > 15 times greater than the Institutional Average »Over a three year period

42 Department of Graduate Medical Education (GME) Stanfords Program A

43 Department of Graduate Medical Education (GME) Stanfords Program A

44 Department of Graduate Medical Education (GME) What does this tell the DIO? –Educational Milieu Intimidation and Fear of Retaliation - Increasing Lack of confidence in the program to confidentially deal with problems or concerns - Increasing Overall resident satisfaction with the program – decreasing Would recommend: extremely low in 2008/09 (lowest of any SHC program) Definitely an early warning….

45 Department of Graduate Medical Education (GME) How Did We Use the Data Validated data with resident interviews (single and group) You know you have a problem when the residents call and ask to meet you at Starbucks…

46 Department of Graduate Medical Education (GME) Setting up an Action Plan Met with program leadership –Shared data –Discussed their interpretation of the data –Brain-stormed with program leadership How can GME help you?

47 Department of Graduate Medical Education (GME) Program View on Situation Need more help…more residents…more MD extenders…. Residents are the problem…not tough and they whine….

48 Department of Graduate Medical Education (GME) GME Evaluation of the Situations Qualitative analysis of every comment on the internal house staff survey Trend analysis of surveys DIO used her training in conflict resolution Report developed to define the problems –Shared with C-Suite

49 Department of Graduate Medical Education (GME) Action Plan Used data regarding duty hours with the C- Suite to: – justify hiring additional physician extenders – reconfigure rotations Used resident program satisfaction data with the Dean to: –leverage program change and motivate program leadership

50 Department of Graduate Medical Education (GME) Results - Success ? Duty Hours have improved Waiting on resident sat from 2012 survey (May)

51 Department of Graduate Medical Education (GME) Program B Setting the stage: –Internal Review in 2008-09 found a program substantially out of compliance »Problems were largely identified as structural ie: formal policies/curriculum »Acting program director »Small, internationally known fellowship »Very competitive »Trainees pleased with program »Trainees interviewed 2 months after start of the program

52 Department of Graduate Medical Education (GME) Program B

53 Department of Graduate Medical Education (GME) Stanfords Program B

54 Department of Graduate Medical Education (GME) Assigned a GME staff professional to work with the program director on the missing structural elements in the program. DIO monitored progress Setting up an Action Plan

55 Department of Graduate Medical Education (GME) Progress… Internal review citations corrected 2010 ACGME review gave 5 years and a single citation

56 Department of Graduate Medical Education (GME) However….the voices of the residents ACGME survey begins to show concerns over training environment DIO starts receiving individual concerns from trainees Program becomes large enough to qualify for its own internal survey (smaller programs are grouped) –Internal survey shows major concern areas on Report Card…a sea of RED

57 Department of Graduate Medical Education (GME) Power of the Data Concerning data is shared with program leadership

58 Department of Graduate Medical Education (GME) Report Card Speaks Changes are implemented –Service is reduced –Associate program director appointed –Protected time for education enforced –Department leadership engaged –Administration support in place

59 Department of Graduate Medical Education (GME) Disconnect Report card allowed GME to implement intervention which successfully addressed the areas of concern BUT…..ACGME (not having access to the most recent data) called for an early site visit.

60 Department of Graduate Medical Education (GME) Program C: Setting the Stage The program was already experiencing difficulties before their most recent ACGME site visit –Tough transition to a new program director when former director abruptly left Problems identified on both the ACGME and internal GME house staff surveys

61 Department of Graduate Medical Education (GME) Program C

62 Department of Graduate Medical Education (GME) Stanfords Program C

63 Department of Graduate Medical Education (GME) Action Items GME assigned a staff member to work closely with program –Note: program director readily accepted any and all assistance-knew he had inherited a train wreck.

64 Department of Graduate Medical Education (GME) Progress? Program continued to spiral down…… –Lack of C-Suite buy in

65 Department of Graduate Medical Education (GME) ACGME Outcomes ACGME site visit –Proposed probation which resulted in: HUGE wake-up call for the C-Suite, residents, and four major affiliates Data invaluable in presenting problems

66 Department of Graduate Medical Education (GME) Action Items Data presented to –Department chair –Dean –DIOs at major affiliates –CEOs –RESIDENTS

67 Department of Graduate Medical Education (GME) Consensus Decision to save the program –C-Suite –Program leadership –GME

68 Department of Graduate Medical Education (GME) Results Appealed the probation –2 year cycle –Program continues to improve

69 Department of Graduate Medical Education (GME) What does this tell the DIO? Report card allows for easy monitoring of multiple factors With longer ACGME accreditation cycles in our futures…imperative to be able to monitor programs frequently

70 Department of Graduate Medical Education (GME) Lessons Learned… Three very different programs –But data drives change in all –In all three cases the final outcome was data driven –In all three cases the C-Suite was needed to resolve issues-and was moved to help ONLY after being presented with data

71 Department of Graduate Medical Education (GME) Lessons Learned (continued) As good as we think our hunches are… –The C-Suite wants facts, figures, stats, etc You need to prove your case The C-Suite is VERY sensitive to the norm, average, scores of other programs, benchmarks, etc. Your argument needs to be evidenced-based Dont forget the outcomes!!

72 Department of Graduate Medical Education (GME) A Tale of Three Programs

73 Department of Graduate Medical Education (GME) Questions


Download ppt "Department of Graduate Medical Education (GME) Now That We Have an Institutional Report Card---How Do We Use It? Conference Session: SES010 2012 ACGME."

Similar presentations


Ads by Google