How Accurate is the ACGME Resident Survey? Comparison Between ACGME and In-House GME Survey Bridget N. Fahy 1, S. Rob Todd 1, Judy L. Paukert 2, Melanie L. Johnson 1, Barbara L. Bass 1 1 Department of Surgery, The Methodist Hospital, Weill Cornell Medical College 2 The Methodist Hospital Education Institute Association of Program Directors in Surgery 20 April 2010

Accuracy of ACGME Survey Disclosure No financial conflict of interest

Accuracy of ACGME Survey Background Purpose of annual Accreditation Council for Graduate Medical Education (ACGME) Resident/ Fellow Survey: – monitor graduate medical education – provide early warning of potential non-compliance Not administered in conjunction with program site visit-- results are used at the time of the visit Used to prompt early site visits for programs with significant areas of non-compliance

Accuracy of ACGME Survey Background 15/32 questions allow only “yes/no” responses Wording of some questions is confusing  resident misunderstanding of the question* Graduate Medical Education (GME) Department administers Annual Survey of Residents – similar content to ACGME survey – responses given using a 5-point Likert scale – opportunities for narrative responses for each question are solicited Differences in the results obtained from the two surveys identified *Holt et al. Acad Med;85, 2010.

How Accurate is the ACGME Resident Survey?

Accuracy of ACGME Survey Purpose of Study Examine systematically the results of 15 questions on the two surveys that assessed similar information – identify areas of significant disparity

Accuracy of ACGME Survey Subjects 26 general surgery residents – 4 categorical residents at each PGY level – 2 preliminary PGY2 residents – 4 preliminary PGY1 residents Tertiary referral academic training center

Accuracy of ACGME Survey Survey Instruments ACGME – – 32-question survey administered annually – Response options: “yes/no” = 15, 3-point scale = 4, 5-point Likert scale = 9 – Residents assured of anonymity

Accuracy of ACGME Survey Survey Instruments GME – 53-question survey administered annually – Administered to residents from all programs – Response options: majority given on 5-point Likert scale + option for narrative responses – Resident assured of anonymity

Accuracy of ACGME Survey Common Survey Items Faculty support 1. Do the faculty spend sufficient time teaching residents/fellow in your program? 2. Do the faculty spend sufficient time supervising residents/fellows in your program? 3. Do your faculty members regularly participate in organized clinical discussions, rounds, journal clubs, conferences? 4. Do you have the opportunity to confidentially evaluate your faculty, in writing, electronically, at least once a year? Rotation / program evaluation 5. Do you receive written or electronic feedback on your performance for each rotation? 6. Do you have the opportunity to confidentially evaluate your program, in writing, electronically, at least once a year? 7. Has your program provided you access to, either by hard copy or electronically, written goals and objectives for the program overall and for each rotation and major assignment? 8. Have residents/fellows had the opportunity to assess the program for the purposes of program improvement? Working / educational environment 9. Does your program and/or institution have a system through which you are able to raise and resolve issues without fear of intimidation or retaliation? 10. How often are you able to access, either in print or electronic format, the specialty specific and other reference materials that you need? 11. To what extent to trainees who are not part of your program (such as residents from other specialties, subspecialty fellows, Ph.D. students and nurse practitioners) interfere, in a negative way, with your education? 12. Do your rotations and other major assignments emphasize clinical education over any other concerns, such as fulfilling service obligations? 13. Does your program offer you the opportunity to participate in research or scholarly activities? Duty hours 14. Have you had sufficient education (from your program, your hospital(s), your institution, or your faculty) to recognize and counteract the signs of fatigue and sleep deprivation? 15. Have you met the ACGME duty hour requirements?

Accuracy of ACGME Survey Subject Demographics GME Survey N=26 Age Yrs. (median)30 Male:Female18:8 Postgraduate year Categorical:Preliminary20:6

Accuracy of ACGME Survey Faculty Support Yes on ACGME (%) Yes on GME (%) P-Value Faculty spend sufficient time teaching 12/25 (48)22/26 (85)0.008 Faculty spend sufficient time supervising 24/25 (96)25/26 (96)NS Faculty participation in didactics25/25 (100)25/26 (96)NS Faculty provide feedback22/25 (88)23/26 (88)NS

Accuracy of ACGME Survey Rotation/Program Evaluation Yes on ACGME (%) Yes on GME (%) P-Value Opportunity to evaluate program25/25 (100)26/26 (100)NS Opportunity to evaluate faculty25/25 (100)25/26 (96)NS Receive goals/objectives for rotation and program 25/25 (100)25/26 (96)NS Opportunity to assess program for improvement 23/25 (92)25/26 (96)NS

Accuracy of ACGME Survey Work / Educational Environment Yes on ACGME (%) Yes on GME (%) P-Value Raise issues in program/institution 23/25 (92)25/26 (96)NS Ability to access reference materials 22/25 (88)25/26 (96)NS Interference by other trainees6/25 (24)21/26 (81) Rotations emphasize education over other concerns 7/25 (28)19/26 (73)0.001 Opportunity for research25/25 (100)25/26 (96)NS

Accuracy of ACGME Survey Duty Hours Compliance Yes on ACGME (%) Yes on GME (%) P-Value Education regarding sleep deprivation and fatigue 22/25 (88)23/26 (88)NS Compliance with ACGME duty hour requirements *24/26 (92) 80 hours/week, averaged over a four week period23/25 (92)N/A 1 day in 7 free from all educational and clinical responsibilities, averaged over a four-week period 24/25 (96)N/A 10-hour time period provided between all daily duty periods and after in-house call 22/25 (88)N/A In-house call must occur no more frequently than every third night, averaged over a four-week period 25/25 (100)N/A Continuous on-site duty must not exceed 24 consecutive hours25/25 (100)N/A No new patients may be accepted after 24 hours of continuous duty23/25 (92)N/A At-home call must not be so frequent as to preclude rest and reasonable personal time 23/25 (92)N/A

Accuracy of ACGME Survey Summary Differences in responses found for 3 items: – Faculty time spent teaching – Interference of other trainees – Rotation emphasis of education over other concerns Responses on ACGME survey were less favorable than on GME survey

Accuracy of ACGME Survey Conclusions Two factors may contribute to inaccurate responses on ACGME survey: – Limited range of available responses – Confusing wording of questions Inaccurate responses elicited on the ACGME survey may result in erroneously elevated rates of non- compliance Further validation of the current ACGME survey is warranted