 Alterman DM, Jones TM, Daley BJ, Goldman MH Department of Surgery

 Optimal markers of achievement and goals for Surgical Education are being defined  Graduate Medical Education is being directed towards a competency-based education and training paradigm by the ACGME  Predictors of Goal Achievement is poorly defined

 We currently document comprehensive resident outcomes that may be equivalent to “Milestones” o “360” performance evaluation o ABSITE in-service examination o ACGME core competency faculty evaluation o Academic productivity (publications, presentations, etc.) o Attrition  We sought to understand predictors of these “Milestone” markers of successful surgical education

 18 year review of all General Surgery residents at a single academic university training center (n=101)  Background and Demographic data obtained for these included extraction from ERAS application, transcripts, and interview evaluations  Outcome data was collected for the above makers o ABSITE scores o Conference attendance o Attrition rate o Academic productivity o ACGME core competency faculty assessments o Operative case numbers o Board certification

 Background Data Extraction (>40 variables for each resident) o Demographics o Personal items listed (such as marital status) o Undergraduate degree / major o Medical School Performance (GPA, Clerkship grades, etc.) o AOA status o USMLE STEP scores o Prior work or research experience o Publications o Interview faculty score (1-5 Likert scale) o “Special Skill” (high level of achievement / performance in non-medical field – such as concert pianist or division I athlete)

 Statistical analysis was performed with SPSS 19 software package  Comparison of multiple means was with ANOVA  Multiple logistic regression analysis was used to create predictive models for each outcome sought  For univariate comparison of means or percentages, t-test or z-test was used, respectively, with an alpha level of 0.05

 USMLE Step 1 (STEP1) score is positively correlated to ABSITE scores PGY1-5 levels (R= , p=0.008)

 STEP1 is positively correlated to faculty ratings of all six ACGME core competencies across all PGY levels Summary data for significant correlations of STEP1-3 with ACGME core competency evaluation ratings across PGY1-5 levels A total of 150 correlations were made between each PGY level (1-5) of each of the six ACGME core competencies and STEP1-3 The number of significant correlations (all positive) are given for each comparison

 Multivariate logistic analysis was performed to evaluate for predictors of each discrete outcome measure  Sub-analysis was repeated for residents distributed among: o Categorical graduates and non-graduates o Non-Graduates with voluntary and involuntary attrition, o Preliminary non-designated residents matched/non-matched

 In multivariate logistic analysis, STEP1 and faculty interview ranks were able to predict categorical matched residents attrition (Wald 8.4, p=0.004)  Analysis of Categorical Residents attrition: o SPECIAL SKILL was predictive of non-attrition ( OR 3.5 )

 In repeated analysis among subgroups, GENDER, GPA, AGE or ETHNICITY was not significant (in contrast to prior data) o * several of these difference were noted in univariate analysis but none remained

 Single Institution Review  The data points which we feel are equivalent to “Milestones” may not be  Some older files required some interpretation and normalization of scores

 Our current resident evaluation is “360 degrees” and evaluates multiple domains of performance – calling into question the need for novel supplemental assessment tools  STEP1 retained strong positive predictive value among academic and clinical outcomes  The importance of STEP1 for screening and prediction of future success in Graduate Surgical Education is becoming increasingly clear

 USMLE STEP1 is a robust data point and has documented predictive value across many specialties o Associated with residency clinical success and board passage rates in specialties such as EM, Ortho, FP, Peds, IM, OB/GYN, GS and Anesthesiology. o The literature and our data support a critical threshold of >215 for USMLE STEP1 evaluation

 Our overall attrition rate 23.7% is comparable to national reported ranges, however if one considers the attrition from just categorical matched residents it would rise to 34%.  If the denominator included all categorical matched residents and preliminary residents that were matched to our system, the attrition rate would then be 30.3%.  If we were to look at attrition as a function of categorical resident’s “failure” or non-voluntary attrition; this number would be 15%.  Clearly, there are many ways to analyze and report attrition and this makes comparison of institutional reports difficult.