In 2014, U.S. Residency Programs including Ob-Gyn fully implemented the Next Accreditation System with the use of Milestone evaluation and reporting. Residency program directors and coordinators were tasked with the immense project of providing information to their Clinical Competency Committees (CCC) that would help them accurately and objectively assess residents. The need for evaluation tools that incorporated the Milestones became paramount to residency programs in all specialties. In this study, we describe the development of a novel evaluation tool that aligns with ACGME Ob-Gyn Milestones and review analysis of the data obtained in it’s inaugural year Purpose: 1) develop a residency evaluation tool in accordance with the ACGME milestones 2) evaluate trends in resident achievements following the inaugural year of the surveys Background: A novel venue and PGY specific evaluation tool that incorporated the ACGME Milestones was initiated to evaluate residents across multiple dimensions, including medical knowledge, procedural skill, teaching, and professionalism in the operating room (OR) and labor and delivery (L&D) venues. Methods: The evaluation tool was built, tested, and implemented. Resident development was captured over time and compared using two-tailed Fisher’s exact and Student’s t-tests (p<0.05 = significant). Results: 822 evaluations were gathered between 9/1/14 and 6/1/15. Among all L&D parameters, interns achieved an “independent” rating 8.4% of the time in the first third of the year, 9.6% in the middle third, and 17.7% in the last third. A significant shift toward evaluations of “with minimal supervision” and “independent” was seen among the intern class (p=0.03) for the management of normal labor. “Independent” L&D scores increased monotonically across the start of all years, from 8.4% for PGY-1s, 60.3% for PGY-2s, 73.7% for PGY-3s, and 87.5% for PGY-4s. For all surgeries combined, PGY-3s demonstrated a notable increase in “able to be primary surgeon,” from 36% of the time in the first 2/3 of the year, to 62.3% in the last 1/3 (p<0.01). Conclusion: ACGME Milestone evaluations are able to capture the growth of residents over time. These tools can demonstrate quantifiable differences in Milestone achievements between PGY classes, and will allow for targeted teaching opportunities for both individual residents and the program overall. Describe the development of a novel evaluation tool that aligns with ACGME Ob-Gyn Milestones and review analysis of data obtained in its first year of use ABSTRACT OBJECTIVE RESULTSMATERIALS AND METHODS BACKGROUND Randi H. Goldman, MD 1, Joan M. Bengtson, MD 1, Ruth E. Tuomala, MD 1,2, Amy R. Stagg, MD 2 1 Department of Obstetrics, Gynecology and Reproductive Biology, Brigham and Women’s Hospital / Harvard Medical School, Boston, MA 2 Vincent Department of Obstetrics and Gynecology, Massachusetts General Hospital / Harvard Medical School, Boston MA CONCLUSIONS A novel PGY and venue specific evaluation tool was built to assess Milestone achievement and the data was analyzed over one year. This study marks the beginning of a continual process of improvement for Milestone assessment tools. As we accrue more data, we can hone these tools to provide accurate feedback to our residents. An evaluation tool using branching logic was built using the REDCap system. Evaluators could access the survey by 1) typing in a URL, 2) clicking a direct survey link sent by , or 3) using smartphone and QR code technology. Evaluations were targeted to a specific venue, surgery, and/or resident-year. A pilot study to test the feasibility of evaluation system was undertaken for “intern boot-camp” during which incoming PGY1s learn basic knot tying, suturing, and the steps of vaginal and cesarean deliveries. The pilot survey was found to assess “level 1” Milestones effectively and the information from this pilot study was useful to the CCC. Following the pilot study, a formal evaluation tool was built. A single tool incorporated all possible evaluation types for both the OR and L&D venues. The tool was used to evaluate residents across multiple dimensions, including medical knowledge, procedural technique, teaching skills, and professionalism. Questions were year, surgery and/or venue specific. Resident development along these dimensions was captured over time. In order to account for different resident rotation schedules, the year was divided into thirds. Resident development was then compared both across years and within the same year using two-tailed Fisher’s exact and Student’s t- tests, where p<0.05 defined significance. How Effective are New Milestones Evaluations at Demonstrating Resident Growth? 1 Year of Data First ThirdMiddle ThirdLast Third PGY1 L&D OR 19/226 = 8.4% 48/75 = 64.0% 27/280 = 9.6% 48/94 = 51.1% 50/282 = 17.7% 99/127 = 78.0% PGY2 L&D OR 38/63 = 60.3% 242/383 = 63.2% 53/98 = 54.1% 243/369 = 65.9% 41/69 = 59.4% 409/563 = 72.7% PGY3 L&D OR 56/76 = 73.7% 161/268 = 60.1% 91/145 = 62.8% 226/325 = 69.5% 103/166 = 62.1% 288/348 = 82.8% PGY4 L&D OR 196/224 = 87.5% 152/169 = 89.9% 310/358 = 86.6% 107/120 = 89.2% 322/357 = 90.2% 184/215 = 85.6% Table 1: “Independent” Scores, All PGY-specific parameters Figure 1: “Independent” Labor and Delivery Scores increase monotonically at the start of each resident year Interns required “more than minimal supervision” on L&D 57% of the time in the first third of the year, which decreased to 40% by the end of the year (p<0.01) There was a significant shift toward evaluations of “with minimal supervision” and “independent” among the PGY1s for ‘management of normal labor’ over the course of the year (p=0.03) PGY2s were able to perform primary C-Sections in a largely independent manner 67% of the time PGY2s required only “minimal direction” among OR parameters 64% of the time in the first two-thirds of the year, which increased significantly to 73% in the last third of the year (p=0.002) PGY-3s were increasingly able to “initiate and complete” operating room tasks throughout the year, from 69% in the beginning of the year to 88% at the end (p<0.01) PGY-3s demonstrated an increase in “able to be primary surgeon,” from 36% of the time in the first two- thirds of the year, to 62.3% in the last third (p<0.01) PGY4s could “independently” organize/lead teams on L&D 94% of the time by the end of the year PGY-4 residents acted “independently” 88% of the time when making intraoperative decisions, and were able to effectively teach junior residents “without direction” 71% of the time 822 evaluations were completed between 9/1/14 and 6/1/15 (range 8-30/resident) Table 1 demonstrates overall trends in “independent” scores for residents of all years Figure 1 shows the overall percentage of “independent” labor and delivery scores