Presentation on theme: "Conflict of Interest Kim Walker – No conflicts of interest to disclose"— Presentation transcript:
1 Conflict of Interest Kim Walker – No conflicts of interest to disclose Ann Dohn – No conflicts of interest to discloseNancy Piro – No conflicts of interest to disclose
2 PC002c Coordinators and Clinical Competency Committees: How to Streamline and Support the Work of your Program’s CCCKim Walker, PhDProgram Manager/Education SpecialistAnn Dohn, MADIO & GME DirectorNancy Piro, PhD
3 Session Outcomes Participants will be able to: Identify new aspects of the coordinators’ evolving role in program administration.Understand and use program requirements as a guide for planning, organizing and implementing educational and assessment tools.Develop and utilize a newly developed comprehensive resident performance profile tool to streamline the work of the CCCs.
5 Evolving Role for Coordinators in Evaluations Education & Evaluation Coordinator/ManagerExtraordinaireAdministrator Scheduler SupremeConstructing new milestone evaluations to pilot/deliverReviewing evaluation completion data for accuracyAggregating data for the CCC from multiple sources and formsMilestone data to ACGMEDeliver evaluationsDevelop evaluation forms for PDs to approveSchedule semi annual evaluationsEnsure summative evaluations completed and filed
6 Now I’m really confused! OutcomesEvaluationsCCCsEPAsClarify and briefly define, especially the different between EPAs and milestones.MilestonesGoals and Objectives
8 Know the NAS Building Blocks: Concepts defined Core CompetenciesMilestonesEPAsCurriculum and EvaluationsClinical Competency Committee (CCC)
9 NAS – Next Accreditation System What is NAS – in a nutshell:“an outcomes-based accreditation process through which the doctors of tomorrow will be measured for their competency in performing the essential tasks necessary for clinical practice in the 21st century.”
10 Major Changes: Accreditation based on… Pre-NASCompetenciesSite Visits – Up to 5+ year cyclesInternal ReviewsADS UpdatesPIFsResident SurveysCurrent (New) NASCompetencies with MilestonesSelf-Studies at ~ 8-10 year intervalsDetailed ADS UpdatesCLER Visits ~ 18 – 24 months (Institution)Resident & Faculty Surveys
11 The New Accreditation System (NAS)… Outcomes Increased Annual reporting by Programs (online)Reduced volume of accreditation demands … but increased attention to accuracy and completeness of information submitted onlinePIF-less Surveyor visits (unless new application)Two Field Surveyors per visitNo Faculty CVs (only PD)….but Faculty & Resident Scholarly Activity required.
12 The Six AGME Core Competencies PatientCareInterpersonal & Communication SkillsMedicalKnowledgeSix CoreCompetenciesFor QualityPatient CarePractice-basedLearning &ImprovementNeed to redo- Should be PBL&IProfessionalismSystems-based Practice
13 What Are Milestones?High Level - Milestones are simply defined as areas of competency/expectations for our traineesLinked to six core competenciesDefined as a continuum of progressive growth/learningAdvanced Beginner PGY-1Competent PGY-2 and 3Proficient Practitioner PGY-4/5Expert Practitioner - OngoingEach trainee assessed with respect to level for each competency
14 Dreyfus Model (1980): Stages of developing expertise Source: Eraut, M. Developing Professional Knowledge and Competencies. (1994)
15 Milestone Level Definitions Level 1: The resident is a graduating medical student/experiencing first day of residency.Level 2: The resident is advancing and demonstrating additional milestones.Level 3: The resident continues to advance and demonstrate additional milestones; the resident consistently demonstrates the majority of milestones targeted for residency.
16 Milestone Level Definitions (continued) Level 4: The resident has advanced so that he or she now substantially demonstrates the milestones targeted for residency. This level is designed as the graduation target – not requirement.Level 5: The resident has advanced beyond performance targets set for residency and is demonstrating “aspirational” goals which might describe the performance of someone who has been in practice for several years. It is expected that only a few exceptional residents will reach this level.
17 Reporting the Milestones V.A.1.b).(1).(b) prepare and assure the reporting of Milestones evaluations of each resident semi-annually to ACGME (Core)Milestones are reported directly through ADSReporting windows are:November 1- December 31May 1- June 15
18 Entrustable Professional Activities (EPA) Professional life activities that define a medical specialty:Ground the competencies in a physician’s everyday workActivities lead to some outcome that can be observedComplexity of the activities requires an integration of knowledge, skills and attitudes across competency domainsHow EPAs Relate to MilestonesSituates competencies in the clinical, authentic context in which they are demonstratedAligns what is assessed with what physicians really do in the realm of patient careAdds meaning to assessment by focusing on integration of competencies in care delivery
19 Examples of EPAsFacilitate handovers to another healthcare provider either within or across settingsContribute to the scholarly work of the subspecialtyCo-manage patients with generalists and other subspecialistsSource: https://www.abp.org/abpwebsite/taskforce/reslib/24.ppt
20 Curriculum: Rotation-specific goals and objectives & links to milestones
21 Milestones Impact on Evaluations: Linking questions to milestones Step Two: Ensure specific evaluation questions are linked to milestonesAdvises the referring health care provider(s) about the appropriateness of a procedure in routine clinical situations
22 Milestones Impact on Evaluation System Allows for more objective methods of assessment and provide better feedbackProvides a process for early identification of residents that are having difficultiesAll old and new evaluations and questions should be aligned with and tracked to milestones
23 Clinical Competency Committee (CCC) V.A.1. The program director must appoint the Clinical Competency Committee.(Core) V.A.1.a) At a minimum the Clinical Competency Committee must be composed of three members of the program faculty.(Core) V.A.1.a).(1) Others eligible for appointment to the committee include faculty from other programs and non- physician members of the health care team.(Detail)ACGME Common Program RequirementsApproved: February 7, 2012; Effective: July 1, 2013Approved focused revision: June 9, 2013; Effective: July 1, 2013
24 Clinical Competency Committee (CCC) V.A.1.b).(1) The Clinical Competency Committee should: V.A.1.b).(1).(a) review all resident evaluations semi- annually; (Core) V.A.1.b).(1).(b) prepare and assure the reporting of Milestones evaluations of each resident semi-annually to ACGME; and, (Core) V.A.1.b).(1).(c) advise the program director regarding resident progress, including promotion, remediation, and dismissal.(Detail)
25 Clinical Competency Committee (CCC) V.A.1.b) There must be a written description of the responsibilities of the Clinical Competency Committee.(Core)The reason for a required written description is so that every participant knows what his or her responsibility is to the CCC, and to ensure a fair process that all the members and the program director agree to follow. The responsibilities may go beyond what is listed in the ACGME Program Requirements. For some programs, the CCC will also be the Curriculum Committee or the Program Evaluation Committee, or may exist with a different name with additional responsibilities.ACGME Common Program RequirementsApproved: February 7, 2012; Effective: July 1, 2013Approved focused revision: June 9, 2013; Effective: July 1, 2013
26 Clinical Competency Committee (CCC): Written description
27 Clinical Competency Committee (CCC) How the CCC does its work can be decided by the Program DirectorSubcommitteesAssigning residents to faculty members for pre-reviewPre-review work will varyScheduling and frequency of meetingsAdvisory to Program DirectorDoes not voteConfidentialAssessment of all residentsData provided includes summary evaluations plus resident self-assessmentProvide narrative to PD
28 What Should a CCC Do First? Understand their specialty Milestones (Posted on acgme.org)Decide how to assess the Milestones – Program Evaluation StrategyIf necessary, identify new evaluation tools from program director associations, societies, colleges
29 NAS and Milestones and YOU The program coordinator will play a crucial role in developing, implementing, collecting data on and reporting of milestone evaluation tools.You are a big part of the equation in how you are able to support your program’s mission at all levels… many balls to juggles requires many hands, many minds. You are the “coordinator” of it all!!!
31 U - R - IT! Understanding RRC program requirements Requirements applied to evaluation methods/processImplementing new evaluation systemTracking completion and accuracy (outliers) for data aggregation
32 1. Understand Your Program’s New Requirements CoreOutcomesDetails
33 1. Understand Your Program’s New Requirements Each standard/requirement is categorized:Outcome - All programs must adhereCore - All programs must adhereDetail – Considered mandatory for new programs and those that fail to meet core requirements. Allows high-performing programs to innovate.The focus in the NAS is on educational outcomes. The common and specialty programsrequirements were categorized, with the expectation that programs that demonstrate goodeducational outcomes will not be assessed for compliance with the “detail” requirements.“Detail” requirements will be considered mandatory for new programs and for programs thathave failed to meet expectations for outcomes (and have an accreditation status of “Probation”or “Continued Accreditation with Warning”), and are intended to offer these programs addedguidance.Allowing high-performing programs the freedom to meet the detailed requirements withalternatives will provide such programs the opportunity to innovate.From : ACGME – NAS FAQsSource: Implementing The Next Accreditation System ACGME WebinarJohn R. Potts, III, M.D.: 4 November 2013
34 1. Understand Your Program’s New Requirements Example Program Requirement:VI.B. Transitions of Care(Core)(Core)(Outcome)
35 1. Understand Your Program’s New Requirements Example Program Requirement: VI.B. Transitions of CareWhen core and outcome not in compliance, then:Details
36 U - R - IT! Understanding RRC program requirements Requirements applied to evaluation methods/processLinking milestones/EPAs and objectives to evaluation questionsUtilizing milestone scalesImplementing new evaluation systemTracking completion and accuracy (outliers) for data aggregation
37 2. Requirements Specific to CCC Review of Trainee Aggregating/compiling multiple evaluations of individual trainees (V.A.1. Formative Evaluation)Tracking trainee participation in conferences, journal clubs, didactics (IV.A.3. Didactic Sessions)Monitoring duty hour compliance (VI.G. Duty Hours)Reviewing involvement in quality improvement and patient safety activities (IV.A.5.c. PBLI)Reviewing scholarly work (IV.B. Scholarly Work)Monitoring and reporting procedure logs (IV.A.5.a)
38 U - R - IT! Understanding RRC program requirements Requirements applied to evaluation methods/processLinking milestones/EPAs and objectives to evaluation questionsUtilizing milestone scalesImplementing new evaluation systemTracking completion and accuracy (outliers) for data aggregation
39 3. Implementing Evaluation systems Milestone-based/EPAs Rotation-specificPatient handoversDefine evaluator groups (faculty, staff, patients)Set up and timing of delivery systems
40 3. Implementing Documentation and reporting systems for: Conference attendanceScholarly work (Learning Portfolios)Quality Improvement and Patient Safety (Learning Portfolios / Safety reporting systems)Duty Hours (recording, monitoring, reporting)Case Logging (if applicable)
41 U - R - IT! Understanding RRC program requirements Requirements applied to evaluation methods/processLinking milestones/EPAs and objectives to evaluation questionsUtilizing milestone scalesImplementing new evaluation systemTracking completion and accuracy (outliers) for data aggregation
42 4. Tracking and Reporting… Start with the end in mind:CCC biannual reporting windows to ACGMENovember 1- December 31 / May 1 - June 15Back track and set calendar events for:Periodic monitoring of evaluation completionRunning aggregate reports and reviewing milestone evaluation dataReviewing case logs, learning portfolios, duty hours
43 Pulling the Data Together Quality Improvement ActivitiesIn-service training examsClinical Competency CommitteeEnd-of-Rotation EvaluationsSafety Incident ReportsCase LogsPatient/ Family EvaluationsClinical Skills AssessmentNursing and Staff / Techs EvaluationsProgress on MilestonesSimLab
46 Creating a Resident Performance Profile Goals to support your CCC Resident performance data that is:ComprehensiveConsolidated / AggregatedEasy for CCC to identify strengths, areas for improvement, opportunities for advancement
47 Creating a Resident Performance Profile: Compiling and centralizing data Complete with auto-fill colors
48 Creating a Resident Performance Profile: Apply visual formatting for trends Complete with auto-fill colors
49 Creating a Resident Performance Profile Step 1 – Defining what to track
50 Creating a Resident Performance Profile Step 1 – Defining what to track
51 Creating a Resident Performance Profile Step 1 – Defining what to track
52 Creating a Resident Performance Profile Step 1 – Defining what to track
53 Creating a Resident Performance Profile Step 1 – Defining what to track
54 Creating a Resident Performance Profile Step 1 – Defining what to track
55 Creating a Resident Performance Profile Step 2 - Link data sources to milestones
56 Creating a Resident Performance Profile Step 3: CCC defines performance ranges Example:For all aggregate milestone evaluation scores for a PGY 3, the CCC has defined these ranges by PGY level in advance of the meeting:At or Above Expectation: and higherBelow Expectation:1.7 – 2.7Remediation:Below 1.7Conditional formatting is a super helpful tool to use for visually presenting color coded data that “at a glance” can provide a general trend over a residents’ progress.STRENGTHWATCHAT RISK
57 Creating a Resident Performance Profile Step 4 – Set conditional formatting Conditional formatting is a super helpful tool to use for visually presenting color coded data that “at a glance” can provide a general trend over a residents’ progress.
58 Creating a Resident Performance Profile Step 4 – Set conditional formatting Example: Aggregate milestone evaluation data cellsConditional formatting is a super helpful tool to use for visually presenting color coded data that “at a glance” can provide a general trend over a residents’ progress.Highlight cells to apply the conditional formatting
59 Creating a Resident Performance Profile Step 4 – Set conditional formatting Set Ranges: > , < , between
60 Creating a Resident Performance Profile Step 4 – Set conditional formatting Select, “Greater Than” “Less Than” or “Between” to Set Value RangesChoose the corresponding fill color (e.g., red, yellow, green)Conditional formatting is a super helpful tool to use for visually presenting color coded data that “at a glance” can provide a general trend over a residents’ progress.
61 Resident Performance Profile: Step 5: Enter in data Complete with auto-fill colors
62 Creating a Resident Performance Profile Visual trends and detailed data Complete with auto-fill colors
64 Leveraging Resident Management System (RMS)Tools, if Available RMSs – becoming more feature richCurriculumGoals and Objectives and learning outcomes by rotationTeaching and Assessment methodologiesEvaluation tool developmentSharing between programs and institutions
65 Leveraging Resident Management System (RMS)Tools, if Available Conference attendance statisticsCore competencies linked to specified conferencesAttaching conference materials for later referenceProcedures and levels; linked procedure evaluations
66 Leveraging Resident Management System (RMS)Tools, if Available Resident portfolio toolsQI participation and outcomesScholarly Activity logs
67 Leveraging Resident Management System (RMS)Tools, if Available Aggregate reporting and graphic summariesPeer or departmental average, individual average, minimum and maximum scores, standard deviation or listing of all scores
68 Leveraging Calendaring and Task Management Software Set “data gathering and reporting” appointments with yourselfRemember to start with the end in mind (e.g., CCC meeting dates)Break down large tasks into smaller tasks to keep it manageable
69 Leveraging Calendaring and Task Management Software
71 When a CCC Meeting… Does go well Doesn’t go well Data completeorganizedaccurateCooperative, collaborative decision makingEfficient use of timeSound valid conclusions aligned with dataDatanot completenot organizednot accuratePD or faculty member dominates meetingProlonged inefficient decision making with inability to gain consensusUnsubstantiated/unreliable conclusions
72 Successful Resident Ranking PGY1- 4: Ready to graduate
73 PGY 1 Ranked at Graduation Level Milestone range for a PGY1 should not be a 4.0, 4.5 or 5.0 …1?
75 We Should not be at this Point Tracker can avoid this situation – PGY 7 being passed along although aggregate scores show areas for concern.
76 Beyond data…Creating a climate of CCC Success Gentle Words of WisdomTight efficient meetingsGround RulesBeware of Negative Group ThinkSchedule firm standing meeting dates in advanceReserve room of appropriate size with required audio-visual tools if needed…..and have snacks
77 Use Technology to Your Advantage… You can be a ‘Rock Star’
78 Session Recap in a Nutshell… Know your program requirements and follow them unconditionallyUse simple spreadsheet, calendaring and task organizational tools to manage, track and present resident performance data to your CCCResident education is a cyclical process – revisit and revise tools and processes each year