Presentation on theme: "Competency Workshop Medbiquitous May 2011 Susan Albright."— Presentation transcript:
Competency Workshop Medbiquitous May 2011 Susan Albright
Workshop Objectives By the end of this workshop, participants will be able to: Describe a process for developing competency frameworks Identify challenges and potential barriers to developing and implementing competency frameworks Identify opportunities available with the implementation of competency frameworks in an electronic system
Agenda Introductions Part 1 – Background on the Tufts Experience Activity Part 2 Assessment Discussion Break Part 3 – Developing the system Discussion Summary
Why Competencies? “Competency-based education is characterized by the development of clearly defined and explicitly published competencies, mapping the curriculum to achieve competencies, and assessment process matched to competencies.” (Ron Harden, Medical Teacher 2002 #24 Vol 2
What is Competency Based Education CBE is an institutional process that moves education from focusing on what academics believe graduates need to know (teacher-focused) to what students need to know and be able to do CBE is focused on outcomes (competencies) that are linked to workforce needs as defined by the profession. CBE necessitates complex assessment through portfolios or experiential learning assessement in field experience Large skill sets are broken down into competencies which sequestial levels of mastery
CBE Competencies may reinforce one another from basic to advanced as learning progresses Continual refinement of competencies is essential as this a process not a product Consistency in expectations across the curriculum Consistency in grading http://www.ceph.org/pdf/Competencies_TA.pdf Council on Education for Public Health
ACGME Competency Domains Patient care Medical knowledge Practice-based learning and improvement Interpersonal and communication skills Professionalism Systems-based practice
AAMC Learning objectives The first of the four attributes is that physicians must be altruistic. There are seven learning objectives, including the objective that before graduation, the student can demonstrate compassionate treatment of patients and respect for their privacy and dignity. The second attribute is that physicians must be knowledgeable; one of the six learning objectives is that the student can demonstrate knowledge of the normal structure and function of the body and of each of its major organ systems. The third attribute is that physicians must be skillful; one of the eleven learning objectives is that the student have knowledge about relieving pain and ameliorating the suffering of patients. The last attribute is that physicians must be dutiful; one of the six learning objectives is that the student have knowledge of the epidemiology of common maladies within a defined population, and the systematic approaches useful in reducing the incidence and prevalence of those maladies.
Planning Process at the Tufts Cummings Veterinary School One year + planning process involving: Educational Dean Chair of Rotation Directors All Rotation Directors Support Workshops provided by the Tufts Center for Teaching and Learning
Part 1 The Tufts Cummings Veterinary School Experience Start with Competency Statements of accrediting body – AVMA Localize to School Wide graduating competencies for clinical rotations Workshops on CBE and competency writing Build rotation/clerkship Competencies Produce Grading Rubrics
Vet School Experience Match each rotation back to the school wide competencies All competencies come back to curriculum committee for approval Informatics had not been identified by any rotation – curriculum committee added these competencies Curriculum committee added the preclinical competencies
Helpful Tips Provide a good model Provide support – workshops Do not insist on uniformity Make sure grading rubrics match back to competencies Identify it as a work in progress Hard part was getting the attention of the rotation directors
Helpful Tips One person brings it all together Work with external bodies – curriculum committee to provide overview
How to write Competencies Understand the level (school-wide, course, content) Action Words Verbs followed by nouns Measureable and/or observable Performance based
How to write…. Consider relationships up and down the framework Do not use evaluative words in the competency (this will come later) Who owns the competency? Who can create? Who can modify? Feel free to use terms like ‘outcomes’, learning objectives
How rubrics help: For Courses: Rubrics enable faculty to clearly communicate expectations for student performance to students. They support more consistent and objective assessment of student work. They also facilitate detailed feedback to students. When areas of strength and weakness in performance are noted, students more easily comprehend the reasoning behind their professor’s assessment of their work. A rubric is a scoring guide that seeks to evaluate a student's performance based on the sum of a full range of criteria rather than a single numerical score. For Student Outcomes Assessment: Rubrics facilitate quality conversations among faculty about student learning, shorten the time it takes to do SOA, and contribute to the validity and reliability of the assessment process. - Stevens and Levi, Ch. 2 www.uni.edu/chfasoa/IntroductiontoRubrics.ppt
Developing a Rubric cont. Scale How well or poorly has the student done on the task? Sophisticated, competent, partly competent, not yet competent. Exemplary, proficient, marginal, unacceptable. Advanced, Intermediate, Novice. Distinguished, proficient, intermediate, novice. Accomplished, developing, beginning. (Huba and Freed, 2000) Note: Consider whether a scale needs more than three levels. Some research indicates that information about student learning obtained from a three-level scale is comparable to that obtained from a five- level scale. More levels typically means more time spent on assessment.
Developing a Rubric cont. Dimensions Dimensions break down a task into components and identify the importance of these components. Dimensions are descriptive, not evaluative (e.g., “organization” not “good organization”). Dimensions help students see that the work they are doing is multidimensional and draws on multiple abilities.
Developing a Rubric cont. Description of the Dimensions Rubrics should contain at least a description of the highest level of performance. With experience and in response to the complexity of the performance, descriptions of all levels can be written. Students need not fit cleanly into a single category. On oral presentation skills, a student might speak in a clear voice but lack eye contact. Descriptions for each level help students see that the work they are doing does have varying levels of achievement. There is a difference between poor and excellent work.
Planning Process After the competencies and grading rubrics TUSK had a series of meetings with : Educational Dean Chair of Rotation Directors All Rotation Directors Iterative process of moving current process to Online version integrated with TUSK Create Powerpoint ”story board” which developer used as functional spec
Process to get to competency assessment in TUSK TUSK meetings with vet school leaders Understand business processes Translate these to the online world Develop powerpoint story board to achieve agreement of functional spec Iterate over 7 months Check in with all rotation directors Test and train Pilot test in two clerkships Release to all clerkships
View the Process through Every Lens Course Administrators Create form Link to competencies Link Faculty to students Faculty Assessors Able to select/deselect students See picture of student Easy access to form Course Director See each assessor’s view as well as summary Able to add comments Able to override scores Student View the results Compilation of all clerkship results Registrar Access final grade (allow resubmit)
Competency Assessment Features Prepare TUSK to accept competencies Link to School-wide/course competencies Configurable form for each course Multiple assessors Access to Registrar Show scores or only feedback Multiple rubrics per form Override score/grade Student retains access to scores over time
Overview I. Admin View a) Populate Competency Data b) Populate Competency Assessment Form & Data c) Other Assessment Form Data II. Faculty View a) Access to the Assessment b) Selecting Additional Students c) Assessment Form III. Director View a) Assessment Results for All Students b) Completion Report III. Director View (con’t) c) Individual Student Results d) Completed Faculty/Staff Form e) Summary Form IV. Student View a) Accessing Assessment Results V. Registrar View IV. View final grades
I. Administrator View Creating and Populating Competency Assessment Data
POPULATE COMPETENCY DATA Done at the School Administrator level
POPULATE COMPETENCY ASSESSMENT FORM DATA Done at the Course Admin level
Step 2: Create Assessment Form(s) Note: The “View Reports” button is only visible to the course director “Balance Weights” still needs to be further discussed/clarified
Populate Assessment Form Metadata Grading is numeric. There will be an option in the grade book to give an equivalent letter or pass/fail grade Scoring Range defaults to ‘No’ Score Display: Score & Feedback Feedback Only Scoring Value: Max value = 25 Frequency: Max No. = 10
Step 3: Populate Fields and Data Question type: Scaling Scaling w/Sub-question Single Select Single Select w/Sub-question Multi-level questions (options with sub- questions) Link competencies to questions Free-text feedback option
First steps A set of clear definitions Use cases A survey of Canadian, US and UK health professional education organizations (24 respondents) See definitions_and_usecases.pdf for details (medbiquitous.org)
Comparison of published frameworks Accreditation Council on Graduate Medical Education (ACGME) Competencies 8 Acute Care Nurse Practitioner Competencies American Association of Critical-Care Nurses (AACN) General Patient Care Competencies American Society of Health-System Pharmacists CanMEDS 2005 9 CanMEDS Specialty Competencies Good Medical Practice (UK) Good Medical Practice (USA) The Scottish Doctor 7 Tomorrow's Doctors Tuning (MEDINE) Women’s Healthcare Competencies
Outcomes for Clinical Skills A competency object... Relates to other competency objects... Which can relate to external resources... Learning Object Assess- ment Performance data Current Specifications
Specs and Schemas Available At: http://www.medbiq.org/working_group s/competencies/index.html http://www.medbiq.org/working_group s/competencies/index.html Open license, membership not required
Competencies – individual competency definition A standard format for a single competency definition, including an identifier, categories and references. Competency Object Working draft Being implemented by Tufts May be implemented by VA Essential for tying curricula to competencies and enabling competency based education. Competencies – an interrelated set of competencies A framework for establishing the relationships among competencies in a single framework Competency Framework Working draft Being implemented by Tufts May be implemented by VA Essential for integrating existing competency sets into curriculum management systems. Competencies – evidence of achievement A format for documenting achievement of a milestone or competency. Educational Achievement Work to begin in July 2010 This will be important for schools tracking competence in a portfolio and for residency programs tracking competence. Competencies – cross-mapping A format for mapping competencies from one framework to competencies from another framework. Informal proposal from competency working group (planned work) Should we arrive at the point of having a national competency framework, this would be important for schools that want to see how local competencies fit into the national picture. Competencies – mapping to external resources A way of mapping competencies to learning activities and resources, curricula, and potentially performance data. Informal proposal from competency working group (planned work) Could be important for the curriculum inventory portal should it be competency based.
Spec needs Analysis of the Problem from all perspectives Student View How am I doing? What do I have to do? How can I show my learning Faculty view How is student doing Grade/Rate/Comment on student View proof of learning See full range of expected learning outcomes Registrar Track grades Admin View What are the competencies Where are competencies taught - content/courses/themes/rotations/clinics Where are the gaps Need tools to Enter, Store, view, Grade
Medbiquitous Competency Object Standard Copyright Medbiquitous consortium 2010 All rights reserved May 20 2010 p.11
What Info Needs to be brought Together School wide competencies Course level competencies/learning objectives Content addressing competencies and learning objectives Element of time – when and how much How are competencies assessed
Where does TUSK get the Data Schedule Content linked to schedule (upon upload) Curriculum hours pulled from schedule Keywords linked to content
School Wide Competencies: Competency Framework School Wide Competencies: Competency Framework Course competencies Content repository w/ metadata Content repository w/ metadata Content competencie s Content competencie s Competency Assessment skills knowledge attitudes Where are they taught ? How are they taught? Where are they assessed? External Competency Framework External Competency Framework lecture s PBL rotation s When? (Links to schedule) Competencies
Methods to Display Complex Data complex data displays complex data displays
References Medical Teacher: Outcomes-Based Education Themed Issue Vol 29 Number 7 September 2007 Lessons Learned from the Competency-Based Curriculum Initiative at Baylor College of Dentistry. McCann, Ann L.; Babler, William J.; Cohen, Peter A., Journal of Dental Education, v62 n2 p197-207 Feb 1998 McCann, Ann L.Babler, William J.Cohen, Peter A. The new formal competency based curriculum at Indiana University School of Medicine, Litzleman and Cottingham, Academic Medicine Vol 82, No. 4 April 2007 An educational blueprint for the Brown Medical School, Smith MD Learning objectives for medical student education--guidelines for medical schools: report I of the Medical School Objectives Project. AAMC Acad Med. 1999 Jan;74(1):13-8.