Competency Workshop Medbiquitous May 2011 Susan Albright

Workshop Objectives By the end of this workshop, participants will be able to: Describe a process for developing competency frameworks Identify challenges and potential barriers to developing and implementing competency frameworks Identify opportunities available with the implementation of competency frameworks in an electronic system

Agenda Introductions Part 1 – Background on the Tufts Experience Activity Part 2 Assessment Discussion Break Part 3 – Developing the system Discussion Summary

Why Competencies? “Competency-based education is characterized by the development of clearly defined and explicitly published competencies, mapping the curriculum to achieve competencies, and assessment process matched to competencies.” (Ron Harden, Medical Teacher 2002 #24 Vol 2

What is Competency Based Education CBE is an institutional process that moves education from focusing on what academics believe graduates need to know (teacher-focused) to what students need to know and be able to do CBE is focused on outcomes (competencies) that are linked to workforce needs as defined by the profession. CBE necessitates complex assessment through portfolios or experiential learning assessement in field experience Large skill sets are broken down into competencies which sequestial levels of mastery

CBE Competencies may reinforce one another from basic to advanced as learning progresses Continual refinement of competencies is essential as this a process not a product Consistency in expectations across the curriculum Consistency in grading Council on Education for Public Health

Competency Frameworks ACGME Scottish Doctors Tomorrow’s Doctors The Tuning Project CanMeds MSOP AAMC

ACGME Competency Domains Patient care Medical knowledge Practice-based learning and improvement Interpersonal and communication skills Professionalism Systems-based practice

AAMC Learning objectives The first of the four attributes is that physicians must be altruistic. There are seven learning objectives, including the objective that before graduation, the student can demonstrate compassionate treatment of patients and respect for their privacy and dignity. The second attribute is that physicians must be knowledgeable; one of the six learning objectives is that the student can demonstrate knowledge of the normal structure and function of the body and of each of its major organ systems. The third attribute is that physicians must be skillful; one of the eleven learning objectives is that the student have knowledge about relieving pain and ameliorating the suffering of patients. The last attribute is that physicians must be dutiful; one of the six learning objectives is that the student have knowledge of the epidemiology of common maladies within a defined population, and the systematic approaches useful in reducing the incidence and prevalence of those maladies.

Planning Process at the Tufts Cummings Veterinary School One year + planning process involving: Educational Dean Chair of Rotation Directors All Rotation Directors Support Workshops provided by the Tufts Center for Teaching and Learning

Part 1 The Tufts Cummings Veterinary School Experience Start with Competency Statements of accrediting body – AVMA Localize to School Wide graduating competencies for clinical rotations Workshops on CBE and competency writing Build rotation/clerkship Competencies Produce Grading Rubrics

Vet School Experience Match each rotation back to the school wide competencies All competencies come back to curriculum committee for approval Informatics had not been identified by any rotation – curriculum committee added these competencies Curriculum committee added the preclinical competencies

Helpful Tips Provide a good model Provide support – workshops Do not insist on uniformity Make sure grading rubrics match back to competencies Identify it as a work in progress Hard part was getting the attention of the rotation directors

Helpful Tips One person brings it all together Work with external bodies – curriculum committee to provide overview

How to write Competencies Understand the level (school-wide, course, content) Action Words Verbs followed by nouns Measureable and/or observable Performance based

How to write…. Consider relationships up and down the framework Do not use evaluative words in the competency (this will come later) Who owns the competency? Who can create? Who can modify? Feel free to use terms like ‘outcomes’, learning objectives

Part 2 Assessment and grading rubrics

How rubrics help: For Courses: Rubrics enable faculty to clearly communicate expectations for student performance to students. They support more consistent and objective assessment of student work. They also facilitate detailed feedback to students. When areas of strength and weakness in performance are noted, students more easily comprehend the reasoning behind their professor’s assessment of their work. A rubric is a scoring guide that seeks to evaluate a student's performance based on the sum of a full range of criteria rather than a single numerical score. For Student Outcomes Assessment: Rubrics facilitate quality conversations among faculty about student learning, shorten the time it takes to do SOA, and contribute to the validity and reliability of the assessment process. - Stevens and Levi, Ch. 2

Developing a Rubric cont. Scale How well or poorly has the student done on the task? Sophisticated, competent, partly competent, not yet competent. Exemplary, proficient, marginal, unacceptable. Advanced, Intermediate, Novice. Distinguished, proficient, intermediate, novice. Accomplished, developing, beginning. (Huba and Freed, 2000) Note: Consider whether a scale needs more than three levels. Some research indicates that information about student learning obtained from a three-level scale is comparable to that obtained from a five- level scale. More levels typically means more time spent on assessment.

Developing a Rubric cont. Dimensions Dimensions break down a task into components and identify the importance of these components. Dimensions are descriptive, not evaluative (e.g., “organization” not “good organization”). Dimensions help students see that the work they are doing is multidimensional and draws on multiple abilities.

Developing a Rubric cont. Description of the Dimensions Rubrics should contain at least a description of the highest level of performance. With experience and in response to the complexity of the performance, descriptions of all levels can be written. Students need not fit cleanly into a single category. On oral presentation skills, a student might speak in a clear voice but lack eye contact. Descriptions for each level help students see that the work they are doing does have varying levels of achievement. There is a difference between poor and excellent work.

Examples of Grading Rubrics handout

Competency Assessment- TUSK mock-ups

Planning Process After the competencies and grading rubrics TUSK had a series of meetings with : Educational Dean Chair of Rotation Directors All Rotation Directors Iterative process of moving current process to Online version integrated with TUSK Create Powerpoint ”story board” which developer used as functional spec

Process to get to competency assessment in TUSK TUSK meetings with vet school leaders Understand business processes Translate these to the online world Develop powerpoint story board to achieve agreement of functional spec Iterate over 7 months Check in with all rotation directors Test and train Pilot test in two clerkships Release to all clerkships

View the Process through Every Lens Course Administrators Create form Link to competencies Link Faculty to students Faculty Assessors Able to select/deselect students See picture of student Easy access to form Course Director See each assessor’s view as well as summary Able to add comments Able to override scores Student View the results Compilation of all clerkship results Registrar Access final grade (allow resubmit)

Competency Assessment Features Prepare TUSK to accept competencies Link to School-wide/course competencies Configurable form for each course Multiple assessors Access to Registrar Show scores or only feedback Multiple rubrics per form Override score/grade Student retains access to scores over time

Overview I. Admin View a) Populate Competency Data b) Populate Competency Assessment Form & Data c) Other Assessment Form Data II. Faculty View a) Access to the Assessment b) Selecting Additional Students c) Assessment Form III. Director View a) Assessment Results for All Students b) Completion Report III. Director View (con’t) c) Individual Student Results d) Completed Faculty/Staff Form e) Summary Form IV. Student View a) Accessing Assessment Results V. Registrar View IV. View final grades

I. Administrator View Creating and Populating Competency Assessment Data

POPULATE COMPETENCY DATA Done at the School Administrator level

Step 1: Create Competency Category

Modify Competency Category

Add Sub-Competency Levels (if applicable)

Sort Competencies

Sort Child Competencies

POPULATE COMPETENCY ASSESSMENT FORM DATA Done at the Course Admin level

Step 2: Create Assessment Form(s) Note: The “View Reports” button is only visible to the course director “Balance Weights” still needs to be further discussed/clarified

Populate Assessment Form Metadata Grading is numeric. There will be an option in the grade book to give an equivalent letter or pass/fail grade Scoring Range defaults to ‘No’ Score Display: Score & Feedback Feedback Only Scoring Value: Max value = 25 Frequency: Max No. = 10

Step 3: Populate Fields and Data Question type: Scaling Scaling w/Sub-question Single Select Single Select w/Sub-question Multi-level questions (options with sub- questions) Link competencies to questions Free-text feedback option

Add Field: Scaling Make text boxes bigger

Linking Competencies

Scaling w/ Sub-Questions

Add Field: Single Select

Single Select w/ Sub-level Questions

OTHER COMPETENCY ASSESSMENT FORM DATA Done at the Course Admin level

Considerations Display student images? Display elective information? Display summary comments Multiple assessors? Who selects assessors?

Considerations? Use Numeric scores? Display scores to students or just comments? Display scores to assessors? 360 degress review? What else?

Set Category Weights

Link Faculty/Student

II. Faculty View Getting to the form

Accessing the Assessment(s)

Accessing the Form Evaluator: J. Berg

Assessment Form – Scaling

Assessment Form – Single Select

III. Director View Viewing assessment results

Assessment Results – All Students

Assessment Results – Individual Student Note: To view a form completed by a faculty/staff click their name Note: When ready, click here to fill out a final assessment form for the student

Completed Form by Faculty/Staff

Summary Form Hover over score to see actual individual scores This is at the bottom of the form Click to view student’s patent logs in a pop-up window

Student’s Patient Log Summary This opens up as a pop-up window

Completion Report

Send Reminders

IV. Student View Access via the Grade Book

Accessing Assessment Form Note: Link to view final assessment form completed by the course director

Accessing Assessment Form J. Berg happens to also be the course director

V. Registrar View View Final Grades

Grades by Course Step 1: Select the course

Grades by Course Step 2: Select the time period

Part 3 Building the Framework

First steps A set of clear definitions Use cases A survey of Canadian, US and UK health professional education organizations (24 respondents) See definitions_and_usecases.pdf for details (medbiquitous.org)

Comparison of published frameworks Accreditation Council on Graduate Medical Education (ACGME) Competencies 8 Acute Care Nurse Practitioner Competencies American Association of Critical-Care Nurses (AACN) General Patient Care Competencies American Society of Health-System Pharmacists CanMEDS CanMEDS Specialty Competencies Good Medical Practice (UK) Good Medical Practice (USA) The Scottish Doctor 7 Tomorrow's Doctors Tuning (MEDINE) Women’s Healthcare Competencies

DEVELOPMENT OF A CONCEPTUAL MODEL

Outcomes for Clinical Skills A competency object... Relates to other competency objects... Which can relate to external resources... Learning Object Assess- ment Performance data

Outcomes for Clinical Skills A competency object... Relates to other competency objects... Which can relate to external resources... Learning Object Assess- ment Performance data Current Specifications

Specs and Schemas Available At: s/competencies/index.html s/competencies/index.html Open license, membership not required

Competencies – individual competency definition A standard format for a single competency definition, including an identifier, categories and references. Competency Object Working draft Being implemented by Tufts May be implemented by VA Essential for tying curricula to competencies and enabling competency based education. Competencies – an interrelated set of competencies A framework for establishing the relationships among competencies in a single framework Competency Framework Working draft Being implemented by Tufts May be implemented by VA Essential for integrating existing competency sets into curriculum management systems. Competencies – evidence of achievement A format for documenting achievement of a milestone or competency. Educational Achievement Work to begin in July 2010 This will be important for schools tracking competence in a portfolio and for residency programs tracking competence. Competencies – cross-mapping A format for mapping competencies from one framework to competencies from another framework. Informal proposal from competency working group (planned work) Should we arrive at the point of having a national competency framework, this would be important for schools that want to see how local competencies fit into the national picture. Competencies – mapping to external resources A way of mapping competencies to learning activities and resources, curricula, and potentially performance data. Informal proposal from competency working group (planned work) Could be important for the curriculum inventory portal should it be competency based.

Spec needs Analysis of the Problem from all perspectives Student View How am I doing? What do I have to do? How can I show my learning Faculty view How is student doing Grade/Rate/Comment on student View proof of learning See full range of expected learning outcomes Registrar Track grades Admin View What are the competencies Where are competencies taught - content/courses/themes/rotations/clinics Where are the gaps Need tools to Enter, Store, view, Grade

Medbiquitous Competency Object Standard Copyright Medbiquitous consortium 2010 All rights reserved May p.11

What Info Needs to be brought Together School wide competencies Course level competencies/learning objectives Content addressing competencies and learning objectives Element of time – when and how much How are competencies assessed

Where does TUSK get the Data Schedule Content linked to schedule (upon upload) Curriculum hours pulled from schedule Keywords linked to content

School Wide Competencies: Competency Framework School Wide Competencies: Competency Framework Course competencies Content repository w/ metadata Content repository w/ metadata Content competencie s Content competencie s Competency Assessment skills knowledge attitudes Where are they taught ? How are they taught? Where are they assessed? External Competency Framework External Competency Framework lecture s PBL rotation s When? (Links to schedule) Competencies

Schedule – Curriculum View

First year courses

Course Summary view

Methods to Display Complex Data complex data displays complex data displays

References Medical Teacher: Outcomes-Based Education Themed Issue Vol 29 Number 7 September 2007 Lessons Learned from the Competency-Based Curriculum Initiative at Baylor College of Dentistry. McCann, Ann L.; Babler, William J.; Cohen, Peter A., Journal of Dental Education, v62 n2 p Feb 1998 McCann, Ann L.Babler, William J.Cohen, Peter A. The new formal competency based curriculum at Indiana University School of Medicine, Litzleman and Cottingham, Academic Medicine Vol 82, No. 4 April 2007 An educational blueprint for the Brown Medical School, Smith MD Learning objectives for medical student education--guidelines for medical schools: report I of the Medical School Objectives Project. AAMC Acad Med Jan;74(1):13-8.