The ABC’s of CDE (Curriculum Development and Evaluation) We’re hoping to provide the building blocks or basics on how to develop and curriculum in medical education. Cliparts.co Teaching and Learning (B) Leslie, Salma and Grace June 11, 2015

Agenda Introduction Goal & objectives Baseline knowledge assessment Overview of curriculum development & evaluation Three approaches to curriculum development Six-step approach Evaluation Summary/Discussion A wise man once told me that I could break rules, just never two at a time – broke one of the rules of 6 x 6 slides – mea culpa Three approaches – mention that we will do a deep dive into one of the three approaches Fun activities interspersed throughout workshop Wrap up – return to goals of workshop and identify how we achieved them (should we ask audience if they agree?)

Workshop Goal To become familiar with the principles of curriculum development and evaluation Our ultimate goal is you can take the principles and build any curriculum needed, big or small Cliparts.co Bitchspot.jadedragononline.com

Workshop Objectives Appreciate the complexity of curriculum development and evaluation Be able to actively participate in applying the six steps of curriculum development Identify the core principles of curriculum development and evaluation Attitude Skill Alphabetblocks.org Taxonomies Attitudes = Krathwohl Skills = Simpson Knowledge = Bloom Knowledge

Overview of Curriculum Development Definition “The means and materials with which students will interact for the purpose of achieving identified educational outcomes.” Ebert, Ebert & Bentley (2013) About intention of teaching and the way it will happen (Dent) Can apply to everything from a large lecture series to a small workshop and everything inbetween

‘Need to Know’ Complex task Knowledge & skill of curriculum development process Commitment by faculty, administration, students Three spheres Formally endorsed curriculum Informal curriculum Hidden curriculum Goldenberg, Andrusyszyn, Iwasiw (2004); Bennett, Lockyer, Mann et al. (2004) Three spheres: formal – is what is written down and approved by the Academy; informal is the adhoc, interpersonal form of teaching and learning that takes place; hidden is the set of influences that operate at the organizational structure level and culture All three spheres have a direct impact not only on learner’s outcomes and decisions about their career paths (impact of hidden curriculum)

Importance of Curriculum Development Fundamental changes in medical practice Shift from hospital-based to community-based clinical settings Advances in information technology Need to encourage interprofessional training Changes in society’s expectations of physicians Key to long term change in medical education More attention on process and organization of medical education Bligh (1999); Goldenberg et al. (2004); Neufeld, Maudsley, Pickering et al. (1998) Rigid to fluid – integrating science teaching with clinical instruction rather than approaching as discrete units Nursing – champions of curriculum development – focus on professional development, personal teaching and learning experiences as both learners and academics – embrace CD

Importance cont’d Curriculum impacts on learning experience No difference in Board exam scores Traditional vs innovative approach: Poor development of moral reasoning skills Increased stress Less desirable mode of learning Bligh (1999) Ripsky, Swanson & Case study 1998 Study compared traditional rigid curriculum to more fluid, innovative ones

Problem-based Learning Approach (PBL) Definition Educational method which is used to: Help learners identify their learning needs Motivate self-directed learning Develop problem-solving & clinical reasoning skills Work effectively in a group Barrows (1985); Walsh (2005)

Advantages Promote self-directed learning Motivate learners Create interest about subject Facilitate team work Barrows (1985)

Disadvantages Resource intensive Lack of traditional structure and progression Lack of depth of acquired knowledge Time consuming Does not fit all learning situations Barrows (1985); Hung (2009); Robert (1997); Bradley-Levine (2014)

How to Build a PBL Curriculum Why? Where? Who? Set the learning goals and objectives Set the stage The scenario (the problems) The location The timing The equipment (resources) The actors Reinout (2002); Barrows (1985); Walsh (2005); Stanford Uni (2001)

PBL Curriculum The Process: The initial activities Introduce the problem Self-directed study Application of the new knowledge Summary and integration of learning Barrows (1985)

PBL Summary Helps learners to develop problem solving and teamwork skills Secret of success: requires unstructured timing Resource intensive High recall of content, surface learning

Experiential Learning Approach NOT at a prescription for how to approach curriculum development IS a theory about adult learning that can be applied to curricula Li, Greenberg & Nicholls (2007) Tenets of ELT which I will be going over shortly are embedded in the curriculum

What is Experiential Learning? Adult learning theory developed by David Kolb Used extensively in many professions Learners take charge of their learning Provides opportunity for deep learning Discovery, self-reflection Second point – used extensively in nursing, business, law, social work, medical residency (internships, practicums, articling etc), Self-reflection is mechanism that transforms experience into learning

4 Tenets of Experiential Learning Theory (ELT) Focus on PROCESS not OUTCOMES Holistic approach All learning is re-learning Moves between reflecting, acting, feeling, thinking Second point: holistic in that it moves beyond cognition to include behaviors, feelings, perceptions Third point: dynamic process

4 Stages of Experiential Learning Cycle Reference: Kolbe (1984), Kolb & Kolb (2005) Kolb (1984); Kolb & Kolb (2005)

Characteristics of Curriculum Based on ELT Learner-centred while respecting professional standards Variety of learning activities Time for reflection on experience Where learning occurs Reflect on learning experience as well on you as a learner Kolb (1984); Kolb & Kolb (2005) Second point – variety of activity so learner can match their learning style with activities

When is it Appropriate to Use ELT in Curriculum Development? Adult learning Need to meet standardized requirements of regulatory body while still addressing learners’ individual needs Courses where theory and practice must come together to complete the learning process

Challenges of Using ELT in Curriculum Development Departure from traditional lecture-style teaching Can be met with resistance More difficult to manage with larger groups Effort intensive theory to put into practice Kolb & Kolb (2005); Li, Greenberg & Nicholls (2007); Smith, Emmett & Woods (2008)

A Six-Step Approach Step 1 Problem Identification and General Needs Assessment Step 2 Needs Assessment for Targeted Learners Step 3 Goals and Objectives Step 4 Educational Strategies Step 5 Implementation Step 6 Evaluation and Feedback Step 1 begins with the identification and critical analysis of a health care need or other problem. A complete problem identification requires an analysis of the current approach of patients, practitioners, medical education system and society to addressing the identified need. This is followed by identification of an ideal approach that describes how patients, practitioners, the medical education system and society should be addressing the need. The difference between the ideal approach and the current approach represents a general needs assessment. Step 2 involves assessing the needs of one’s targeted group of learners and their learning environment Once the needs of targeted learners have been identified, step 3 involves writing goals and objectives for the curriculum, starting with broad or general goals then moving the specific, measurable objectives. This step is critical because it helps to determine curricular content, learning methods and help to focus the learners. Once objectives have been clarified, curriculum content is chosen and educational methods that most likely achieve he educational objectives are selected. Step 5, implementation of a curriculum has several components including obtaining political support, getting resources, addressing barriers to implementation, introducing and administering and refining the curriculum. This is the step that converts a mental exercise into reality Step 6 also has several components. It is to assess the performance of both individuals (individual assessment) and the curriculum (called program evaluation). Evaulation drives the ongoing learning of particpants and improvemet of the curriculum. It can also be used to gain support and resources for the curriculum.

Advantages & Disadvantages Systematically designed Practical, feasible approach Generic Timeless Disadvantages Concept is a ‘step-wise’ approach

Step 1: Problem Identification and General Needs Assessment Identify the current approach Brainstorm the ideal solution General needs assessment Dr Gillian Hawker (chair, DOM UofT) said everything we do in medicine ultimately is to serve our patients, thus medical education has to fulfill a need for our patients as well. The first step is to identified the health problem or learning problem that needs to be addressed in your practice. There are 4 substeps

Step 1a: Problem Identification What is the health or learning problem? Whom does the problem affect? What does the problem affect? How big is the problem? What is the problem? This could be a disease state that has emerged in your population, a national epidemic, or a well known health problem that you feel is not being satisfactorily addressed in your current curriculum. Then you have to ask, whom does the problem affect? For example, does the problem affect people with a particular disease (e.g. frequent respiratory exacerbations requiring hospitalization in patients with asthma), learners (e.g. physicians inadequately prepared to provide ambulatory care to teach in the ambulatory setting) or health care organizations (e.g. failure to provide cost-effective medicine?). Does the problem affect patients, learners, medical educators, health care organizations or society? What does the problem affect? Clinical outcomes, educational effectiveness, quality of life, health care resources? To what extent is this a problem currently? High priority, medium or low priority?

Step 1b: Identify the Current Approach What is currently being done about it? Is there existing medical curricula? Does anyone else have the same problems? How are they resolving them? Where are sources of possible solutions? What is the current approach to addressing the problem or identified need? On a broader scope, what are patients, practitioners, medical education system and society currently doing to address the need? Are there medical education curricula already in place to address the problem?

Step 1c: Brainstorm the Ideal Solution In the perfect world, How would patients, health care professionals, educators and society solve this problem? How would you solve this curricular problem? Now that you have identified the current approach to the problem, including current medical education curricula, think of what an ideal solution would be. In other words, with unlimited resources, students, faculty, materials, time, money, how could you solve problem? To what extent should patients be involved in handling the problem themselves? Which health care professionals should deal with the problem and what should they be doing? Is there a specific type of health care professionals that see this problem more? What role should medical educators have? What are some past educational strategies? What role should society have? A curriculum can choose to address societal changes that might alleviate problem This step gives you the basic idea of resources that you will need to solve the problem even if all of them are not within your reach.

Step 1d: General Needs Assessment The difference between the actual and ideal curriculum model is the general needs assessment Patients Health Care Professionals Medical Educators Society Current Approach Ideal Approach Use top row of table to list some things that groups are doing to currently address the problem Based on current knowledge, use bottom row to list things that groups should be ideally doing to address the problem, filling the gap This provides an idea of the needed resources for your curriculum

What are the Barriers to Solution? What are the barriers between current and ideal solution? Time Personnel Money Knowledge Community issues Knowing the problem with your curriculum is not enough, one must know the circumstances that makes it a problem

Step 2: Targeted Needs Assessment Describes the learners and their environment Learners Previous training Existing proficiencies Past and current performance Attitudes Learning styles Environment Past or current curricula Enabling factors and barriers to curriculum development Stakeholders Salma talks Leslie writes

Step 3: Set Goals and Objectives Goal = broad educational objective Objective = specific measurable outcomes Definition of goal/objective: an end toward which an effort is directed Goal = broad educational objective (i.e. overall purpose of curriculum) Objective = specific measurable objective Why this step important: helps direct choice of curricular content, suggests what learning methods most effective, enables evaluation of learners and curriculum, clearly communicates to others what curriculum hopes to achieve, and benchmark to evaluate against

Writing Objectives – 5 Elements Who? Will do? How Much? Of What? By When? Each resident will demonstrate at least once of obtaining PAP smear by end of course

Writing Learner Objectives Attitude Skill Alphabetblocks.org Knowledge

Competency-Based Curriculum Objectives related to CanMEDS roles framework Push in North America towards competency-based outcomes, where all objectives must be related to the core competencies which describe the expertise of modern clinical practice

Other Types of Objectives Process objectives Outcome objectives Process objectives relate to implementation of the curriculum, and getting an outcome of that (different from ELT process) (e.g. by the academic year, 90% of PGY2 will have participated in the patient safety initiative) Outcome objectives – referring to health, healthcare and patient outcomes (the impact of the curriculum beyond those delineated in its learner and process objective) – some of these are the metrics that you want to build into evaluation

Step 4: Educational Strategies What How Curriculum content Curriculum methods Education strategies is about choosing curriculum content and methods Content is the specific material to be included, methods are the way in which the content is presented Choosing curriculum content and education methods that most likely achieve educational objectives Why important: educational strategies are the heart of the curriculum, the educational intervention itself. Natural tendency to think of the curriculum in terms of this step alone, but groundwork is through steps 1-3

Guidelines to Choose Educational Methods Understand the principles of adult learning Feasibility Congruence between objectives and educational methods Guidelines to choose educational methods: Understand adult learning theory as curriculum developers are to facilitate learning (goal-oriented, motivated, self-directed, bring experiences, experiential learning , etc.) Curriculum content flows from learning objectives. Should maintain congruence between objectives and educational methods Use multiple educational methods to maintain learner interest and reinforce learning Choose educational methods that are feasible in terms of resources – what are some elements of feasibility that need to be considered? consider time, space, availability of material, cost and availability of learner time

Congruence Between Objectives and Educational Methods Attitudes Skills Knowledge Exposure Supervised clinical experience Readings Facilitation of reflection Simulations Lecture Role models Artificial models Audiovisual materials Discussion Problem-based learning Programmed learning Pg 62 in book

Step 5: Implementation Converts a mental exercise to reality Resources Political support Administration and support staff Barriers Obtaining political support both internally and externally Consider the details of administrating the curriculum: operations staff, communication, administration staff Anticipate barriers – are they financial, people or other resources?

Piloting and Phasing In What are some advantages/disadvantages to piloting a curriculum? What are some advantages/disadvantages to phasing in a curriculum? Piloting Advantages: Introduce to friendly audience Getting critical feedback to make revisions Piloting Disadvantages: Resources Need to find friendly audience Phasing Advantages: Focusing of initial efforts as teaching staff learn new curriculum Lessen resistance that new cultural shifts or curriculum brings Critical feedback Phasing Disadvantages: Pressing need for full curriculum by learners

Step 6: Evaluation and Feedback Closes the loop in the curriculum development cycle Evaluation: identification, clarification and application of criteria to determine the merit or worth of what is being evaluated Evaluation and feedback: is important because it closes the loop in the curriculum development cycle. Helps make a decision or judgment about the curriculum. Answers the critical question “were goals and objectives met”. Guides ongoing improvement. Can garner support for curriculum, to assess student achievement, satisfy external requirements, document accomplishments of curricular developers. Keepcalm-o-matic.co.uk

Curriculum Evaluation - Definition “The systematic documentation of the consequences of a curriculum and the determination of their worth in order to make decisions about them.” Green (1977)

Purposes of Curriculum Evaluation Determine the outcome Decide the future of the program Future development Improve teaching methods Cost-benefit analysis Knowledge dissemination Green (1977); Herman (1987); Kern (2009)

What to Evaluate Learners Instructors Curriculum material Scope and sequence of content Environment and facilities The overall curriculum Green (1977)

Types of Evaluations Final feedback Summative evaluation Formative evaluation Final feedback Evaluates overall effectiveness and impact of curriculum Helps to decide about future of the program Occasional or continuous feedback Used to continually improve the curriculum during its implementation Green (1977); Herman (1987)

How to Evaluate - The Evaluation Cycle Rowel (1999)

Who Benefits? Decision-makers Employers Evaluators Scholars Students The profession Academic setting Green (1977); Herman (1987)

Models of Evaluation Bradley's effectiveness model (1985) Tyler's objectives centred model (1950) Stufflebeam Context, Input, Process, Product (CIPP) 1971 Glatthorn (2015)

Key Principles Complex process Begins with clear goals and objectives Meets needs of learners and communities they serve Contains evaluation/feedback mechanism Foundation to successful learning Highlight a few approaches to CD and evaluation – all have things in common: Complex: requires knowledge of the topic, learners, context (institutional, community) Feedback loop makes it an iterative process