1. The ABC’s of CDE (Curriculum Development and Evaluation)
We’re hoping to provide the building blocks or basics on how to develop and curriculum in medical education.
Cliparts.co
Teaching and Learning (B)
Leslie, Salma and Grace
June 11, 2015

2. Agenda
Introduction Goal & objectives Baseline knowledge assessment Overview of curriculum development & evaluation Three approaches to curriculum development Six-step approach Evaluation Summary/Discussion
A wise man once told me that I could break rules, just never two at a time – broke one of the rules of 6 x 6 slides – mea culpa
Three approaches – mention that we will do a deep dive into one of the three approaches
Fun activities interspersed throughout workshop
Wrap up – return to goals of workshop and identify how we achieved them (should we ask audience if they agree?)

3. Workshop Goal To become familiar with the principles of
curriculum development and evaluation
Our ultimate goal is you can take the principles and build any curriculum needed, big or small
Cliparts.co
Bitchspot.jadedragononline.com

4. Workshop Objectives
Appreciate the complexity of curriculum development and evaluation
Be able to actively participate in applying the six steps of curriculum development
Identify the core principles of curriculum development and evaluation
Attitude
Skill
Alphabetblocks.org
Taxonomies
Attitudes = Krathwohl
Skills = Simpson
Knowledge = Bloom
Knowledge

5. Overview of Curriculum Development Definition
“The means and materials with which students will interact for the purpose of achieving identified educational outcomes.”
Ebert, Ebert & Bentley (2013)
About intention of teaching and the way it will happen (Dent)
Can apply to everything from a large lecture series to a small workshop and everything inbetween

6. ‘Need to Know’ Complex task
Knowledge & skill of curriculum development process
Commitment by faculty, administration, students
Three spheres
Formally endorsed curriculum
Informal curriculum
Hidden curriculum
Goldenberg, Andrusyszyn, Iwasiw (2004); Bennett, Lockyer, Mann et al. (2004)
Three spheres: formal – is what is written down and approved by the Academy; informal is the adhoc, interpersonal form of teaching and learning that takes place; hidden is the set of influences that operate at the organizational structure level and culture
All three spheres have a direct impact not only on learner’s outcomes and decisions about their career paths (impact of hidden curriculum)

7. Importance of Curriculum Development
Fundamental changes in medical practice
Shift from hospital-based to community-based clinical settings
Advances in information technology
Need to encourage interprofessional training
Changes in society’s expectations of physicians
Key to long term change in medical education
More attention on process and organization of medical education
Bligh (1999); Goldenberg et al. (2004); Neufeld, Maudsley, Pickering et al. (1998)
Rigid to fluid – integrating science teaching with clinical instruction rather than approaching as discrete units
Nursing – champions of curriculum development – focus on professional development, personal teaching and learning experiences as both learners and academics – embrace CD

8. Importance cont’d Curriculum impacts on learning experience
No difference in Board exam scores
Traditional vs innovative approach:
Poor development of moral reasoning skills
Increased stress
Less desirable mode of learning
Bligh (1999)
Ripsky, Swanson & Case study 1998
Study compared traditional rigid curriculum to more fluid, innovative ones

9. Problem-based Learning Approach (PBL) Definition
Educational method which is used to:
Help learners identify their learning needs
Motivate self-directed learning
Develop problem-solving & clinical reasoning skills
Work effectively in a group
Barrows (1985); Walsh (2005)

10. Advantages Promote self-directed learning Motivate learners
Create interest about subject
Facilitate team work
Barrows (1985)

11. Disadvantages Resource intensive
Lack of traditional structure and progression
Lack of depth of acquired knowledge
Time consuming
Does not fit all learning situations
Barrows (1985); Hung (2009); Robert (1997); Bradley-Levine (2014)

12. How to Build a PBL Curriculum
Why? Where? Who?
Set the learning goals and objectives
Set the stage
The scenario (the problems)
The location
The timing
The equipment (resources)
The actors
Reinout (2002); Barrows (1985); Walsh (2005); Stanford Uni (2001)

13. PBL Curriculum The Process: The initial activities
Introduce the problem
Self-directed study
Application of the new knowledge
Summary and integration of learning
Barrows (1985)

14. PBL Summary
Helps learners to develop problem solving and teamwork skills
Secret of success: requires unstructured timing
Resource intensive
High recall of content, surface learning

15. Experiential Learning Approach
NOT at a prescription for how to approach curriculum development
IS a theory about adult learning that can be applied to curricula
Li, Greenberg & Nicholls (2007)
Tenets of ELT which I will be going over shortly are embedded in the curriculum

16. What is Experiential Learning?
Adult learning theory developed by David Kolb
Used extensively in many professions
Learners take charge of their learning
Provides opportunity for deep learning
Discovery, self-reflection
Second point – used extensively in nursing, business, law, social work, medical residency (internships, practicums, articling etc),
Self-reflection is mechanism that transforms experience into learning

17. 4 Tenets of Experiential Learning Theory (ELT)
Focus on PROCESS not OUTCOMES
Holistic approach
All learning is re-learning
Moves between reflecting, acting, feeling, thinking
Second point: holistic in that it moves beyond cognition to include behaviors, feelings, perceptions
Third point: dynamic process

18. 4 Stages of Experiential Learning Cycle
Reference: Kolbe (1984), Kolb & Kolb (2005)
Kolb (1984); Kolb & Kolb (2005)

19. Characteristics of Curriculum Based on ELT
Learner-centred while respecting professional standards
Variety of learning activities
Time for reflection on experience
Where learning occurs
Reflect on learning experience as well on you as a learner
Kolb (1984); Kolb & Kolb (2005)
Second point – variety of activity so learner can match their learning style with activities

20. When is it Appropriate to Use ELT in Curriculum Development?
Adult learning
Need to meet standardized requirements of regulatory body while still addressing learners’ individual needs
Courses where theory and practice must come together to complete the learning process

21. Challenges of Using ELT in Curriculum Development
Departure from traditional lecture-style teaching
Can be met with resistance
More difficult to manage with larger groups
Effort intensive theory to put into practice
Kolb & Kolb (2005); Li, Greenberg & Nicholls (2007); Smith, Emmett & Woods (2008)

22. A Six-Step Approach
Step 1 Problem Identification and General Needs Assessment
Step 2 Needs Assessment for Targeted Learners
Step 3 Goals and Objectives
Step 4 Educational Strategies
Step 5 Implementation
Step 6 Evaluation and Feedback
Step 1 begins with the identification and critical analysis of a health care need or other problem. A complete problem identification requires an analysis of the current approach of patients, practitioners, medical education system and society to addressing the identified need. This is followed by identification of an ideal approach that describes how patients, practitioners, the medical education system and society should be addressing the need. The difference between the ideal approach and the current approach represents a general needs assessment.
Step 2 involves assessing the needs of one’s targeted group of learners and their learning environment
Once the needs of targeted learners have been identified, step 3 involves writing goals and objectives for the curriculum, starting with broad or general goals then moving the specific, measurable objectives. This step is critical because it helps to determine curricular content, learning methods and help to focus the learners.
Once objectives have been clarified, curriculum content is chosen and educational methods that most likely achieve he educational objectives are selected.
Step 5, implementation of a curriculum has several components including obtaining political support, getting resources, addressing barriers to implementation, introducing and administering and refining the curriculum. This is the step that converts a mental exercise into reality
Step 6 also has several components. It is to assess the performance of both individuals (individual assessment) and the curriculum (called program evaluation). Evaulation drives the ongoing learning of particpants and improvemet of the curriculum. It can also be used to gain support and resources for the curriculum.

23. Advantages & Disadvantages
Systematically designed
Practical, feasible approach
Generic
Timeless
Disadvantages
Concept is a ‘step-wise’ approach

24. Step 1: Problem Identification and General Needs Assessment
Identify the current approach
Brainstorm the ideal solution
General needs assessment
Dr Gillian Hawker (chair, DOM UofT) said everything we do in medicine ultimately is to serve our patients, thus medical education has to fulfill a need for our patients as well.
The first step is to identified the health problem or learning problem that needs to be addressed in your practice.
There are 4 substeps

25. Step 1a: Problem Identification
What is the health or learning problem?
Whom does the problem affect?
What does the problem affect?
How big is the problem?
What is the problem? This could be a disease state that has emerged in your population, a national epidemic, or a well known health problem that you feel is not being satisfactorily addressed in your current curriculum.
Then you have to ask, whom does the problem affect? For example, does the problem affect people with a particular disease (e.g. frequent respiratory exacerbations requiring hospitalization in patients with asthma), learners (e.g. physicians inadequately prepared to provide ambulatory care to teach in the ambulatory setting) or health care organizations (e.g. failure to provide cost-effective medicine?). Does the problem affect patients, learners, medical educators, health care organizations or society?
What does the problem affect? Clinical outcomes, educational effectiveness, quality of life, health care resources?
To what extent is this a problem currently? High priority, medium or low priority?

26. Step 1b: Identify the Current Approach
What is currently being done about it?
Is there existing medical curricula?
Does anyone else have the same problems?
How are they resolving them?
Where are sources of possible solutions?
What is the current approach to addressing the problem or identified need? On a broader scope, what are patients, practitioners, medical education system and society currently doing to address the need?
Are there medical education curricula already in place to address the problem?

27. Step 1c: Brainstorm the Ideal Solution
In the perfect world,
How would patients, health care professionals, educators and society solve this problem?
How would you solve this curricular problem?
Now that you have identified the current approach to the problem, including current medical education curricula, think of what an ideal solution would be.
In other words, with unlimited resources, students, faculty, materials, time, money, how could you solve problem?
To what extent should patients be involved in handling the problem themselves?
Which health care professionals should deal with the problem and what should they be doing? Is there a specific type of health care professionals that see this problem more?
What role should medical educators have? What are some past educational strategies?
What role should society have? A curriculum can choose to address societal changes that might alleviate problem
This step gives you the basic idea of resources that you will need to solve the problem even if all of them are not within your reach.

28. Step 1d: General Needs Assessment
The difference between the actual and ideal curriculum model is the general needs assessment
Patients
Health Care Professionals
Medical Educators
Society
Current Approach
Ideal Approach
Use top row of table to list some things that groups are doing to currently address the problem
Based on current knowledge, use bottom row to list things that groups should be ideally doing to address the problem, filling the gap
This provides an idea of the needed resources for your curriculum

29. What are the Barriers to Solution?
What are the barriers between current and ideal solution?
Time
Personnel
Money
Knowledge
Community issues
Knowing the problem with your curriculum is not enough, one must know the circumstances that makes it a problem

30. Step 2: Targeted Needs Assessment
Describes the learners and their environment
Learners
Previous training
Existing proficiencies
Past and current performance
Attitudes
Learning styles
Environment
Past or current curricula
Enabling factors and barriers to curriculum development
Stakeholders
Salma talks
Leslie writes

31. Step 3: Set Goals and Objectives
Goal = broad educational objective Objective = specific measurable outcomes
Definition of goal/objective: an end toward which an effort is directed
Goal = broad educational objective (i.e. overall purpose of curriculum)
Objective = specific measurable objective
Why this step important: helps direct choice of curricular content, suggests what learning methods most effective, enables evaluation of learners and curriculum, clearly communicates to others what curriculum hopes to achieve, and benchmark to evaluate against

32. Writing Objectives – 5 Elements
Who?
Will do?
How Much?
Of What?
By When?
Each resident
will demonstrate
at least once
of obtaining PAP smear
by end of course

33. Writing Learner Objectives
Attitude
Skill
Alphabetblocks.org
Knowledge

34. Competency-Based Curriculum
Objectives related to CanMEDS roles framework
Push in North America towards competency-based outcomes, where all objectives must be related to the core competencies which describe the expertise of modern clinical practice

35. Other Types of Objectives
Process objectives
Outcome objectives
Process objectives relate to implementation of the curriculum, and getting an outcome of that (different from ELT process) (e.g. by the academic year, 90% of PGY2 will have participated in the patient safety initiative)
Outcome objectives – referring to health, healthcare and patient outcomes (the impact of the curriculum beyond those delineated in its learner and process objective) – some of these are the metrics that you want to build into evaluation

36. Step 4: Educational Strategies
What
How
Curriculum content
Curriculum methods
Education strategies is about choosing curriculum content and methods
Content is the specific material to be included, methods are the way in which the content is presented
Choosing curriculum content and education methods that most likely achieve educational objectives
Why important: educational strategies are the heart of the curriculum, the educational intervention itself. Natural tendency to think of the curriculum in terms of this step alone, but groundwork is through steps 1-3

37. Guidelines to Choose Educational Methods
Understand the principles of adult learning
Feasibility
Congruence between objectives and educational methods
Guidelines to choose educational methods:
Understand adult learning theory as curriculum developers are to facilitate learning (goal-oriented, motivated, self-directed, bring experiences, experiential learning , etc.)
Curriculum content flows from learning objectives. Should maintain congruence between objectives and educational methods
Use multiple educational methods to maintain learner interest and reinforce learning
Choose educational methods that are feasible in terms of resources – what are some elements of feasibility that need to be considered?
consider time, space, availability of material, cost and availability of learner time

38. Congruence Between Objectives and Educational Methods
Attitudes
Skills
Knowledge
Exposure
Supervised clinical experience
Readings
Facilitation of reflection
Simulations
Lecture
Role models
Artificial models
Audiovisual materials
Discussion
Problem-based learning
Programmed learning
Pg 62 in book

39. Step 5: Implementation Converts a mental exercise to reality Resources
Political support
Administration and support staff
Barriers
Obtaining political support both internally and externally
Consider the details of administrating the curriculum: operations staff, communication, administration staff
Anticipate barriers – are they financial, people or other resources?

40. Piloting and Phasing In
What are some advantages/disadvantages to piloting a curriculum?
What are some advantages/disadvantages to phasing in a curriculum?
Piloting Advantages:
Introduce to friendly audience
Getting critical feedback to make revisions
Piloting Disadvantages:
Resources
Need to find friendly audience
Phasing Advantages:
Focusing of initial efforts as teaching staff learn new curriculum
Lessen resistance that new cultural shifts or curriculum brings
Critical feedback
Phasing Disadvantages:
Pressing need for full curriculum by learners

41. Step 6: Evaluation and Feedback
Closes the loop in the curriculum development cycle
Evaluation: identification, clarification and application of criteria to determine the merit or worth of what is being evaluated
Evaluation and feedback: is important because it closes the loop in the curriculum development cycle. Helps make a decision or judgment about the curriculum. Answers the critical question “were goals and objectives met”. Guides ongoing improvement. Can garner support for curriculum, to assess student achievement, satisfy external requirements, document accomplishments of curricular developers.
Keepcalm-o-matic.co.uk

42. Curriculum Evaluation - Definition
“The systematic documentation of the consequences of a curriculum and the determination of their worth in order to make decisions about them.”
Green (1977)

43. Purposes of Curriculum Evaluation
Determine the outcome
Decide the future of the program
Future development
Improve teaching methods
Cost-benefit analysis
Knowledge dissemination
Green (1977); Herman (1987); Kern (2009)

44. What to Evaluate Learners Instructors Curriculum material
Scope and sequence of content
Environment and facilities
The overall curriculum
Green (1977)

45. Types of Evaluations Final feedback
Summative evaluation
Formative evaluation
Final feedback
Evaluates overall effectiveness and impact of curriculum
Helps to decide about future of the program
Occasional or continuous feedback
Used to continually improve the curriculum during its implementation
Green (1977); Herman (1987)

46. How to Evaluate - The Evaluation Cycle
Rowel (1999)

47. Who Benefits? Decision-makers Employers Evaluators Scholars Students
The profession
Academic setting
Green (1977); Herman (1987)

48. Models of Evaluation Bradley's effectiveness model (1985)
Tyler's objectives centred model (1950)
Stufflebeam Context, Input, Process, Product (CIPP) 1971
Glatthorn (2015)

49. Key Principles Complex process Begins with clear goals and objectives
Meets needs of learners and communities they serve
Contains evaluation/feedback mechanism
Foundation to successful learning
Highlight a few approaches to CD and evaluation – all have things in common:
Complex: requires knowledge of the topic, learners, context (institutional, community)
Feedback loop makes it an iterative process