Translating calls for competencies-oriented, evidence-based education into achievable and meaningful curricular reform initiatives Frank J Papa, DO, PhD Assistant Dean Curricular Design and Educational Technologies UNTHSC

Introduction A Time of Change A Time of Change Competence Competence What are our current beliefs and practices regarding competence? What are our current beliefs and practices regarding competence? What does the literature say about competence? What does the literature say about competence? Instruction to Competence Instruction to Competence Assessment of Competence Assessment of Competence Building a competencies-oriented curriculum Building a competencies-oriented curriculum

A Time of Change Patient Care arena Patient Care arena Adoption of Patient Care Competencies: Adoption of Patient Care Competencies: Outcomes Outcomes Expectations Expectations Compliance Compliance Norms Norms Error rates Error rates EBM EBM

A Time of Change MEDED arena MEDED arena Adoption of Educational Competencies: Adoption of Educational Competencies: Medical Knowledge Medical Knowledge OMM OMM Patient Care Patient Care Professionalism Professionalism Interpersonal and Communication Skills Interpersonal and Communication Skills Practice based Learning and Improvement Practice based Learning and Improvement Systems based Practice Systems based Practice BEME BEME

A Time of Change A Time of Convergence A Time of Convergence Merging patient care and educational competencies Merging patient care and educational competencies EBM literature guides patient care competencies EBM literature guides patient care competencies Will tradition guide the development of educational competencies or will evidence? Will tradition guide the development of educational competencies or will evidence? What is known about educational competencies? What is known about educational competencies? Competence has an evidence base in selected areas Competence has an evidence base in selected areas When will BEME really happen? When will BEME really happen?

A Time of Change A Timetable for Change A Timetable for Change Major curricular reform initiatives: Major curricular reform initiatives: 1950: Systems-based 1950: Systems-based 1970: Problem-based 1970: Problem-based 1990: Presentation-based 1990: Presentation-based 2010: ? 2010: ? Competencies-Oriented, Competencies-Oriented, Evidence-Based Evidence-Based 2010: ___COEB 2010: ___COEB How will ___COEB curricula be defined/implemented? How will ___COEB curricula be defined/implemented?

A Time of Change A Time for Leadership and Commitment A Time for Leadership and Commitment Who will define COEB curricular reform? Who will define COEB curricular reform? Will Osteopathic Medicine lead or follow? Will Osteopathic Medicine lead or follow? Reality check Reality check Lead? (e.g., competencies initiative, standardized patient/skills testing) Lead? (e.g., competencies initiative, standardized patient/skills testing) Follow? (e.g., competencies initiative, standardized patient/skills testing) Follow? (e.g., competencies initiative, standardized patient/skills testing) Internal dialogue of the uncommitted (silent majority/politically correct/astute) Internal dialogue of the uncommitted (silent majority/politically correct/astute) Who are we kidding? Who are we kidding? Were too small, have too few resources, etc, etc, etc Were too small, have too few resources, etc, etc, etc Anything you can do (first) I can do better (later) (me-too syndrome) Anything you can do (first) I can do better (later) (me-too syndrome) Isnt my primary job to keep my job? Isnt my primary job to keep my job? Was AT Still the first and last Osteopathic leader? Was AT Still the first and last Osteopathic leader? Is the Osteopathic educational community willing to commit to the task of leading COEB reform? Is the Osteopathic educational community willing to commit to the task of leading COEB reform?

A Time of Change The Time for Leadership and Commitment is now The Time for Leadership and Commitment is now Someone will define and implement COEB reform! Someone will define and implement COEB reform! Timetable will be around 2010! Timetable will be around 2010! Will Osteopathic Medicine lead or follow? Will Osteopathic Medicine lead or follow? Who wants to commit themselves to the task of leading? Who wants to commit themselves to the task of leading? Osteopathic educators can define and implement the first COEB reform Osteopathic educators can define and implement the first COEB reform

Competence COMPETENCE COMPETENCE Beliefs and Practices Beliefs and Practices Evidence Evidence

Competence Beliefs and Practices Beliefs and Practices Competence happens somewhere along a continuum Competence happens somewhere along a continuum Naïve - Novice - Intermediate – Expert - Master Naïve - Novice - Intermediate – Expert - Master Competence is Intellectual Skills-based (heavily predicated upon development of problem solving skills (PSS)) Competence is Intellectual Skills-based (heavily predicated upon development of problem solving skills (PSS)) Time (length of training) determines competence Time (length of training) determines competence Competence is norm referenced (adequately assessed via traditional exams) Competence is norm referenced (adequately assessed via traditional exams)

Competence Evidence Evidence Competence: The Donts Competence: The Donts Dont have to happen Dont have to happen Dont have much to do with the development of PSS Dont have much to do with the development of PSS Dont depend upon length of training time Dont depend upon length of training time Dont have to norm reference it Dont have to norm reference it

Competence Evidence: Evidence: Development of Intellectual Skills (IS) such as PSS in the absence of problem and task-specific knowledge will not lead to competence (IS are merely competency antecedents) Development of Intellectual Skills (IS) such as PSS in the absence of problem and task-specific knowledge will not lead to competence (IS are merely competency antecedents) Competence is predominately knowledge based Competence is predominately knowledge based Knowledge based competencies are problem/task-specific Knowledge based competencies are problem/task-specific Develops optimally with deliberate, focused problem/task- specific training and feedback Develops optimally with deliberate, focused problem/task- specific training and feedback To be properly assessed, competencies must be measured one problem/task at a time To be properly assessed, competencies must be measured one problem/task at a time

Competence Impediments to COEB Impediments to COEB MEDED is OK the way it is! MEDED is OK the way it is! Weve been doing it this way for decades (comfort in traditions) Weve been doing it this way for decades (comfort in traditions) Everyone else does it this way (comfort in numbers) Everyone else does it this way (comfort in numbers) Same arguments against the adoption of EBM in practice Same arguments against the adoption of EBM in practice Weve been doing patient care this way for decades Weve been doing patient care this way for decades Everyone else practices this way Everyone else practices this way COEB Accelerators COEB Accelerators Understanding literature Understanding literature Technology/simulations Technology/simulations

Competence Competence fundamentals Competence fundamentals Intellectual Skill antecedents Intellectual Skill antecedents Knowledge Base antecedents Knowledge Base antecedents

Competence Blooms Taxonomy Blooms Taxonomy A Hierarchy of Intellectual Skills and their development A Hierarchy of Intellectual Skills and their development (competence antecedents) (competence antecedents) Information acquisition Information acquisition Comprehension Comprehension Application Application Analysis Analysis Synthesis Synthesis Generation Generation

Competence Competence and Blooms Taxonomy Competence and Blooms Taxonomy Information acquisition (Naïve student) Information acquisition (Naïve student) Comprehension (Novice student) Comprehension (Novice student) Application Application Analysis Analysis Synthesis Synthesis Three (Intermediate) Competence antecedents Three (Intermediate) Competence antecedents Generation (Expert – Master antecedent) Generation (Expert – Master antecedent)

Competence NaiveNoviceIntermediate(Competence)ExpertMaster Info Acquisition XXXX ComprehensionXXX ApplicationXX AnalysisXX SynthesisXX GenerationX

Competence Using Blooms taxonomy to evaluate curricular reforms Using Blooms taxonomy to evaluate curricular reforms What percentage of curricular time is devoted to supporting the development of IS-related competence antecedents? What percentage of curricular time is devoted to supporting the development of IS-related competence antecedents?

Competence Discipline/ Systems PBLPresenCOEB Info Acquisition Comprehension Application Analysis Synthesis Generation

Competence Discipline/ Systems PBLPresenCOEB Info Acquisition 85 – 90% Comprehension5-10% Application5% Analysis0% Synthesis0% Generation0%

Competence Discipline/ Systems PBLPresenCOEB Info Acquisition 85 – 90% 80-85% Comprehension5-10%10-15% Application5%10% Analysis0%0% Synthesis0%0% Generation0%0%

Competence Discipline/ Systems PBLPresenCOEB Info Acquisition 85 – 90% 80-85%65-75% Comprehension5-10%10-15%15-20% Application5%10%15% Analysis0%0%2-5% Synthesis0%0%0% Generation0%0%0%

Competence Discipline/ Systems PBLPresenCOEB Info Acquisition 85 – 90% 80-85%65-75%40-50% Comprehension5-10%10-15%15-20%20-30% Application5%10%15%20-25% Analysis0%0%2-5%10-15% Synthesis0%0%0%5-10% Generation0%0%0%2-5%

Competence Knowledge Base antecedents to the development of competence Knowledge Base antecedents to the development of competence Competence is heavily predicated upon how the knowledge base is organized or structured Competence is heavily predicated upon how the knowledge base is organized or structured Knowledge-Based competencies appears to be organized around: Knowledge-Based competencies appears to be organized around: Specific Problems Specific Problems Specific tasks associated with the solving of each problem Specific tasks associated with the solving of each problem

Competence Hundreds of medical problems Hundreds of medical problems Chest pain, dyspnea, melena, vertigo, etc Chest pain, dyspnea, melena, vertigo, etc Three core medical tasks associated with every problem Three core medical tasks associated with every problem Differential Diagnosis Differential Diagnosis Explanation Explanation Treatment Treatment

Competence Competence-related IS (application, analysis & synthesis) are optimally developed in the context of problem and task - specific learning opportunities Competence-related IS (application, analysis & synthesis) are optimally developed in the context of problem and task - specific learning opportunities That is, CASE solving activities are the perquisite for applying, analyzing and synthesizing their evolving problem and task-specific knowledge bases That is, CASE solving activities are the perquisite for applying, analyzing and synthesizing their evolving problem and task-specific knowledge bases CASES are the catalyst, the grist for the intellectual mill, the context within which IS and problem/task-specific knowledge bases are organized/structure CASES are the catalyst, the grist for the intellectual mill, the context within which IS and problem/task-specific knowledge bases are organized/structure

Competence No cases, no IS, no PTSKB No cases, no IS, no PTSKB The more cases, the greater the IS, the better organized the PTSKB The more cases, the greater the IS, the better organized the PTSKB Cases give birth to competence Cases give birth to competence

Competence Antecedents to the development of competence Antecedents to the development of competence To support the development of both the IS and Knowledge-Based antecedents to competence, you need to have: To support the development of both the IS and Knowledge-Based antecedents to competence, you need to have: Problem and task-specific cases sufficient to support the development of competence-related IS antecedents Problem and task-specific cases sufficient to support the development of competence-related IS antecedents The curricular content (the learners future knowledge base) organized around problems and tasks The curricular content (the learners future knowledge base) organized around problems and tasks What percentage of curricular time is devoted to supporting the development of problem and task- specific competence antecedents? What percentage of curricular time is devoted to supporting the development of problem and task- specific competence antecedents?

Competence Application, Analysis and Synthesis Disc/SystemsPBLPresCOEB A case in isolation A case in the context of a problem Multiple cases in the context of a problem Multiple cases in the context of multiple problems and tasks

Competence Application, Analysis and Synthesis Disc/SystemsPBLPresCOEB A case in isolation A case in the context of a problem Multiple cases in the context of a problem Multiple cases in the context of multiple problems and tasks 5%0%0%0%

Competence Application, Analysis and Synthesis Disc/SystemsPBLPresCOEB A case in isolation A case in the context of a problem Multiple cases in the context of a problem Multiple cases in the context of multiple problems and tasks 5%0%0%0%5%5%0%0%

Competence Application, Analysis and Synthesis Disc/SystemsPBLPresCOEB A case in isolation A case in the context of a problem Multiple cases in the context of a problem Multiple cases in the context of multiple problems and tasks 5%0%0%0%5%5%0%0%5%5%5%5%

Competence Application, Analysis and Synthesis Disc/SystemsPBLPresCOEB A case in isolation A case in the context of a problem Multiple cases in the context of a problem Multiple cases in the context of multiple problems and tasks 5%0%0%0%5%5%0%0%5%5%5%5%0%0%15-25%20-30%

Competence Constructing a COEB curriculum Constructing a COEB curriculum Significant reduction in time dedicated to information acquisition Significant reduction in time dedicated to information acquisition Significant increase in time dedicated to IS development Significant increase in time dedicated to IS development Application, Analysis, Synthesis opportunities provided via the context of multiple cases representing multiple problems and tasks Application, Analysis, Synthesis opportunities provided via the context of multiple cases representing multiple problems and tasks Course content organized around problems and tasks Course content organized around problems and tasks Problems: 100 – 150 Primary care-oriented (chest pain, dyspnea, melena, etc) Problems: 100 – 150 Primary care-oriented (chest pain, dyspnea, melena, etc) Tasks: DDX, explanation, treatment Tasks: DDX, explanation, treatment Problem/Task-specific learning modules Problem/Task-specific learning modules

Competence Result: Result: Problem/Task-specific, competencies-oriented, evidence-based curriculum Problem/Task-specific, competencies-oriented, evidence-based curriculum PTSCOEB curriculum PTSCOEB curriculum

Instruction to Competence Three core Knowledge-Based Competencies Three core Knowledge-Based Competencies DDX DDX Explanation Explanation Treatment Treatment DDX DDX The most difficult and resource intensive competence to train and learn The most difficult and resource intensive competence to train and learn

Instruction to Competence DDX DDX The most difficult and resource intensive competence The most difficult and resource intensive competence Deliberate practice involving multiple problem and disease- specific case exposures as basis for refinement of requisite IS (application, analysis and synthesis) Deliberate practice involving multiple problem and disease- specific case exposures as basis for refinement of requisite IS (application, analysis and synthesis) DDX competence requires development of additional IS (pattern matching and pattern discrimination) DDX competence requires development of additional IS (pattern matching and pattern discrimination) Training cases selected at appropriate level of difficulty Training cases selected at appropriate level of difficulty Case-specific feedback oriented to refining Problem and Disease-specific Knowledge Base and IS Case-specific feedback oriented to refining Problem and Disease-specific Knowledge Base and IS Opportunities for correction of errors Opportunities for correction of errors

Instruction to Competence DDX DDX Technological Accelerators Technological Accelerators Computer-based simulations Computer-based simulations High fidelity simulations High fidelity simulations High cost/resource intensive/limited training opportunities High cost/resource intensive/limited training opportunities Demonstrated psychomotor benefits Demonstrated psychomotor benefits Case availability and appropriateness very limited Case availability and appropriateness very limited Significant feedback limitations Significant feedback limitations Faculty have significant limitations in defining learning objectives Faculty have significant limitations in defining learning objectives Problem-specific, Knowledge-Based benefits not documented Problem-specific, Knowledge-Based benefits not documented

Instruction to Competence DDX DDX Technological Accelerators Technological Accelerators Computer-based simulations Computer-based simulations Low fidelity simulations Low fidelity simulations Low cost/less resources/expanded training opportunities Low cost/less resources/expanded training opportunities Demonstrated problem-specific, knowledge-based benefits Demonstrated problem-specific, knowledge-based benefits Case availability and appropriateness virtually limitless Case availability and appropriateness virtually limitless Feedback virtually unlimited and tailored to specific student errors Feedback virtually unlimited and tailored to specific student errors Faculty have few limitations in defining DDX learning objectives Faculty have few limitations in defining DDX learning objectives Psychomotor benefits - none Psychomotor benefits - none

Instruction to Competence DDX DDX Technological Accelerators Technological Accelerators KBIT Computer-based, low fidelity DDX simulations provide: KBIT Computer-based, low fidelity DDX simulations provide: Faculty the opportunity to define problem and task (DDX)- specific training objectives Faculty the opportunity to define problem and task (DDX)- specific training objectives Multiple cases readily available Multiple cases readily available Objective and subjective means of selecting appropriate cases Objective and subjective means of selecting appropriate cases IS and Knowledge-Based feedback inherent to KBIT training modules IS and Knowledge-Based feedback inherent to KBIT training modules Student have opportunities to correct and immediately learn from errors Student have opportunities to correct and immediately learn from errors

Instruction to Competence KBIT KBIT Evidence Evidence KBIT training: KBIT training: Improves student DDX performance up to 25% greater than faculty trained students Improves student DDX performance up to 25% greater than faculty trained students Effect size-based performance improvements of one standard deviation Effect size-based performance improvements of one standard deviation Appears to correlate with, and perhaps support, performance improvements in non-DDX tasks (explanation and treatment) (Dr Dubin) Appears to correlate with, and perhaps support, performance improvements in non-DDX tasks (explanation and treatment) (Dr Dubin) Augments traditional coursework leading to increases in student satisfaction indices Augments traditional coursework leading to increases in student satisfaction indices Perceived as significantly contributing to students evolving sense of DDX competence Perceived as significantly contributing to students evolving sense of DDX competence Provides instructional approaches and benefits not possible via traditional classroom instruction Provides instructional approaches and benefits not possible via traditional classroom instruction

Instruction to Competence KBIT DDX training simulator KBIT DDX training simulator Resource allocations Resource allocations Minimal time investment of faculty (5-6 hours per module) Minimal time investment of faculty (5-6 hours per module) Heavily leverages faculty effort /student training ratio (ten to hundreds of times) Heavily leverages faculty effort /student training ratio (ten to hundreds of times) If one faculty member at one school develops a module (6 hours development time), all students across all osteopathic schools can have access to the module (thousands of hours of individualized instruction occur) If one faculty member at one school develops a module (6 hours development time), all students across all osteopathic schools can have access to the module (thousands of hours of individualized instruction occur) Provides opportunity for participating faculty and schools to engage in leading edge educational research Provides opportunity for participating faculty and schools to engage in leading edge educational research Accelerates osteopathic educators opportunity to create and implement COEB curricula Accelerates osteopathic educators opportunity to create and implement COEB curricula

Assessment of Competence KBIT KBIT Evidence Evidence Provides objective measurement of problem and disease- specific performance to competence (Dr Oglesby) Provides objective measurement of problem and disease- specific performance to competence (Dr Oglesby)

Building a PTSCOEB curricula One problem at a time One problem at a time Small reduction of information acquisition in a given discipline or system course (approximately 2-3 days) Small reduction of information acquisition in a given discipline or system course (approximately 2-3 days) Each school inserts one Problem-Specific module Each school inserts one Problem-Specific module Training to: Training to: DDX via KBIT DDX via KBIT Explanation Explanation Treatment Treatment Defining and assessing competence performance levels in terms of: Defining and assessing competence performance levels in terms of: DDX via KBIT DDX via KBIT Explanation Explanation Treatment Treatment Sharing module, methods and results across all schools Sharing module, methods and results across all schools

A Time of Change A Timetable for Change A Timetable for Change Major curricular reform initiatives: Major curricular reform initiatives: 1950: Systems-based 1950: Systems-based 1970: Problem-based 1970: Problem-based 1990: Presentation-based 1990: Presentation-based 2010: PTSCOEB 2010: PTSCOEB

A Time of Change A Time for Leadership and Commitment A Time for Leadership and Commitment Who will define COEB curricular reform? Who will define COEB curricular reform? Will Osteopathic Medicine lead or follow? Will Osteopathic Medicine lead or follow? Uncommitted / Silent Majority Uncommitted / Silent Majority Who are we kidding? Who are we kidding? Were too small – have too few resources Were too small – have too few resources D.O.s always follow! D.O.s always follow! Isnt my primary job to keep my job? Isnt my primary job to keep my job?

A Time of Change A Time for Leadership and Commitment A Time for Leadership and Commitment Is the Osteopathic educational community willing to commit to the task of leading COEB reform? Is the Osteopathic educational community willing to commit to the task of leading COEB reform?

A Time of Change COEB initiative COEB initiative One problem module to be developed per school One problem module to be developed per school Twenty modules available to all schools after one year Twenty modules available to all schools after one year 100 modules available to all schools by modules available to all schools by 2010 SOME publishes results of COEB initiative SOME publishes results of COEB initiative

A Time of Change Come to workshop this Friday evening Come to workshop this Friday evening Ask questions Ask questions Dream dreams Dream dreams Make commitment to create tomorrow's reality Make commitment to create tomorrow's reality Academic Medicine Academic Medicine Proceedings of the RIME Conference: November, 2010 Proceedings of the RIME Conference: November, 2010 Creation and Implementation of Competencies-Oriented, Evidence-Based Curricula. Authors – SOME Creation and Implementation of Competencies-Oriented, Evidence-Based Curricula. Authors – SOME New York Times & Wall Street Journal articles state: Osteopathic educators step forward and create first 21 st century curricula; allopathic educators acknowledge preliminary data looks promising New York Times & Wall Street Journal articles state: Osteopathic educators step forward and create first 21 st century curricula; allopathic educators acknowledge preliminary data looks promising