Presentation on theme: "Critical Thinking in Medical Education: Assessing What We Mean and What We Know Ed Krupat, PhD Director, Center for Evaluation Medical Education Grand."— Presentation transcript:
Critical Thinking in Medical Education: Assessing What We Mean and What We Know Ed Krupat, PhD Director, Center for Evaluation Medical Education Grand Rounds Dec. 5, 2008
Components of Talks that I Have Liked 1. A bit of context or framing 2. A touch of theoretical/conceptual background 3. A good portion of research and data 4. A bounty of concern for practical application 5. A strong dose of provocative discussion
Context: The Origins of These Efforts Derek Bok--to--Richard Hersh –Evaluation of programs Confusion-to-clarity-to-confusion-to-??? Why is critical thinking so important –In life –In medicine Desire to be data-driven
Conceptual Analysis: Just What Is Critical Thinking?? Overlapping concepts –Analytic reasoning –Problem solving –Decision making –Clinical/diagnostic reasoning/judgement –Habits of mind –Meta-cognition –Adaptive expertise
Some Definitions: Critical thinking is…...the intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication as a guide to belief and action. (Scriven, 1996) …the art of thinking about your thinking while you are thinking in order to make your thinking better: more clear, more accurate, and more defensible. (Paul et al, 1989)
Research & Data Two projects being conducted simultaneously 1.Qualitative: Do physicians agree about just what critical thinking is? 2.Quantitative: Can we assess critical thinking among medical students?
What is Critical Thinking: The Responses of Physicians Survey of practicing MD faculty at 5 medical schools –HMS, UCSF, Case Western, U Mass, Baylor (total n=73) –Convenience sample
Two Tasks 1.Define critical thinking 2.Think of a clinical scenario in which critical thinking was important describe it state what a good critical thinker would do or say in that situation state what a poor critical thinker would do or say state how the outcome would differ if one or the other would have been the physician involved
Coding of the Definitions. 1 Definition specifically included: –Collection of information/data: 27% –Making sense of information: 96% –Utilization for decision making: 74% –Utilization for action: 14% –Specific ties to medical context: 43% –Necessity of building upon knowledge base: 27%
Coding of the Definitions. 2 Critical thinking characterized as –A process –A skill or ability –A disposition
Critical Thinking as a Process …a process of reflective reasoning that uses objective evidence, a deliberate weighing of options and alternatives, and clinical judgement and experience to guide decision making. …the process by which one is able to rationally acknowledge different choices, processes and outcomes in the clinical encounter. a process in which problems are being analyzed from different angles and connected to pre-existing knowledge before any conclusions are being drawn 44% of all responses
Critical Thinking as a Skill or Ability …the ability to think through a problem using reasoning. Also the ability to judge the credibility of sources. …the ability to rigorously weigh the validity of evidence and then to effectively synthesize this evidence to reach a clinical decision. …the ability to effectively problem solve using known data or under conditions of uncertainty. 50% of all responses
Critical Thinking as a Disposition …careful attention to what you know, vigilance for what you do not, and the courage to question both of the above categories. …thinking about an topic, issue, or challenge in a way that sets aside my immediate ’gut’ response,so that I can be open and reflective to other possible ways of viewing the challenge… …thinking deeply, keenly, flexibly, openly, reflectively, with an awareness of self and others, with attention to what is known and unknown, and with humility. 10% of all responses
Breakdown by Specialty
Clinical Situations: An Interim Report Looking for –What situations, choices, challenges differentiate critical and non-critical thinkers –What is it that critical thinkers do or say –What is it that non-critical thinkers do or say –What outcomes differ when critical thinking is or is not in evidence Bottom line: –What are the key differences in the thoughts, words, and actions of critical thinkers
Clinical Contexts and Tasks Not surprising –Diagnosis and treatment Surprising –Collaboration and interpersonal issues How to protect patient’s rights and autonomy How to deal with difficult patient requests How to provide patient with appropriate options
The Differences Critical thinkers: –Do more data gathering –Avoid premature conclusions –See inconsistencies in information –Utilize knowledge more extensively and explicitly to make decisions –Are aware of limitations and doubts –Monitor and evaluate their own decisions –Involve patients more fully –Provide patients with options
A Few Conclusions, Many Questions Physicians are not necessarily all speaking the same language when they discuss critical thinking with one another Is critical thinking a skill, a process, or a way of looking at the world? –Are skill and disposition both necessary? –Do you teach/encourage/foster skills in the same way as dispositions??? –Do you assess skills in the same way that you assess dispositions? Where does building upon a knowledge base come in?
How do we assess critical thinking? Collegiate Learning Assessment (CLA) –Used in over 200 colleges and universities –Assesses higher order outcomes –Students work on 90 minute performance tasks –System has been devised to generate reliable scores Question: Is this a valid measure of critical thinking for medical students?
Catfish: One of the Performance Tasks Used Grotesquely mutated catfish has been found in the local lake that supplies the town’s water You will serve on the mayor’s advisory panel You are provided with 6 documents to read –Newspaper article –Editorial by environmental activist –Radio interview with a biologist from a nearby college –State report on water testing from lake –Area map –Journal article about similar discoveries
Task Open-ended written questions ask students to: –Identify main hypotheses to explain phenomenon –Identify strengths and weakness of each –State and defend most likely explanation –Suggest course of action
Scoring Evaluation of evidence –What is relevant, what is valuable Analysis and synthesis of evidence – Connections, inconsistencies, flaws in reasoning Conclusion drawing –Acknowledging alternatives and options Presentation of arguments –Concise, evidence-based, logically structured
Research Design Recruit multiple schools Recruit and compare students at two (or more) points in time –Entering Year 1 students took two tasks –Students at end of Year 3 took one task Look for associations between CLA scores and other performance indicators –MCAT scores –Gender –Age –English as first language –Step scores Compare scores between first and third year students
Analysis Problems Among year 1 students –scores on the two tasks not as highly correlated as would be expected –Brain Boost task always given first Year 1 students spend consistently less time and do consistently worse on Catfish (always second task) than on Brain Boost Among Year 3 students--who only took one task, either Brain Boost or Catfish--students take more time and perform better than on Brain Boost Small n’s
Findings 1. Medical school students have good critical thinking skills at baseline –Across the 3 schools, year 1’s fell at the 87th percentile among graduating seniors nationally were 1.25 standard deviations above the national mean Is that good news or, with selectivity of the medical schools, should we expect to be even higher??
Table 1. CLA performance by task, year, and school ** p<.01 First YearThird Year TaskSchoolMeanSDnMeanSDn Mean Difference t AllHMS AllS AllS ** BBHMS BBS BBS **
Table 2. Correlations among tests and demographic variables CLABBMCAT Total MCAT Phys MCAT Bio MCAT Verbal MCAT Writing AgeGenderEnglishStep I CLA--.77***.22***.17**.15*.20**.25*** Brian Boost--.18**.16*.13.23***-.17** MCAT Total--.82***.70***.38*** **.03.61*** MCAT Phys--.57***.31***.32*** *** *** MCAT Bio--.37***.25*** ** *** MCAT Verbal--.31*** **.42*** MCAT Writing *** Age ***.01 Gender English Step I-- Significance codes: * p<.05, ** p<.01, *** p<.001 Note: Significance not adjusted for multiple significance tests
Findings to Come What is the correlation between CLA scores and –Step II CK –Step II CS –HMS Comprehensive Exam
What have we learned? Critical thinking has many dimensions, and physicians may be talking past one another if they don’t share meaning when they speak. Critical thinking can manifest itself in how to deal with patients, not just in generating diagnoses and treatment plans. Critical thinking involves being –Sensitive –Inclined –Able (Perkins & Ritchhart 2002)
What have we learned? 2 Medical students enter school with solid critical thinking skills Critical thinking is somewhat, but not greatly, associated with other measured outcomes It is still unclear the extent to which medical education fosters or hinders critical thinking We have more questions at the end of this process than we did than when it began, but the questions are –more sophisticated –more tied to possible action
Some Big Questions If we think that critical thinking is so important, why don’t we explicitly design the curriculum to assure that we “teach” it? If we wanted to do so, how and when would it be taught? Should measures of critical thinking, as a skill or disposition, be built into our admissions screening process? Should critical thinking be a core competency that is assessed throughout the curriculum? as a graduation requirement? As a requirement for licensure?