Presentation on theme: "Conflict of Interest Nancy Piro, PhD - No Conflicts of Interest to Declare Ann Dohn, MA - No Conflicts of Interest to Declare."— Presentation transcript:
1 Conflict of InterestNancy Piro, PhD - No Conflicts of Interest to DeclareAnn Dohn, MA - No Conflicts of Interest to Declare
2 PC002 BR04 The Basics of Effective Evaluations in the Era of NAS Nancy Piro, PhDProgram Manager & Education SpecialistAnn Dohn, MADIO, Director of GMEGraduate Medical Education
3 Workshop Objectives Understand how: Evaluation impacts NASNAS impacts evaluationsDesign and implement a valid & reliable resident evaluation process.Unintended cognitive biasThe impact of self-fulfilling prophesyEvaluation questions
4 What Are Milestones ? Defined areas of competency/expectations We know what we expect of traineeTrainee knows what is expected of themPublic knows what a ‘pathologist’, for example, should know and be able to do when certified as … “competent to practice pathology without supervision”
5 Milestones Levels (from Dreyfus) Novice PGY-0Advanced Beginner PGY-1Competent PGY-2 and 3Proficient Practitioner PGY-4/5Expert Practitioner - OngoingEach trainee assessed with respect to level for each competency
6 Milestones List of competencies to be attained by trainee Some milestones are still in developmentMilestones are within the ACGME existing 6 core competenciesEach milestone is defined as continuum with respect to each level of competency from level 1 to level 5Rating Scales: Levels 1 to 5
7 Where do these level 1-5 come from? Dreyfus Model in a Nutshell… “In acquiring a skill by means of instruction and experience, the trainee normally passes through five developmental stages:NoviceCompetenceProficiencyExpertiseMastery”1980-UC Berkeley-Mental ActivitiesBased on analysis of careful descriptions of skill acquisition, as the trainee becomes skilled, s/he depends less on abstract principles and more on concrete experience.Dreyfus systematized and illustrated the progressive changes in a trainee's ways of ‘seeing’ the task environment.Dreyfus concluded that any skill training procedure must be based on some model of skill acquisition, so that it can address, at each stage of training, the appropriate issues involved in facilitating advancement.”
8 Dreyfus Model (1980): Characteristics of each stage of developing expertise “In acquiring a skill by means of instruction and experience, the student normally passes through five developmental stages which we designate novice, competence, proficiency,expertise and mastery. We argue, based on analysis of careful descriptions of skill acquisition, that as the student becomes skilled, he depends less on abstract principles and more on concrete experience. We systematize and illustrate the progressive changes in a performer's ways of seeing his task environment. We conclude that any skill training procedure must be based on some model of skill acquisition, so that it can address, at each stage of training, the appropriate issues involved in facilitating advancement.”Source: Eraut, M. Developing Professional Knowledge and Competencies. (1994)
9 Milestone Level Definitions Level 1: The resident is a graduating medical student/experiencing the first day of residency.Level 2: The resident is advancing and demonstrating additional milestones.Level 3: The resident continues to advance and demonstrate additional milestones; the resident consistently demonstrates the majority of milestones targeted for residency.
10 Milestone Level Definitions (continued) Level 4: The resident has advanced so that he or she now substantially demonstrates the milestones targeted for residency. This level is designed as the graduation target – not requirement.Level 5: The resident has advanced beyond performance targets set for residency and is demonstrating “aspirational” goals which might describe the performance of someone who has been in practice for several years. It is expected that only a few exceptional residents will reach this level.
11 MilestonesMaking decisions about readiness for graduation is the purview of the residency program director.Study of milestone performance data will be required before the ACGME and its partners will be able to determine whether Level 4 milestones and milestones in lower levels are in the appropriate level within the developmental framework and whether milestone data are of sufficient quality to be used for high stakes decisions.
12 What is NAS Driving Us to Do? Change our EvaluationsTagging current evaluation questions to core competencies and milestonesUse direct milestone evaluationsOr use a combinationEnsure our evaluations are valid and reliable measures
13 Overall Milestone Evaluation Approach: Examples CurrentUse Current EvaluationsTag questions to milestonesEXAMPLE #2:DirectDirect milestone evaluationEXAMPLE #3:ComboMap questions to milestonesUse Current EvaluationsDirect milestone evaluation
14 Milestones Impact on Evaluations: Linking questions to milestones Step Two: Ensure specific evaluation questions are linked to milestonesAdvises the referring health care provider(s) about the appropriateness of a procedure in routine clinical situations
15 Example #2 DirectMany milestones may already be in the Residency Management Systems…
16 Example # 3- Combined Evaluation Questions Tagged To Milestones Milestones Evaluated Directly
17 Milestone Evaluations demand valid, reliable evaluations as component measurements Unintended bias can impact both direct and indirect milestone evaluations.Bias in questions & response scales (existing evaluations)Bias in evaluators (all evaluations)Halo EffectDevil EffectSimilarity EffectCentral TendencySelf-Fulfilling Prophesy
18 Step 1: Question and Response Scale Construction Two Major Goals:Construct unbiased, focused and non-leading questions that produce valid dataUse valid unbiased response scales
19 Step 1: Create the Evaluation-Eliminating Unintended Cognitive Bias What is cognitive bias?Cognitive response bias is a type of bias which can affect the results of an evaluation if evaluators answer questions in the way they think they are designed to be answered, or with a positive or negative bias toward the trainee.Cognitive bias is distortion in the way we perceive reality / information.
20 Step 1: Create the Evaluation - Where does response bias occur? Response bias can be in the evaluators themselves: Central Tendency, Similarity Effect, First Impressions, Halo Effect, Devil Effect, Self-Fulfilling ProphecyResponse bias occurs most often in the wording of the question.Response bias is present when a question contains a leading phrase or wordsResponse bias can also occur in the rating scales.Response biasa type of cognitive bias which can affect the results of an evaluation if respondents answer questions in the way they think they are designed to be answered, or with a positive or negative bias rather than according to their true perceptions of the person or program being evaluated.Response bias is present when a question contains a leading phrase.Response bias can, however, also occur in rating scales.More positive or negative choicesFive point scalesResponse bias in the raterCentral TendencyHalo EffectDevil Effect
21 Response Bias in the Raters/Evaluators - Beware the Halo Effect The “halo effect” refers to a type of cognitive bias where the perception of a particular behavior or trait is positively influenced by the perception of the former positive traits in a sequence of interpretations.Thorndike (1920)In a study published in 1920, Thorndike asked commanding officers to rate their soldiers; Thorndike found high cross-correlation between all positive and all negative traits.
22 The Halo Effect and Expectations The halo effect is discussed in Kelley's implicit personality theory…“the first traits we recognize in other people influence our interpretation and perception of later ones because of our expectations…”
24 Could this impact our evaluations ? A majority of trainees at Stanford believe that:“The general feeling in my program is that your ability will be labeled based on your initial performance.”If someone makes a mistake, it is often held against them.Agree Strongly Agree Moderately Agree Slightly - REDDisagree Slightly - YELLOWDisagree Moderately Disagree Strongly GREENGME House Staff Survey
25 “Reverse Halo Effect”A corollary to the halo effect is the “reverse halo effect” (devil effect)Individuals or brands which are seen to have a single undesirable trait are later judged to have many poor traits…- i.e., a single weak point (showing up late, for example) influences others' perception of the person or brand
26 Blind SpotsIn the 1970s, the social psychologist Richard Nisbett demonstrated that we typically have no awareness of when the halo effect influences us (Nisbett, R.E. and Wilson, T.D., 1977)The problem with Blind Spots is that we are blind to them…now on to self-fulfilling prophesy effect …
27 The Self-Fulfilling Prophecy (SFP) What is the Self-Fulfilling Prophecy?First defined by Robert Merton in 1948 to describe:‘A false definition of a situation evoking a new behavior which makes the originally false definition come true.’…a belief which has consequences that make reality conform to the initial belief.Has also been called the “Pygmalion Effect” …
28 The Self-Fulfilling Prophecy (SFP) What is the Self-Fulfilling Prophecy?Has been observed in many processes (Henshel, 1978)Within an individual, as with the placebo responseWhile the placebo effect occurs “within” the individual undergoing treatment, it is important to note that the individual’s belief derives from situational encounters with someone assumed to have expertise.…a belief which has consequences that make reality conform to the initial belief.Has also been called the “Pygmalion Effect” …Underlying financial institutions e.g., banks and markets
29 Rosenthal’s Research on SFP For more than 50 years, Robert Rosenthal has conducted research on the role of self-fulfilling prophecies in everyday life and in laboratory situations.Faculty expectations on trainees’ academic and clinical performanceThe effects of experimenters’ expectations on the results of their research (hence double-blinds)The effects of clinicians’ expectations on their patients’ mental and physical health.
30 The Pygmalion Effect: (Self-Fulfilling Prophesy) Sum it up: the greater the expectation placed upon people, the better they performRosenthal and Jacobson (1968) reported and discussed the Pygmalion effect in the classroom at length.In their study, they demonstrated that if teachers were led to expect enhanced performance from some children, then the children did indeed show that enhancement.
31 Pygmalion in Education The purpose of the experiment was to support the hypothesis that reality can be influenced by the expectations of others.This influence can be beneficial as well as detrimental depending on which label an individual is assigned.The observer-expectancy effect, which involves an experimenter's unconsciously biased expectations, is tested in real life situations.Rosenthal posited that biased expectancies can essentially affect reality and create self-fulfilling prophecies as a result.
32 Pygmalion Effect in Education All students in a single California elementary school were given a disguised IQ test at the beginning of the study. These scores were not disclosed to teachers.Teachers were told that some of their students (about 20% of the school chosen at random) could be expected to be "spurters" that year, doing better than expected in comparison to their classmates.The spurters' names were made known to the teachers. At the end of the study all students were again tested with the same IQ-test used at the beginning of the study.
33 Pygmalion EffectYou will be successfulALL six grades in both experimental and control groups showed a mean gain in IQ from pretest to post-test.First and second graders showed statistically significant gains favoring the experimental group of "spurters”.Conclusion: Teacher expectations, particularly for the youngest children, can influence student achievement.Paul Harvey
34 Why ???Anthropologists have postulated that accurate predictions in our human evolution came to be valued in and of themselves….
35 Other extensions… “Dani is my shy child…” “Mike never eats his vegetables…”“I can’t see them making much progress…”“Michelle isn’t very coordinated….”“You will feel comfortable and heal very well after this surgery”“I don’t expect Chris to make it through residency, do you?”
36 Quotes…James Rhem, executive editor for the online National Teaching and Learning Forum, commented:"When teachers expect students to do well and show intellectual growth, they do; when teachers do not have such expectations, performance and growth are not so encouraged and may in fact be discouraged in a variety of ways.""How we believe the world is and what we honestly think it can become have powerful effects on how things will turn out."
39 Applying these principles to assessments and evaluations…
40 Step 1: Create the Evaluation Question Construction - Exercise One Review each question and share your thinking of what makes it a good or bad question.
41 Question Construction: Exercise 1 Example 1:"I can always talk to my Program Director about residency related problems.”Example 2:“Sufficient career planning resources are available to me and my program director supports my professional aspirations .”Example 3:“Incomplete, inaccurate medical interviews, physical examinations; incomplete review and summary of other data sources. Fails to analyze data to make decisions; poor clinical judgment.”Example 4:"Communication in my sub-specialty program is good.“Example 5:"The pace on our service is chaotic."
42 Question Construction - Test Your Knowledge Example 1: "I can always talk to my Program Director about residency related problems."Problem: Terms such as "always" and "never" will bias the response in the opposite direction.Result: Data will be skewed.
43 Question Construction - Test Your Knowledge Example 2: “Career planning resources are available to me and my program director supports my professional aspirations."Problem: Double-barreled ---resources and aspirations… Respondents may agree with one and not the other. Researcher cannot make assumptions about which part of the question respondents were rating.Result: Data is useless.
44 Question Construction - Test Your Knowledge Example 3: "Communication in my sub-specialty program is good."Problem: Question is too broad. If score is less than 100% positive, researcher/evaluator still does not know what aspect of communication needs improvement.Result: Data is of little or no usefulness.
45 Question Construction - Test Your Knowledge Example 4: “Evidences incomplete, inaccurate medical interviews, physical examinations; incomplete review and summary of other data sources. Fails to analyze data to make decisions; poor clinical judgment.”Problem: Multi-barreled ---Respondents may need to agree with some and not the others. Evaluator cannot make assumptions about which part of the question respondents were rating.Result: Data is useless.
46 Question Construction - Test Your Knowledge Example (5):"The pace on our service is chaotic.“Problem: The question is negative, and broadcasts a bad message about the rotation/program.Result: Data will be skewed, and the climate may be negatively impacted.
47 Evaluation Question Design Principles Avoid ‘double-barreled’ or multi-barreled questions. Eliminate them from your evaluations.A multi-barreled question combines two or more issues or “attitudinal objects” in a single question.More examples….
48 Avoiding Double-Barreled Questions Example: COMPETENCY 1 – Patient Care“Resident provides sensitive support to patients with serious illness and to their families, and arranges for on-going support or preventive services if needed.”
49 Evaluation Question Design Principles Combining the two or more questions into one question makes it unclear which object attribute is being measured, as each question may elicit a different perception of the resident’s performance.RESULTRespondents are confused and results are confounded leading to unreliable or misleading resultsTip: If the word “and” or the word “or” appears in a question, check to verify whether it is a double-barreled question.
50 Evaluation Question Design Principles Avoid questions with double negatives…When respondents are asked for their agreement with a negatively phrased statement, double negatives can occur.Example: Do you agree or disagree with the following statement?Attendings should not be required to supervise their residents during night call.
51 Evaluation Question Design Principles If you respond that you disagree, you are saying you do not think attendings should not supervise residents. In other words, you believe that attendings should supervise residents.Phrase the questions positively if possible.If you do use a negative word like “not”, consider highlighting the word by underlining or bolding it to catch the respondent’s attention.
52 Evaluation Question Design Principles Because every question is measuring “something”, it is important for each to be clear and precise.Remember…Your goal is for each respondent to interpret the meaning of each survey question in exactly the same way.Pre-testing questions is definitely recommended!
53 Evaluation Question Design Principles If your respondents are not clear on what is being asked in a question, their responses may result in data that cannot or should not be applied to your evaluation results…Example: "For me, further development of my medical competence, it is important enough to take risks.”Does this mean to take risks with patient safety, risks to one's pride, or something else?
54 Evaluation Question Design Principles Keep questions short.Long questions can be confusing.Bottom line: Focus on short, concise, clearly written statements that get right to the point, producing actionable data.Short concise questions take only seconds to respond to and are easily interpreted.
55 Evaluation Question Design Principles Do not use “loaded” or “leading” questions.A loaded or leading question biases the response the respondent gives. A loaded question is one that contains loaded words.For example: “I’m concerned about doing a procedure if my performance would reveal that I had low ability”Disagree AgreeHow can this be answered with agree or disagree if you think you have good abilities in appropriate tasks for your work? It's like a question "I have stopped beating my wife" -- answer STRONGLY AGREE to STRONGLY DISAGREE
56 Evaluation Question Design Principles "I’m concerned about doing a procedure if my performance would reveal that I had low ability"How can this be answered with “agree or disagree” if you think you have good abilities in appropriate tasks for your area?How can this be answered with agree or disagree if you think you have good abilities in appropriate tasks for your work? It's like a question "I have stopped beating my wife" -- answer STRONGLY AGREE to STRONGLY DISAGREE
57 Evaluation Question Design Principles A leading question is phrased in such a way that suggests to the respondent that a certain answer is expected:Example: Don’t you agree that nurses should show more respect to residents and attendings?Yes, they should show more respectNo, they should not show more respectHow can this be answered with agree or disagree if you think you have good abilities in appropriate tasks for your work? It's like a question "I have stopped beating my wife" -- answer STRONGLY AGREE to STRONGLY DISAGREE
58 Evaluation Question Design Principles Do use Open-Ended QuestionsDo use comment boxes after negative ratingsTo explain the reasoning and target areas for focus and improvement.General, open-ended questions at the end of the evaluation can prove very beneficialOften it is found that entire topics have been omitted from the evaluation that should have been included.However, it is difficult to quantify the comments received.This qualitative data can be used to improve future evaluations.
59 Quick Exercise #2Please rate the anesthesia resident’s communication and technical skills.Rate the resident’s ability to communicate with patients and their families.Rate the resident’s abilities with respect to case familiarization; effort in reading about patient’s disease process and familiarizing with operative care and post op care
60 Quick Exercise #2Residents deserve higher pay for all the hours they put in, don’t they?Explains and performs steps required in resuscitation and stabilization.Do you agree or disagree that residents shouldn’t have to pay for their meals when on-call?
61 Quick Exercise #2Demonstrates an awareness of and responsiveness to the larger context of health care.Demonstrates ability to communicate with faculty and staff.
62 Quick Exercise #2 “Post Test” (1) 1. Please rate the anesthesia resident’s communication and technical skills- Please rate the anesthesia resident’s communication skills.- Please rate the anesthesia resident's technical skills.2. Rate the resident’s ability to communicate with patients and their families- Rate the resident’s ability to communicate with patients- Rate the resident’s ability to communicate with families
63 Quick Exercise #2 “Post Test” (2) 3. Rate the resident’s abilities with respect to case familiarization; effort in reading about patient’s disease process and familiarizing with operative care and post op careRate the resident’s ability with respect to case familiarizationRate the resident’s ability with respect to effort in reading about patient disease processRate the resident’s ability with respect to familiarizing with operative careRate the resident’s ability with familiarizing with post op care
64 Quick Exercise #2 “Post Test” (3) 4. Residents deserve higher pay for all the hours they put in, don’t they?To what extent do you agree that residents are adequately paid?5. Explains and performs steps required in resuscitation and stabilizationExplains steps required in resuscitationExplains steps required in stabilization following resuscitation
65 Quick Exercise #2 “Post Test” (4) 6. Do you agree or disagree that residents shouldn’t have to pay for their meals when on-call?To what extent do you agree that residents need to pay for their own on-call meals?7. Demonstrates an awareness of and responsiveness to the larger context of health care.Demonstrates an awareness of the larger context of health careDemonstrates responsiveness to larger context of health care8. Demonstrates ability to communicate with faculty and staff.Demonstrates ability to communicate with facultyDemonstrates ability to communicate with staff
66 Bias in the Rating Scales for Questions The scale you construct can also skew your data, much like we discussed about question construction but luckily, for us…
67 Milestone Scales use Balanced Scaled Responses…
68 Take a look at the evaluations you use. Review your Program’s evaluations with the goal of eliminating any unintentional bias from them.Take a look at the evaluations you use.Identify questions which may have unintended bias.Modify questions to eliminate bias
69 Gentle Words of Wisdom - Avoid large numbers of questions…. Respondent fatigue – the respondent tends to give similar ratings to all items without giving much thought to individual items, just wanting to finishIn situations where many items are considered important, a large number can receive very similar ratings at the top end of the scaleItems are not traded-off against each otherMany items that not at the extreme ends of the scale or that are considered similarly important are given a similar ratingRespondents quit answering questions at all…Total Started Survey: 578Total Completed Survey: 473 (81.8%)
70 Gentle Words of Wisdom - Begin with the End Goal in Mind What do you want as your outcomes?Be prepared to put in the time with pretesting.The faculty member, nurse, patient, resident has to be able to understand the intent of the question - and each must find it credible and interpret it in the same way.
71 Gentle Words of Wisdom: Relevancy/Accuracy – Collect data in a valid and reliable way If the questions aren’t framed properly, if they are too vague or too specific, it’s impossible to get any meaningful data.Question miswording can lead to skewed data with little or no usefulness.If you don't plan or know how you are going to use the data, don't ask the question!
72 Summary: Evaluation Do’s and Don’ts Keep Questions Clear, Precise and Relatively Short.Use a balanced response scaleMilestone scales are balancedUse open-ended questionsDON’TsDo not use Double+ Barreled QuestionsDo not use Double Negative QuestionsDo not use Loaded or Leading Questions.
73 QuestionsPlease feel free to contact us with anystanford.edu orStanford University Medical CenterGraduate Medical Education