1. Graduate Medical Education Stanford University Medical Center
Improving Academic Program Quality by Standardizing and Streamlining the Evaluation Process for Programs and Faculty
Graduate Medical Education
Stanford University Medical Center
Ann M. Dohn, MA
Designated Institutional Official
Nancy Piro, PhD
Education Specialist/ Program Manager
Ann

2. Session Objectives
The Situation and Problems/Issues: Why Did we Do This?
What Did We Do?
How Did We Do It?
Results
Ann

3. Why Did We Do This? The Situation:
The largest number of citations for our programs were in the area of evaluations
Program Directors’ Needs Assessment Results
One of the Top Five Requests for the Department of GME = Templates for Evaluation
GME Review of Evaluations Results
Ann

4. What Did We Find? Poor quality questions Inappropriate response scales
Questions not core competency based
Evaluations not even being conducted
Program Director frustration with the whole process
Ann

5. Setting the Stage PROGRAM EVALUATION 82 Programs 14 Citations
What Can We Do???
Ann

6. What Did We Do? – STEP 1 Step 1. Standardized Questions
Collaboratively developed Prototype Questions for Programs to Use for Evaluation of Faculty and Residents
Core Competency Based
Unbiased
Educated Faculty and Program Directors on Removing Unintended Cognitive Bias from their Evaluations and Evaluating
Ann

7. Tools Were Not Sufficient
Used group consensus to finalize the questions
Distributed the questions to the Program Directors
Educated them on removing bias
but…. the frustration still remained
Ann

8. GME now administers program evaluations for all 82 programs
What Could We Do? – STEP 2
Step 2. Centralized the Annual Program Evaluation Process.
GME now administers program evaluations for all 82 programs
Used the Standardized Program Evaluation for :
Faculty
Residents/Fellows
Ann

9. How Is This Implemented?
GME electronically delivers the evaluations to all their faculty and residents/fellows…
Programs can choose one of two dates –
Feb or May
Nancy

10. Standardized Program Evaluation by Faculty Form
Nancy

11. Standardized Program Evaluation by Residents/Fellows Form
Nancy

12. Generate reports for each program: 1) Program Eval by Faculty
Nancy

13. Generate reports for each program: 2) Program Evaluation by the Residents/Fellows
Nancy

14. But still not enough
Program Directors had the data but didn’t use it…...
SIGH…..
Ann

15. The Total GME Annual Program Review Package
Aggregate Data Delivered “to you”
1. Evaluation results posted for all Program Directors on their Residency Management System website
Program Eval by Faculty
Program Eval by Residents/Fellows
Internal GME House Staff Annual Survey
ACGME Survey (if available)
Ann

16. The Total GME Annual Program Review Package
2. Review Checklist – All the essentials that need to be covered
Nancy

17. GME Website Support
Nancy

18. The Total GME Annual Program Review Package
3. Template Agenda for the Annual Review Meeting
Nancy

19. The Total GME Annual Program Review Package
Sign-In Sheet /Documentation for web
Nancy

20. Standardized Program Evaluation
Minutes (Meeting Documentation) Template
Nancy

21. Standardized Program Evaluation
As soon as completed, we post the aggregated summary reports for both Faculty and Residents to our online Residency Management System for each Program:
Nancy

22. Standardized Program Evaluation
Action Plan Documentation Template /Example of an action plan from one of our Programs….
Nancy

23. Each program is reviewed by GME
Ensures all the requirements of the APR are completed and documented: …
Nancy

24. Outcomes
100% compliance in 2010!!
Ann

25. Faculty Evaluations
At the Dean’s request, GME developed standardized questions for residents and fellows to evaluate their faculty

26. Annual Academic Evaluation of Faculty

27. Roadblock Could not get PD consensus
Some faculty did not want the results going directly to the Dean’s office without Chair ‘review’…
Ann

28. Where are we? Some programs are using it
The Dean to present to Program Directors
Ann

29. Summative Evaluation What is summative evaluation?
A summative evaluation is a method of judging the competence of a trainee at the end of the program (summation). The focus is on the outcome.
“Assessment with the primary purpose of establishing whether or not performance measured at a single defined point in time meets established performance standards, permanently recorded in the form of a grade or score.”
Ann

30. Summative Evaluation Why do we do this?
A program director must provide timely verification (written documentation) of residency education by completing summative performance evaluations:
for all graduating residents/fellows
for residents who leave the program prior to completion.
Continued…
Ann

31. Summative Evaluations
3) ACGME also requires this for
residents who are transferring into your program from another program.
“The program director must obtain written or electronic verification of previous educational experiences and a summative competency-based performance evaluation of the transferring resident/intern.”
Ann

32. Summative Evaluations
GME created Templates for the programs for Program to use in completing Summative Evaluations for their trainees graduating or leaving the program / transferring…
Nancy

33. Step 2. Continued…
Provided Faculty/Program Director Education on the tools and processes
Reviewed the requirements of Summative Evaluation…
Nancy

34. Summative Evaluation Why do we do this?
A program director must provide timely verification (written documentation) of residency education by completing summative performance evaluations:
for all graduating residents/fellows
for residents who leave the program prior to completion.
Continued…
Nancy

35. Summary We have found that:
Standardizing Forms and Developing Templates
Saves a lot of time for our Programs/PDs
Improves feedback to the Programs
Standardizing Program Evaluations in the GME Office
Not only saves time for the programs, but assures GME that there will be no citations for not having program evaluations
Our faculty have been engaged and very positive with development sessions from GME on eliminating bias from their evaluation questions and response scales
Nancy

36. Questions?
CONTACT INFORMATION ~
Ann M. Dohn, MA
Designated Institutional Official
Nancy Piro, PhD
Education Specialist/ Program Manager

38. Faculty Development Session EVALUATIONS
Our focus today:
How do we eliminate unintended bias from our evaluation process?

39. What is cognitive bias…
Cognitive bias is distortion in the way we perceive reality / information.
Response bias is a type of cognitive bias which can affect the results of an evaluation if evaluators answer questions in the way they think they are designed to be answered, or with a positive or negative bias toward the fellow being evaluated

40. Where does response bias occur?
Response bias most occurs most often in the wording of the question.
Response bias is present when a question contains a leading phrase or words.
Response bias can also occur in rating scales.
Response bias can be in the raters themselves
Central Tendency
Halo Effect
Similarity Effect
Response bias
a type of cognitive bias which can affect the results of an evaluation if respondents answer questions in the way they think they are designed to be answered, or with a positive or negative bias rather than according to their true perceptions of the person or program being evaluated.
Response bias is present when a question contains a leading phrase.
Response bias can, however, also occur in rating scales.
More positive or negative choices
Five point scales
Response bias in the rater
Central Tendency
Halo Effect
Devil Effect

41. Examples of Question Bias
Example 1: "I can always talk to my Program Director about residency related problems."
Problem: Terms such as "always" and "never" will bias the response in the opposite direction.
Result: Data will be skewed.

42. Examples of Question Bias
Example 2: “Career planning resources are available to me and my program director supports my professional aspirations."
Problem: Double-barreled ---resources and aspirations… Respondents may agree with one and not the other. Evaluator cannot make assumptions about which part of the question respondents were rating.
Result: Data is useless.

43. Examples of Question Bias
Example 3: "Communication in my program is good."
Problem: Question is too broad. If score is less than 100% positive, researcher/evaluator still does not know what aspect of communication needs improvement.
Result: Data is of little or no usefulness.

44. Rating Scale Bias Competence and knowledge in general medicine.
Poor Fair Good Very Good Excellent
The data will be artificially skewed in the positive direction with this scale because there are far more (4:1) positive than negative rating options.

45. Response bias can be in the evaluators themselves
Rater/Evaluator Bias
Response bias can be in the evaluators themselves
Central Tendency
Similarity Effect
Halo Effect

46. Beware the Halo Effect
The halo effect refers to a cognitive bias whereby the perception of a particular behavior or trait is influenced by the perception of the former traits in a sequence of interpretations.
Thorndike (1920) was the first to support the halo effect with empirical research.
People seem not to think of other individuals in mixed terms; instead we seem to see each person as roughly good or roughly bad across all categories of measurement.
In a study published in 1920, Thorndike asked commanding officers to rate their soldiers; Thorndike found high cross-correlation between all positive and all negative traits.

47. The Halo Effect and Expectations
The halo effect is involved in Kelley's implicit personality theory
the first traits we recognize in other people influence our interpretation and perception of later ones because of our expectations.

48. Halo Effect Extends to Products / Marketing
The iPod has had positive effects on perceptions of Apple’s other products…

49. Could this impact our evaluations here?
Empirical evidence from our HouseStaff….
A question from the most recent GME HouseStaff survey:
“The general feeling in my program is that your ability will be labeled based on your initial performance.”
Overall SHC-LPCH
Overall Peds Fellows
If someone makes a mistake, it is often held against them
.Agree Strongly Agree Moderately Agree Slightly - RED
Disagree Slightly - YELLOW
Disagree Moderately Disagree Strongly GREEN

50. Reverse Halo Effect
A corollary to the halo effect is the reverse halo effect (devil effect)
individuals, brands or other things judged to have a single undesirable trait are subsequently judged to have many poor traits, allowing a single weak point or negative trait to influence others' perception of the person, brand or other thing in general.

51. Blind Spots
In the 1970s, the social psychologist Richard Nisbett demonstrated that we may have no awareness of when the halo effect influences us (Nisbett, R.E. and Wilson, T.D., 1977)
The problem with Blind Spots is that we are blind to them…