1. Development and Application of a Simulation-Based Assessment Center for Non-Cognitive Attributes: Screening of Candidates to Tel-Aviv University Medical School Presented to: National Examinations Centre (NAEC) Tbilisi, Georgia 25 September, 2007

2. 2 Tel Aviv University (TAU) Sackler Faculty of Medicine Israel Center for Medical Simulation (MSR) National Institute for Testing and Evaluation (NITE)

3. 3 Authors & Researchers: NITE –Naomi Gafni –Orit Rubin –Avital Moshinsky –Avi Allalouf MSR –Amitai Ziv TAU –Moshe Mittelman –Dov Lichtenberg

4. 4 Background Despite growing acknowledgment of the importance of non-cognitive factors, most medical schools currently rely primarily, and sometimes exclusively, on cognitive factors in student selection. The tool most commonly used in an attempt to consider non-cognitive factors is the interview, which: has strong face validity suffers from rater biases, context biases, unsatisfactory reliability & validity

5. 5 The Motivation Behind the Project Growing dissatisfaction among TAU faculty members: “While the selected candidates show extremely high academic capabilities, some of them have problematic personal and interpersonal characteristics.” Medical School Leadership – Ready for a revolutionary change… In the pipeline for over 2 years… The Task: to select the 100 candidates most suited to the study and practice of medicine from among some 1,700 candidates Given: A Psychometric Entrance Test Score (PET) High School Grade Point Average (GPA) Disqualifying Interviews (“Lip-Service”) Missing: Non-cognitive measures Value message to candidates

6. 6 The Goal of the Assessment Center To improve the screening of medical school candidates by introducing non-cognitive measurements into the equation

7. 7 Assessment Center: Essential Elements  Job analysis  Definition of behavioral profile  Simulations  Multiple methods and multiple assessments  Multiple raters  Raters’ training  Data integration

8. 8 1.Ethical attitude 2.Honesty 3.Professional commitment and responsibility 4.Empathy 5.Service awareness 6.Commitment for the patient 7.Inter-personal communication skills 8.Self confidence 9.Sensitivity 10.The ability for detail observation 11.The ability to identify a need for help, search for help and accept help 12.Openness 13.Initiative 14.Attitude towards authority 15.Self awareness 16.Maturity 17.The ability to function under stress “Job Analysis” – Behavioral Profile

9. 9 Development of structure, content (cases and tasks), rating scales and rating workshops –4 months work –Test Development Committee - Admission committee members, MSR experts, NITE experts Faculty (MDs & PhDs) recruitment and training –Half a day “ Train the rater ” workshop –>150 faculty members per year SP training – behavioral (roles) & rating (similar workshop as to faculty members) Developmental & Logistical Milestones

10. 10 The Structure of the Assessment Center Simulations Biographical Questionnaire Judgment and Decision-Making Communication skills Handling of stress Initiative & responsibility Consciousness & self-awareness 21 Questions 3 Dilemmas 120 minutes 90 minutes 45 minutes 8 Behavioral stations Essay questions related to candidate’s past experiences Short descriptions of dilemmas that require candidate to make decisions

11. 11 Half a day – mandatory for participation Groups of 20 faculty each Include: –Overview of new admission process –Awareness to Biases (Halo, Cultural etc) –Introduction of Rating Scales and Behavioral anchors –Actual rating exercises based on videos of ( “ standardized ” ) candidates prepared in advance –“ Calibration ” of Raters through open discussion of metrics and reference to group ratings Train the Raters workshops

12. 12 Logistics - 98 candidates a day (X 3-4 days) –Two sessions - 5 hours each –Two parallel modules in each session (24 candidates each) –Test Security – Different modules for different days –32 SPs – all day (1 hour work / 1 hour rest) –36 faculty members per session (72 per day) –25 staff members for logistics and administration tasks Data analysis system & score reporting –By NITE – 3-4 weeks Developmental & Logistical Milestones

13. 13 Broad-spectrum simulation modalities (#s) –Simulated Patient section –High-tech simulators –Task trainers –Skills lab Clinical environments –Field, ER, OR, ICU, Clinic Debriefing capabilities –One-way mirrors –Digital A-V equipment –Debriefing software Multidisciplinary staff The Location: MSR – Virtual Hospital

14. 14 Floor Plan 9 16 2 The Location: MSR – Virtual Hospital Location: MSR – Virtual Hospital Floor Plan

15. 15 Behavioral (“OSCE Like”) Stations – Rationale Simulation is the component that makes the assessment center unique in comparison with conventional tests, traditional interviews and questionnaires Observing people’s present behavior is a better predictor of future behavior than their own subjective account of how they would behave Since the candidates do not yet possess any professional knowledge, the simulation should not be based on such knowledge The simulations reflect common situations similar to those encountered by doctors, through which inter-personal and communication skills can be assessed

16. 16 Behavioral Stations – Structure 8 behavioral assessment stations 6 individual stations (7 minutes per station) 2 group stations (25 minutes per station) Candidate’s behavior was observed by faculty members who score behavior according to an assessment form consisting of four dimensions of personal characteristics: Communication skills Handling of stress Initiative and responsibility Consciousness and self-awareness Scoring was on a scale of 1-6.

17. 17 Behavioral Stations – Examples Examples Station Type 1. Aggressive patient 2.You are going to meet your friend, who is a school bus driver, just after you discovered he was drinking on the job. A challenging encounter with an actor Simulation “ Describe the situation ” “ What would you have done differently? ” A structured interview regarding the candidate ’ s performance in the simulation Debriefing “ Why do you want to become a doctor? ” “ What is your opinion on abortions? ” Interview Divide a budget among 3 different sectors in a hospital ward. Group Description A structured mini- interview Groups of 6 candidates perform a task together (* MMI - Reiter H, Eva K et al, McMaster, Canada)

18. 18 Judgment and Decision-Making Questionnaire – Rationale One of the characteristics medical professionals are expected to possess is a well-developed capacity for moral reasoning The goal is to examine the candidate’s ability to contend with moral dilemmas… And to measure the candidate’s ability to comprehend all aspects of a complex situation and reach a considered decision as to how to act

19. 19 Judgment and Decision-Making Questionnaire – Structure 3 short scenarios, each describing a real-life situation that raises questions or doubts vis-à-vis the appropriate decision to be made. The dilemmas have no correct solutions. Candidates should state the reasons for and against the decision to be made. After providing a detailed account of their considerations, they should state how they would act, and explain their decision. The score is based on the number of arguments and their quality: complexity, reference to the conflict between law and morality, application of professional and moral considerations simultaneously, and the ability to reach a final, justifiable decision. Scoring conducted by two independent / trained psychologists

20. 20 Judgment and Decision-Making Questionnaire – Example Jane, 32, is a new teacher at a boarding school. After spending a short time at the school she discovers that the older students are conducting humiliating initiation ceremonies for new students and even having them do their chores. Jane reports this to the school principal, who tells her that this is a longstanding tradition at the school and that she should not be concerned. He implies that should she make this information public, she would no longer be trusted by the students or veteran teachers. What would you advise Jane to do with the information she possesses? Why? What considerations should she take into account?

21. 21 The Biographical Questionnaire – Rationale The most reliable and valid predictor of future behavior is past behavior. In an attempt to estimate qualities and attitudes such as motivation, the tendency to help, consistency, curiosity and leadership, it is reasonable to look to evidence from the candidate’s biographical details, such as hobbies, studies, military service, social activities, and voluntary activities. The questionnaire is standardized and objective. It is less biased than an interview and is scored according to a detailed scoring guide (by two independent / trained psychologists)

22. 22 The Biographical Questionnaire – Structure 21 questions divided into two sections: (1) Past experience – questions regarding experiences and activities during and after high school (military service, job experience, volunteer activities etc.) (2) Emotional awareness – questions regarding past experience in coping with challenging emotional situations

23. 23 Briefly describe a situation in which someone approached you for help/advice and you provided it. What was the problem for which you were asked to provide help/advice? Why were you the person approached? Describe the situation in which you provided help/advice. Describe how you felt in this situation. The Biographical Questionnaire – Example

24. 24 1 st and 2 nd Year Data

25. 25 283 tested in first year (3 days) and 280 tested in second year (4 days) –Allotted randomly between days (proved to be equal) Mean age: ~20 years ~50/50 male/female ratio Mother tongue – –75% Hebrew speakers –20% Arabic speakers –5% other languages Candidates Demographics

26. 26 Scoring the Each item was evaluated on a scale of 1 (lowest) to 6 (highest). In cases with two observers the average of the two evaluations was computed. 4 scores on the 4 aspects: Sum of evaluations relating to each of the factors. Scoring of simulation stations

27. 27 Simulations Biographical Questionnaire Judgment and Decision-Making 1 2 3 4 Communication skills Handling stress Initiative & responsibility Consciousness & self-awareness 21 Questions Dilemma 1 Dilemma 2 Dilemma 3 The Assessment Center – 3 Major Components

28. 28 The Scoring Process Preparatory workshops for raters – Faculty / SPs and Psychologists Simulations Biographical Questionnaire Judgment and Decision-Making For 3 out of 8 stations: 2 Assessors (Real time assessment) Two Assessors: (10 assessors overall). A third assessor is added if there is a large discrepancy Two assessors: (9 assessors overall). A third assessor is added if there is a large discrepancy 90 minutes of evaluation per candidate 65 minutes of evaluation per candidate

29. 29 The Scoring Process - Weights We assumed that the populations participating in the three days were similar. Simulations Biographical Questionnaire Judgment and Decision-Making 60% weight in the final score 20% weight in the final score Standardized scale: Mean = 200; SD = 20

30. 30 What Do the Simulations Assess? 1. Communication skills a. Ability to convey a message clearly and coherently b. Maintenance of boundaries (respectful attitude towards others) c. Candidate ’ s ability to engender trust (sincerely convey intentions and limitations) d. Ability to listen (be attentive without interrupting) e. Openness to the other person ’ s opinion and position -- flexibility f. Ability to behave sensitively and express empathy towards others Attributes and Behaviors Assessed by Simulation Stations

31. 31 What Do the Simulations Assess? 2. Handling stress a. Coping with the situation (low score – nervous, non-functioning, pressured by time) b. Coping with the task (involved, contends with frustration, does not blame others, non-judgmental) c. Extent to which the candidate maintained a high level of functioning throughout the task Attributes and Behaviors Assessed by Simulation Stations

32. 32 What Do the Simulations Assess? 3. Initiative and responsibility a. Responsibility and initiative (takes personal responsibility and tries to solve the problem, initiates) b. Extent to which the candidate controls the situation (leads, plans, organizes) 4. Consciousness and self-awareness a. Capacity for introspection (to describe own behavior, emotions and feelings) b. Ability to recognize the ethical complexity of the situation Attributes and Behaviors Assessed by Simulation Stations

33. 33 9 Scores Calculated for Each MOR Candidate 1. Communication skills (39 items) 2. Handling stress (15 items) 3. Initiative and responsibility (10 items) 4. Consciousness and self-awareness (7 items)

34. 34 9 Scores Calculated for Each MOR Candidate 5. General score for the simulation stations, based on the weighted average of the four dimensions 6. Judgment and decision-making questionnaire score 7. Personal/biographical questionnaire score 8. General MOR score based on agreed upon weights of the scores in the three previous items (60/20/20) 9. Final score, calculated as a simple average (50/50) of the general MOR score and the candidate ’ s aggregate score (GPA + PET)

35. 35 SDMeanDimension Assessment Day 1 13.8190.1 Communication skills (N = 96) 8.870.0 Handling stress 6.245.6 Initiative & responsibility 5.332.0 Consciousness & self-awareness Assessment Day 2 15.3191.3 Communication skills (N = 96) 8.671.5 Handling stress 7.146.0 Initiative & responsibility 6.831.2 Consciousness & self- awareness Mean and SD of raw scores for each dimension for each assessment day

36. 36 Max.Min.SDMeanNVariable 24015119200283 MOR (150 – 250) 79768025740276 PET (200-800) 75770211722271 Aggregate score (200-800) Number of examinees, mean, SD, minimum, and maximum standard MOR scores for all three assessment days in first year (2004) MOR – Assessment Center, PET - Psychometric Entrance Test Score GPA - High School Grade Point Average

37. 37 Max.Min.SDMeanNVariable 23915419200280 MOR (150 – 250) 80067526738279 PET (200-800) 76869910728280 Aggregate score (200-800) Number of examinees, mean, SD, minimum, and maximum standard MOR scores for all three assessment days in second year

38. 38 PETMORBQJ & DStations 1.00 Stations 1.000.06 J & D 1.000.290.17 BQ 1.000.480.370.91 MOR 1.000.130.050.150.12 PET 0.400.050.020.050.08 Aggregate score Correlation between MOR components for the group of examinees taking the Hebrew-language examination over two years

39. 39 Reliability - Consistency Literature: –Reliability of simulations around 0.7 – 0.8 Estimation methods: Test-retest 0.70 (N=34) Internal consistency (Cronbach Alpha) Inter-rater reliability Methods based on inclusion criteria

40. 40 Reliability Day 3 (n=91) Reliability Day 2 (n=96) Reliability Day 1 (n=96) WeightComponent 0.870.830.78 6 Simulation stations 0.620.770.58 2 J & D 0.640.730.61 2 Biographical Questionnaire 0.870.880.81 MOR Cronbach Alpha Reliability Estimates for MOR Components (1 st Year)

41. 41 20052004WeightComponent 0.630.67 6 Simulation stations 0.610.66 2 J & D 0.630.66 2 Biographical Questionnaire 0.750.78 MOR Cronbach Alpha Reliability Estimates for MOR Components

42. 42 Inter-Rater Reliability Stations –Median inter-rater correlation = 0.58 –Corrected for two evaluators (Spearman-Brown) = 0.72 J&D (3 dilemmas X 3 days) –Median inter-rater correlation = 0.72 Biographical Questionnaire (3 Days) –Median inter-rater correlation = 0.94

43. 43 Results from MOR amongst men and women (1 st Year) SDAverageSDAverageVariable 2119718202Simulation stations 2020117199J&D questionnaire 2019718202Personal biographic questionnaire 2119717202MOR 2574424736Psychometric exam 1172310723Aggregate score Men N=121 Women N=132

44. 44 Results from MOR amongst men and women (1 st Year) Small discrepancies were found favoring women in all the factors included in MOR (0.1-0.25 SD) excluding the J&D questionnaire The greatest discrepancy was found in the general grade for simulation stations A discrepancy of one third of a SD was found in favor of men in the psychometric exam No difference was found between the groups for the aggregate score

45. 45 Participants ’ Feedback 2.9To what extent did you feel you were able to express your abilities as a candidate in the assessment center as a whole? 1 3.0To what extent was the selection process for medical school using the MOR system fair? 2 3.7How satisfied are you with the organizational aspects of the assessment day? 3 2.9To what extent was the preparation booklet you received in advance of the assessment day useful and satisfactory? 4 1- not at all, 2- to a minor extent, 3- to a great extent, 4- to a very great extent

46. 46 2.4To what extent did you feel you were able to express your abilities as a candidate in the biographical questionnaire? 1 2.9To what extent did you feel you were able to express your abilities as a candidate in the J&D questionnaire? 2 3.0To what extent did you feel you were able to express your abilities as a candidate in the stations with the actors? 3 2.9To what extent did you feel you were able to express your abilities as a candidate in general at the evaluation center? 4 3.0To what extent was the selection process for medical school using the MOR system fair? 5 1- not at all, 2- to a minor extent, 3- to a great extent, 4- to a very great extent Participants’ Feedback

47. 47 To what extent is each of the following measures fair as a selection tool for candidates to medical schools? 3.1High School Grade Point Average1 3.1Psychometric Entrance Test2 3.4Personal Interview3 2.9Simulations using actors4 2.6Group Stations5 1- not at all, 2- to a minor extent, 3- to a great extent, 4- to a very great extent

48. 48 The Scoring Process: Conclusions 1.The scoring process and score reporting for 300 candidates lasted 3-4 weeks. 2.Reliability measures - Good inter rater reliability ( “ train the rater ” ) and High internal consistency measures 3.The make-up of the student body changed by 20% 4.Validity measures – current & future research: a.Already in process: comparing qualities of students accepted based on MOR and students accepted by previous selection process. b.Planned: a longitudinal validity study in several milestones down the road – pre clinical, clinical, internship …

49. 49 The Assessment Center – Summary 1.The Assessment Center – very complex endeavor - successfully conducted for 2 consecutive years (600 candidates). 2.Very smooth recruitment and enthusiastic collaboration with faculty members – (The Hidden Agenda … ) 3.Both candidates and faculty members expressed high satisfaction as regards the fairness and implementation of the new selection process – High Face Validity 4.Very high national interest – Technion Medical School – joined the process in 2006. (HU modifying to MMI) 5.High international interest - encourages more elaborated discussions and research regarding non-cognitive admission methods.

50. 50 Potential shortcomings Small and very select group of candidates which might present low variability in different measures The validation process is long and difficult Potential cultural biases Cost… The Assessment Center – Discussion

51. 51 The Assessment Center – Discussion Strengths Theoretical basis Multiple methods Multiple independent ratings (20) Standard measurements Significant weights (50%) Other consequences Social message: to Candidates, Faculty & Public The make-up of the student body changed by 20% Dramatic Change in atmosphere at TAU Non-Cognitive Factors Are Important Reliability & Validity