Presentation is loading. Please wait.

Presentation is loading. Please wait.

Multiple Mini Interviews in the UK

Similar presentations

Presentation on theme: "Multiple Mini Interviews in the UK"— Presentation transcript:

1 Multiple Mini Interviews in the UK
Jon Dowell - University of Dundee Aileen O’Brien - St George's University of London Multiple Mini Interviews in the UK Welcome, thanks for coming, thanks to Martin Hope you enjoy hearing more about changes Presented in the spirit of ‘work in progress’. Every school struggling with same issues and we do not claim to have ‘the answer’ 1

2 Rationale Personal qualities considered ‘important’ but not very reliably assessed by interview and without much predictive validity. (Goho + Blackman cognitive r 0.06, clinical r 0.17) Introduced for assessing medical school applicants at McMaster University, Canada. (Reiter, Eva et al) Piloted 2002 – 12 x 8 min. Generalisability since +/- 0.7 Predicted OSCE performance ß 0.44 (interview nil) Clerkship ratings ß 0.57 (interview nil) MCC Part 1 (selected components) ß (interview nil or negative) Supporting evidence from Canberra - Harris and Owen 2007 Calgary - Brownell et al 2007 Dundee - Ponnamperuma 2008 / 4 station trial 2

3 Domains assessed Dundee Georges Interpersonal and communication skills
Critical thinking, problem solving Team work Robustness against stress Ethical/moral reasoning Integrity/honesty Preparation and motivation for medical school Georges Empathy Communication Organisation and problem solving Team work Initiative and resilience Insight and integrity Effective learning style Academic ability & intellect ability & intellect (already tested) Suggest Aileen presents 1-1/2 way through this one. Jon then describes Dundee content and process in slides 5-7. 3

4 Dundee Stations 2010 Role play OCD discussion 1:1
Professional Integrity Task Sort it out - cards Issues in Medicine Puzzle Fitness to Practice Medicine as a career About you - UCAS Medications Muddle Moral Dilemmas

5 How it runs 5

6 Marking Training 30 min general 20 min station specific
Optimal criteria for each domain defined Name Label 6

7 Stations St George’s Standard interview questions Prioritising
Scenario/ breaking bad news Presentation Observation Communication/ giving instructions

8 George’s Marking Scheme
Candidates name……………………………………………… Motivation to do medicine What are you looking forward to about being a doctor? Points for guidance: Personal satisfaction from helping people Satisfaction from achieving something worthwhile Working with a diverse range of people Combining an interest in science with caring for people Variety of career opportunities once qualified Job status and security Good income The opportunity to work abroad Intellectual rigour Lifelong learning and clear professional career structure Camaraderie from clinical teams Research collaboration Please mark out of 5 Poor motivation good motivation Please also assign an overall impression of the candidate’s suitability to be offered a place at medical school based on their performance at this station (please circle and comment as appropriate). EXCELLENT ACCEPTABLE CAUSE FOR CONCERN Red card Comments:

9 George’s results Graduates and non graduates did equally well
No correlation with Gender Age 4 year and 5 year course UKCAT and GAMSAT scores Reliable (kappa 0.68)

10 George’s Conclusions Successful pilot 2009
Ran this year with 1000 candidates Smooth (two breakdowns!) Graduates and school leavers same cut off score for admission

11 Dundee results 2009 n = 452 Cronbach’s Alpha = .70
No correlation in 2009 with Gender Age Overseas candidates School type Small age gender interaction

12 2010 MMIs: Gender Females performed significantly better than males.
t(475) = 3.19, p = < .01 Effect size: 0.14

13 2010 MMIs: Applicant Group Significant differences between groups.
F(4) = 2.50, p < .05 Graduates/Mature perform significantly better than all other groups

14 Domain Reliability Number of Stations Number of Scores
Cronbach’s alpha Communication 9 12 .767 Critical Thinking 6 .453 Integrity 1 - Moral Reasoning 4 .170 Motivation 3 .515 Teamwork 2 .395

15 Correlations between scales
No correlation between UKCAT and MMI scores No correlation between our UCAS form and MMI scores

16 Dundee MMI - Conclusions
Good reliability in last two years. All stations were reliable enough. No negative Item- total correlations; no reason to reject any station. Successfully ran for over 900 candidates with no significant problems (no breakdowns, tears etc!) Putting more weight on MMI may increase female and graduate/mature candidates compared to the Pre-Interview Score

17 MMI/Pre-Interview 40/60 60/40 80/20 88 Selection Difference Male 129
40/60 60/40 80/20 Male 129 122 121 Female 201 208 209 A-Level 81 79 Scottish 189 182 179 G/M 27 35 38 EU 16 17 OS 18 48 40 Selection Difference 88

18 So what do the punters think?

19 Assessors’ Survey Dundee
116 /157 assessors (88% of students and 50% of staff) completed online survey to find out how they viewed MMI. 91% thought the process fair. 88% thought stress on applicants ‘moderate’ or less. 19

20 Candidates’ Survey Dundee
324 /433 (75%) of Candidates completed online survey. 94% felt MMI was ‘fair’. 90% felt it is a ‘valid way to assess candidates’. 71% preferred MMI to traditional interview. ‘Student Counselor’ (role play) cited as the most enjoyable (153/172). 20

21 Assessors’ Survey George’s
Equally fair Equally accurate Equally able to pick best candidate Interviews “coached” MMIs can be boring MMIs good distinguishing tool

22 Candidates’ survey George’s
5 years- MMI more fair -MMI more accurate -interview more difficult 4 years- MMI more difficult -equally fair and accurate

23 Conclusions and next steps
MMIs becoming an interview tool at many schools. Is it justifiable not to use them? How to develop new stations? Collectively? Reliability potentially increased (above 0.8) using Rasch analysis – alters decision for 10% Predictive validity in UK? – could UKCAT help? Questions...

24 ? Rasch Modeling Options to improve?

25 Examiner Track A B C D E Station
Variation in outcomes can be enhanced by using multiple tracks Examiner 1 2 3 4 5 6 7 8 9 10 A B C D E 4 Track 5 7 4 25

26 Yardstick

27 Effect of using Rasch Reliability based on ‘fair scores’ 0.84
10% different candidates would be given offers. (+ 10% not) Only the top 169 ‘stable’. Could not be 100% confident in process this year, so not applied. (complex black box stuff)

Download ppt "Multiple Mini Interviews in the UK"

Similar presentations

Ads by Google