What can aptitude tests such as the UK Clinical Aptitude Test (UKCAT) offer undergraduate medical selection? Issues and solutions Dr Sandra Nicholson

Issues A Level inflation causing problems with discrimination Image from /media/images/ / gif/_ /media/images/ / gif/_

Issues Underrepresentation of some groups in medicine because ?A-level grades reflect educational background and social class Can aptitude tests widen participation? Stringer Principle –what applicants might have achieved had similar educational resources been available, without losing sight of what each one has actually achieved –Stringer N. Aptitude tests versus school exams as selection tools for higher education and the case for assessing educational achievement in context. Res Papers Educ 2008;23 (1):53–68.

Issues A systematic review of all the factors of the selection process associated with success at medical school showed that personal statements and teacher references –form a significant component of many medical schools’ current selection processes –have a low predictive value of success –contain insufficient detail to be a useful selection tool –Ferguson E, James D, Madeley L: Factors associated with success in medical school: systematic review of the literature. BMJ 2002, 324:

Issues Desire to select on non-cognitive or personal attributes (blueprinting) Consensus statement, MSC/GMC highlighted: –Decision making, leadership And if you don’t –one UK medical school identified10 –15% of each intake (over 5 years) as ‘strugglers’ (Yates & James, BMJ332 (2006) ) –Doctor distress Willcock et al.,MJA, 181, 2004,

Solutions There is an urgent need to reduce the number of candidates by initial screening that is appropriate, fair and transparent but also gives added value to the process Intellectual ability or aptitude rather than achievement may be a valuable means to widen participation Some aptitude tests aim to assess more than intellectual ability assessing some of the desirable non cognitive characteristics in potential medical students.

So, what’s the evidence? Fernando N, Prescott G, Cleland J, Greaves K, McKenzie H: A comparison of the United Kingdom Clinical Aptitude Test (UK-CAT) with a traditional admission selection process. Med Teach 2009 Turner and Nicholson, 2011, Can the UK Clinical Aptitude Test (UKCAT) select suitable candidates for interview? Medical Education 2011: 45: 1041–1047

So, what’s the evidence? McManus, I., et al., Intellectual aptitude tests and A levels for selecting UK school leaver entrants for medical school. BMJ, : p McManus, I., et al., A levels and intelligence as predictors of medical careers in UK doctors: 20 year prospective study. BMJ, : p Outcome of students in relation to A level grades in all subjects are based on best three. A=10, B=8, C=6, D=4, and E=2 points

However is the field the same? Significance of A level inflation? Characteristics of newer aptitude tests? McManus I, Woolf K, Dacre J: Even one star at A level could be "too little, too late" for medical student selection. BMC Medical Education 2008, 8:16. Performance of third year UCL medical students in MCQ and OSCE assessments by top three A level points.

So, what’s the evidence? BMAT has shown predictive validity between test scores and medical school progression –Emery JL, Bell JF The predictive validity of the BioMedical Admissions Test for pre-clinical examination performance. Med Educ 43:557–654 For GAMSAT a significant correlation with Year 2 examination results only for the biological and physical sciences section assessing achievement, and not for the sections assessing “more general skills in problem solving, critical thinking and writing” –Groves MA, Gordon J, Ryan G. Entry tests for graduate medical programs: is it time to re- think? Med J Aust 2007; 186: UKCAT initially disappointing but small scale studies measuring possibly incomplete outcomes –Lynch B, MacKenzie R, Dowell J, Cleland J, Prescott G: Does the UKCAT predict year 1 performance in medical school? Med Educ 2009, 43: –Janet Yates and David James (2010) The value of the UK Clinical Aptitude Test in predicting pre-clinical performance: a prospective cohort study at Nottingham Medical School. BMC Medical Education 2010

So, what’s the evidence? Wright and Bradley, 2010, Has the UK Clinical Aptitude Test improved medical student selection? Med Education, Nov;44(11) that previous school type did not predict the UKCAT scores of their accepted students at their university. UKCAT scores were predictive of Year 1 and 2 examination performance including significant predictors of knowledge examination performance Preliminary results from an additional medical school confirm that UKCAT scores can predict performance in 1st and 2nd year exams. Abstract Reasoning is the single best predictor of Year 1 examination Decision Analysis is the single best predictor of Year 2 examination performance Verbal Reasoning also being a useful predictor (Adam et al, submitted for publication). James D, Yates J, Nicholson S: Comparison of A level and UKCAT performance in students applying to UK medical and dental schools in 2006: cohort study. BMJ 2010 Concerns over no benefit in increasing participation although also showed that UKCAT scores were less subject to bias than A level results alone Non-cognitive qualities have been shown to predict professional behaviour in medical students and medical school assessment outcomes (if appropriately structured) (Adam et al., submitted for publication) Non-cognitive test have satisfactory reliability and scores have been shown to be unaffected by social class, gender and ethnicity (Wu, 2009).

Selection encapsulates most of the important questions that medical education faces. The problems are not insoluble, and the intellectual tools mostly exist. The problem is in finding the collaborative will to solve them, and to confront negative answers if and when they arise –Testing medical school selection tests Chris McManus and David Powis MJA 2007; 186 (3):