Presentation on theme: "Improving Selection to the Foundation Programme"— Presentation transcript:
1 Improving Selection to the Foundation Programme Parallel Recruitment Exercise
2 Selection to the Foundation Programme – improving and evolving FP 2005 Foundation Programme introduced: local paper-based applications FP 2006 National timetable and standard application introduced: 50% paper-based applications + 50% online applications FP 2007 – 100% of applications online using ‘white present space’ application form questions and academic quartiles
3 Evolution2009 DH commissioned a review of selection methods. The Improving Selection to the Foundation Programme project was set up and overseen by the Medical Schools Council 2010/2011 New selection methods were piloted successfully FP 2012 Full-scale Parallel Recruitment Exercise (PRE) FP 2013 New selection methods implementedThe Department of Health commissioned a review of the current selection methods, not because they didn’t work, but because there were concerns about the long term availability of suitably different ‘white space’ questions, and to ensure that the most current research evidence and selection methods are used. The review included academic literature reviews of high-stakes selection, extensive stakeholder consultation, advice from an international panel of experts and an independent cost benefit analysis.The new selection methods were piloted with 1100 students in , in 15 UK and two non-UK schools. The pilots have demonstrated the evidence that the new selection methods are the most fair and effective method of selection to the Foundation Programme.Applicants to start the Foundation Programme in August next year will apply using ‘white space’ questions and quartiles; the new methods will be used for applications to FP2013 onwards
4 What are the new selection methods? Situational Judgement Test (SJT)SJTs will replace the ‘white space’ application form questionsThis is an invigilated, machine markable test in exam conditionsThe SJT will consist of around 70 questions in 2 hours 20 minsEducational Performance Measure (EPM)The EPM will replace the academic quartile scores. The EPM score is comprised of three elements:medical school performance in deciles [SEE accompanying ppt on basket of assessments, quartiles and deciles]additional degreesacademic achievementsThe EPM and SJT will each be worth 50 points from a 100 point application scoreThe EPM reflects academic performance at medical school, and additional academic achievements – evolving from academic quartiles and Q1 of the ‘white space’. There are no points for extra-curricular activities – it would be impossible to compare care for a family member, with committee leadership, with volunteering - however the personal gains from taking part in extra-curricular should be apparent through answers to the SJT.The SJT assesses the attributes of the national person specification – as the current white space questions doThe evidence shows that the SJT and EPM in combination assess the clinical and non-clinical knowledge, skills, and personal attributes to be effective in the role of an FY1 doctorThe change is about evolution, not revolution. The EPM and SJT are changes to the way that the application score is produced; but applicants will continue to declare their preferences for foundation schools, make linked applications, special circumstances and so on.
5 How will recruitment work in the future? FP 2012FPAS runs as normal + new recruitment process trialledthrough the Parallel Recruitment Exercise (PRE)SJT: All students are expected to participate in a shortened version of theSJT– the exam will only be one hour and will consist of 30 questionsEPM: Medical school staff will calculate EPM deciles for each studentIMPORTANT: Scores from the PRE EPM and SJT will not impact your FPASscore - and you will receive feedbackFP 2013All applicants to the Foundation Programme, including to AcademicProgrammes, will take the SJT, and will receive an EPM scoreThe Parallel Recruitment Exercise (PRE) will not affect the FPAS score for final year students applying this year to the Foundation Programme 2012.There will continue to be a separate selection and appointment process for Academic Foundation Programmes; however ALL applicants will be required to complete the national application process, including the SJT and EPM.
6 What is the PRE?New selection methods will be trialled alongside the normal selection methods during this year’s national FP recruitment roundAll final year medical students are expected to participate in the SJT, in addition to completing their FPAS formsThe PRE is the final step in ensuring the selection methods can be consistently and robustly applied before implementation for FP2013Parallel Recruitment Exercise – running in all UK medical schools during the 2011/2012 academic year. Final year medical students only – expected to take part in a shortened one-hour SJTThe PRE is an opportunity to trial the SJT and EPM on a national scale, and to provide additional evidence that the SJT and EPM are the way forward (n.b. the pilots have demonstrated that the SJT is a fair and effective measure of the aptitude for the FY1 role, with 1100 participants – the PRE is on a national scale) – students will be asked for their views and feedback
7 What is a Situational Judgement Test? SJTs are:a test of aptitudedesigned to assess the professional attributes expected of a Foundation doctorbased on a detailed job analysis of an FY1 doctorSJT questions assess your judgement by presenting you withchallenging situations you are likely encounter at work during the firstyear of an integrated Foundation ProgrammeSJTs are an invigilated assessment of professional judgement and the attributes expected of the Foundation doctor.The SJT is not something students can revise for, but these are skills and attributes that they should exhibit, and that they will have been made aware of throughout their time at medical school, on placements and through e.g. Tomorrow’s Doctors.SJTs are used in selection to GP training, trialled for selection to five other specialties, and used by FBI, Fire Service, police, civil service
8 Job analysis of FY1 doctor Commitment to professionalismCoping with pressureEffective communicationLearning and professional developmentOrganisation and planningPatient focusProblem solving and decision-makingSelf-awareness and insightWorking effectively as part of a teamThe findings of a detailed Job Analysis (including shadowing of FY1s, 50+ interviews with FY1s and colleagues working with FY1s, as well as literature reviews) will be published.The Job Analysis identified more than 100 behaviours which can be grouped into the nine ‘domains’ listed on the slide in alphabetical order. The individual SJT items are mapped against these domains, and the test will include questions across the breadth of these domains.
9 Why change from ‘white space’ questions? What students say:Some applicants get help writing their answers from parents/friends who are doctors and it is not fair to those who don’t get helpModel answers can be bought on the internetVery time-consuming and stressful to complete during a busy time‘White space’ questions are an exercise in creative writingPerceived variability in scoringStudents have raised concerns during the recruitment feedback surveys about ‘white space’ questions, particularly:risk of plagiarism – with a 2 week window within which applicants can answer the questionsAnecdotal feedback that ‘white space’ questions are an exercise in creative writing
10 Why change from ‘white space’ questions? What the UKFPO and deaneries say:Difficult to ask questions about teamwork, professionalism, etc in sufficiently different waysThe more similar the questions are, the more model answers and ‘answer writing’ workshops will be available resulting in a narrow spread of scoresDifficult to differentiate between candidates with similar scores for selection into the Foundation ProgrammeScoring is very resource intensiveWhilst the current system currently works well, there are a number of concerns that need to be addressed including:Longevity of white space questions – there’s a limit to how many variations of questions can be askedCostly in terms of consultant time to score white space questions – raises the question ‘is it cost effective?’
11 Why change to SJTs? Exam conditions – therefore fair for everyone Research evidence to support use in this contextSufficient questions can be developed for long-term useAccess to example questions and answers prior to taking SJTMarks for near missesAble to differentiate between applicantsWithin medical selection, SJTs have been shown to be a reliable and valid method of selectionEvidence shows ability to predict performance in the roleSJTs are increasingly popular in large-scale selection (in medical selection in the UK and in other disciplines – police, FBI, fire service)Students will take the SJT in exam conditions and so everyone will have an equal chance to do well.There is research evidence to support the use of SJTs and the range of different scenarios an FY1 might encounter means that a sufficient number of questions can be developed in order to use SJTs in the long term.Students and schools will have access to example questions and answers to prepare them with the format of questions, and how the answers are scoredThe scoring key allows for near misses for the ranking questions – so for example if ABCDE is right answer, and student answers BACDE, would score nearly full marksPilots so far have shown that SJTs are able to differentiate between students – some students felt that would result in everyone getting similar score but this was shown not to be the case.
12 Example SJT Questions There are two question formats: Rank the five responses in the most appropriate orderChoose the three most appropriate responses from eightYou should answer what you ‘should’ do in the scenariodescribed, not what you ‘would’ doSJTs are a test of aptitude and are designed to assess the professional attributes expected of a Foundation doctor. There are two question formats: Rank five possible responses in the most appropriate orderSelect the three most appropriate responses for the situation (from eight possible options)Students must answer what they ‘should’ do in the scenario described, not what they ‘would’ do. This is because research into SJT shows that questions asking an applicant what they ‘would’ favours the more ‘test wise’ – plus the GMC has an emphasis on probity
13 Example Question 1 – rank response Mr Johnson is admitted with a minor groin abscess requiring surgical drainage although he is otherwise well and has full mental capacity. Four hours prior to surgery Mr Johnson informs a nurse that he wishes to self discharge as he says he is due in court. Mr Johnson’s next of kin are aware that he has been admitted for surgery and that he is due in court. The nurse asks you to speak to him. Rank in order the following actions in response to this situation (1= most appropriate; 5= least appropriate). A Allow Mr Johnson to leave but advise him to see his General Practitioner if there are further problems B Prevent Mr Johnson from leaving by phoning security C Explain to Mr Johnson the risks of leaving without treatment D Telephone Mr Johnson’s next of kin to ask them to try and persuade him not to leave E Allow Mr Johnson to leave but ask him to return to the hospital as soon as possible
14 Answer to Question 1C Explain to Mr Johnson the risks of leaving without treatment E Allow Mr Johnson to leave but ask him to return to the hospital as soon as possible D Telephone Mr Johnson’s next of kin to ask them to try and persuade him not to leave A Allow Mr Johnson to leave but advise him to see his General Practitioner if there are further problems B Prevent Mr Johnson from leaving by phoning security Communicating to Mr Johnson the risks of leaving is important as he may not be aware of these risks (C). However, it is the patient’s choice as to whether he remains in hospital. Informing him that he should return to the hospital is important (E). Mr Johnsons’ next of kin are aware of the situation and may be able to persuade him to stay (D). By asking Mr Johnson to see his GP, instead of returning to the hospital, you would be passing responsibility for his care (A), however this is more appropriate than forcing a patient to stay against his wishes (B).Correct answer (CEDAB) would receive a maximum of 20 marks – 4 marks per answer in the right orderIf answered ECDAB (only first two the wrong way round), would score almost full marksIf answered B as the most appropriate (when it is the least) but all others in order i.e. BCEDA, would receive 0 points for answer ‘B’, but 3 out of 4 for each of the others i.e. a total of 12 out of 20 marks.n.b. the total marks available for an SJT paper will be scaled so that the SJT score will be worth 50 points on the FPAS application form, out of the total 100 points
15 Example Question 2 – choose best 3 You review a patient on the surgical ward who has had an appendicectomy done earlier on in the day. You write a prescription for strong painkillers. The staff nurse challenges your decision and refuses to give the medication to the patient. Choose the THREE most appropriate actions to take in this situation A Instruct the nurse to give the medication to the patient B Discuss with the nurse why she disagrees with the prescription C Ask a senior colleague for advice D Complete a clinical incident form E Cancel the prescription on the nurse’s advice F Arrange to speak to the nurse later to discuss your working relationship G Write in the medical notes that the nurse has declined to give the medication H Review the case again
16 Answer to Question 2B Discuss with the nurse why she disagrees with the prescription C Ask a senior colleague for advice H Review the case again Ensuring patient safety is key to this scenario. It is important to discuss the nurse’s decision with her as there may be something that you have missed when first reviewing the patient (B). Therefore it would also be important to review the patient again (H). Also relating to this is the importance of respecting the views of colleagues and maintaining working relationships, even if there is disagreement. As there has been a disagreement regarding patient care, it is important to seek advice from a senior colleague (C).In this example, students will receive 4 marks for each correct answer. However, if they put more than 3 options are selected they will get 0 marks as they have not answered the question. i.e. a score of 0, 4, 8 or 12 for this question.n.b. the total marks available for an SJT paper will be scaled so that the SJT score will be worth 50 points on the FPAS application form, out of the total 100 points
17 Taking the PRE SJT at your school All final year medical students are expected to complete their FPAS form PLUS a one hour invigilated SJTSJTs will:take place at Nottingham Medical School on Friday 9 Dec. Your school will confirm the date, time and venuegive you the opportunity to practice taking an SJT, which you are likely to encounter for specialty selectionprovide you with your score in relation to all other students who have taken the test (feedback in April 2012)NOT have any bearing on your FPAS score, nor on your allocation to foundation school or programmeAll students are expected to take part in a one hour SJT in addition to completing their FPAS applicationApplicants will be given feedback on their score in April 2012 via the FPAS system, although this will have no effect on their application score or foundation school allocationThere should be provision for students with additional needs e.g. extra time
18 Benefits of taking part in the PRE SJT It is good preparation for specialty selectionYou will get feedback on how you scoredYou are helping to form national recruitment policyYou are expected to take part unless you have permission to be exemptYou may win an i-pad!PRACTICE! Taking part in the PRE SJT provides students with an excellent chance to familiarise themselves and practice this style of question format before encountering it for real during specialty selectionThey will receive feedback on their scoreThey will be helping to form the national recruitment policy – and their views and input are extremely important to ensure a fair and evidenced based system is implementedAll final year students are expected to take part – permission may be granted if they are away on elective – have to explain to a senior medical school staff member know in writingThere are 5 prizes of an i-pad (drawn at random from all participants)
19 SJT for selection to FP2013 onwards All applicants to the Foundation Programme will be required to complete an SJT in the UKThe SJT will be run by UK medical schools on two national dates, to allow for electives and extenuating circumstancesThe SJT will last 2 hours 20 minutes, and will consist of around 70 questions, including a small number of ‘pilot’ questions which will not count towards the application scoreThe SJT will be completed on paper, with the possibility of computer delivery in futureThere will be provision for access arrangements e.g. extra timeThe two national dates will be between November and January – to be announced
20 Why change to EPM?A clear framework with agreed principles will be used to calculate the EPM, ensuring that it is fair, transparent and consistent across the schools of the UKSplitting cohorts into deciles rather than quartiles provides a wider spread of scores, which makes it easier to differentiate between applicants, and will be more fair for applicants at the marginsIt makes more sense for all the academic components of the application to be one part of the application, rather than being split between ‘white space’ questions (Q1) and academic quartile rankingIt is not possible to compare performance of students in different medical schools, as schools offer different assessments at different times throughout the course – and the FPAS application is completed before finals. The most fair way to recognise achievements at medical school is thus relative to applicants from the same medical school.The way that quartiles are calculated does vary between schools; in future all schools will be required to follow an agreed framework, and to publish the way that they calculate points for performance at medical school. The EPM will improve consistency and transparency.Deciles instead of quartiles – more fairBrings together points for additional academic achievements, which is currently Q1 of the ‘white space’ questionsRecommendations on the best way forward, using evidence gathered from pilots during the 2010/2011 academic year.
21 How is the EPM calculated? Score produced by applicant’s medical school to reflect achievementand performance compared to rest of cohortEPM = 3 parts (maximum 50 points):Medical school performance in deciles (34 – 43 points)E.g. Top 10% = 43; Top 20% = 42; etcAdditional degrees (max 5 points)Educational achievements (presentations, prizes and publications (max 2 points)Schools will consult with students about how the decile points for the EPM will be calculated. This will then be published onlineThe EPM reflects a student’s performance at medical school on summative assessments, with all schools (UK and non UK) using the same framework which outlines the rules that each school must follow when calculating the EPM decile score. Schools will publish the way that they calculate deciles according to this framework, specifying the assessments and weightings used locally.Deciles will be worth points from the 50 points for the EPMAs with the current application form, applicants are each eligible for additional points to recognise degrees, presentations, prizes and publications – up to 7 points in total, from the 50 points for the EPMThere are no points for extra-curricular activities – it would be impossible to compare caring for a family member, with committee leadership, with volunteering - however the personal gains from taking part in extra-curricular should be apparent through answers to the SJT.
22 1. Medical school performance (34-43 points) Your medical school will use your results from a number ofassessments up to the end of the penultimate year at medical schoolto calculate your decile score, as outlined in the EPM Frameworkpublished onThe EPM Framework sets out the principles for calculating deciles,which are listed below. Assessments must:be summative(in combination) cover clinical knowledge, skills and performance(in combination) cover non-clinical performancecover all aspects of the curriculum assessed up to the end of the penultimate year at medical schoolrepresent the average performance of applicants over timeinclude written and practical assessmentsStudents will be assessed and ranked on their medical school performance, according to the EPM framework.Schools must consult with students, and will publish the agreed way that they calculate deciles according to this framework, specifying the assessments and weightings used locally.
23 2. Additional degrees (max 5 points) A maximum of 5 points for degrees awarded by time of application to Foundation Programme:Previous degreeNumber of pointsDoctoral degree (PhD, DPhil, etc)5Masters degree1st class honours degreeBachelor of Dental Science (BDS)B Vet Med42.1 class honours degree1st class intercalated degree which does not extend the degree programme32.2 class honours degree2.1 class intercalated degree which does not extend the degree programme23rd class honours degreeUnclassified or ordinary degree2.2 class intercalated degree which does not extend the degree programme1Primary medical qualification only3rd class intercalated degree which does not extend the degree programmeIf an applicant holds more than one degree at the time of application to the Foundation Programme, they should provide evidence of the degree that will achieve the highest number of points.
24 3. Educational achievements (max 2 points) A maximum of 2 points will be awarded for national prizes, presentations and publications:Educational achievementsNumber of pointsPrizes1st prize – National /international educational prize1PresentationsOral presentation at a national or international conference1st named author in a poster or presentation at a national or international conferencePublicationsEducational research paper published in a peer-reviewed journalMaximum number of points available2If an applicant has more than one publication, prize or presentation, they will receive a maximum of one point for any of the three categories individually; a maximum of two points in total.
25 Parallel Recruitment Exercise - EPM Schools will consult with students about how the decile scores will be calculated SEE accompanying PPTYour medical school administrator will calculate your decile score whichwill be analysed alongside your Question 1 scores (additional degreesand educational achievements). You will be told your score in April 2012.Your decile score will:be calculated by your medical school using the methods they are planning to use for FP2013be used confidentially for analysisNOT have any bearing on your FPAS application score, nor will the information be used to determine your allocation to foundation school or to programmeIn the event of failed finals/re-applying to the FP2013, the EPM decile score calculated this year will countSchools will be consulting with students during the autumn term 2011, to agree the way that decile points will be calculated.In the same way that it is up to medical school staff to calculate quartiles, it is the schools who will calculate deciles – students don’t have to do anything extra.The EPM score will not affect applications to the Foundation Programme in any way – however students will be told their decile points for information, in April 2012 (when they will also receive feedback on their SJT performance)If a student fails the year or re-applies to the FP2013, their EPM decile score calculated this year will be used (as their performance up to the penultimate year will not change) – but points for additional degrees, prizes etc will be at the point of application.
26 Selection to FP2012 FPAS application to be completed as normal Participation in the Parallel Recruitment Exercise will not affect your FPAS application in any waySituational Judgement Test (SJT)You are expected to take part in a shortened 1 hour SJTYou will receive feedback on your SJT score in April 2012Educational Performance Measure (EPM)Your school will consult on how deciles will be calculated in futureYou will receive feedback on your decile score in April 2012SUMMARY
27 Selection to FP2013 onwards Situational Judgement Test (SJT) – max 50 pointsSJTs will replace the ‘white space’ application form questions as a measure of meeting the national person specificationSJTs assess aptitude for the FY1 roleA machine markable, multiple choice test taken in exam conditionsThe SJT will consist of around 70 questions in 2 hours 20 minsEducational Performance Measure (EPM) – max 50 pointsThe EPM will replace academic quartiles as a measure of clinical and non-clinical skills, knowledge and performanceThe EPM score is comprised of three elements:medical school performance in deciles (34-43 points)additional degrees (up to 5 points)academic achievements (up to 2 points)SUMMARY
28 More information ISFP website – www.isfp.org.uk Discussion forum E-bulletin updatesProject handbookFAQsFacebookYouTube (background info)UKFPO student representative at your schools from medical schools – keep a look out in your inbox!Student views are hugely important. These are some of the different ways students can keep update with the project, find out more and send us their comments.
29 Nottingham contact: firstname.lastname@example.org Any questions?If you cannot answer any questions, please contact us with these and we will respond directly to you.Nottingham contact: