Changes to the Final FRCA Exam

Changes to the Final Written Constructed Response Questions (CRQs) replacing Short Answer Questions (SAQs) Reduce possibility of misinterpretation More focussed marking system Awarding of marks more transparent Increase in reliability and validity Easier to adapt to Computer Based Testing The examiner board felt that moving the FRCA SAQ examination to a more structured form, for both questions and answers, would reduce the possibility of misinterpretation of either which will be beneficial to both examiner and candidate. The marking system would be more focussed having tiered sub-questions with clear mark allocations alongside which would allow greater transparency in how marks have been awarded. This type of system is considered good practice in further and higher educational boards. The adoption of this format should see an increase in reliability and validity of this part of the Final Written examination. This change of question format will continue to reflect the knowledge required in clinical practice and allow whole Curriculum assessment. The more direct question and answer format would also be more easily adaptable to computer based testing which should see an increase in effectiveness and improve the service offered to candidates.

Final Written timeline Constructed Response Questions (CRQs) replacing Short Answer Questions (SAQs) First exam September 2019: 6 x CRQ (1-6) and 6 x SAQ (7-12) in 3 hours Both sets of questions carry 20 marks each Plan move to all CRQ in March 2020 Chair’s report and example questions on website.

changes to the Final SOE Dec 18 1. Introduction of an extra table change so that candidates will see six examiners 2. SOE1 - AM - Four clinical short cases each one linked to an applied clinical science question 3. SOE2 - PM - Two part long case and two stand-alone short cases

Reason for change More examiners reduces possibility of subjectivity and increases validity and reliability Short with linked applied clinical science more intuitive Shorter long case allows greater breath of the curriculum overall

SOE1 - Short with linked Science (AM) Two parts A and B taken in one sitting Each part consists of two short cases each with linked science questions in 26 mins Normally, clinical short case followed by a linked science question – 13 minutes Occasionally the linked science question will be covered first

Part A Short 1 science 1 Science 3 Short 3 Short 4 Science 4 Part B Short 2 science 2

SOE 2 – Clinical Anaesthesia - PM 10 minutes to review material in long case Long case (13 minutes) two parts: Pre-op – assessment Intra-op – Management Two clinical short cases (13 minutes) Two further examiners – six in total

Marks available – giving results 2 marks for each question/section 8 question SOE1 and 4 section/questions SOE2 X 2 examiners = 48 marks No fixed pass mark – pass mark assessed for each day using Borderline regression (BLR) and examiner review of borderline candidate scores. Sense check using Hofstee Results on fixed date using pass-fail list – followed by result letters.

Standard setting process The borderline regression method will be the primary standard setting method A sense check will be carried out using the Hofstee method Examiner board review borderline candidates The ultimate responsibility for setting the pass mark will reside with the chair and board of examiners. In all cases, it is essential that the process is transparent and clearly recorded

marking and global scoring The marking sections will be split across four marking sheets: SOE1 - Part A (16 marks), SOE1 - Part B (16 marks), SOE2 - Long case (8 marks), SOE2 - Short cases (8 marks), Therefore, there will be four global scores allocated for each candidate.

Global scoring Descriptors Global rating Descriptor Excellent [5] Excellent knowledge, application and confidence. I.e. worthy of a prize. Clear Pass [4] Very good knowledge. Any minor omissions did not detract from the overall performance. Examiner more than satisfied that candidate was proficient across topics covered. Pass [3] Demonstrated an adequate level of knowledge and understanding. Some omissions and inaccuracies occurred in their performance. Some evidence of flexibility of approach with acceptable interaction with examiners. Safe to progress. Borderline [2] Patchy knowledge overall. Demonstrated only just enough level of essential skill and understanding. However, omissions and inaccuracies occurred in their performance. Often formulaic in approach and at times hesitated or prompted to answer. Examiner undecided whether to pass or fail candidate. Fail [1] Failed to demonstrate competence with a clinical approach that is incompatible with accepted practice. Their performance may show inadequate reasoning and incorrect management or treatment likely to result in serious and untoward patient outcome. Appeared disorganised. Unsafe and unsuitable to progress.

Advantages of BLR Reliable results: BLR uses real exam data Used widely by all Medical Colleges and the GMC Identifies questions issues: BLR is an effective way of identifying problems. Poorly performing questions will be identified the cut score will be lower. The awarding of global scores for candidates is not time consuming and once familiar with the global score descriptors it will become an intuitive process. Regular training and benchmarking exercises will be given.

Contact the examinations department Any questions? Contact the examinations department 020 7092 1520 exams@rcoa.ac.uk