Presentation on theme: "Computerised Free-text Assessment of Key Feature Problems Adrian Molyneux Steve Capey School of Medicine."— Presentation transcript:
Computerised Free-text Assessment of Key Feature Problems Adrian Molyneux Steve Capey School of Medicine
Project aims Fulfil student requests for more formative assessment Fulfil need for practice of summative exam formats Provide quality feedback Achieve this without a heavy marking load placed on staff
Key-feature problems (KFPs) …an innovative assessment format that tests clinical decision making skills by focussing only on the critical steps - or 'key features' - of each decision, thereby increasing the number of problems that can be answered in an examination. - Liz Farmer, Royal Australian College of GPs
SCENARIO - Jennifer Evans, a 38-year-old statistician, presents to the accident and emergency department with a two hour history of sudden onset of leftsided pleuritic chest pain shortness of breath and an episode of haemoptysis. She had returned from a conference in Malaga on a flight 48 hours earlier. On examination she was found to be slightly febrile, tachycardic and in atrial fibrillation. Her blood pressure is 98/64mmHg. Examination of the chest showed decreased expansion on the left and localised decreasing breath sounds with a pleural rub. A plain radiograph of the chest shows a very small trace of fluid in the left costophrenic angle to the lungs are clear. A presumptive diagnosis of pulmonary embolism is made.
Example questions from scenario 1. What initial investigations should be requested? Name only two (2) 2. Long distance travel is considered a minor risk factor for venous thromboembolism – what are the major risk factors? Name only four (4) 3. What is the mechanism of action of heparin? 4. Shortly after the heparin was commenced, Jennifer was started on warfarin. What endogenous compound does warfarin inhibit to produce its anticoagulant effect? List only one (1) 5. How is warfarin therapy monitored?
Research questions How do we train a computer system to mark an open-ended freetext response? How to deliver the assessment How to feed back the results to students How to author succinct KFPs Can we develop a method for the most efficient creation of a computer-marked KFP?
Methods Generate 10 KFPs and marking guidelines Generate computer markscheme Build web interface Recruit students and run exam Computer-mark responses Tutor-mark responses Refine markschemes (iterative process) Statistical analysis
Exam delivery system and results Exam delivery Results
Computer – Tutor comparison
Markscheme improvement Make first attempt at markscheme and run exam Examine student answers for further valid responses Include these in computer markscheme and mark a second time Examine computer markscheme for badly-performing questions and adjust (maybe loosen). Mark a third time.
Run NumberMean Mark% of Human Mark 10.939% 21.147% 32.086%
Conclusions – practical limitations Succinct answers should be encouraged – perhaps limit text entry Expect unexpected answers from students and markscheme modification Standardisation of human marking Adherence of humans to marking guidelines Evolution of computer markschemes to cover the whole range of student answers
Conclusions - positives Computer marking ideal for questions with narrow range of possible responses Reduce marking hours spent on formative questions, whilst providing quality feedback in a realistic examination format Perfect for low-stakes, on-demand assessment
Future direction of work Run exams with full year groups Refine question authoring Evaluate other freetext question formats currently used within the medical school Investigate the possibility of distributing tests asynchronously throughout the curriculum