1. Blind double marking Aim: to raise some issues contingent on the introduction of blind double marking of essay exams

2. BChD ( Bachelor of Dental Surgery ) O 5 year course O non-modular (curriculum largely determined by General Dental Council) O degree exams in 1st, 2nd, 3rd and 5th year plus ‘Progress to Finals’

3. The written exam in ‘Finals’ O takes place in June of 5th year O = 40% of marks for Finals O 2 three-hour papers O 4 compulsory essay questions in each paper O each essay marked out of 25 O traditionally double-marked O first double-marked ‘blind’ in 1999

4. Double-marking vs. blind double-marking O traditionally 2nd marker agreed with 1st mark 70% - 80% of time and disagreements nearly always only 1 - 2 marks (out of 25) O marking blind, absolute agreement dropped to between 48% and 17% with, in one question, 35% of marks being >2 marks apart

5. Extent of agreement between markers in 1999 final written exam

6. Analysis of markers’ consensus process (all questions, 48 students)

7. Inappropriate use of correlation statistics 5 Correlation statistics show association, not agreement e.g. correlation would be perfect if all marks given by 2nd marker are exactly 10 points higher than those given by 1st marker 4 therefore use kappa, which measures absolute agreement with reference to the frequency with which marks would be expected to agree by chance Note: agreement can be evaluated as: Poor (<0.20), Fair (0.21 - 0.40), Moderate (0.41 - 0.60), Good (0.61 - 0.80) or Very Good (0.81 - 1.00)

8. Standard ways of improving inter-marker reliability 4 re-visit assessment criteria/marking scheme 4 have standardisation meetings with markers 4 downgrade importance of essays in assessment and use more objective methods 4 continue to monitor marking S

9. Action taken between June and November 1999 4 Issues raised and discussed at Undergraduate Dental Education Committee and in Dental School newsletter ‘Teaching Quality Matters’ 4 Assessment criteria reviewed, re-formatted and published in Nathan Bodington 4 Markers for Operative Dentistry prize exam (December 1999) given verbal briefing about assessment criteria S Result = a marked improvement in reliability !! (but….)