Presentation on theme: "Electronic Marking of Examinations Pete Thomas Open University"— Presentation transcript:
Electronic Marking of Examinations Pete Thomas Open University email@example.com http://mcs.open.ac.uk/eap
The Context 1.Distance education – Open University, UK 2.On-line study 3.Improving assessment 4.Enhancing the student experience 5.Computing
Distance Education: OU Style Open University (HQ) Marker Exam Centre Exam Centre Exam Centre Exam Centre Exam Centre Typical Computing course has 2 – 4 thousand students
Motivation The gap between the environment in which students learn and how they are assessed is widening … there is a radical discontinuity in their educational experience. Race et al. (1999)
Electronic Assessment Project Aims for Examinations To put the whole examination process on-line. To design an appropriate exam paper. To define the client and server functions. To investigate invigilation issues. To provide automatic marking of free text. To provide useful feedback on answers. To provide induction to the process. To provide support for examiners.
Marking Exam Scripts The objects in the examination system: –Questions and sub-questions –Specimen (sample) solutions –Mark schemes –Rubrics (rules) –Student answers Marking consists of –Comparing an answer with a specimen solution and assigning a mark –Applying the rubric
Issues Problems with questions –Incomplete, erroneous or inconsistent data –Asking for an inappropriate conclusion –Ambiguity Checking used to remove defects Errors come to light: –When candidates read the question –When examiners attempt to grade answers
Issues Problems with answers –Understanding the question –Poorly expressed ideas –Lack of knowledge –Poor language skills (spelling) –Use of abbreviations of own devising –Lack of typing skills Actions to address the problems –Standardization between markers
Issues Problems with specimen solutions –Do not match the question (as perceived by candidates) –Numerous acceptable alternatives –Incomplete –Incorrect Actions to address problems –Expect markers to use professional judgement –Standardization between markers
Issues Problems with mark schemes –Incorrect –Inappropriate for the question as perceived by candidates –Inappropriate for the difficulty of the question Actions to address problems –Revise in the light of experience –Standardization between markers
Electronic Marking A specimen solution can consist of more than one solution. Each solution can have a different mark. A solution can be split into parts with each part having its own marks. In general, an electronic solution can be composed of a set of alternatives and each alternative is composed of parts.
Solution Representation Suppose that the solution to a (trivial) question consists of the following three phrases: range memory locations, bounds registers, address outside range exception AND (3, 6.0) bounds registersrange memory locationsaddress outside range exception storage protection keys bounds registers OR (2, 1, 3.0) 1.5 3.0 Synonyms? Thesaurus
First Experiment 20 student scripts from a conventional written exam 3 independent human markers Question 712 (a)12 (b)12 (c)12 (d)12 TotalOver all Marks allocated 482822024 Marker averages 126.96.36.199.81.1313.014.6 Tool averages 1.755.50.984.80.812.414.2
Second Experiment Electronic mock examination (3 hours) Full exam scripts (10 Part 1 and 3 Part 2 questions) 11 students 3 independent human markers Automatic ScoreMarkers mean score Mean44.1856.25 St.Dev.8.2810.09 Automatic score is lower and there is a smaller standard deviation
Deficiencies DescriptionNumber Number of significant spelling errors in answers8 Number of deficient solution trees (out of 39)15 Number of thesaurus deficiencies (synonyms)12 Number of deficient specimen solutions2 Number of lexical deficiencies (abbreviations)1 Number of language parsing errors (software errors)2 Number of deficient questions2
Deficiency Reduction Examined one script taken at random and corrected/amended the solution trees and the thesaurus. Repeated the process with 3 more scripts. Automatic MarkerHuman Marker Mean 01234 Mean 44.246.050.552.853.255.85
Marking Diagrams (April 03) What features should a drawing tool provide? How familiar should students be with the tool prior to the examination? How should the tool be provided to students in order to be used under examination conditions? How should a diagram be represented for transmission to the server? How should a diagram be represented for grading purposes? How to grade a diagram?
Exam Question Use the drawing tool to draw a diagram that illustrates how the data hazard inherent in the execution of the pair of instructions ADD R2, R3, R1 SUB R1, R5, R4 by a 4-stage pipeline, can be overcome.