1. Developing Electronic Marking and Management Solutions Brian Carbarns International Business Development Manager, DRS Developing Electronic Marking and Management Solutions Brian Carbarns International Business Development Manager, DRS EALTA CONFERENCE MAY 2006

2. Summary of presentation Presentation will cover:  an update on the DRS programme;  validity, reliability and bias in assessment;  quality control;  value-added worth of e-Marker®;  further work. Presentation will cover:  an update on the DRS programme;  validity, reliability and bias in assessment;  quality control;  value-added worth of e-Marker®;  further work.

3. Update on DRS programme Delivery in 2005:  marking across the UK and internationally;  over 2.5 million candidates marks and 30 million item marks collected;  over 400,000 scripts scanned and marked remotely with 12.6 million items marked individually;  secure hosting centre established and providing access to marking worldwide. Delivery in 2005:  marking across the UK and internationally;  over 2.5 million candidates marks and 30 million item marks collected;  over 400,000 scripts scanned and marked remotely with 12.6 million items marked individually;  secure hosting centre established and providing access to marking worldwide.

4. Update on DRS programme Delivery to a variety of ABs:  worked with ABs of differing sizes to deliver benefits;  includes professional bodies and specialist ABs;  key benefits of modernising processes and improving speed and quality. Delivery to a variety of ABs:  worked with ABs of differing sizes to deliver benefits;  includes professional bodies and specialist ABs;  key benefits of modernising processes and improving speed and quality.

5. Update on DRS programme Plans for 2006:  extend marking internationally;  up to 4 million candidates marks to be captured with 1.5 million scanned;  review of logistics to deliver increased volumes;  improvement of the marking system to facilitate better feedback on marking standards to examiners. Plans for 2006:  extend marking internationally;  up to 4 million candidates marks to be captured with 1.5 million scanned;  review of logistics to deliver increased volumes;  improvement of the marking system to facilitate better feedback on marking standards to examiners.

6. Validity, reliability and bias Discussion is included because:  these factors determine how well an examination is viewed;  much research has been undertaken, but in the context of marking conventionally;  electronic marking brings a degree of flexibility not available previously;  some factors can be addressed through electronic marking. Discussion is included because:  these factors determine how well an examination is viewed;  much research has been undertaken, but in the context of marking conventionally;  electronic marking brings a degree of flexibility not available previously;  some factors can be addressed through electronic marking.

7. What do they mean? Meanings:  validity; Meanings:  validity; Examination is valid when the results provide information about the knowledge, skills and behaviours that the examination purports to measure.

8. What do they mean? Meanings:  validity;  reliability (between and within examiner); Meanings:  validity;  reliability (between and within examiner); Examination is reliable if results can be replicated on a subsequent occasion. Interested here in reliability in an individual examiners’ marking batch and between examiners marking the same subject.

9. What do they mean? Meanings:  validity;  reliability (between and within examiner);  bias. Meanings:  validity;  reliability (between and within examiner);  bias. Examination outcome relates only to the candidate’s performance in the examination and not to other, personal or social factors.

10. What do they mean? Meanings:  validity;  reliability (between and within examiner);  bias. Meanings:  validity;  reliability (between and within examiner);  bias. Conventionally, the balance between validity and reliability has been difficult to strike (ie MCQ on the one hand, completely free-response on the other).

11. Validity and electronic marking Electronic marking does not affect validity. However, electronic marking could improve the reliability of marking in ways that cannot be done very easily under conventional, script-based marking processes.

12. Reliability and electronic marking Examiner reliability is addressed through:  removal of clerical errors;  sending individual items to examiners, not whole scripts; Examiner reliability is addressed through:  removal of clerical errors;  sending individual items to examiners, not whole scripts; By sending individual items to examiners that do not relate to an individual candidate, marking bias is reduced as no individual candidate’s result is determined by the standards applied by a single examiner.

13. Reliability and electronic marking Examiner reliability is addressed through:  removal of clerical errors;  sending individual items to examiners, not whole scripts; Examiner reliability is addressed through:  removal of clerical errors;  sending individual items to examiners, not whole scripts; When marking complete scripts an examiner’s judgement can be affected as more scripts are marked – as the true calibre of the entry is revealed. Some evidence supports the view that markers adjust standards as marking progresses.

14. Reliability and electronic marking Examiner reliability is addressed through:  removal of clerical errors;  sending individual items to examiners, not whole scripts;  sending items to marking methods most suited to the type;  regular sampling of examiner standards against items of known standards;  double-marking of all items (shown to be one of the most effective ways of increasing reliability). Examiner reliability is addressed through:  removal of clerical errors;  sending individual items to examiners, not whole scripts;  sending items to marking methods most suited to the type;  regular sampling of examiner standards against items of known standards;  double-marking of all items (shown to be one of the most effective ways of increasing reliability).

15. Reliability and electronic marking Automated items could support improved reliability by:  keying all candidates’ answers and storing them for marking;  enabling one senior examiner to mark all unique answers to all candidates;  marking all occurrences of answers automatically. Automated items could support improved reliability by:  keying all candidates’ answers and storing them for marking;  enabling one senior examiner to mark all unique answers to all candidates;  marking all occurrences of answers automatically.

16. Automated item example

17. Reliability and electronic marking General and expert items could support improved reliability by:  maintaining consistent marking judgement over a period of time for each item;  not allowing marking to be affected by answers to other questions. General and expert items could support improved reliability by:  maintaining consistent marking judgement over a period of time for each item;  not allowing marking to be affected by answers to other questions.

18. Expert item example

19. Bias Increasing factor to take into account in examination marking:  prevent marking judgements being influenced by name, location or social factors;  candidate details physically removed or covered elsewhere;  paper methods are cumbersome and wasteful;  electronic methods only present the answers managed digitally. Increasing factor to take into account in examination marking:  prevent marking judgements being influenced by name, location or social factors;  candidate details physically removed or covered elsewhere;  paper methods are cumbersome and wasteful;  electronic methods only present the answers managed digitally.

20. Manual anonymity methods

22. Quality control in e-Marker® Quality control is maintained by the following methods:  use of seeded items where the mark for an item is already known;  adjudication, where any marking differences are subject to set business rules or senior examiner intervention. Quality control is maintained by the following methods:  use of seeded items where the mark for an item is already known;  adjudication, where any marking differences are subject to set business rules or senior examiner intervention.

23. Use of seeded items Seeded items are used in the following ways:  as part of the qualification process which an examiner or marker has to pass for each item on each new day’s marking, and  as part of the marking process where seeded items are introduced to the examiner’s or marker’s queue of items to be marked (once qualification has been completed). Seeded items are used in the following ways:  as part of the qualification process which an examiner or marker has to pass for each item on each new day’s marking, and  as part of the marking process where seeded items are introduced to the examiner’s or marker’s queue of items to be marked (once qualification has been completed).

24. Marking strategies A number of different marking strategies can be supported:  single-marking;  double-marking. This strategy also allows for adjudication where the values of the marks differ between markers and are outside tolerance. A number of different marking strategies can be supported:  single-marking;  double-marking. This strategy also allows for adjudication where the values of the marks differ between markers and are outside tolerance.

25. Adjudication Adjudication can be set to:  manual – that is a senior marker views the item and previous markers’ marks and makes a final decision;  highest or lowest of the values given by the markers, when compared;  the average of the two values given by the markers either rounded up or rounded down. Adjudication can be set to:  manual – that is a senior marker views the item and previous markers’ marks and makes a final decision;  highest or lowest of the values given by the markers, when compared;  the average of the two values given by the markers either rounded up or rounded down.

26. Valued-added worth of e-Marker® So what is the value-add?  addressing validity, reliability and bias is all about fair assessment;  conventional methods can be used but are extremely cumbersome, can only be used on a small-scale and are not flexible;  electronic methods erode the conventional paradigms and enable practical steps to be taken to address these factors that were impossible previously. So what is the value-add?  addressing validity, reliability and bias is all about fair assessment;  conventional methods can be used but are extremely cumbersome, can only be used on a small-scale and are not flexible;  electronic methods erode the conventional paradigms and enable practical steps to be taken to address these factors that were impossible previously.

27. Further work Further work can be carried out to provide empirical evidence for the discussion above. DRS hope to discuss this further at the IAEA in Singapore.

28. Conclusion e-Marker®:  brings a range of practical and economic benefits;  has a vital and important place in improving the quality of examining;  should lead to underpinning confidence in national and international assessments. e-Marker®:  brings a range of practical and economic benefits;  has a vital and important place in improving the quality of examining;  should lead to underpinning confidence in national and international assessments.