ICCS Field Trial Marking of open-ended response items Hamburg, July 2007

ICCS Marker Training, Hamburg July 2007 Recruiting Markers Preferable to use people with teaching experience (at the Grade level if possible) Preferable to use people who know and understand CCE Current teachers can be used (as part of professional development) Need to be decisive, cooperarative (know when to speak and when not to) and sympathetic to assessment (can be a danger with teachers).

ICCS Marker Training, Hamburg July 2007 Recruiting Markers Approximately 3000 items to mark (including double marking) At 15 items per hour (slowish) = 200 marker hours 8 markers would take approximately 1 full week (25 hours) of marking time (including training time). Consider employing markers with a view to them becoming ‘leading markers’ in the main study.

ICCS Marker Training, Hamburg July 2007 Leading Markers Leading markers can be used primarily to resolve marking problems – e.g. ‘What should I give this?’… in the trial marking the leading marker may well be you.

ICCS Marker Training, Hamburg July 2007 Training Procedures 1.Brief introduction of project including reference to assessment framework 2.Item-by-item training with discussion of item and scoring guide followed by practice marking of student responses – make sure that the scoring system is clear for each item 3.Following is a suggested item training and booklet marking order:

ICCS Marker Training, Hamburg July 2007 TrainMark 1 CI2PDO1 CI2ET01 Booklet 6 2 CI2VSO1 CI2RRO1 Booklet 3 3 CI2WFO1 CI2WFO2 Booklet 4 4 CI2BIO1 CI2MBO1 Booklet 2 5 Mark Booklet 1 Booklet 5 Suggested Training and Marking Order

ICCS Marker Training, Hamburg July 2007 Training and Marking Order Cluster Booklet CI2PDO1 2 CI2BIO1 2 CI2MBO1 3 CI2ET01 4 CI2VSO1 4 CI2RRO1 5 CI2WFO1 5 CI2WFO2 Suggested Training and Marking Order

ICCS Marker Training, Hamburg July 2007 Reliability Marking/Quality Control Outstanding marker training Leading marker to act as final adjudicator of queries (e.g. ‘What score do I give this?’ Double marking of booklets 1,2 & 3 as outlined by DPC

ICCS Marker Training, Hamburg July 2007 Item Codes For the open-response items the permissible codes will be: 0,1,2 – conventional score categories 4 – unaccounted responses (additional conceptual category worth consideration) [optional] 9 – Missing [the item has remained untouched at all by the student] 8 – Not scored or out of range

ICCS Marker Training, Hamburg July 2007 Unaccounted responses & refining scoring guides for main study Use of Code 4 (Unaccounted responses) –Please record the allocation of Code 4 response and communicate this to the ISC (we will provide a form). Please also record any other observations/suggestions for the open-ended response items to the ISC for use in final item & scoring guide revisions.

ICCS Marker Training, Hamburg July 2007 The ‘missing’ code The missing code (9) is used in two ways 1.The missing code is not included in the item calibration because we cannot be certain that the student saw/had time to complete the item. (It is scored as 0 for the student on that item) 2.The missing code can be used to determine items that are ‘not reached’ which can help confirm the appropriateness of the test length.

ICCS Marker Training, Hamburg July 2007 Not Scored or out of range items Codes written in the box to the left of the ‘Q’. Not scored or out of range items will be picked up only at data entry. Minimise/avoid not scored (and out of range) items by having markers re-check each booklet for the scores after it is marked. One easy way is to count the number of open-ended response items per booklet (e.g. 3) and then just count the scores in boxes for that booklet.

ICCS Marker Training, Hamburg July 2007 Not scored or out of range If a not scored or out of range item still makes it through to data entry then the code 8 should be entered by the data entry person (unless someone is there to resolve the problem at that moment) but NOT recorded on the booklet. You will need a system to resolve problems at the data entry level. The data entry people to contact a person who can read/hear the response and provide a score when there is a not scored or out of range score code. Individual systems will need to meet local circumstances (i.e. do what works for you).

ICCS Marker Training, Hamburg July 2007 Questions or comments?