ICCS Main Survey Marker Training The scoring guides Day 1 Hamburg, July 2008

ICCS Marker Training, Hamburg July 2008 Since the Field-Trial Reduction of OER items from 8 to 6OER –CI2MBO1 (awful) –CI2VSO1 (data acceptable BUT concerns about the overlap between cognition and values tipped this item out ahead of one of the two items dealing with money paid to the unemployed) Text revision to three items –CI2ETO1 (ambiguity removed) –CI2WFO1 and CI2WFO2 (“unemployed” explicated to prevent students providing examples of circumstances that might lead to unemployment)

ICCS Marker Training, Hamburg July 2008 Purpose of Marker Training To develop a common, shared understanding of the purpose of each item and the corresponding item codes for scoring student responses.

ICCS Marker Training, Hamburg July 2008 The Open-response item scoring guides The only true common element across countries. Crucial aspects: –the descriptions of acceptable responses for each score category –the scored sample responses The main purposes of the initial day(+) of the seminar are to confirm: 1.The conceptual description of each response category 2.The score (and justification for this score) allocated to each sample response

ICCS Marker Training, Hamburg July 2008 The Open-response item scoring guides Item-by-item, small group discussions to answer the questions: 1.Do the conceptual categories for each score make sense, is there anything “missing” ? ALL GROUPS 2.Are you confident that sample response represents the score AND can you explain to others why it does? Please record comments for those sample responses that you feel need to be moved or are confusing. BY RESPONSE CATEGORY (for some items) AND Please flag any responses that you would like to have included in a scoring training sheet.scoring training sheet. Please take notes from your group on a scoring guide

ICCS Main Survey Marker Training Marking Operations Hamburg, July 2008 “Lamb’s lettuce” is vegetarian (apparently it refers to a salad eaten often by sheep).

ICCS Marker Training, Hamburg July 2008 Recruiting Markers Preferable to use people with teaching experience (at the Grade level if possible) Preferable to use people who know and understand CCE Current teachers can be used (as part of professional development) Need to be decisive, cooperarative (know when to speak and when not to) and sympathetic to assessment (can be a danger with teachers).

ICCS Marker Training, Hamburg July 2008 Leading Markers Leading markers can be used primarily to resolve marking problems – e.g. ‘What should I give this?’… in the trial marking the leading marker may well be you.

ICCS Marker Training, Hamburg July 2008 Training Procedures 1.Brief introduction of project including reference to assessment framework 2.Item-by-item training with discussion of item and scoring guide followed by practice marking of student responses – make sure that the scoring system is clear for each item.

ICCS Marker Training, Hamburg July 2008 Considerations when using the guides Spelling, quality of expression, grammar and punctuation are not assessed. It does not matter where students write responses. HOWEVER – Civics and Citizenship Education provides an unusual set of challenges when considering the language of student responses.

ICCS Marker Training, Hamburg July 2008 Considerations when using the guides In many countries students have not had extensive formal classroom experience of CCE. Many, many students do not have the explicit CCE vocabulary to express conceptual understandings efficiently. One significant challenge in assessing written student responses is to ‘unpack’ student attempts to express conceptual understandings using their ordinary (i.e. non-technical) language.

ICCS Marker Training, Hamburg July 2008 Considerations when using the guides This challenge has two dangers that can affect marker reliability. 1.Some markers can mistakenly believe that because students do not have the language skill to express answers succinctly that all answers are correct. 2.Some markers believe that unless students use appropriate CCE language they cannot be credited with providing correct responses.

ICCS Marker Training, Hamburg July 2008 Considerations when using the guides BOTH misconceptions must be dealt with in the marker training. Student responses can only be scored on the basis of what they have written – markers should not try to infer student responses on the basis of what has not been written. To this end, the students’ choice of the words they use MUST be given careful attention.

ICCS Marker Training, Hamburg July 2008 Considerations when using the guides Many students provide ‘feel-good’ responses (expressions of value rather than understanding) that can be very well expressed but still without substance. Many students provide complex re- wordings of the question stem without answering the question – these can be deceptive as often they are eloquent.

ICCS Marker Training, Hamburg July 2008 TrainMark 1 CI2PDO1 CI2ET01 Booklet 6 2 CI2VSO1 CI2RRO1 Booklet 3 3 CI2WFO1 CI2WFO2 Booklet 4 4 CI2BIO1 CI2MBO1 Booklet 2 5 Mark Booklet 1 Booklet 5 Suggested Training and Marking Order

ICCS Marker Training, Hamburg July 2008 Training and Marking Order Cluster Booklet CI2PDO1 2 CI2BIO1 2 CI2MBO1 3 CI2ET01 4 CI2VSO1 4 CI2RRO1 5 CI2WFO1 5 CI2WFO2 Suggested Training and Marking Order

ICCS Marker Training, Hamburg July 2008 Item Codes For the open-response items the permissible codes will be: 0,1,2 – conventional score categories 9 – Missing [the item has remained untouched at all by the student] 8 – Not scored or out of range

ICCS Marker Training, Hamburg July 2008 Unaccounted responses & refining scoring guides for main study Use of Code 4 (Unaccounted responses) –Please record the allocation of Code 4 response and communicate this to the ISC (we will provide a form). Please also record any other observations/suggestions for the open-ended response items to the ISC for use in final item & scoring guide revisions.

ICCS Marker Training, Hamburg July 2008 Not Scored or out of range items Codes/scores should be written by markers in the box to the left of the ‘Q’. Not scored or out of range items will be picked up only at data entry. Minimise/avoid not scored (and out of range) items by having markers re-check each booklet for the scores after it is marked. One easy way is to count the number of open-ended response items per booklet (e.g. 3) and then just count the scores in boxes for that booklet.

ICCS Marker Training, Hamburg July 2008 Not scored or out of range If a not scored or out of range item still makes it through to data entry then the code 8 should be entered by the data entry person (unless someone is there to resolve the problem at that moment) but NOT recorded on the booklet. You will need a system to resolve problems at the data entry level. The data entry people to contact a person who can read/hear the response and provide a score when there is a not scored or out of range score code. Individual systems will need to meet local circumstances (i.e. do what works for you).

ICCS Marker Training, Hamburg July 2008 Reliability Marking/Quality Control Double marking of booklets as outlined by DPC Reliability marking provides only post-hoc information, it is more important that issues be resolved and monitored during the marking process. Outstanding marker training Leading marker to act as final adjudicator of queries (e.g. ‘What score do I give this?’

ICCS Marker Training, Hamburg July 2008 Reliability Marking/Quality Control Additional thoughts/possibilities Pre score some real student responses in your own language and use them as additional training scores… and/or as “warm-up” or “control” responses. (Good especially given the rotation of items through booklets). Have markers “flag” interesting, difficult to score responses and use them as a basis for ongoing discussions of the marking.

ICCS Marker Training, Hamburg July 2008 Item Code Question Number by Booklet Number B1B2B3B4B5B6B7 CI2PDO CI2BIO CI2ETO CI2RRO CI2WFO CI2WFO

ICCS Marker Training, Hamburg July 2008 Questions or comments?