1. 1 Scoring Provincial Large-Scale Assessments María Elena Oliveri, University of British Columbia Britta Gundersen-Bryden, British Columbia Ministry of Education Kadriye Ercikan, University of British Columbia 1

2. Objectives Describe and Discuss –Five steps used to score provincial large-scale assessments (LSAs) –Advantages and challenges associated with diverse scoring models (e.g., centralized versus decentralized) –Lessons learned in British Columbia when switching from a centralized to a decentralized scoring model

3. 3 LSAs are administered to collect data to evaluate efficacy of school systems, guide policy-making make decisions regarding improving student learning An accurate scoring process examined in relation to the purposes of the test, and the decisions the assessment data are intended to inform are key to obtaining useful data from these assessments Scoring Provincial Large-Scale Assessments

4. Accuracy in Scoring Essential to having accurate & meaningful scores is the degree to which scoring rubrics: –(1) appropriately and accurately identify relevant aspects of responses as evidence of student performance, –(2) are accurately implemented –(3) are consistently applied across examinees Uniformity in scoring LSAs is central to achieving comparability of students’ responses: ensure differences in results are attributable to differences among examinees’ performance rather than due to biases introduced by the use of differing scoring procedures A five-step process is typically used 4

5. Step One: “Test Design Stage” Design of test specifications –That match the learning outcomes or construct(s) assessed –Include particular weights & number of items needed to assess each intended construct 5

6. Step Two: “Scoring Open-Response Items” Decide which model to use to score open-response items: Centralized models are directly supervised by provincial Ministries or Departments of Education in a central location Decentralized models often take place across several locations & are performed by a considerably greater number of teachers; used for scoring medium to low-stakes LSAs 6

7. Step Three: “Preparing Training Materials” Identify common tools to train scorers, including: –Exemplars of students’ work demonstrating each of the scale points in the scoring rubric –Illustrate potential biases arising in the scoring process (e.g., differences in scores given to hand- vs. type-written essays) 7

8. Step Four: “Training of Scorers” Training occurs prior to scoring and can recur during the session itself, especially if the session spans more than one day A “train the trainer” approach is often used –a small cadre of more experienced team leaders are trained first, then they train other scorers who will actually score the responses Team leaders often make final judgement calls on the assignment of scores differing from exemplars Serves to reinforce common standards and consistency in the assignment of scores and leads to having fair and accurate scores 8

9. Step Five: “Monitoring Scores” Includes checks for inter-marker reliability, wherein a sample of papers is re-scored to check consistency in scoring across raters May serve as re-training or “re-calibration” activity, with raters discussing scores and rationales for their scoring procedures 9

10. The Foundation Skills Assessment The Foundation Skills Assessment (FSA) will be used as a case study to illustrate advantages and challenges associated with switching from a centralized and decentralized scoring model The FSA assess Grade 4 and 7 students’ skills in reading, writing and numeracy Several changes made to the FSA in 2008 as a response to stakeholders’ demands to have more meaningful LSAs that informed classroom practice 10

11. Changes to the FSA Earlier administration – from May to February Online administration of closed-response sections Parents or guardians received child’s open-response test portions & summary statement of reading, writing and numeracy skills Scoring model changed from a centralized to a decentralized model Ministry held “train the trainer” workshops to prepare school district personnel to organize and conduct local scoring sessions School districts could decide how to conduct scoring sessions –score individually, in pairs or in groups –double-score only a few, some or all the responses 11

12. Advantages of a Decentralized Model Professional Development –A decentralized model allowed four times as many teachers to work with scoring rubrics and exemplars –Educators were able to develop a deeper understanding of provincial standards and expectations for student achievement –If scorers are educators, they may later apply knowledge of rubrics and exemplars in their classroom practice and school environments and consider the performance of their own students in a broader provincial context 12

13. Advantages of a Decentralized Model Earlier return of test results & earlier provision of feedback to teachers, students and the school –More immediate feedback may lead to improving learning and guiding student teaching –Data informs teachers about students’ strengths and areas of improvement in relation to provincial standards –May be helpful in writing school plans and targeting the areas upon which particular schools may focus 13

14. Challenges of a Decentralized Scoring Model Increased difficulty associated with –Less time allocated to implementing cross-check procedures –Decreased standardization of scoring instructions given to raters –Increased costs (higher number of teachers scoring) –Reduced training time 14

15. Potential Solutions Provide teachers with adequate training time –e.g., one to two days of training prior to scoring the assessments Increase discussion among teachers, which may involve reviewing exemplars falling in between scale points in the rubric Have table leaders –e.g., teachers with prior scoring experience Re-group teachers to verify difficulties or uncertainties related to the scoring process 15

16. Final Note Closer collaboration among educators and Ministries and Departments of Education may lead to improved tests as educators bring their professional experience of how students learn in the classroom to bear on test design itself Strong alignment between the overall purposes of the test, the test design and the scoring model used may add value to score interpretation and subsequent use of assessment results 16

17. 17 Thank you