1. Holistic Scoring in Social Studies Curriculum, Instruction, and Instructional Technology Team New York State Education Department Albany, NY 2005

2. NYS Education Department 2005 Contact Information Curriculum & Instruction Jo Ann Larson jlarson@mail.nysed.govlarson@mail.nysed.gov Lawrence Paska lpaska@mail.nysed.govpaska@mail.nysed.gov Phone: (518) 474-5922 State Assessment Donna Merlau dmerlau@mail.nysed.gov Gary Warren gwarren@mail.nysed.gov Phone: (518) 474-3860

3. NYS Education Department 2005 Purposes of the Regents Accountability: To measure student attainment of the Learning Standards for Social Studies at the commencement level Diagnostic: To help school districts identify those students who are in need of Academic Intervention Services (AIS)

4. NYS Education Department 2005 Goal To achieve consistent and accurate scores Note: not simply to have everyone assign the same scores, but to assign them for the right reasons.

5. NYS Education Department 2005 Purpose of This Workshop Make the rating task as simple and painless as possible

6. NYS Education Department 2005 Cast of Characters PRINCIPAL / SOCIAL STUDIES COORDINATOR Appoints a Scoring Coordinator who will manage the training and logistics of the scoring process Appoints a Trainer

7. NYS Education Department 2005 Cast of Characters SCORING COORDINATOR Manages the training and logistics of scoring Organizes the movement of papers throughout the entire scoring process Records scores on Part I answer sheets

8. NYS Education Department 2005 Cast of Characters TRAINER Responsible for the overall training session, including time management (starting on time, moving from step to step of training procedures, keeping to schedule) Responsible for the substance of scoring Leads review of: specific rubric, procedures for assigning holistic scores, each anchor paper and commentary, practice papers

9. NYS Education Department 2005 Cast of Characters TRAINER Follows the procedures and adapts them when necessary to ensure that everyone can assign valid and consistent scores Answers questions that arise over how to evaluate papers or assign scores Monitors scoring session progress; helps keeps Raters on schedule Assists the Table Leaders with back-reading

10. NYS Education Department 2005 Cast of Characters TABLE LEADERS Distributes the bundles of papers to be scored (with Support Staff help) Leads review of the generic rubric and specific rubric with reference to the task by matching evidence from the student responses to the rubric Leads review of each anchor paper and commentary

11. NYS Education Department 2005 Cast of Characters TABLE LEADERS Leads discussion until Raters feel confident to move on to actual rating; discusses any discrepancies Helps the Trainer with the substantive issues that arise during discussion (e.g., why a paper received a certain score, how to explain a distinction between score levels)

12. NYS Education Department 2005 Cast of Characters TABLE LEADERS Back-reads several papers for each scorer, to confirm that Raters assign scores consistently Serves as a third Rater to resolve scores that differ by more than one point Calculates a final score for each student’s essay Ensures that Raters fill out score sheets correctly Assembles all score sheets and training materials at end of scoring session

13. NYS Education Department 2005 Cast of Characters RATERS Scores the five practice papers without looking at the scores and commentaries provided Records own individual rating for each student’s essay on the rating sheet provided (not directly on the student’s paper/answer sheet), when the actual rating begins; the scores of each scaffold question may be recorded in the student’s examination booklet

14. NYS Education Department 2005 Cast of Characters RATERS Should not correct the student’s work by making insertions or changes of any kind Note: Each essay must be scored by two different Raters; scaffold questions only need scoring by one Rater

15. NYS Education Department 2005 Cast of Characters RATERS: QUALIFICATIONS Teachers of Grades 7-12 social studies Special Education Teachers who are knowledgeable about the GHG or USHG social studies curriculum Should have previous school, district, and/or regional training on scoring social studies essays or scaffold questions as part of the turnkey training process

16. NYS Education Department 2005 Cast of Characters SUPPORT STAFF Distributes the bundles of papers to the tables Moves the papers so that all parts and all papers are completely scored Collects the score sheets and training materials, moving them to shipping centers or a storage facility, at the end of the session

17. NYS Education Department 2005 The Rating Process 1. BEGIN Select a Table Leader. Review each scaffold question and/or essay task. Discuss possible student responses. Review and discuss the specific rubric. Discuss the nature of responses that would be expected at each score point of the rubric. Remember: This is a holistic scoring. A student does not need to achieve a “5” for each bullet in the rubric in order to be awarded a “5” on the essay. A “5” is not necessarily a perfect paper.

18. NYS Education Department 2005 The Rating Process 2. REVIEW THE SET OF ANCHOR PAPERS Read the student anchor papers individually. Starting with the first anchor paper, the group should discuss the scores that were awarded. Using the specific rubric, explain why each anchor paper earned the score that it did. Read the commentaries at the back of the packet for each of the anchor papers, to find the rationale for each paper’s score.

19. NYS Education Department 2005 The Rating Process 3. REVIEW THE PRACTICE PAPERS Read the student practice papers individually. Using the specific rubric, each Rater at the table awards a score to each practice paper without looking at the scores and commentaries provided after the papers. After each Rater has separately scored the practice papers, the Table Leader charts the scores and leads discussion on the scores that were awarded.

20. NYS Education Department 2005 The Rating Process 3. REVIEW THE PRACTICE PAPERS Where there are discrepancies, the Table Leader should ask individuals with the most discrepant scores to discuss their rationale for awarding credit. The group should continue to discuss the scores until consensus is reached. (See the commentaries at the back of the packet for the scores awarded for each practice paper.) Discuss any discrepancies between the SED commentaries and the scores awarded above.

21. NYS Education Department 2005 The Rating Process 3. REVIEW THE PRACTICE PAPERS If additional practice is required to reach scoring consensus, a sample of actual student answer papers from the test may be used.

22. NYS Education Department 2005 Setting the Scene … Provide frequent breaks. Provide incentives (e.g., water, snacks, meals). Conduct the scoring session in a pleasant environment. Check for understanding of rating procedures frequently. Review the rubric after any long-term break (e.g., a lunch period, overnight). Work toward exact agreement in awarding credit.

23. NYS Education Department 2005 Materials Needed Photocopiers Fax machines Computers and printers Paper Chart paper Paper clips and rubber bands Pens, pencils, markers, and highlighters

24. NYS Education Department 2005 Checking for Accuracy Do the Raters share a common interpretation of the rubric, or do some Raters have their own personal construct of the rubric? Are Raters applying the rubric appropriately, or are they influenced by irrelevant factors when scoring student performance? Are Raters using all of the criteria, or are they ignoring some aspects? Are Raters assigning scores reflective of the rubric?

25. NYS Education Department 2005 Other Reminders “Scoring Key and Rating Guide” may not be used until after the Uniform Statewide Admission Deadline (45 mins. after exam start time). Set aside one room as a central rating room for collecting, storing answer sheets/essay booklets, and for preparing/maintaining records, for the exam. Avoid having a teacher rate his/her own students’ responses. Provide adequate time for training and for rating.