1. Assessment Literacy Series 1 -Module 4- Scoring Keys and Rubrics

2. Participants will: 1. Develop scoring keys for all multiple choice items outlined with in the blueprint. 2. Develop scoring rubrics for constructed response items/tasks that reflect a performance continuum. 2 Objectives

3. Participants may wish to reference the following: Guides Handout #6 – Scoring Key Example Handout #7 – Rubric Examples Templates Template #5-Scoring Key-Rubric Stuff  Performance Task Framework 3 Helpful Tools

4. 4 Outline of Module 4 Module 4: Scoring Keys and Rubrics Scoring Key for MC Rubrics for Constructed Response Sample Answers Scoring Criteria Process Steps

5. 5 Scoring Keys

6. Scoring Keys typically contain elements such as the following: Performance measure name or unique identifier Grade/Course Administration timeframe (e.g., fall, mid-term, final examination, spring, etc.) Item tag and type Maximum points possible Correct answer (MC) or rubric with sample answers or anchor papers (SCR & ECR) 6

7. Scoring Key Example [Handout #6] 7 Assessment NameGrade/CourseAdministration UAI (Unique Assessment Identifier) Algebra I End-of-CourseAlgebra IPost-test (Spring) 01.003.1112 Item # Item TagItem Type Point Value Answer 1 0001.MTH.ALGI.POST.MC-LV1-8EE2MC 1C 20002.MTH.ALGI.POST.MC-LV1-ACED1 MC 1B 30003.MTH.ALGI.POST.MC-LV1-8EE1 MC 1C 40004.MTH.ALGI.POST.MC-LV1-ASSE1 MC 1D 50005.MTH.ALGI.POST.MC-LV2-ASSE1 MC 1A 60006.MTH.ALGI.POST.MC-LV2-7RP3 MC 1C 70007.MTH.ALGI.POST.MC-LV2-ACED1 MC 1D 80008.MTH.ALGI.POST.MC-LV2-ACED1 MC 1D 90009.MTH.ALGI.POST.MC-LV1-ACED1 MC 1B 100010.MTH.ALGI.POST.MC-LV1-AREI3 MC 1A 110011.MTH.ALGI.POST.MC-LV1-FIF1 MC 1D 120012.MTH.ALGI.POST.MC-LV2-ACED1 MC 1A 130013.MTH.ALGI.POST.ECR-LV2-NRN2 SCR 2See Scoring Rubric 140014.MTH.ALGI.POST.ECR-LV2-FIF1 ECR 4See Scoring Rubric

8. Scoring Keys Scoring keys for MC items: Answers within the key must represent a single, correct response. Answers should be validated once the key is developed to avoid human error. Validating answers should be done prior to form review. Items changed during the review stage must be revalidated to ensure the scoring key is correct. 8

9. Process Steps [Template #4] 9 1. Enter the assessment information at the top of the Scoring Key. 2. Record the single, correct answer during item development. For SCR and ECR items/tasks, the scoring rubrics should be referenced in the answer column and put in the correct rubric table on the Rubric Template. 3. Record the item number, item tag, item type, and point value. 4. Record the MC answers in the answer column. For each CR item, include the general scoring rubric and sample response for each point value. 5. Repeat Steps 1-4 until all items/tasks on the blueprint are reflected within the Scoring Key.

10. QA Checklist All items/tasks articulated on the blueprint are represented within the Scoring Key. MC items have been validated to ensure only one correct answer among the possible options provided exists. MC answers do not create a discernible pattern. MC answers are “balanced” among the possible options. Scoring Key answers are revalidated after the final operational form reviews are complete. 10

11. 11 Scoring Rubrics

12. Holistic vs. Analytic Rubric Scoring Holistic Scoring Provides a single score based on an overall determination of the student’s performance Assesses a student’s response as a whole for the overall quality Most difficult to calibrate different raters Analytic Scoring Identifies and assesses specific aspects of a response Multiple dimension scores are assigned Provides a logical combination of subscores to the overall assigned score 12

13. Rubric Scoring Considerations Describe whether spelling and/or grammar will impact the final score. Avoid using words like “many”, “some”, and “few” without adding numeric descriptors to quantify these terms. Avoid using words that are subjective, such as “creativity” or “effort”. Avoid subjective adjectives such as “excellent” or “inadequate”. 13

14. SCR Rubric Example [Handout #7] 14 General Scoring Rubric 2 points The response gives evidence of a complete understanding of the problem. It is fully developed and clearly communicated. All parts of the problem are complete. There are no errors. 1 point The response gives evidence of a reasonable approach but also indicates gaps in conceptual understanding. Parts of the problem may be missing. The explanation may be incomplete. 0 points There is no response, or the work is completely incorrect or irrelevant.

15. SCR Rubric Example [Handout #7] 15 Sample Response: “In two complete sentences, explain why people should help save the rainforests.” 2 points The student’s response is written in complete sentences and contains two valid reasons for saving the rainforest. “People must save the rainforest to save the animals’ homes. People need to save the rainforest because we get ingredients for many medicines from there.” 1 point The student’s response contains only one reason. “People should save the rainforest because it is important and because people and animals need it.”

16. Rubrics for ECR Tasks Create content-based descriptions of the expected answer for each level of performance on the rubric. Provide an example of a fully complete/correct response along with examples of partially correct responses. Reference the item expectations in the rubric. Make the rubric as clear and concise as possible so that other scorers would assign exact/adjacent scores to the performance/work under observation. 16

17. ECR Rubric Example [Handout #7] 17 General Scoring Rubric 4 points The response provides all aspects of a complete interpretation and/or a correct solution. The response thoroughly addresses the points relevant to the concept or task. It provides strong evidence that information, reasoning, and conclusions have a definite logical relationship. It is clearly focused and organized, showing relevance to the concept, task, or solution process. 3 points The response provides the essential elements of an interpretation and/or a solution. It addresses the points relevant to the concept or task. It provides ample evidence that information, reasoning, and conclusions have a logical relationship. It is focused and organized, showing relevance to the concept, task, or solution process. 2 points The response provides a partial interpretation and/or solution. It somewhat addresses the points relevant to the concept or task. It provides some evidence that information, reasoning, and conclusions have a relationship. It is relevant to the concept and/or task, but there are gaps in focus and organization. 1 point The response provides an unclear, inaccurate interpretation and/or solution. It fails to address or omits significant aspects of the concept or task. It provides unrelated or unclear evidence that information, reasoning, and conclusions have a relationship. There is little evidence of focus or organization relevant to the concept, task, and/or solution process. 0 points The response does not meet the criteria required to earn one point. The student may have written on a different topic or written "I don't know."

18. ECR Rubric Example [Handout #7] 18 Sample Response: “List the steps of the Scientific Method. Briefly explain each one.” 4 points 1.Ask a Question- Ask a question about something that you observe: How, What, When, Who, Which, Why, or Where? 2.Do Background Research- Use library and Internet research to help you find the best way to do things. 3.Construct a Hypothesis- Make an educated guess about how things work. 4.Test Your Hypothesis- Do an experiment. 5.Analyze Your Data and Draw a Conclusion- Collect your measurements and analyze them to see if your hypothesis is true or false. 6.Communicate Your Results- Publish a final report in a scientific journal or by presenting the results on a poster. 3 points 1.Ask a Question 2.Do Background Research-Use library and Internet research. 3.Construct a Hypothesis- An educated guess about how things work. 4.Test Your Hypothesis- Do an experiment. 5.Analyze Your Data and Draw a Conclusion 6.Communicate Your Results 2 points 1.Ask a Question 2.Do Background Research 3.Construct a Hypothesis 4.Test Your Hypothesis 5.Analyze Your Data and Draw a Conclusion 6.Communicate Your Results 1 point Ask a Question, Hypothesis, Do an Experiment, Analyze Your Data 0 points “I don’t know.”

19. Process Steps [Template #4] 19 1. Create the item/task description for the student. 2. Using a “generic” rubric, begin by modifying the language using specific criteria expected in the response to award the maximum number of points. 3. Next, determine how much the response can deviate from “fully correct” in order to earn the next (lower) point value. [Continue until the full range of possible scores is described] 4. Using the “sample” rubric, create an example of a correct or possible answer for each level in the rubric. 5. In review, ensure the item/task description for the student, the scoring rubric, and the sample rubric are aligned.

20. QA Checklist CR items/tasks have scoring rubrics that reflect a performance continuum. CR items/tasks include sample responses for each level of performance. CR scoring rubrics are clear and concise. CR scoring rubrics include all dimensions (aspects) of the tasks presented to the students. CR scoring rubrics avoid including non-cognitive (motivation, timeliness, etc.) or content irrelevant attributes. 20

21. Summary Module 4: Scoring Keys & Rubrics Developed a scoring key and rubrics for all items. Next Steps Module 5: Operational Forms & Administrative Guidelines Given the items/tasks developed, create an operational form with applicable administrative guidelines. 21 Summary & Next Steps