1. Rubrics and Other Scoring Methods Sue Brookhart January 26, 2015

2. Introductions Sue Brookhart, Ph.D. Juliette Lyons-Thomas, Ph.D. (Fellow, Regents Research Fund) 2

3. Webinar Norms All phones will be placed on mute If you have a question, you can type into the chat box, and your question will be addressed during a break  The chat box icon is located at the top right hand corner of your screen (remember to direct your chat to “Everyone”) At the end of the webinar, you will be asked to fill out a survey based on your experience today 3

4. Learning Outcomes Participants will learn to create (or select and adapt rubrics) that match intended learning outcomes. Participants will learn to check for consistency among scorers. Participants will learn to use checklists, rating scales, and point schemes for special purposes. 4

5. Performance Assessment Task(s) or assignment must tap the student learning outcome  Knowledge  Skills  Level of Thinking Rubrics or other scoring scheme must accurately describe student performance along a continuum of quality 5

6. Criteria Important for all scoring methods  Rubrics, Checklists, Rating Scales, Point Schemes These are what connect student performance to the learning outcomes they indicate 6

7. RubricsCriteria and Performance Level Descriptions Criteria: What qualities need to be in the work? Performance level descriptions: How well does the work exhibit these qualities? 7

8. Example – Choral Music Rubric ABCDF AccuracyAlways on pitch Rhythms correct Only a few incorrect notes Mostly accurate rhythm, stable beat Usually on pitch Some minor rhythm problems, generally a steady beat Rarely on pitch, many incorrect notes Some rhythmic errors and changes in beat Not on pitch Many rhythmic errors and changes in beat TextFully memorized Always clear with precise sounding consonants Mostly memorized Mostly clear with attempted ending consonants Some memorization Usually clear but missing consonants Little memorization Unclear or mumbled No evidence of memorization Not clear Part WorkKnows all parts Well balanced sound (both parts heard equally) Knows most parts Mostly balanced sound One part not well known Adequate balance between parts Both parts not well known Parts not balanced Parts mostly unknown Not able to hear two parts Tone QualityClear and projected Even tone between head & chest voice Consistent and effective breath support Mostly clear & projected Some difference in tone between head & chest voice Breath support effective most of the time Beginning to be clear & projected Much difference in tone between head & chest voice More breath support needed Unclear and unsupported (lack of projection), or a harsh tone in many sections Weak (no projection) Tone harsh, unsupported, and/or uncontrolled InterpretationConsistent and correct use of dynamics and expression Effective use of dynamics and expression Some dynamic variation Adequate expression Very few dynamics Little expression No dynamics No expression 8

9. A Counter-Example - What happens when bad rubrics are applied to good work? 9

10. Student A – 7 th grade inclusion 10

11. 11

12. Student B – 7 th grade inclusion 12

13. 13

14. Sometimes rubrics can be very concrete Example – third grade Concepts of Multiplication poster Present a multiplication problem, a repeated addition problem, a picture with equal groups, and an array for the same multiplication fact Poster 14

15. 15

16. Example 16

17. 17

18. Example 18

19. More Complex Rubrics for More Complex Tasks Several criteria  Analytic rubrics – criteria considered one at a time  Holistic rubrics – criteria considered simultaneously Criteria should be qualities that work should exhibit (e.g., position is supported, steps are explained with mathematical vocabulary, cites appropriate sources) Criteria should NOT be about directions (e.g., has 4 sentences, cites 3 sources) 19

20. Example 20 210 Understanding the Problem Complete understanding of the problem Part of the problem misunderstood or misinterpreted Complete misunderstanding of the problem Planning a Solution Plan could have led to a correct solution if implemented properly Partially correct plan based on part of the problem being interpreted correctly No attempt, or totally inappropriate plan Getting an Answer Correct answer and correct label for the answer Copying error, computational error, partial answer for a problem with multiple answers No answer, or wrong answer based on an inappropriate plan (NCTM, 1987, 1991)

21. Example Jan's Snack Shop has 3 flavors of ice cream—vanilla, chocolate, and strawberry. The ice cream can be served in a dish, a sugar cone, or a regular cone. There are 9 people who choose 1 dip of ice cream in a dish, or in a sugar cone, or in a regular cone, and all of their choices are different. List or show the 9 different choices. Could another person have a choice that is different from one of these 9 choices? Why or why not? (NAEP 2003, grade 4, Block M10 #17) 21

22. Example to Score 22

23. Example to Score 23

24. Example to Score 24

25. Consider a Template 4Task is complete and correct, plus it includes an extension, exceptional insight, or some other feature that is “above and beyond” what was required or intended. 3Task is complete and correct. Response shows understanding of intended learning outcome. 2Task is partly complete and correct. Response shows some understanding of intended learning outcome. 1Task is not complete or correct. Response shows little/no understanding of intended learning outcome. 25 NOTE: DON’T use this “rubric” as is – it’s a template. It lacks CRITERIA. Replace the “boilerplate” language with criteria specific to your learning outcome of interest.

26. Example – 6 th ELA – Writing a well-structured paragraph in response to a prompt Direct instruction and practice, four weeks Learning outcome = can write a paragraph that  Answers the prompt question  Has a focus/topic sentence  Has detail sentences that relate to the focus  Has an effective ending sentence 26

27. Example – 6 th grade writing 4Task has a focus sentence, details, and conclusion, plus it includes a particularly compelling response to the prompt, for example with especially vivid details. 3Task is has a focus sentence, details, and conclusion. 2Task is has some attempt at a focus sentence, details, and conclusion. 1Task does not have a focus sentence, details, and conclusion. 27

28. Example – 6 th grade writing 28

29. Example – 6 th grade writing 29

30. For Simple Tasks 3Task is complete and correct. Response shows understanding of intended learning outcome. 2Task is partly complete and correct. Response shows some understanding of intended learning outcome. 1Task is not complete or correct. Response shows little/no understanding of intended learning outcome. 30 NOTE: DON’T use this “rubric” as is – it’s a template. It lacks CRITERIA. Replace the “boilerplate” language with criteria specific to your learning outcome of interest.

31. Kindergarten Example Making Equal Sets 3Elements of each set are organized in a single line. There is a one-to-one correspondence between elements of each set. 2There is an attempt at organizing elements of each set and showing correspondence between elements of each set. These attempts are not entirely successful. 1Elements of each set are not organized. No correspondence is shown between elements of each set. 31

32. Kindergarten Example Making Equal Sets 32 AB

33. Ensuring Scoring Consistency Use a clear scoring guide (e.g., rubric) Score one question/task at a time Score content separately from other factors (e.g., grammar, spelling) Score questions/tasks anonymously 33

34. Ensuring Scoring Consistency Select a few papers (questions, tasks) from different classes Double-score this set of papers (questions, tasks) 34

35. A Tale of 8 th Grade ELA Portfolios 35

36. Checklists Most useful for self or peer assessment of following directions for the assignment Useful for basic skills (e.g., capital and period for each sentence) or concrete behaviors (e.g., checklist for washing hands) and procedures (e.g., making a wet mount slide). 36

37. Example – 3 rd grade learning support 37

38. 38

39. Example – Checklist for preparing a wet mount slide  Assemble materials (slides, cover slips, pipet, sample and any solutions or chemicals)  Place a drop of the sample or staining solution in the center of a clean slide  Add cells to be viewed under the microscope (unless they are already in the sample)  Angle cover slip over the sample with one edge touching the slide  Lower cover slip gently, to avoid bubbles in the solution  Remove any excess solution by touching a paper towel to the edge of the cover slip  Place slide on the stage of a microscope and view at low power to check slide contents 39

40. Checklists and Scoring Typically, checklists are NOT scored, but used for feedback. When a score is needed for a procedure that can be governed by a checklist, it is possible to use the checklist with 1-0 scoring (check = 1 point, no check = 0 points). Note that using a checklist assumes that the only measure of “quality” is that something was done – not how well – and this is why rubrics are usually preferred for scoring. 40

41. Rating Scales Rating scales are useful for behavior, learning skills, and citizenship.  Frequency scales (e.g., “always, frequently, sometimes, never”) are usually better than evaluative scales (e.g., “excellent, good, fair, poor”) Rating scales are useful for measuring attitudes, interests, and other survey-type questions. Not recommended for scoring student school work – can be useful for self-assessment 41

42. Example High school Chemistry 42

43. Point Schemes Best used to score brief, focused constructed response test questions  Essay questions  Show-the-work math problems Need a new point scheme for every question Cannot share with students 43

44. Example Fill in the chart below with the name of each of the three branches of government and the main purpose of each branch. Total possible points = 6 1 point each for naming Executive, Legislative, & Judicial 1 point each for main purpose, at minimum: Enforces laws (Exec), Makes laws (Legislative), Interprets laws (Judicial) 44 Brand of GovernmentMain Purpose

45. Example [NAEP, Grade 8, 2005; Block S13 #10] Some scientists think that the Earth's climate is getting warmer. If these scientists are correct and the Earth keeps getting warmer for the next 50 years, what will happen to the oceans? Explain why this would happen. [space given for answer] If these scientists are correct, what things about the Earth's weather will change? Explain why this would happen. [space given for answer] If these scientists are correct, what will happen to plants? Explain why this would happen. [space given for answer] Scoring: Response consists of six correct parts: a prediction and an explanation for each of oceans, weather, and plants. 45

46. Summary Use rubrics to match LEARNING OUTCOMES, not directions. Use other scoring methods for special purposes. Check consistency for any scoring that has consequences for students, teachers, or schools 46

47. Thank You The slides and a video of this webinar will be posted at https://www.engageny.org/resource/teaching- core-assessment-literacy-series-materials https://www.engageny.org/resource/teaching- core-assessment-literacy-series-materials Please provide feedback on your webinar experience, or suggestions for future topics, by filling out this survey: https://www.surveymonkey.com/s/rubricswebi nar https://www.surveymonkey.com/s/rubricswebi nar 47