1. Research and Theory, Introduction to the Scale or Rubric, Writing Measurement Topics, Score 3.0 and Sample Tasks
Ravalli County Curriculum Consortium
Marzano Research Laboratory

2. Research and Theory behind the Standards Rubric or Scale

3. Why are we doing this?
Typical State and National Standards Documents simply have too much content to cover effectively and with any depth
Standards lack unidimensionality
The use of formative assessment data allows for the greatest gain in student achievement
Scales or rubrics provide the best feedback to students and are the most accurate way to assess student knowledge and growth

4. Typical Standards documents cover too much content!
If you wanted to teach all of the standards in the national documents, you would have to change school from K-12 to K-22!
255 standards across 14 subject areas
3,500 benchmarks
15,500 hours of instruction needed to cover the 3,500 benchmarks (Kendall and Marzano, 2000)
9,000 hours of instruction available (Marzano, Kendall and Gaddy, 1999)

5. Typical Standards documents lack Unidimensionality
Standards documents are not written with assessment in mind
Benchmarks typically cover more than one skill or piece of knowledge
In order to assess each individual dimension, standards and benchmarks must be “unpacked” so that they measure only one thing at a time

6. The use of formative assessment data can dramatically improve student achievement
Black and Williams (1998) in a synthesis of 250 studies, concluded that the use of formative assessment data, done properly, “…does improve learning. The gains in achievement appear to be quite considerable, and as noted earlier, amongst the largest ever reported for educational interventions.”

7. Characteristic of Feedback from Classroom Assessment
Types of Feedback from Formative Assessments
Bangert-Drowns, Kulik, Kulik, & Morgan, 1991
# of studies
Characteristic of Feedback from Classroom Assessment
Percentile Gain/Loss 6
Right/wrong -3 39
Provide correct answers
8.5 30
Criteria understood by student vs. not understood 16 9
Explain 20 4
Student reassessed until correct

8. Characteristic of Feedback from Classroom Assessment
Fuchs & Fuchs 1988
# of studies
Characteristic of Feedback from Classroom Assessment
Percentile Gain/Loss 89
Displaying results graphically 26 49
Evaluation by rule
[uniform way of interpreting results of classroom assessments using a tight logic) 32

9. Why use a rubric or scale for formative assessment and to assess student progress over time?
Formative assessment using rubrics provides the best form of feedback to students
Well written and specific scales or rubrics are more accurate than a set of tests or formal assessments graded on a 100 point scale

10. In a study conducted by Marzano in 2002, a group of science teachers independently developed a 100 point scoring scheme for a test and scored it. The reliability coefficient was only .294 (out of 1.0).
The resulting scores were so variable that the test could not reliably be used to assess student knowledge and/or skills

11. The same teachers scored the same test using a 4 point rubric or scale.
The reliability coefficient was .719
This means that the resulting scores were significantly more consistent, and as a result the rubric came much closer to measuring what the student actually knows or is able to do than the test scored using the 100 point scale.

12. Using Standards Rubrics or Scales Will…
Create greater reliability in assessment of standards
Create more consistency in teaching, assessment and learning throughout a district or state
Allow for more consistent and better feedback throughout the course of learning
Allow for more focused instruction in appropriate depth
Make true assessment possible due to the creation of unidimensionality
IMPROVE LEARNING!

13. Questions? Comments? Discussion?

14. Writing Standards Scales or Rubrics

15. The Standards Scale or Rubric

16. The Scale 4
In addition to exhibiting level-3 performance, in-depth inferences and applications that go BEYOND what was taught in class 3
No major errors or omissions regarding any of the information and/or processes (SIMPLE OR COMPLEX) that were explicitly taught
The Learning Goal: what you expect the student to know and be able to do 2
No major errors or omissions regarding the SIMPLER details and processes BUT major errors or omissions regarding the more complex ideas and processes
The simpler or foundational knowledge that is necessary as a step to mastery of the Score 3.0 1
With HELP, a partial knowledge of some of the simpler and complex details and processes
Even with help, no understanding or skill demonstrated

17. 4 Easy(?) Steps
Unpack/Cut Down Standards/Organize /Write Measurement Topics
Specify expected learning by writing Score 3.0, and write sample tasks to further specify content
Identify steps towards mastery of the content by writing Score 2.0, write sample tasks to further specify content
Pilot and Use! Design Assessments, Assess student learning, Report student progress

18. Strands Broad categories within each standard
Sample strands
Math:
Number Sense
Geometry
Algebraic Concepts
Social Studies
Geography
History
Economics
Civics
Science
Life Science
Earth Science
Scientific Inquiry
Language Arts
Reading
Writing

19. Measurement Topics Categories of learning within each strand
Organize your standards into more precise categories of study (Measurement Topic) within each strand
The Measurement Topic represent the critical or essential content that you want to measure and record student progress on
You may need to leave some things out!
You will write scales or rubrics for each of the MT’s
Each MT will assess only one specific piece of information or skill for each grade level

20. Writing Measurement Topics
MRL highly recommends limiting the number of MT’s to no more than per grade level
Write each measurement topic specific enough that it can be described by only 1 or 2 elements
Combine similar or covarying elements
Covarying: as the ability in one increase, so does ability in the second
Split up statements that are not unidimensional
Unidimensional: measuring only one concept or skill
Delete content that is not essential
Delete content that is not measureable
Reduce! You can’t assess it all!

21. Other Options
Consider including “Life Skills” to assess school behaviors
Examples: Participation, Work completion, Behavior, Working in Groups
Reorganize at the High School level
Organize by class and write measurement topics for each class
Consider including a Teacher Choice rubric allowing teachers to assess something other than what was included

22. Measurement topics do not have to be assessed at every grade level!
They can disappear and reappear at various levels

23. MT’s for Ravalli County
MRL took the Montana Standards for Mathematics and wrote 33 DRAFT MT’s as a starting point
As the local content experts, the final decision on the MT’s is up to you!
Add, delete, modify as you wish, BUT…
Keep in mind the total goal of no more than per grade level!

24. Measurement Topic Review
Divide into 6 teams. Each team will take one strand
Review the list of topics. Does this list represent the areas that you want to measure student progress in? Do any MT’s need to be renamed or deleted? Are there any topics that need to be added?
Take a first cut at the grade levels for each topic. When will each topic be taught?
Complete chart with topic and grade levels
Review the work of each of the other teams and make suggestions
REMEMBER: This list is not final! Topics can and will change once you begin defining the learning goals or elements!

25. Writing Score 3.0

26. Writing Score 3.0 elements
What is a Score 3.0 element?
3.0’s represent all of content that is explicitly taught in class under a measurement topic
This is what you expect a student to know and be able to do
This represents the parts of the curriculum that are being assessed

27. General Guidelines for Writing 3.0’s
3.0 elements:
have a procedural or declarative focus
Are unpacked – measure only one piece of knowledge or skill
Are learning goals, not tasks
Are measureable!
Are specific!
Are differentiated by grade level

28. Learning Goals vs. Learning Tasks
Score 3.0 elements are written in the form of learning goals, not learning tasks or activities
Learning goals specify the content or skills that the student is expected to know or do
Learning tasks or activities are the classroom activities that a student does in order to accomplish or be assessed on the learning goal.
Examples
Goal
Task or activity
The student describes the division of the country during the Civil War
The student creates a map showing the states that formed the Confederacy and those that stayed in the United States

29. Measurability 3.0 elements must be measureable Measureable:
Observable or Quantifiable
Makes predictions about reading
The teacher can observe a student doing this
Performs 10 push-ups with proper form
The teacher can count the number of pushup and observe proper form
Not Measureable
The student enjoys independent reading

30. Specificity of Elements
Elements must be written so that a teacher who is unfamiliar with the process and unfamiliar with the curriculum, knows what the student is expected to know or be able to do and can accurately assess a student!
Avoid terms that can’t be easily defined or allow for subjective interpretation
Terms can be defined within the element
Examples can also be used to provide specificity
Examples
Non-Specific
Specific
Monitors own reading for comprehension
Employs self-monitoring strategies while reading (e.g., predicting, confirming, rereading)

31. Differentiation of Elements
Elements must be written to show a increasing level of complexity as the performance level increases
If the element remains the same through several performance levels, examples must be provided to differentiate each level
For Example – ELA will frequently assess the same reading strategies (e.g., inference) at multiple performance levels, but… the complexity of the expectation or text will change. Give examples of the type of inference, or level appropriate text to differentiate

32. Unidimensional Rubrics
For each measurement topic, we are going to write one (maybe 2 if you beg!) Score 3.0 element
Each element should measure only one individual piece of knowledge or skill
If the topic requires more than one (or 2) elements, then break the topic into two or more new topics to maintain unidimensionality
Remember, you can’t measure it all! You will have to make decisions about what to include and measure and what not to include and measure.

33. Writing Sample Tasks

34. What are sample tasks?
Sample Tasks are examples of assessment items or tasks that teachers can use to construct formative assessments
Sample Tasks are written to present a model of an assessment task to help clarify the intent of the element, not as a definitive list of assessment tasks
Assessment items that are written to address the appropriate taxonomic level for each element and score level

35. The Purpose of Writing Sample Tasks
Help to clarify the expectations of each element
Provide instructional guidance to the teacher
Suggest ways to assess an element
Can be used to design formative assessment
Can be used to construct a common assessment item bank

36. Writing Sample Tasks Ask yourself some basic questions:
How can a student demonstrate success for this element?
What kinds of learning activities or tasks would assess success in this element?
What taxonomic operation would be appropriate for this task at this grade level?

37. General Rules
When explaining or other higher level processes are involved in sample tasks, ensure that the students are explaining how or why something is happening and not just that it happens. If the elements are worded correctly this should not be a problem.
If an element refers to memorization, a sample task must say something like, produce from memory.
Each bulleted element will generally have a corresponding sample task.
Be specific!!!!

38. Organizing the Work
We suggest that you split the work up into several teams, each with representatives from lower elementary, upper elementary and middle school. High school reps are helpful, if possible.
Each team would be responsible for writing the scales or rubrics K-8 within a measurement topic. This helps to ensure vertical alignment. A high school rep is helpful to make sure that the student ends up where they need to be!
High school teams can work to create rubrics by course.

39. Our Job today:
Review Draft MT’s and take a first cut at revising them. You can and will continue to revise as you develop the rubrics
Develop the Learning Goal (3.0) for each of the MT’s at each grade level
YOU WILL NOT FINISH THIS TODAY!