1. District Programs for Measuring Student Growth in Teacher Evaluation
Megan Lovinguth, Denver Public Schools
Peter Leonard, Chicago Public Schools
Department of Student Assessment

2. Session Objectives Participants will:
Understand how two large urban districts are approaching the design and measurement challenges posed by student growth-based teacher evaluation systems
Connect the experiences of Denver & Chicago to a problem of practice in their own work
Turn & talk (2 minutes)
Why did you decide to attend this morning’s session?
Department of Student Assessment

3. Turn & Talk (2 mins)
Why did you decide to attend this morning’s session?
What do you hope to take away from the session to inform your own work?
Turn & talk (2 minutes)
Why did you decide to attend this morning’s session?
Department of Student Assessment

4. STUDENT GROWTH IN DPS
FOCUSING ON SLOS

5. WHY “FOCUSING ON SLOS?”
Why do I say, student growth focusing on SLOs? Well, in CO we’ve been through a bit of a rollercoaster over the last few years in regard to using student growth in teacher evaluation. So, in 1314 a state bill passed that indicated that teacher evaluations needed to include an individual measure of attribution for each teacher, which was SLOs. Then, we got a year reprieve to figure out how we, as a district were going to do that. So we only had professional practice scores. This year, we needed to add student growth back in, but another state bill was passed barring districts from using the prior year state measures (because we had piloted PARCC and legislators were unsure that it was a good measure). Which left us with SLOs. So, as you can see here, SLOs, this year, were 45% of a teacher’s evaluation.

6. EVOLUTION OF SLOS IN DPS
1999 SGOs
2013 SLO Pilot
2014 whole district SLO “learning year)*
2015 SLOs count as 45% teacher’s overall LEAP**
2016 SLOs count as 30% or 40% teacher’s overall LEAP***
*adjustments to SLO process made based on teacher/school leader feedback AND external evaluations by CADRE/CU-Boulder
**adjustments to SLO process anticipated
***proposed, but not yet agreed upon

7. Right now, you’re thinking, “SLOs. We all know about SLOs
Right now, you’re thinking, “SLOs?! We all know about SLOs.” Put aside for a bit how you’re doing SLOs in your state, or your district, if possible, and hear me out. Because we’re doing SLOs a little differently than most others.

8. MOST IMPACTFUL FORMATIVE ASSESSMENT PRACTICES (a.k.a. The SLO Process)
Where are students going?
Students know their own learning goals.
[SLO Objective Statement and Learning Progression Rubric]
Where are students now?
Identify strengths and learning needs.
Provide descriptive feedback.
[SLO Baseline data and Preparedness Levels, Ongoing Data-driven Instruction (DDI)]
How do we help students meet learning goals?
Focus instruction.
Practice with feedback.
Students track and reflect on their own learning.
[Using the Learning Progression Rubric and information collected for the SLO Body of Evidence, adjust instruction appropriately, ongoing DDI]

9. WHAT’S DIFFERENT ABOUT OUR SLOs?
Body of Evidence
Multiple measures give a truer sense of what the student knows
Collaboration and Support
Improve the quality of data collection and data use
Scoring
Based on movement from baseline to end of course along a learning progression
All Students
Every child succeeds

10. BODY OF EVIDENCE
Teachers select a minimum of two pieces of evidence to determine baseline levels for students:
Prior year data source, if possible
Beginning of the year data source if possible
District defined beginning of the year baseline preparedness levels
Over the course of the year, teachers collect a minimum of 4-6 pieces of evidence to demonstrate student command levels
District defined learning progression and command levels, if teachers used a model SLO
Learning progressions were designed to describe what students can do
They are NOT designed to be used to score pieces of evidence
District model performance based tasks for end of course
Moved from a pre-test/post-test model to asking teachers to draw on pieces of evidence that they’re already using. We no longer wanted teachers to rely on outside measurements for their estimates of student learning. So this is a key place where two things I’ve heard so much about this week come into play: 1) assessment literacy (teachers and more importantly school leaders) and 2) “formative assessments”.
Show WIKI here to demonstrate what teachers get to work with
We are asking for a ton of professional judgment on the teacher and the school leader

11. SLO SCORING

12. GENERAL APPROACH TO SCORING SLOS
Beginning of Course:
Teachers select SLO objective and establish students’ baseline preparedness to meet the objective*
Evaluators (school leaders or teacher leaders) approve objectives and student baseline designation
Midyear or Mid-Course:
Teachers and school leaders check in on SLO progress during (i.e., Midyear Conversations)
End of Course:
Teachers identify students’ end-of-course command of standards in the SLO
Evaluators approve student command levels and finalize the SLO
SLO score is generated based on students’ baseline preparedness and end-of-year command
While the SLO process is centered on collaboration, progress monitoring, and formative assessment, scoring is based on the comparison between students’ baseline preparedness levels and end-of-course command levels.
SLOs are all about good instruction, teaching and learning. While today’s session is focused on the scoring of SLOs (the accountability piece), let’s not lose sight on the rationale for utilizing SLOs in student growth. That is, we believe SLOs are the most authentic measure of student growth, the closest to the classroom, that we currently have.
SLO scoring for teachers is rooted in two key components of SLOs: 1) students’ baseline preparedness levels and 2) students’ end of course command levels. An SLO score will be generated based on the movement between the two levels.
Teachers and school leaders should be completing each component of the SLO process in collaboration and collecting a body of evidence throughout the course/year to determine student progress. Scoring will then be calculated by comparing each student’s movement from baseline preparedness to command and assigning a point value to each student’s amount of movement.

13. COMBINING PREPAREDNESS AND COMMAND FOR SCORING
Below Limited Command
Limited Command
Moderate Command
Strong Command
Distinguished Command
Significantly Underprepared
Underprepared
Somewhat Prepared
Prepared
Ahead
Students can be plotted in the cell in the table that represents their preparedness and command levels.
Each cell is assigned a point value based on the amount of growth it represents.
Points are totaled across all students included in the SLO based on their amount of growth.
A percentage of points earned out of points possible for that teacher and SLO is calculated.
This percentage of points possible is the score for the SLO.
Once a determination has been made for each student regarding their baseline preparedness level and his/her end of course command level, that student can be plotted on the matrix above. Each cell in the matrix is then assigned a point value based on the amount of growth it represents. Points will be totaled across all students included in that SLO, and then a percentage of points earned out of the total amount of points possible for that teacher will be calculated. That is the score for the SLO.

14. SLO SCORING MATRIX
Below Limited Command
Limited Command
Moderate Command
Strong Command
Distinguished Command
Significantly
Underprepared
Teacher & Evaluator Decision:
0, 1, or 2 3
0 or 1 2
Somewhat Prepared 1
Prepared
NA*
Ahead
2 or 3
Teacher & Evaluator Decision Cells:
Growth can look different for individual students falling in these cells. For example, a Significantly Underprepared student can demonstrate substantial growth, but still not meet the criteria for Limited Command of the current year standards.
In these cells, teachers and evaluators determine the student’s growth based on the individual student’s body of evidence.
Possible Points = # students X 3
In order to determine the point values for each of the cells on this matrix, we began with the movement of a prepared student. A prepared student should leave the course/year at strong command. This amount of movement would signal approximately one year’s worth of growth. Then, using this standard, point values for other cells were given. So, a somewhat prepared student should be leaving that course/year at least at moderate command. An underprepared student should at least be leaving that course at limited command. Anything above this green diagonal would be considered higher growth, and therefore worth an additional point. Anything below this line would be considered less than expected growth, and therefore be worth less points.
Please note the three gray decision boxes on this matrix. These boxes indicate opportunities for a teacher and evaluator team to make a collaborative decision about the point value appropriate for that student’s growth. So, for example, a student entering the course significantly underprepared (more than 2 years behind grade level) could still make more than a year’s worth of growth, or even up to 2 year’s worth of growth and not hit limited command. In a case like this, it would be up to the teacher and school leader team to determine what point value the growth of that student is worth (0, 1, or 2), based on strong evidence. Likewise, a student who enters underprepared may make some growth and still not be at limited command. A student who entered ahead, may be expected to leave the course at distinguished, but we also allow for the fact that distinguished may not capture the true amount of growth of that student. If the evidence supports, a teacher and school leader team may decide to award 3 points for a student in that situation. *Right now, in order to be preparing for making these decisions when the time comes, you, as the evaluator, should be speaking with your teachers regarding the students they’re assigning to the significantly underprepared, underprepared and ahead preparedness levels. Remember that for the underprepared and significantly underprepared categories teachers will need to enter additional information in the SLO Application. We will be working with LEAP to define your evaluator actions for these decision boxes even further.
STOP: Take a minute to read and reflect on this matrix. How might you explain it to someone else (or put it into your own words)? There is a notecatcher available on your table for you to use.
* For students starting a course Prepared or Ahead, Below Limited Command represents less mastery than they begin the course with. This is based on Limited Command representing a level of mastery expected around the beginning of a course. Thus, Below Limited Command is not an option for these students. Limited Command is the lowest level of mastery they can attain.

15. POINTS TO EARN FOR EACH STUDENT GROWTH RATING
Student Growth Points Earned
Student Growth Rating
% SLO Pts. Earned
SLO Points +
District Growth Points =
Total SG Pts.
3.75 – 18.59
Not Meeting → 0%
3.75
18.6 – 32
Approaching
33%
14.85
18.6
32.1 – 45.14
Effective
63%
28.35
32.1
45.15 – 50
Distinguished
92%
41.4
45.15
Ways Teachers Can Get to Each Rating
Not Meeting
Approaching
Effective
Distinguished
100% of students with 0 points
100% of students with 1 point
100% of students with 2 points
100% of students with 3 points
50% of students with 0 points &
50% with 1 point
50% of students with 1 point &
50% with 2 points
90% students with 2 points &
10% with 1 point
75% students with 3 points
25% with 2 points
25% with 1 point &
25% of students with 0 points &
50% with 1 point &
25% of students 0 points &
50% with 2 points &
25% with 3 points
5% of students with 0 points &
15% with 2 points &
80% with 3 points
Teachers can earn the % possible SLO points in multiple ways. This table shows a few ways that teachers can earn each rating.
There are many different ways that teachers can get to each of the different Student Growth ratings given their SLO points earned and then the additional of the district growth points (3.75 points for every teacher).
In order to determine appropriate cut points, it was important to consider the actual situations that would happen in order for each number of points to be earned. This helped to determine what seemed most appropriate and just for each rating. So, for example, in order to earn a rating of ‘not meeting’ a teacher could have 100% of their students with 0 points. Which would mean that none of their students made any significant growth. In order to get a rating of ‘approaching’ a teacher could have 100% of their students only earning 1 point (falling in the yellow boxes). A rating of ‘effective’ could be earned with a teacher who moved all of their students to the 2 point cells. It’s also important to note that this rating of ‘effective’ could be earned if a teacher had 90% of their students with 2 points, and 10% of their students with 1 point. Lastly, a teacher may earn the student growth rating of ‘distinguished’ if they had 100% of their students with 3 points or 75% of their students with 3 points and 25% of their students with 2 points.
Now go back to the example teacher you were looking at earlier. Where would they fall in their ratings?

16. LESSONS LEARNED AND CHANGES PROPOSED FOR 2016-17

17. LESSONS LEARNED AND CHANGES PROPOSED FOR 2016-17
Teachers and school leaders misunderstanding the definitions of preparedness and command levels for students, and therefore incorrectly determining levels
Important information not reaching our target audience (teachers) correctly and/or in a timely fashion
Teachers and school leaders approach SLOs as an additional process for compliance purposes, rather than integrating into data-driven instruction and classroom assessment.
School leader evaluation of SLOs and corresponding inflation of scores

18. dpsare.com
leap.dpsk12.org

19. REACH Performance Tasks Chicago Public Schools19

20. Agenda Program Overview Task Design Growth Measurement
Teacher Perceptions
Problems of Practice
What’s Next?
Welcome and introduction.
Icebreaker – 30 sec think about why you want to be a task writer & what you expect to get out of this training
30 sec turn & talk

21. Program History & Objectives
REACH Performance Task Objectives
Measure of Student Growth
For teachers, by teachers
Aligned to curriculum
Communicate expectations
Opportunity for collaboration
Instructional Tool
Build assessment creation skills
Provide data on student performance
Here we have the main objectives of REACH Performance Tasks. First and foremost, PTs are a measure of student growth, but what’s important about this is that they are both specifically designed to be for CPS teachers, by CPS teachers, as well as to be closely aligned to classroom curriculum and instruction.
As we previously discussed, they are also designed to communicate expectations both around content and rigor, including for traditionally non-tested grades and subjects
Provide a unique opportunity for collaboration between teachers in the task writing process
Finally, serve as an instructional tool, both to build capacity among teachers to write good assessments, but also to provide teachers with data on how students perform in the beginning and end of the year along a powerful standard in the discipline
***In case this comes up, CTU worked with policy framework around REACH PT

22. REACH Educator Percentages
REACH Performance Tasks are one of two student growth measures used in teacher evaluation.
Educators
Professional Practice
Performance Tasks
Value-Added
Elementary Pre-K - 2
70%
10%
20% (Individual)
Elementary 3- 8 of tested subjects
15%
Elementary 3 – 8 of non-tested subjects
20%
10% (Schoolwide)
High school
Counselors, Related Service Providers, Education Support Specialists
100%

23. Overview Definition: Performance Task Why performance tasks? 23
A written or hands-on demonstration of mastery, or progress towards mastery, of a particular skill or standard.
Beginning of year and end of year administration measures student mastery over the course of an academic year on a set of identified knowledge, skills, or understanding (depth over breadth).
Why performance tasks?
Narrow but in-depth
Reflect long-term, key outcomes
Require higher-level and extended thinking
Most may know what a PT is, but can I have someone read the definition here for those who are new to this process?
Why Performance Tasks? They offer us a narrow but deep measurement of 1-2 standards that are central to classroom learning and instruction. In doing this, they reflect expectations for what students should know, as well as set expectations for rigor by focusing on higher-level thinking and questioning 23

24. Design Commonalities across Performance Tasks: 24 3 task components
Teacher administration document
Student document
Rubric
Measure 1-2 key standards (depth over breadth)
Must be completed in one class period
Differences across Performance Tasks:
Standards assessed (CCSS + ILS + other national standards)
Design type (text-based, action-based, or hybrid)
Still 4 buckets, but slightly different organization from last year, and we’ll talk more about that on the next slide.
All PTs measure 1-2 primary standards
All consider variation in student seat time for every day classes vs. classes that may only meet once a week or once a month
All must be completed in one class period
Some key differences include the fact that they assess different standards depending on the class, and that some tasks are designed to be more text-based, while others are more action based depending on the context of the classroom and how instruction is typically given 24

25. Principles of Swagg

26. PT Design Principles Design Principles Key Considerations Relevance
Incorporates materials and activities that are authentic to instruction and student experience in a given discipline
Rigor
Maintains high cognitive demand throughout the task
Stretch
Includes items that address foundational and advanced skills/content within the standard
Alignment
Measures identified objectives (skills/content) that are aligned with the standard(s)
Accessibility
Incorporates text and/or activities that all students can engage with, or that can be easily adapted for diverse learners
Here are five major design principles that we want to keep in consideration during the design of the task and that we want to check for after the task is written. During this part of the training, we’ll walk through each of the design principles one by one and then have a chance to apply them.

27. Science 14-15 Performance Task

28. Science 14-15 Performance Task (Cont.)

29. Science 14-15 Performance Task (Cont.)

30. Science 14-15 Performance Task (Cont.)
Explain why and why not

31. Science 14-15 Performance Task (Cont.)
What did you notice about the assessment

32. Rubric & Growth Measurement
Each objective scored along four performance levels
Objectives weighted according to complexity
Growth a binary measure based on a student advancing at least one performance level from BOY to EOY

33. Teacher Perceptions of REACH PTs
UChicago CCSR Research Brief (November 2014)

34. Problems of Practice Data Integrity Locally scored
PT Study (summer 2014)
PT Audit (summer 2015)
Compliance mindset
Another required assessment
What’s Next? Tool
Collaborative Scoring Protocol
Task Differentiation
Only students who qualify for DLM-AA can receive modified test form
Universal accommodations that do not interfere with the construct; formation of teacher working group

35. What’s Next? Tool Delivered with every BOY PT order
How did students perform?
What are students’ strengths and challenges?
How will these strengths and challenges inform your instructional planning?
How will I monitor progress?

36. You should do this training at every school.
Collaborative Scoring Protocol
Exit Survey Results
It was very helpful to talk with other teachers about how and why they assessed an answer the way they did.
What Teachers Said:
It's always great to get together with teachers from other parts of the city and different grade and content expertise to talk about education and improve our practice.
You should do this training at every school. 36 36

37. What’s Next?
Making the connections explicit between the work of PTs, other district systems, and instructional priorities
Providing more structured support to improve teacher’s interaction with student work in performance tasks
Building quality control systems to hold teachers accountable in the PT administration process

38. How’d We Do? Participants will:
Understand how two large urban districts are approaching the design and measurement challenges posed by student growth-based teacher evaluation systems
Connect the experiences of Denver & Chicago to a problem of practice in their own work
Turn & talk (2 minutes)
Department of Student Assessment