Using Data for Improvement Riverside County Assessment Network Noah Bookman May 18, 2018

What’s New in the CORE Data System?

On Track for College and Career Analytics Metrics track 3rd through 12th grade student progress toward graduation and college readiness assessments, behavior, attendance, course credit grades, and SEL survey responses Supports goal setting, and developing and monitoring interventions for students Pilot in Winter 2019 Rollout in Spring 2019

Partnerships that Support College Readiness, Enrollment and Persistence Data on students’ success into higher education (National Student Clearinghouse) College and career pathway data (Linked Learning Alliance) SAT and AP data (College Board – later in 2018)

Partnerships that Support College Readiness, Enrollment and Persistence

Streamlined Student Surveys Streamlined student surveys include four constructs measured in school culture and climate and four in social-emotional learning Allows district to insert tailored survey items

Multiple Capabilities and Measures System offers LCAP views, scatterplots, metric trends, gap analyses, school distributions, and school and district comparisons Individual school profiles available for distribution (pilot in winter 2019, larger rollout in spring 2019, additional cost TBD)

Index Options Performance on each measure is compared with similar schools across California A large holistic data set ensures a clear view for continuous improvement

A Sample Project Underway Helping Districts and Schools with ELA and Math Goal Setting

Academic Growth Highlights the impact school teams are having on student outcomes compared to similar schools Substantially different than any other indicator and a critical tool for continuous improvement

Academic Growth - The Basics

Academic Growth - The Basics

Academic Growth - The Basics

Academic Growth - The Basics

Measuring change in status is NOT the same thing as measuring growth. Metric Example What it Tells You for the Example Status: Measures performance at a particular point in time. 45% of students met grade level standards in math in 2016-17 The degree to which students know math. Change: Measures the change in status from one year to the next or over multiple years There was a 5% decrease in students who met grade level standards in 2016-17 (compared to the fact that 50% of students met grade level standards in 2015-16) How much math one year’s students knew compared to another year’s students. (The key distinction is that these are different sets of students). Simple Growth (Average Gain): Measures how much the average student has improved from one year to the next. The average student improved 20 scale score points in math when we compared each individual student’s 2015-16 math score to his/her 2016-17 math score. How much math knowledge students gained in one year. Growth [Impact]: Estimates the impact of school and grade level teams on student learning (by comparing each student’s results to what we would have predicted for that student had he/she been in an average impact school). Growth in math was at the 85th percentile, representing above average impact. The impact of a district, a school or grade level team on improving math learning.

Change versus Scale Score Gain – Elementary Math Preliminary Analysis Change versus Scale Score Gain – Elementary Math All of these schools have a change level of “maintained,” but they range for highest to lowest levels of average student scale score gain (and relatedly of growth/impact).

Change versus Scale Score Gain – Elementary Math Preliminary Analysis Change versus Scale Score Gain – Elementary Math Change is providing a false signal to negative change, high growth schools. All of these schools have a change level of “declined” or “declined significantly,” but they range from high to low levels of average student scale score gain (and relatedly of growth/impact).

Change versus Scale Score Gain – Elementary Math Preliminary Analysis Change versus Scale Score Gain – Elementary Math All of these schools have a change level of “increased” or “increased significantly,” but they range for high to low levels of average student scale score gain (and relatedly of growth/impact). Change is providing a false signal to positive change, low growth schools.

Change versus Growth – Elementary Math Preliminary Analysis Change versus Growth – Elementary Math The relationship between change and growth is even weaker than change and scale score gain. Growth estimates impact. Impact and one year change are only modestly correlated.

Growth versus Scale Score Gain – Elementary Math Preliminary Analysis Growth versus Scale Score Gain – Elementary Math The relationship between growth and average scale score gain is tighter and more predictable.

Growth versus Scale Score Gain – Elementary Math Preliminary Analysis Growth versus Scale Score Gain – Elementary Math That said, the schools in this box are in the range of 30 to 40 points in average scale score gain. Most are considered average impact. Some are high impact. A few are low impact. This variation is due to differences in where there students started the year, student demographics, average grade level demographics, and the number of students in particular grade levels.

Discussion: Which metric is best for school level goal setting Discussion: Which metric is best for school level goal setting? For what purposes? Tradeoffs? Metric Example What it Tells You for the Example Status: Measures performance at a particular point in time. 45% of students met grade level standards in math in 2016-17 The degree to which students know math. Change: Measures the change in status from one year to the next or over multiple years There was a 5% decrease in students who met grade level standards in 2016-17 (compared to the fact that 50% of students met grade level standards in 2015-16) How much math one year’s students knew compared to another year’s students. (The key distinction is that these are different sets of students). Simple Growth (Average Gain): Measures how much the average student has improved from one year to the next. The average student improved 20 scale score points in math when we compared each individual student’s 2015-16 math score to his/her 2016-17 math score. How much math knowledge students gained in one year. Growth [Impact]: Estimates the impact of school and grade level teams on student learning (by comparing each student’s results to what we would have predicted for that student had he/she been in an average impact school). Growth in math was at the 85th percentile, representing above average impact. The impact of a district, a school or grade level team on improving math learning.

Preliminary Analysis Average Student Scale Score Gain by Growth Level and Grade Level - Math The plus/minus piece is based upon one standard deviation from the mean.

Preliminary Analysis Average Student Scale Score Gain levels are associated with high impact – Elementary School Math Overall Elementary schools (with 250+ analyzed students) with an average scale score gain of 43 points were all identified as having above average impact.

Preliminary Analysis Average Student Scale Score Gain levels are associated with high impact – math by grade 4th grade 5th grade 6th grade Grade Average Scale Score Gains Consistently Associated with High Impact* 4th 52+ 5th 40+ 6th 45+ 7th 34+ 8th 31+ 7th grade 8th grade *Based upon growth from 2016 to 2017, and only including grade level cases with 100 or more students.

Contextualizing growth by looking at 4th to 8th grade growth for students tested in 2017 The typical 4th grader grew 37 scale score points from 3rd grade. That said, he/she would have needed to grow 49 points to stay at the same performance level as he/she was in 3rd grade. In other words, the typical 4th grader fell twelve points further behind in distance to meeting standards. What’s more, the average 4th grader would have needed to growth 73 points to be meeting grade level standards.

Discussion: Which metric is best for school level goal setting Discussion: Which metric is best for school level goal setting? For what purposes? Tradeoffs? Metric Example What it Tells You for the Example Status: Measures performance at a particular point in time. 45% of students met grade level standards in math in 2016-17 The degree to which students know math. Change: Measures the change in status from one year to the next or over multiple years There was a 5% decrease in students who met grade level standards in 2016-17 (compared to the fact that 50% of students met grade level standards in 2015-16) How much math one year’s students knew compared to another year’s students. (The key distinction is that these are different sets of students). Simple Growth (Average Gain): Measures how much the average student has improved from one year to the next. The average student improved 20 scale score points in math when we compared each individual student’s 2015-16 math score to his/her 2016-17 math score. How much math knowledge students gained in one year. Growth [Impact]: Estimates the impact of school and grade level teams on student learning (by comparing each student’s results to what we would have predicted for that student had he/she been in an average impact school). Growth in math was at the 85th percentile, representing above average impact. The impact of a district, a school or grade level team on improving math learning.

Time Permitting, Let’s Explore Some College Going Analytics Go to http://bit.ly/rcoeshiny Username: rcoe@coredistricts.org Password: RCOE2017 Go to preliminary college analytics → Bar Plot Distributions

CORE’s Next Improvement Community - A Breakthrough Series on 9th Grade On Track for Graduating College Ready

CORE’s Next Improvement Community - A Breakthrough Series on 9th Grade On Track for Graduating College Ready Cohort 1 Cohort 2 joins Cohort 1 Who 5 districts, 2-3 schools per district +/-12 districts, +/- 45 schools Application Timing Fall 2018 Fall 2019 Launch Winter 2019 Winter 2020 Close Summer 2022 More information coming soon….

Appendix

Average Student Scale Score Gain by Growth Level and Grade Level - ELA Preliminary Analysis Average Student Scale Score Gain by Growth Level and Grade Level - ELA The plus/minus piece is based upon one standard deviation from the mean.

Preliminary Analysis Average Student Scale Score Gain levels are associated with high impact – ELA by grade 4th grade 5th grade 6th grade Grade Average Scale Score Gains Consistently Associated with High Impact* 4th 57+ 5th 52+ 6th 37+ 7th 36+ 8th 7th grade 8th grade *Based upon growth from 2016 to 2017, and only including grade level cases with 100 or more students.