1. Using Data for Improvement
Riverside County Assessment Network
Noah Bookman
May 18, 2018

2. What’s New in the
CORE Data System?

3. On Track for College and Career Analytics
Metrics track 3rd through 12th grade student progress toward graduation and college readiness
assessments, behavior, attendance, course credit grades, and SEL survey responses
Supports goal setting, and developing and monitoring interventions for students
Pilot in Winter 2019
Rollout in Spring 2019

4. Partnerships that Support College Readiness, Enrollment and Persistence
Data on students’ success into higher education (National Student Clearinghouse)
College and career pathway data (Linked Learning Alliance)
SAT and AP data (College Board – later in 2018)

5. Partnerships that Support College Readiness, Enrollment and Persistence

6. Streamlined Student Surveys
Streamlined student surveys include four constructs measured in school culture and climate and four in social-emotional learning
Allows district to insert tailored survey items

7. Multiple Capabilities and Measures
System offers LCAP views, scatterplots, metric trends, gap analyses, school distributions, and school and district comparisons
Individual school profiles available for distribution (pilot in winter 2019, larger rollout in spring 2019, additional cost TBD)

8. Index Options
Performance on each measure is compared with similar schools across California
A large holistic data set ensures a clear view for continuous improvement

9. A Sample Project Underway
Helping Districts and Schools with ELA and Math Goal Setting

10. Academic Growth
Highlights the impact school teams are having on student outcomes compared to similar schools
Substantially different than any other indicator and a critical tool for continuous improvement

11. Academic Growth - The Basics

12. Academic Growth - The Basics

13. Academic Growth - The Basics

14. Academic Growth - The Basics

15. Measuring change in status is NOT the same thing as measuring growth.
Metric
Example
What it Tells You for the Example
Status: Measures performance at a particular point in time.
45% of students met grade level standards in math in
The degree to which students know math.
Change: Measures the change in status from one year to the next or over multiple years
There was a 5% decrease in students who met grade level standards in (compared to the fact that 50% of students met grade level standards in )
How much math one year’s students knew compared to another year’s students. (The key distinction is that these are different sets of students).
Simple Growth (Average Gain): Measures how much the average student has improved from one year to the next.
The average student improved 20 scale score points in math when we compared each individual student’s math score to his/her math score.
How much math knowledge students gained in one year.
Growth [Impact]: Estimates the impact of school and grade level teams on student learning (by comparing each student’s results to what we would have predicted for that student had he/she been in an average impact school).
Growth in math was at the 85th percentile, representing above average impact.
The impact of a district, a school or grade level team on improving math learning.

16. Change versus Scale Score Gain – Elementary Math
Preliminary Analysis
Change versus Scale Score Gain – Elementary Math
All of these schools have a change level of “maintained,” but they range for highest to lowest levels of average student scale score gain (and relatedly of growth/impact).

17. Change versus Scale Score Gain – Elementary Math
Preliminary Analysis
Change versus Scale Score Gain – Elementary Math
Change is providing a false signal to negative change, high growth schools.
All of these schools have a change level of “declined” or “declined significantly,” but they range from high to low levels of average student scale score gain (and relatedly of growth/impact).

18. Change versus Scale Score Gain – Elementary Math
Preliminary Analysis
Change versus Scale Score Gain – Elementary Math
All of these schools have a change level of “increased” or “increased significantly,” but they range for high to low levels of average student scale score gain (and relatedly of growth/impact).
Change is providing a false signal to positive change, low growth schools.

19. Change versus Growth – Elementary Math
Preliminary Analysis
Change versus Growth – Elementary Math
The relationship between change and growth is even weaker than change and scale score gain. Growth estimates impact. Impact and one year change are only modestly correlated.

20. Growth versus Scale Score Gain – Elementary Math
Preliminary Analysis
Growth versus Scale Score Gain – Elementary Math
The relationship between growth and average scale score gain is tighter and more predictable.

21. Growth versus Scale Score Gain – Elementary Math
Preliminary Analysis
Growth versus Scale Score Gain – Elementary Math
That said, the schools in this box are in the range of 30 to 40 points in average scale score gain. Most are considered average impact. Some are high impact. A few are low impact. This variation is due to differences in where there students started the year, student demographics, average grade level demographics, and the number of students in particular grade levels.

22. Discussion: Which metric is best for school level goal setting
Discussion: Which metric is best for school level goal setting? For what purposes? Tradeoffs?
Metric
Example
What it Tells You for the Example
Status: Measures performance at a particular point in time.
45% of students met grade level standards in math in
The degree to which students know math.
Change: Measures the change in status from one year to the next or over multiple years
There was a 5% decrease in students who met grade level standards in (compared to the fact that 50% of students met grade level standards in )
How much math one year’s students knew compared to another year’s students. (The key distinction is that these are different sets of students).
Simple Growth (Average Gain): Measures how much the average student has improved from one year to the next.
The average student improved 20 scale score points in math when we compared each individual student’s math score to his/her math score.
How much math knowledge students gained in one year.
Growth [Impact]: Estimates the impact of school and grade level teams on student learning (by comparing each student’s results to what we would have predicted for that student had he/she been in an average impact school).
Growth in math was at the 85th percentile, representing above average impact.
The impact of a district, a school or grade level team on improving math learning.

23. Preliminary Analysis
Average Student Scale Score Gain by Growth Level and Grade Level - Math
The plus/minus piece is based upon one standard deviation from the mean.

24. Preliminary Analysis
Average Student Scale Score Gain levels are associated with high impact – Elementary School Math Overall
Elementary schools (with 250+ analyzed students) with an average scale score gain of 43 points were all identified as having above average impact.

25. Preliminary Analysis
Average Student Scale Score Gain levels are associated with high impact – math by grade
4th grade
5th grade
6th grade
Grade
Average Scale Score Gains Consistently Associated with High Impact*
4th
52+
5th
40+
6th
45+
7th
34+
8th
31+
7th grade
8th grade
*Based upon growth from 2016 to 2017, and only including grade level cases with 100 or more students.

26. Contextualizing growth by looking at 4th to 8th grade growth for students tested in 2017
The typical 4th grader grew 37 scale score points from 3rd grade. That said, he/she would have needed to grow 49 points to stay at the same performance level as he/she was in 3rd grade. In other words, the typical 4th grader fell twelve points further behind in distance to meeting standards. What’s more, the average 4th grader would have needed to growth 73 points to be meeting grade level standards.

27. Discussion: Which metric is best for school level goal setting
Discussion: Which metric is best for school level goal setting? For what purposes? Tradeoffs?
Metric
Example
What it Tells You for the Example
Status: Measures performance at a particular point in time.
45% of students met grade level standards in math in
The degree to which students know math.
Change: Measures the change in status from one year to the next or over multiple years
There was a 5% decrease in students who met grade level standards in (compared to the fact that 50% of students met grade level standards in )
How much math one year’s students knew compared to another year’s students. (The key distinction is that these are different sets of students).
Simple Growth (Average Gain): Measures how much the average student has improved from one year to the next.
The average student improved 20 scale score points in math when we compared each individual student’s math score to his/her math score.
How much math knowledge students gained in one year.
Growth [Impact]: Estimates the impact of school and grade level teams on student learning (by comparing each student’s results to what we would have predicted for that student had he/she been in an average impact school).
Growth in math was at the 85th percentile, representing above average impact.
The impact of a district, a school or grade level team on improving math learning.

28. Time Permitting, Let’s Explore Some College Going Analytics
Go to
Username:
Password: RCOE2017
Go to preliminary college analytics → Bar Plot Distributions

29. CORE’s Next Improvement Community -
A Breakthrough Series on 9th Grade On Track for Graduating College Ready

30. CORE’s Next Improvement Community - A Breakthrough Series on 9th Grade On Track for Graduating College Ready
Cohort 1
Cohort 2 joins Cohort 1
Who
5 districts, 2-3 schools per district
+/-12 districts, +/- 45 schools
Application Timing
Fall 2018
Fall 2019
Launch
Winter 2019
Winter 2020
Close
Summer 2022
More information coming soon….

31. Appendix

32. Average Student Scale Score Gain by Growth Level and Grade Level - ELA
Preliminary Analysis
Average Student Scale Score Gain by Growth Level and Grade Level - ELA
The plus/minus piece is based upon one standard deviation from the mean.

33. Preliminary Analysis
Average Student Scale Score Gain levels are associated with high impact – ELA by grade
4th grade
5th grade
6th grade
Grade
Average Scale Score Gains Consistently Associated with High Impact*
4th
57+
5th
52+
6th
37+
7th
36+
8th
7th grade
8th grade
*Based upon growth from 2016 to 2017, and only including grade level cases with 100 or more students.