1. California Educational Research Association
Analysis, Measurement, and Accountability Reporting Division
November 30, 2016
Printed by the Riverside County Office of Education

2. Polling Questions
Throughout this presentation we will be asking you a variety of questions to help guide further development of the accountability system. You will need to use your phone to respond to the questions.
The polling URL is:

3. Polling Question What organization are you affiliated with?
County Office
District Office
Research Organization
Charter School
CDE
Other

4. Agenda
Framework for the New Accountability and Continuous Improvement System
Overview of the New Accountability System
Local Control Funding Formula (LCFF) Evaluation Rubrics : Top Level Data Display
Overview of State Indicators
Status of the Academic Indicator

5. Agenda (Cont.)
Status, Change, and Assigning a Color: The California Model
An In-depth Look into State Indicators:
Graduation Rate Indicator
Suspension Rate Indicator
English Learner Indicator
An In-depth Look into the College/Career Indicator (local indicator for initial release)

6. Agenda (Cont.) In-Depth Look at Performance Categories
Overview of Local Indicators
Timeline

7. Framework for the New Accountability and Continuous Improvement System

8. Accountability Framework
Performance
Equity
Improvement
Transparency
One single cohort system to meet local, state, and federal needs
Preparing All Students for College, Career, Life, and Leadership in the 21st Century, Superintendent’s Task Force on Accountability and Continuous Improvement Report, May 2016.

9. Accountability and Continuous Improvement System: Components
Local Control and Accountability Plan (LCAP) and Annual Update
Address eight state priorities (ten for counties)
Outline plans for expenditures
Report on progress
LCFF Evaluation Rubrics
State and local indicators
Performance standards
Support and Assistance System
Designed to support continuous improvement

10. Overview of the New Accountability System

11. New Accountability System
On September 8, 2016, California’s State Board of Education (SBE) approved a new multiple measures accountability system.
Rather than having two separate accountability systems (state and federal) as in prior years, the new integrated system captures local, state, and federal requirements.

12. New Accountability System (Cont.)
The new accountability system is based on multiple measures which will provide a more complete picture of what contributes to a positive educational experience for students.

13. New Accountability System (Cont.)
The multiple measures are based on the LCFF state priorities and are divided into two types of indicators:
State indicators
Local indicators

14. LCFF Evaluation Rubrics
Both state and local indicators will be reported through the LCFF Evaluation Rubrics.
The statutory purposes of these rubrics:
Support LEAs in identifying strengths, weaknesses, and areas for improvement
Assist in determining whether LEAs are eligible for technical assistance
Assist the State Superintendent of Public Instruction in determining if LEAs are eligible for more intensive state support/intervention
LEAs = local educational agencies

15. Polling Question
Would you like to see the Top-Level Data Display for the new multiple measures accountability system?
Yes No

16. Local Control Funding Formula Evaluation Rubrics: Top Level Data Display

17. Currently There are Four Reports
Equity Report
All Student Performance Report
Status/Change Report
Indicator Cluster Report

18. Overview of State Indicators

19. State Indicators Academic Indicator: Graduation Rate Indicator
English language arts/literacy (ELA) assessment
Mathematics assessment
Graduation Rate Indicator
Suspension Rate Indicator
English Learner Indicator (ELI)—measures progress of English learners (ELs)

20. Future State Indicators
Chronic Absenteeism Rate
College/Career Indicator (measures postsecondary preparedness)

21. Status of the Academic Indicator

22. November SBE Meeting
At the November 2016 SBE meeting, the proposed performance standards recommended for the Academic Indicator were based on the percent of students who scored “Standard Met” or “Standard Exceeded” on the Smarter Balanced Assessments for grades three through eight.
Grade eleven assessment results are captured in the College/Career Indicator.
SBE=State Board of Education

23. November SBE Meeting (Cont.)
The SBE decided not to release the Academic Indicator using “Standard Met” or “Standard Exceeded” voicing concern that this closely paralleled the Adequate Yearly Progress (AYP) methodology, which would reward schools focusing only on those students who were closest to proficient.

24. November SBE Meeting (Cont.)
Rather, the SBE wants to encourage districts and schools to improve the academic achievement of all students.
Therefore, the SBE requested CDE staff to work on a methodology that uses scale scores.
CDE= California Department of Education

25. Next Steps
CDE staff met with the testing vendor, Educational Testing Service (ETS), to ensure the validity of using scale scores to measure distance from a standard point.
Based on the information obtained from ETS staff, CDE staff will continue to work with the Technical Design Group (TDG) to explore options for this proposed methodology.

26. Next Steps (Cont.)
In January 2017, the CDE plans to bring the a proposed methodology to the SBE with four possible options for the Academic Indicator.
Distance from Met
Distance from Nearly Met
Distance from the Statewide Average Score (by grade)
Distance from the Lowest Possible Scale Score

28. Next Steps (Cont.)
Pending SBE’s decisions at this meeting, the CDE will move forward with the methodology and one of proposed criteria or pursue other options.

29. Next Steps (Cont.)
In addition, the SBE will be discussing the definition of the EL student group in the Academic Indicator at their January 2017 meeting.
In prior accountability reports, the ELA and mathematics results for the EL student group included results for both EL students and reclassified students.

30. Next Steps (Cont.)
However, the SBE has asked CDE staff to explore removing reclassified fluent English proficient (RFEP) students from the EL student group.
CDE staff are exploring multiple options.

31. Polling Question
For the Academic Indicator, which criteria do you prefer for the proposed methodology?
Distance From Met
Distance From Nearly Met
Distance from the Lowest Possible Scale Score
Distance from the Statewide Average Score (by grade)

32. Polling Question
For the Academic Indicator, which EL student group definition do you prefer?
ELs plus 4-years of RFEP (maximum amount of time allowed under the Every Student Succeeds Act (ESSA)
ELs plus 2-Years of RFEP
ELs only

33. Polling Question
On a scale of one to five, with five being the highest, how familiar are you with the 5X5 color grid to determine the five performance categories (red, orange, yellow, green, and blue)?

34. Status, Change, and Assigning a Color: The California Model

35. The California Model
The California Model combines five “Status” and five “Change” levels which results in a performance category (or color) for each of the state indicators.
The model provides equal weight to both Status and Change.

36. Five Status and Five Change Levels
Five Status Levels
Five Change Levels
Very High
Increased Significantly
High
Increased
Medium
Maintained
Low
Declined
Very Low
Declined Significantly

37. Status and Change Status is based on current year performance.
Change is the difference between prior year performance and current year performance.
Exception! The Graduation Rate Indicator is the only state indicator that uses a multi-year average rather than prior year data.

38. Methodology
To determine overall performance, the percentile distributions for Status and Change were examined for each indicator:
For Status, LEAs and schools were ordered from highest to lowest and four cut points were selected based on the distribution of all LEAs and schools. These cut points created the five Status Levels.

39. Methodology (Cont.)
For Change, LEAs and schools were ordered separately from highest to lowest for positive change and lowest to highest for negative change.
Cut points were determined separately for positive and negative change. A total of four cuts were selected which created five Change Levels.

40. Methodology (Cont.)
Cut points will remain in place for a select number of years (e.g., 3 to 5 years) to be determined by the SBE.

41. Performance Category
The combination of Status and Change results in a performance category that is assigned a color for each indicator:
Blue
Highest
Green
Yellow
Orange
Red
Lowest

42. Who Gets a Performance Category?
The model will be applied to all LEAs, schools, and student groups with 30 or more students.
The data used to determine “30 or more” differs for each indicator.
While a performance category (or color) will not be determined for LEAs, schools, or student groups with fewer than 30 students, the Status and Change data will be reported for groups with students.

43. Who Gets a Performance Category? (Cont.)
In these instances, an asterisk (*) will be reported to indicate that because there are fewer than 30 students, the LEA/school will not receive a performance category.
Data for fewer than 11 students will not be reported due to privacy reasons and a “- - -” will be displayed.
Data that is not yet available will be noted with an N/A.

44. An In-Depth Look into the State Indicators
One of the goals of this presentation is to provide the necessary information for you to calculate performance category (or color) for each state indicator.

45. Before We Dive In…. LEA Data Charter Schools Alternative Schools
Because all charter schools are treated as LEAs under the LCFF, charter school data are not rolled up to their authorizing agency’s data.
Alternative Schools
Since a separate accountability system will be developed for alternative schools, all alternative schools (i.e., Alternative Schools Accountability Model [ASAM]) are also excluded from LEA-level data.

46. Before We Dive In… (Cont.)
Therefore, the LEA-level data for graduation and suspension rates will not match the data reported on the CDE DataQuest Web page.

47. Polling Question
Should alternative schools’ results be included in the LEA-level data even though an alternative accountability system will be developed?
Yes, include
No, do not include

48. Graduation Rate Indicator

49. Who Will Receive a Graduation Rate Indicator?
This indicator applies to LEAs, schools, and student groups that have 30 or more students in the four-year graduation cohort.
Note: Students who earn a Special Education Certificate of Completion or a General Education Degree (GED) are not counted as high school graduates but are included in the denominator.

50. Status
For the initial release of the LCFF Evaluation Rubrics, the 2014–15 cohort graduation rate, or the class of 2015 graduation data, will be used to determine Status.

51. Change
Prior three-years of four-year cohort graduation data were used to calculate the three-year weighted average to determine Change.
Note: This is the only state indicator that uses a three-year weighted average to calculate Change.

52. Three-Year Weighted Average Formula
Class of 2012 Graduates + Class of 2013 Graduates + Class of 2014 Graduates
divided by
Students in the Class of
Students in the Class of
Students in the Class of 2014

53. Three-Year Weighted Average Formula (Cont.)
It is important to note that if the LEA, school, or student group does not have cohort data for all prior three graduating classes, then the weighted average for Change was calculated using the one or two years of available cohort data.

54. Current Status (Class of 2015) minus Three-Year Weighted Average
Change Formula
Current Status (Class of 2015) minus Three-Year Weighted Average

55. Determine Performance Category (Cont.)
Status: 89.4%
Change: +4.6%

56. Future Plans
At the SBE’s request, CDE staff will work with the TDG to analyze and investigate the incorporation of the five-year cohort graduation rate into this indicator.
Based on the current timeline, implementation of the four-and five-year cohort graduation rates may occur in 2018–19.

57. Suspension Rate Indicator

58. Who Will Receive a Suspension Rate Indicator?
LEAs, schools, and student groups that have 30 or more students enrolled will receive a Suspension Rate Indicator.
The enrollment data are obtained from the California Longitudinal Pupil Achievement Data System (CALPADS).

59. Suspension Rate Rules
If a student is suspended multiple times (at the same school or district), the student is counted as being suspended only once.
LEA Example: If a student was suspended:
Five times at School A,
Twice at School B, and
Twice at School C
The student would be counted as being suspended once at the LEA.

60. Status and Change Status:
The 2014–15 suspension rate will be used for Status for the initial release of the LCFF Evaluation Rubrics.
Change:
Change uses current and prior year suspension rates.
Change Formula:
2014–15 rate minus 2013–14 rate

61. Key Differences in Suspension Cut Scores
Data simulations revealed that suspension data varies widely among LEA and school type. Therefore, multiple suspension cut scores were set for LEAs and schools based on their type.
This resulted in six different sets of cut scores:
Three at LEA-level: Elementary, High, and Unified
Three at School-level: Elementary, Middle, and High

62. Key Differences in Suspension Cut Scores (Cont.)
Having six different sets of cut scores also means six separate 5 x 5 color grids based on district or school type.

63. Impact of Multiple Cut Scores
Since cut scores were set separately for LEAs and schools, charter schools and single school districts could receive two performance categories (or two colors): (1) at the LEA-level and (2) at the school-level because:
Charter schools are treated as LEAs under the LCAP, and
Single school districts are treated as schools under the ESSA
LCAP= Local Control and Accountability Plan
ESSA= Every Student Succeeds Act

64. Impact of Multiple Cut Scores (Cont.)
Because the LEA and school performance categories (colors) could be inconsistent, the SBE approved holding charter schools and single school districts accountable for the suspension rate cut scores using the school-level cut scores.
Therefore, charter schools and single school districts will only be held accountable for their school-level performance category (color).

65. Key Differences in the Suspension Goal
It is important to remember that for this indicator, the goal is reversed.
For all other state indicators, the desired outcome and goal is to achieve a high percent for Status and Change.

66. Key Differences in the Suspension Goal (Cont.)
However, the desired outcome and goal for the Suspension Rate Indicator is to have a low suspension rate, which translates to a low percent for Status and Change.
For this reason, the Status and Change levels on the 5 x 5 color grids are in reverse order compared to the grids for the other indicators.

67. Key Differences in 5 x 5 Grid
Level
Increased Significantly
Increased
Maintained
Declined
Declined Significantly
Very Low
Yellow
Green
Blue
Low
Orange
Medium
High
Red
Very High

68. Assignment of Performance Category
Schools that did not certify (or submit) suspension data in the CALPADS are automatically assigned the Orange performance category.

69. Determine Performance Category (Cont.)
Status: 0.5%
Change: +0.2%

70. Polling Question
Currently, a student who is suspended multiple times at the same school (or same district) is only counted once in the suspension rate.
How should suspensions be counted in the Suspension Indicator?
Include only the student’s first suspension
Include all of the student’s suspensions

71. English Learner Indicator (ELI)

72. Who Will Receive an ELI?
This indicator applies to LEAs and schools that have 30 or more students who took the annual California English Language Development Test (CELDT).
Note: Because 86.2% of schools have no significant, or only one significant race/ethnic student group within the EL group, student group data will not be reported for the ELI.

73. ELI Data Sources
The ELI determines progress through the use of two data sources:
Annual CELDT results, and
EL reclassification

74. CELDT Data The CELDT has five overall performance levels: Beginning
Early Intermediate
Intermediate
Early Advanced
Advanced

75. CELDT Data (Cont.)
Because the CELDT Intermediate performance level has a large range of scale scores, many students stay in the intermediate level for multiple years. As a result, stakeholders advised, and the SBE approved, that this level be divided into two, for accountability purposes only, to recognize the substantial growth that can be made within this particular level.

76. CELDT Data (Cont.)
Therefore, the ELI uses six overall CELDT performance levels:
Beginning
Early Intermediate
Low Intermediate
High Intermediate
Early Advanced
Advanced

77. ELI Model
Annual CELDT takers must advance at least one CELDT performance level from prior year to current year to be included in the numerator of the ELI calculation.
Examples:
Prior Overall CELDT Performance
Current Overall CELDT Performance
Does the LEA or School Receive Credit?
Early Intermediate
Low Intermediate
Yes
High Intermediate
Advanced
Beginning No

78. ELI Model (Cont.)
Students who scored Early Advanced or Advanced Proficient in the prior year and maintained that performance level for the current year will be included in the numerator for the ELI calculation. This is the only set of annual CELDT test takers who are not required to advance one CELDT performance level.
ELs who were reclassified in the prior year will also be included in the numerator and denominator for the ELI calculation.

79. Students Reclassified in Prior Year
ELI Model (Cont.)
Previous CELDT
Overall Level
Current CELDT
Beginning
Early Intermediate
Low Intermediate
High Intermediate
Early Advanced
Early Advanced or
Adv Not Proficient
Adv Proficient +
Students Reclassified in Prior Year

80. ELI Formula: Numerator
Annual CELDT test takers who:
Increased at least one CELDT level compared to the prior year
Maintained Early Advanced/Advanced English Proficient
ELs who were reclassified in the prior year

81. ELI Formula: Denominator
Total number of annual CELDT test takers
ELs who were reclassified in the prior year

82. Status and Change Status
The percent of ELs who moved up a performance level from the 2014 to 2015 CELDT plus ELs who were reclassified between July 1, 2013 to June 30, 2014.
Change
Difference in Status from current year to prior year.

83. Assignment of Performance Category
Schools that did not test at least 50 percent of their EL population in the CELDT are automatically assigned an Orange performance category.
Determination of the 50 percent is based on the EL demographic data reported for mathematics in the Smarter Balanced Assessment file from the testing vendor.

84. Example 210/250 = 84% 20 students 230/270 = 85%
Step 1: Percent of annual CELDT test takers who advanced at least one performance level (or maintained Early Advanced/Advanced English Proficient) on the 2015 overall CELDT compared to the 2014 overall CELDT
210/250 = 84%
Step 2: Number of ELs who were reclassified in prior year (2013–14)
20 students
Step 3: Add reclassified students to the numerator and denominator of Step 1 and calculate the rate.
230/270 = 85%

85. Determine Performance Category
Status: 85%
Change: +2%

86. Activity You have 10 minutes!
Each table has a sheet of paper containing the directions for your task.
Your task is determine the performance category based on the data provided, and to populate the LCFF Evaluation Rubrics posted around the room with your performance category (color) results.
You have 10 minutes!

87. An In-Depth Look into the College/Career Indicator

88. What is the Purpose of the CCI?
The College/Career Indicator (CCI) was designed to emphasize that a high school diploma should represent completion of rigorous course work that prepared students for postsecondary.

89. CCI Measures The CCI currently contains the following measures:
Advanced Placement (AP) exam results
Dual Enrollment
Early Assessment Program (EAP) results for ELA and mathematics (Grade 11)
a-g completion
Career Technical Education (CTE) pathway completion

90. CCI Measures (Cont.) Potential measures for the release in 2017–18:
International Baccalaureate (IB)
State Seal of Biliteracy
Golden State Seal Merit Diploma
IB Career–related Programme
Articulated CTE Pathway

91. Students with Disabilities
Students with the most severe cognitive disabilities (i.e., students who take the California Alternate Assessment) are removed from the calculation of the CCI.

92. First a Local Indicator
Although the CCI was planned to be reported as a state indicator, the SBE approved:
Reporting the CCI as a local indicator for the initial release of the LCFF Evaluation Rubrics based on Status only because the current CCI does not contain the Smarter Balanced results.
Note: The CDE will pre-populate the CCI data.

93. Then a State Indicator
When the LCFF Evaluation Rubrics are released in fall 2017, the CCI will be reported as a state indicator based on Status only because only one year of Smarter Balanced results will be available.
The first time grade eleven Smarter Balanced Assessment results will be available to calculate both Status and Change for the CCI is in 2018–19 (Class of 2016 and 2017).

94. CCI Formula
Students Who Receive a High School Diploma and Meet the CCI
Benchmark of “Prepared”
divided by
Current Year Graduation Cohort

95. Status Only
For the initial release of the CCI (as a local indicator), Status will be calculated based on the Class of The following years of data will be used for the CCI measures:
CCI Measure
Data Source
Year of Data
AP exam
The College Board
2014
Grade 11 EAP results
STAR file
From Educational Testing Service (ETS)
2013
a-g completion
CALPADS
(Yes/No filled in by LEA )
CTE pathway completion
Last 3 yrs in high school
(2012, 2013, 2014)
Dual enrollment

96. In-Depth Look at Performance Categories

97. Green is the Target
The target performance category (or color) is Green for all LEAs, schools, and student groups.
Performance categories (or colors) tell more than just the current year’s status; they reflect information about status and change. In other words, if your school is improving, that’s already reflected in the performance category.

98. The Performance Categories Already Reflect Change…..
For some indicators, earning Red one year and Orange the next year may not reflect improvement!

99. Moving From RED to ORANGE Does Not Always Mean Improvement!
Example: An LEA has a 2015–16 graduation rate of 82.0%. The LEA’s prior three-year average graduation rate is 89.0% (Change = -7%). As a result, the performance category is:
The 2016–17 graduation rate is 79.0%. The updated prior 3-year average graduation rate is 83.0% (Change = -4). As a result the performance category is:
Red
Orange

100. The Reverse is also True
Example: An LEA has a 2015–16 graduation rate of 92.0%. The district’s prior 3-year average graduation rate is 85.0% (Change = +7%). As a result, the performance category is:
The 2016–17 graduation rate is 94.0%. The updated prior 3-year average graduation rate is 91.0% (Change = +3). As a result, the performance category is:
Blue
Green

101. Takeaways From These Examples
Performance category already reflects change, so:
Talking about how a performance category changes over time may not make sense.
Discussions about trends should focus on trends in the underlying data.
A red, orange, or yellow performance category means that there is more work to be done. A green or blue performance category means that the trajectory of performance is fine.

102. Overview of Local Indicators

103. Local Indicators Basic (Priority 1):
Appropriately assigned teachers
Access to curriculum-aligned instructional materials
Safe, clean, and functional school facilities
Implementation of State Academic Standards (Priority 2)
Parent Engagement (Priority 3)
School Climate (Priority 6)

104. Local Indicators (Cont.)
For county offices of education (COEs) only:
Coordination of Services for Expelled Students (Priority 9)
Coordination of Services for Foster Youth (Priority 10)

105. Other Local Indicators
The following local indicators will be populated by the CDE:
College/Career Indicator (initially will be reported in the rubrics as a local indicator for informational purposes only.)
Grade eleven Smarter Balanced Assessment results.

106. Meeting Standards For each local indicator, LEAs are responsible for:
Using locally available information to measure progress relative to performance standards and criteria adopted by the SBE.
Reporting the results at a regularly scheduled meeting of the local governing board.
Reporting the results through the Evaluation Rubrics.

107. Meeting Standards (Cont.)
LEAs will use the following standards to assess performance on the local indicators:
Met
Not Met
Not Met for Two or More Years
These results will be populated by the LEAs in the LCFF Evaluation Rubrics.

108. SBE November Decisions
At the November 2016 SBE meeting, the SBE approved the following tools for the Local indicators:
Basic (Priority 1):
These data are collected through the School Accountability Report Card (SARC).

109. SBE November Decisions (Cont.)
LEAs will use locally available information, including data reported through the SARC, to provide evidence of progress.
The Web-based interface system for the Evaluation Rubrics is being developed based on the same data system that supports the SARC template.

110. SBE November Decisions (Cont.)
The goal is for the rubrics system to auto-populate the necessary SARC data for LEAs that use the SARC template.

111. SBE November Decisions (Cont.)
School Climate (Priority 6):
At least once every other year, LEAs administer a survey in at least one grade within each grade span served (K-5, 6-8 and 9-12).
Provide a narrative summary of the local survey and the analyses of the survey in a text box provided in the Evaluation Rubrics Web-based user interface.

112. SBE November Decisions (Cont.)
Coordination of Services for Expelled Students (Priority 9):
COEs determine progress using the self-reflection tool adopted by the SBE.
COEs reports results through the local data selection option in the Evaluation Rubrics Web-based user interface.

113. SBE November Decisions (Cont.)
Coordination of Services for Foster Youth (Priority 10)
COEs determine progress using the self-reflection tool adopted by the SBE.
COEs reports results through the local data selection option in the Evaluation Rubrics Web-based user interface.

114. Next Steps for Local Indicators
The CDE will bring recommended standards for the Implementation of State Academic Standards (Priority 2) and Parent Engagement (Priority 3) to the SBE in January 2017.

115. Timeline

116. Timeline
The LCFF Evaluation Rubrics will be released in 2017 to LEAs first for their preview.
LEA superintendents have been notified by the CDE to designate a “LCFF Rubrics Coordinator” to compile and coordinate feedback and/or responses from stakeholders.

117. Timeline (Cont.)
The LCFF Rubrics Coordinators will be provided access to the LCFF Evaluation Rubrics when they are made available.
After LEAs preview the LCFF Evaluation Rubrics, the public release will occur.
The CDE is releasing the data to provide LEAs and the public an opportunity to become familiar with the LCFF Evaluation Rubrics before they are used to determine technical assistance or intervention.

118. Timeline (Cont.)
Beginning in the 2017–18 school year, the data reported in the LCFF Evaluation Rubrics will be used to determine LEA eligibility for technical assistance or interventions.
In addition, the release of the Evaluation Rubrics in the fall each year will make data available for LEAs to use in the development of their LCAPs.

119. Timeline (Cont.)
Because one of the goals of the new accountability system is continuous improvement, the SBE has established an annual review cycle beginning in March 2017, with any potential action taken no later than September of each year, to continually evolve the Evaluation Rubrics based on user experiences and stakeholder feedback, as well as the availability of new data.

120. Questions?

121. Contact Information
Questions regarding the Evaluation Rubrics should be directed to the Academic Accountability Unit by phone at or by