Accountability Directors and Test Coordinators Fall Meeting

Accountability Directors and Test Coordinators Fall Meeting
Accountability Services North Carolina Department of Public Instruction September 23, 2014

Agenda Introductions EVAAS Growth Read to Achieve/Grade 3 Reading
Online Testing School Performance Grades Alternative School Accountability Model Accountability/Testing Updates

Tom Tomberlin Director, District HR Support
Teacher Evaluation and EVAAS Tom Tomberlin Director, District HR Support

Teacher Evaluation in North Carolina

Teacher Evaluation in NC
The North Carolina Educator Effectiveness System (NCEES) has six standards of performance for teachers and eight standards for principals. NC has a conjunctive model, meaning that teachers and principals must be proficient (or better) on all standards in order to receive an overall effectiveness rating. We do not average or index these standards. Unlike the observational standards, student growth (standard 6 for teacher, standard 8 for principals), requires three years of valid data in order to generate a rating.

1 2 3 4 5 6 1 2 3 4 5 6 7 8 Standards 6 & 8 – The Basics Teachers
Contribute to Academic Success Demonstrate Leadership Establish Environment Know Content Facilitate Learning Reflect on Practice Principals (and other Administrators) 1 2 3 4 5 6 7 8 Human Resource Leadership Strategic Leadership Instructional Leadership Cultural Leadership Managerial Leadership External Development Leadership Micro- political Leadership Academic Achievement Leadership

6 6 6 3-Year Rolling Average Teacher = -0.3 1.0 + (-2.5) + 1.2 3
Rating from 2 years ago Rating from 1 year ago Rating from this year 1.0 + (-2.5) + 1.2 Standard 6 Standard 6 Standard 6 3 Contribute to Academic Success Contribute to Academic Success = -0.3 Met Expected Growth 3- year average rating on standard 6 for determining status 1.0 Met Expected Growth -2.5 Did not meet Expected Growth 1.2 Met Expected Growth On this slide, you see an example of how the rolling average works. Each year, the teacher received a sixth standard rating based on student growth during that school year. The three values roll up into a three-year average that is used as part of the status determination. Note: A similar methodology applies to principals as well.

/ ) 3 ) 1 2 3 4 5 and/or 6 6 6 Teacher Status
1. In Need of Improvement Standards 1-5 In the year 1 2 3 4 5 Demonstrate Leadership Establish Environment Know Content Facilitate Learning Reflect on Practice Any rating lower than proficient and/or A teacher is in need of improvement if any rating on standard 1 – 5 is developing or not demonstrated, or if the rating for standard 6 is “does not meet expected growth.” Of course, if a teacher is not meeting expected growth, we expect that this will be reflected in one of the other standards. ) / Standards 6 Three year rolling average 3 ) 2 years ago 6 + 6 1 year ago + 6 This year Does Not Meet Expected Growth

/ ) 3 ) 1 2 3 4 5 and 6 6 6 Teacher Status 2. Effective
Standards 1-5 In the year 1 2 3 4 5 Demonstrate Leadership Establish Environment Know Content Facilitate Learning Reflect on Practice Proficient or Higher on Standards 1 - 5 and ) / Standard 6 Three year rolling average 3 ) 2 years ago 6 + 6 + 6 1 year ago This year Meets or Exceeds Expected Growth

/ ) 3 ) 1 2 3 4 5 and 6 6 6 Teacher Status 3. Highly Effective
Standards 1-5 In the year 1 2 3 4 5 Demonstrate Leadership Establish Environment Know Content Facilitate Learning Reflect on Practice Accomplished or Higher on Standards 1 - 5 and This is a very high bar. Please remember that proficiency is what we expect for ratings on standards 1 – 5, and meeting expected growth is what we expect for standard 6. Effective teachers are doing what they need to do for their students. Highly effective teachers are those going above and beyond expectations. ) / Standard 6 Three year rolling average 3 ) 2 years ago 6 + 6 1 year ago + 6 This year Exceeds Expected Growth

Teacher Status – First Status
For all teachers (and principals) the first status for Standard 6 will be generated from the best two out of three valid Standard 6 ratings. School-level growth that has been assigned to a teacher as a result of a waiver (from NCFEs or ASW) will function as a valid Standard 6 rating. School-level growth that has been assigned as a result of a lack of data for a teacher (i.e., not from a waiver) will not count as a valid Standard 6 rating.

6 6 6 Status Scenarios 1.0 Met Expected Growth
Rating from Rating from Rating from 6 6 6 Teacher has individual-level data for three years. Standard 6 from the school year is the lowest of the three ratings. Teacher’s Standard 6 status is 1.1 – “Meets Expected Growth”. 1.0 Met Expected Growth -2.5 Did not meet Expected Growth 1.2 Met Expected Growth

Rating from Rating from Rating from 6 6 6 Teacher has individual-level data for the first two years. The data is school-level growth from a waiver. Standard 6 from the school year is the lowest of the three ratings. Teacher’s Standard 6 status is 1.1 – “Meets Expected Growth”. 1.0 Met Expected Growth -2.5 Did not meet Expected Growth 1.2 Met Expected Growth

Rating from Rating from Rating from Teacher has individual-level data for the final two years. The data is school-level growth because teacher did not have individual-level data. The teacher does not receive a status in the fall of 2015 because teacher does not have 3 years of valid data. First status in Fall 2016 (provided teacher has valid data in SY ). 6 6 6 1.0 Met Expected Growth -2.5 Did not meet Expected Growth 1.2 Met Expected Growth

Status Scenarios – Second Year
Rating from Rating from Rating from Rating from Teacher receives second status in fall of 2016. Rating from “rolls off”. Rating from returns to the rolling average (even though it was dropped from prior year’s calculation). Teacher’s status is “Meets Expected Growth” with an average of 0.9. 6 6 6 6 1.0 Met Expected Growth -2.5 Did not meet Expected Growth 1.2 Met Expected Growth 4.0 Exceeded Expected Growth

Student Growth and Teacher Effectiveness

Weight of Standards The six standards (eight for principals) are weighted equally in the determination of teachers’ effectiveness ratings. In practice, however, student growth carries much more weight in differentiating teachers in terms of effectiveness. Nominal vs. Effective Weighting

Weight of Standards Status (Observation) → Needs Improvement Effective
Highly Effective Total Status (Obs + Growth) ↓ 1416 4658 2648 8722 (22.8%) 14049 10106 24155 (63.1%) 5383 (14.1%) (3.7%) 18707 (48.9%) 18137 (47.4%) 38260

Weight of Standards Standard 6, student growth, plays a greater role in determining teacher effectiveness ratings than observational data. The disproportional effect of student growth is an artifact of the lack of variation in observational data, not a value judgment. More accurate assessment of teacher performance can improve this phenomenon.

Correlations Between Standards and Growth
SY SY

What this tells us: Student Growth, as measured using value-added data, and ratings on Standards 1-5 continue to be very, very loosely correlated. Additionally, it appears that we are measuring one, holistic element of “good teaching” rather than five discrete standards of practice.

Results from Fall READY Principals

Understanding EVAAS Growth

General Method of Estimation
Student Raw Score (26/50) Conversion to Scale Score (250) Difference Between Expectation and Actual Scale Score (240 vs. 250) Positive 10 Scale Score Points is compared to distribution at the state level for that grade and subject. (e.g., 65th %tile) Percentile Rank is converted to NCE (~ 58 NCE) Deviation from the mean (50th NCE) for each student is aggregated at the teacher (or school, or district) level. The mean difference is the teacher effect which has an associated standard error The teacher effect is divided by the standard error to create the index. Index>=2 – Exceeds 2>=Index>2 – Meets Index <-2 – Does Not Meet

Teacher- and School-Level Growth
Student A -1.9 -2.2 -0.4 Student B -1.2 -1.6 -0.5 Student C -0.7 1.6 Student D -1.0 Student E 0.5 -1.5 Mean -0.9 Std Dev 0.9 1.5 Std Error Index -1.7 -2.0 -1.8

Teacher- and School-Level Growth
School A Teacher 1 Teacher 2 Teacher 3 Student A -1.9 -2.2 -0.4 Student B -1.2 -1.6 -0.5 Student C -0.7 1.6 Student D -1.0 Student E 0.5 -1.5 Mean -0.9 Std Dev 1.0 Std Error 0.3 Index -3

Proficiency and EVAAS How can a school increase proficiency rates by X percentage points but not meet growth? Prior Year (Expected) Score Prior Year NCE Current Year Score Current Year NCE Growth Student A 229 54 231 55 1 Student B 250 62 238 57 -5 Student C 255 64 236 56 -8 Student D 226 53 230 Student E 228 232 Student F 243 59 235 -3 Student G 225 52 2 Student H -1 Student I 227 220 50 Student J -2 Prior Year Current Year Mean -1.7 Proficiency = 230 % Proficient = 50% %Proficient = 90% Std Dev 3.2 Std Error 0.8 Index -2.1

Does Evaluation Matter?

Does Evaluation Matter?
Taylor, E.S, & Tyler, J.H. (2012). The effect of evaluation on teacher performance . American Economic Review, 102(7), See more at:

Does Growth Matter? Two groups of students captured:
Younger Cohort: began Grade 5 in SY Older Cohort: began Grade 6 in SY Low1 High2 Younger Math 1935 1825 Reading 184 221 Older 1560 1547 183 301 1Students in the “Low Growth” category have consecutive years of teachers in the Does Not Meet Expected Growth category 2Students in the “High Growth” category have consecutive years of teachers in the Exceeds Expected Growth category

Evaluation and Change in Growth

Questions, Comments, and Feedback
Thomas R. Tomberlin Director, District HR Support

Reducing the minimum length of reading camps
HB 230 Clarifications Reducing the minimum length of reading camps Camps now referred to as reading camp Reading camp shall: - Offer at least 72 hours of reading instruction to yield positive reading outcomes for participants - Allow volunteer mentors to read with students at time other than during the 72 hours of reading instruction -72 hours of reading instruction shall be provided over no less than 3 weeks for students in schools using calendars other than year-round calendars -Funding reverts on October 31, 2014 The law struck the reference as summer reading camps and now they are just called reading camps. The camps must still fulfill the hour requirement of 72 hours and has to be at least 3 weeks in length in traditional schools

Student Attendance in Summer Reading Camps
HB 230 Clarifications Student Attendance in Summer Reading Camps parents or guardians of students not demonstrating reading proficiency shall be encouraged to enroll their student in a reading camp Parent/guardian shall make the final decision regarding student reading camp attendance LEAs shall provide a least one opportunity for students not participating in a reading camp to demonstrate reading proficiency appropriate for third grade students Students are not required but are encouraged to attend. As it was before, parents decide whether to send their children to reading camp. Even if students do not attend camp, the LEA should give that child an opportunity to take an alternative test (RtA or local) on a designated day. The LEA decides the day. The DPI portfolios can only be completed at an LEA or charter reading camp.

Revises Good Cause Exemptions
HB 230 Clarifications Revises Good Cause Exemptions Limited English Proficient: students with less than two school years of instruction in an ESL program Students who demonstrate reading proficiency appropriate for third grade students on an alternative assessment approved by the SBE (took out wording on when to administer) Multiple retention language stays the same Portfolio: Student reading portfolio and review processes used by LEAs shall be approved by the SBE LEP – this adds the words “school” to the description. DPI had already provided for that flexibility this past year. Alternative assessments – this gives the LEAs the flexibility to give their local alternatives before the 10 day period at the end of the year. The DPI RtA test would still be given after the administration of the EOG in the last 10 days because of the standardized method of administering this test. Multiple retention Good Cause stayed the same. Portfolio – Good Cause – a local portfolio shall be approved by the SBE. A process will be developed for reviewing any local processes that are submitted. The review process will be developed by representatives from the field. The DPI portfolio will still be available for districts that want to use it. There are more details about the portfolio later in the presentation.

Revises Good Cause Exemptions
HB 230 Clarifications Revises Good Cause Exemptions Students with disabilities whose IEPs indicate: - the use of NCEXTEND 1 - at least a two school year delay in educational performance (guidance from EC division) - receipt of intensive reading interventions for at least two school years Of course, this is going forward. Extend 2 children were included this year, but there will not be an Extend 2 after this year. The EC division has developed guidelines for looking at students with a two year school delay. (Provide hard copies here) Last bullet – two years with an IEP that includes intensive reading interventions (the child should have had an IEP for two years)

Ways to Show Proficiency
BOG EOG EOG retest RtA Alternative Test Reading 3D at Level P (with comprehension) SBE approved Local Alternative Assessment Completed Portfolio (state or local)

Retained Student Placement
HB 230 Clarifications Retained Student Placement Retained in third grade Accelerated reading class – third grade standards and curriculum Placed in 3/4 Transition Class with Retained Reading label – fourth grade standards and curriculum Placed in 4th Grade Accelerated reading class with Retained Reading label – fourth grade standards and curriculum At the end of summer, principals and teachers most familiar with the students should decide on the placement of students for the following year. There are 3 choices for placement for those students not proficient in reading. Retained in 3rd grade and placed in an accelerated reading class – these children are being taught third grade standards and curriculum and are pulled out for 90 minutes of intense reading instruction ¾ Transition Class – these students are being taught 4th grade standards and curriculum and have a 90 minute uninterrupted ELA block within the class taught by a teacher who has demonstrated student outcomes in reading proficiency. We can go into further detail here about our PD that we can offer on how to structure these 90 minute blocks within the class. It is going to be important to emphasize that these classes CANNOT be whole group instruction for 90 minutes but MUST include small reading group work. We also recommend that the retained reading students have an additional time during the day (25-30 min) of intervention time. 4th grade Accelerated reading class – these students are being taught 4th grade standards and curriculum. It is a pull out situation where students receive 90 minutes of uninterrupted intensive reading instruction outside of the classroom. This requires a lot of collaboration and coordination between the regular classroom teacher and the reading teacher. Also, a principal can decide to retain a child NOT BASED SOLELY ON THE READING LAW OR READING PERFORMANCE. Principals can make a decision to retain a child for reasons other than reading. These children would not necessarily have to be in an accelerated reading class unless they are non-proficient in reading from the year before. The decision for placement should not just be based on one score or reading camp attendance. It should be a comprehensive look at the student from their previous year in third grade – formative, diagnostic results, benchmarks, summatives, report card grades, classroom work, classroom observations and reading camp performance (if applicable)

Local Portfolio SBE approved policy on process for submission in Sept.
Proposals for local alternative portfolios must follow requirements of policy Reviewed by Portfolio Advisory Committee Recommendations to SBE by PAC

Numbers and Percentages
Must be reported to SBE by Sept 1 and posted on district website Results will be included in the October state board report This report goes to the General Assembly on October 15

What is Included Demonstrating and not demonstrating proficiency on EOG Exempt for Good Cause Passed Alternative Assessment (state or local) Students retained (placement in third grade, 3/4 Transition or 4th grade accelerated) Charter – retained students who did not return

Mid-Year Promotion November 1(SBE policy)
RtA alternative test, local alternative test, completed portfolio, Reading 3D at Level P (can use all, if needed) After Nov. 1, principal can grade and classify MUST still show proficiency of third-grade standards (local alternative, portfolio, or Reading 3D at Level P)

Classes Continue After Nov. 1, transition and accelerated classes continue No matter the results 90 minutes of uninterrupted reading instruction Recommended 30 more of interventions outside of the 90 minutes

Exempt for Good Cause “…but shall continue to receive instructional supports and services and reading interventions…” Definition – means intentional strategies used with a majority of students to facilitate reading development and remediate emerging difficulty with reading development Include but not limited to – small group instruction, reduced teacher/student ratios, frequent progress monitoring, and extended learning time

Tammy Howard, Director Accountability Services
Read to Achieve: Local Alternative Assessments Tammy Howard, Director Accountability Services

HB 230 “The State Board of Education shall (i) provide several valid and reliable alternative assessments to local school administrative units upon request, (ii) approve valid and reliable alternative assessments submitted by local school administrative units, and (iii) establish achievement level ranges for each approved alternative assessment. The State Board of Education shall annually review all alternative assessments to ensure ongoing relevance, validity, and reliability."

Process Reviewed assessments submitted by LEAs for the school year Reviewed technical information Identified Lexiles as a meaningful way to compare the assessments Linking studies completed for most of the assessments Allows comparison to the Level 3 cut point on EOG (439) with a Lexile of 725

Recommendations to SBE
Recommending assessments with Lexiles Recommending assessments without Lexiles as pending the completion of a linking study Case Assessments by TE21 Discovery Education Recommending not including item banks SchoolNet, ClassScape and Study Island However, districts may arrange a Lexile linking study and submit for approval Also, selections may be used to build a local portfolio (must be approved)

Garron Gianopulos, Psychometrician Test Development
Grade 3 Expectancy Table Garron Gianopulos, Psychometrician Test Development

Beginning of Grade 3 & EOG3 Expectancy Table 2013-14
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

BOG3 Score Distribution
All third graders with BOG3 scores and EOG3 scores within a school were included. Level 1 was divided into three roughly equal sized groups. Total count of 103,394. (2013) Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways.

Highlighted cells indicated most likely achievement level given a student’s BOG3 scale score.
Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways.

69% of students from the scale score range 408 to 421 on the BOG3 stayed within Achievement Level 1 on the EOG. Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways.

23% of students from the scale score range 408 to 421 on the BOG3 reached Achievement Level 2 on the EOG. Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways.

8% of students from the scale score range 408 to 421 on the BOG3 became grade level proficient.
Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways.

Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways. *The total count excludes records that did not have an EOG3 score and students who transferred schools.

Example Questions: 1. What was the most likely EOG achievement level for a student scoring a level 2 on the BOG3? Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways. *The total count excludes records that did not have an EOG3 score and students who transferred schools.

Example Questions: 1. What was the most likely EOG achievement level for a student scoring a level 2 on the BOG3? Answer: Level 4 Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways. *The total count excludes records that did not have an EOG3 score and students who transferred schools.

Example Questions: 2. What was the most likely EOG level for a student scoring at level 3 on the BOG3? Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways. *The total count excludes records that did not have an EOG3 score and students who transferred schools.

Example Questions: 2. What was the most likely EOG level for a student scoring at level 3 on the BOG3? Answer: Level 4 Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways. *The total count excludes records that did not have an EOG3 score and students who transferred schools.

Example Questions: 3. What was the likelihood that a randomly chosen student who scored a level 2 on the BOG3 reached a level 4 or 5 on the EOG3? Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways. *The total count excludes records that did not have an EOG3 score and students who transferred schools.

3. What was the likelihood that a randomly chosen student who scored a level 2 on the BOG3 reached a level 4 or 5 on the EOG3? Answer: 62% Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways. *The total count excludes records that did not have an EOG3 score and students who transferred schools.

Example Questions: 4. What is the probability that a randomly chosen student who scored a level 2 on the BOG3 in 2014 will reach level 4 or 5 on the EOG3 in 2015? Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways. *The total count excludes records that did not have an EOG3 score and students who transferred schools.

4. What is the probability that a randomly chosen student who scored a level 2 on the BOG3 in 2014 will reach level 4 or 5 on the EOG3 in 2015? Answer: If instruction remains unchanged, then the expectation remains the same as : 62% Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways. *The total count excludes records that did not have an EOG3 score and students who transferred schools.

Summary: 1. The percentages under the EOG3 achievement levels sum to 100 and describe only that row. Do not sum the columns. 2. BOG3 Level 1 was disaggregated to help differentiate levels of instructional need. 3. Expectations for defined by this table assume students are randomly chosen within a BOG3 score range and that instruction remains unchanged from 4. If schools reduce effort for level 2 students, the 83% proficiency may drop for those students. Therefore, we need to maintain or increase instructional support for level 2 and above while at the same time increasing instructional support for students at level 1. Talking points: Overview: 1. Explain how the table was made, 2. Explain how to read it, 3. Summarize take aways. *The total count excludes records that did not have an EOG3 score and students who transferred schools.

Hope Lung, Section Chief Test Development
Online Testing Update Hope Lung, Section Chief Test Development

2014–15 NCTP Summative Online Assessments Memo (9/5/14)
English II online mandated NCFEs available online Fall 2014: Eng III, Math II, Math III, G7 Sci, Phy Sci, Earth/Env, G8 SS, C&E, USH, WH Spring 2015: All EOG Grade 7 ELA/Reading & Math available online Projected Online Summative Assessment Participation Survey (9/29) NCTest

Secure Platforms Required for NCTest v5 (2014–15)
Required for online tutorials, released test forms/items, and summative assessments Options include: Secure browser for MS Windows & Apple OS X (version same as 2013–14) Native app for iPad Chromebook (Chrome OS) version with special restrictions NEW NCTest Chrome App for Chromebooks, Windows, and Macintosh workstations/laptops (alternative to secure browser)

NCTest v5 New Features Spring 2014 End-of-Year Survey (Thank you!)
Test Window Scheduler completed by LEA TC/TA Enter start/end test window dates Select test(s) by “test type”: EOC, NCFE ELA, NCFE math, NCFE science, NCFE social studies or EOG Generates enrollment School TC can review but not edit

NCTest v5 New Features (continued)
SIQ Tab All student SIQs visible on SIQ tab Reminder: Schools must check SIQ and record SIQs for LEP and 504 only Start a test Select test type Filter by alphabetical range (e.g., A–C, D–F)

NCTest v5 Technical Requirements
ourse/view.php?id=361

Training and Resources
Webinar being scheduled for week of October 13 LEA Technology Coordinator NCEducation Guides (screen shots for completing various tasks within NCEd) Regional Accountability Coordinators/Regional Computer Consultants

Top Help Desk Requests Username/password assistance
Test administrators not linked to school Setting test session on day of test Enrollments Failing to mark Computer Reads Test Aloud accommodation

Curtis Sonneman, Education Consultant Accountability Services
School Performance Grades Curtis Sonneman, Education Consultant Accountability Services

School Performance Grades Assignment of Grades
North Carolina General Assembly legislation requires assignment of School Performance Grades beginning with 2013–14 results Presented to State Board of Education for first time February 5, 2015 Legislation (budget bill) states “no earlier than January 15, 2015.” Available publicly in new NC School Report Card

School Performance Grades Highlights of Legislation
Calculated using 80% Achievement and 20% Growth Schools with no growth use 100% achievement Schools with no data receive no designation 15 Point Scale 2013–14 (2014–15 and beyond uses 10 Point Scale) A separate achievement score and grade for reading and math must be reported for schools serving grades K–8 Regional and charter schools assigned a “D” or “F” SPG must inform parents via a letter (§115C and §115C F)

School Performance Grades Highlights of Legislation
“If a school has met expected growth and inclusion of the school’s growth score reduces the school’s performance score and grade, a school may choose to use the school achievement score solely to calculate the performance score and grade.” May need to provide an opt in/out document for Superintendents/Charter Directors to sign

School Performance Grades Achievement Indicators
Elem/Middle EOG Mathematics EOG ELA/Reading EOG Science Math I (when applicable) Biology (when applicable) High Schools Math I English II Biology The ACT Math Course Rigor ACT WorkKeys Graduation Rate

School Performance Grades Achievement Indicator Standards
Summative Assessment Scores: Percent of students who score at or above Level 3 (Grade Level Proficiency) End-of-Grade Tests End-of-Course Tests Graduation Rate: Percent of students who graduate in four years (4-Year Cohort Rate) Passing Math III: Percent of graduates who successfully complete Math III The ACT: Percent of 11th grade students who score 17 or above (UNC System’s minimum composite score requirement) ACT WorkKeys: Percent of CTE concentrator graduates who achieve a Silver Certificate or above

School Performance Grades Growth Indicator and Reporting
EVAAS School Accountability Growth Composite Index Score Generated Using End –of- Grade and End-of-Course Assessments Assigned an EVAAS Growth Status reported with School Performance Grades Exceeds Expected Growth Meets Expected Growth Does Not Meet Expected Growth Converted to 100 Point Scale for School Performance Grades Composites Capped Point Scale

School Performance Grades
Calculating the Grade (Middle School Example) School Achievement Score Add all indicator numerators (met indicators standard) and divide by all indicator denominators (students included in indicators) to determine School Achievement Score Indicator Numerator Denominator Score Total Achievement Score EOG Reading (3–8) 200 300 66.7 EOG Math (3–8) 180 60.0 EOG Science (5 & 8) 60 100 EOC Math I 30 35 85.7 EOC English II EOC Biology The ACT (UNC System 17) ACT WorkKeys (Silver or Better) Passing Math III (Math Course Rigor) 4-year Graduation Rate Total 470 735 63.9

Calculating the Grade (Middle School Example) Growth Score EVAAS School Accountability Growth Index score is converted to a 100 point scale and given a designation Converted Score and Designation are reported on School Report Card Composite Index Score Converted Score Status Growth Score 10.00 100.0 Exceeds Expected Growth

Calculating the Grade (Middle School Example) Final Score and Grade 80% Achievement Score and 20% Growth Score Set on a 15 Point Scale Calculated using data rounded to the nearest 10th. Prior to assignment of grades and for reporting purposes scores are rounded to the nearest whole (except for growth, kept at 10th for reporting.) Score Multiply by Input for Final Grade School Achievement 63.9 .80 51.1 Growth 100.0 .20 20.0 Final Score and Grade Reported 71 B Reading Score and Grade Reported 67 C Math Score and Grade Reported 60

Calculating the Grade (High School Example) School Achievement Score Add all indicator numerators (met indicators standard) and divide by all indicator denominators (students included in indicators) to determine School Achievement Score Indicator Numerator Denominator Score Total Achievement Score EOG Reading (3-8) EOG Math (3-8) EOG Science (5 & 8) EOC Math I 175 230 76.1 EOC English II 223 240 92.9 EOC Biology 198 229 86.5 The ACT (UNC System 17) 162 254 63.8 ACT WorkKeys (Silver or Better) 50 56 89.3 Passing Math III (Math Course Rigor) 244 250 97.6 4-year Graduation Rate 238 95.2 Total 1,290 1,509 85.5

Calculating the Grade (Middle School Example) Growth Score EVAAS School Accountability Growth Index score is converted to a 100 point scale and given a designation Converted Score and Designation are reported on School Report Card Composite Index Score Converted Score Status Growth Score -1.99 70.0 Meet Expected Growth

Calculating the Grade (Middle School Example) Final Score and Grade 80% Achievement Score and 20% Growth Score Set on a 15 Point Scale Calculated using data rounded to the nearest 10th. Prior to assignment of grades and for reporting purposes scores are rounded to the nearest whole (except for growth, kept at 10th for reporting.) School meets (exceeds) growth and growth lowers final score and grade, use achievement score (school option) Score Multiply by Input for Final Grade School Achievement 85.5 .80 68.4 Growth 70.0 .20 14.0 Final Score with Growth (Grade) 82 (B) Final Score no Growth (Grade) 86 (A) Final Score and Grade Reported 86/A 93

Additional Notes All accountability business rules are applied to school performance grades Banked scores for high school EOC as part of achievement calculation After 2013–14 reporting in January 2015, future reporting is expected to occur at beginning of school year in conjunction to release of accountability results. SPG will be one piece of the reporting. All accountability results will be reported through NC School Report Card.

Alternative School Accountability Model Curtis Sonneman, Education Consultant Accountability Services

Alternative School Accountability
Current Status According to General Statue for alternative schools “As part of its evaluation of the effectiveness of these programs, the State Board shall, through the application of the accountability system developed under G.S. 115C and G.S. 115C , measure the educational performance and growth of students placed in alternative schools and alternative programs. If appropriate, the Board may modify this system to adapt to the specific characteristics of these schools.”

Current Status Current model for alternative schools is Local Options Model for use in ABCs (GCS-C-013 and GCS-C-022) With elimination of ABCs model current policy is out of alignment with READY and School Performance Grades With creation of School Performance Grades a new model better suited for providing designations to alternative schools is warranted

Alternative School Accountability Timeline of model development
August State Board Meeting Discussion Item Gathered feedback from SBE regarding the intent for evaluating alternative schools Agreed in principle that these schools should have a different evaluation other than A–F letter grades Acknowledged that another model should be designed for alternative schools August-September Gathered input on an optimal model

Present model ideas Accountability Coordinators Fall Meeting September 23, 2014 North Carolina Association of Alternative Educators Annual Conference September 26, 2014 Alternative Schools Webinar September 29, 2014

November State Board Meeting Propose new alternative schools accountability model effective with 2014–15 Receive approval to remove alternative schools from A–F model for 2013–14 December-January If approved, Local Options will not be collected Data collection processes will be reviewed and any changes will be made and provided to field

New Model Proposal New model will create three options for alternative schools to be evaluated Participate in A–F system in exact same way as all other schools Return all results back to sending schools and receive no designations Participate in Alternative Progress Model

New Model Proposal Participate in A–F system Schools elect to participate in A–F system and are evaluated in the same way as all other “traditional” schools in letter grade system School/Superintendent/Local Board decides at beginning of each school year (designated date) Return all test scores back to base school(s) School receives no designations, reporting notes all data sent back to base school

New Model Proposal 3. Participate in Alternative Progress Model School/Superintendent/Local Board decides at beginning of each school year (designated date) Alternative Progress Model measures performance of school compared to itself in the previous year School given a designation for reporting purposes Progressing School increases score by at least specified amount over previous year Maintaining School maintains score between a specified amount +/- over previous year Declining School decreases score by at least specified amount over previous year

Alternative Progress Model Components Student Persistence (20%) Student still in school at the end of the school year after attending an alternative school any time during the school year Measured as percent of students still enrolled in a school anywhere in North Carolina at the end of the school year after appearing in any data collection at the alternative school Student Achievement (20%) Percent students proficient based on 3-year average Total students numerator and denominator for 3-year Must be assessed at alternative school to be included in calculation Includes all available data used for calculating School Performance Grades

Alternative Progress Model Components Growth (60%) EVAAS Growth Working with EVAAS team to develop an alternative growth model Remove 140 day membership rule and calculate as a percentage for all students taking assessments at alternative school Use student enrollment data to provide percentage of student growth to alternative school if student is enrolled in another school in same LEA (remaining percentage is not attributed to other school) Use 3-years of or 3-year average growth measure to create index First 2 years will likely need to be 1-year result then 2-year average, before becoming 3-year rolling average

Alternative School Accountability Calculating Total Score
In order to find the baseline and to calculate future numbers use following methodology Calculation of Data for Current Year Component Component Score Multiply By Adjusted Points Student Persistence 85.6 .20 17.1 Student Achievement 14.7 2.9 Growth 64.8 .60 38.9 Total 41.8 Calculation of Change for Designation Year Points Rating 2014 38.4 * 2015 41.8 Improving Change +3.4

Tammy Howard, Director Accountability Services
General Updates Tammy Howard, Director Accountability Services

February 5, 2015 Release School Performance Grades
Based on EOGs/EOCs, The ACT, ACT WorkKeys, 4-Year Cohort Graduation Rate, Passing Math III Based on 80 % Achievement and 20 % Growth Reported on NC Report Card Schools with Grades 3–8 will also have a reading and mathematics grade reported NC Report Card will be released

Released Data Format (September 4)
2013–14 State, District, School Level EOG and EOC Report LEA Name School Name School Code Subject Percent Level 1 Percent Level 2 Percent Level 3 Percent Level 4 Percent Level 5 2013–14 State, District, School Level Other Indicators Report (Schools without grades 9–12 will only have data on EVAAS Growth Status) LEA Name School Name School Code Percent AMO Targets Met EVAAS Growth Status The ACT Percent ACT WorkKeys Percent Passing Math III Percent 4- Year Cohort Graduation Rate Percent 5- Year Cohort Graduation Rate Percent Graduation Project 2013–14 State, District, School Level AMO Targets Report LEA Name School Name School Code Subject Subgroup Target Denominator Percent Met Target Used Alternate Used Improvement Used Confidence Interval

Implementation of Five Achievement Levels: Effective with 2013–14 Data
Two Standards for Reporting Grade Level Proficiency (Level 3 and above) Used in School Performance Grades Reported on NC School Report Card except Annual Measurable Objective Targets (AMOs) College and Career Readiness (Level 4 and above) Used for AMO targets (Baseline 2012–13) Reported on NC School Report Card

Results Graduation Rate: 83.8 Comparing CCR (2012–13 to 2013–14)
Continuing to increase Also increased for all subgroups Comparing CCR (2012–13 to 2013–14) Increases for 10 of the 14 EOGs Math I and English II increased; Biology decreased The ACT and ACT WorkKeys increased

Results Growth Increased Questions about growth
A 3.2 increase; 71.5 to 74.7 met and exceeded growth Questions about growth How can a school have high achievement and not meet growth? Is it a given that a certain number of schools will have to be designated as not meeting growth

Details to Remember NC School Report Card and Data Reports will include banked scores (Math I and Biology) for schools starting with 9th grade Inclusion of Beginning-of-Grade 3 assessment in accountability reporting if achievement level is higher than End-of-Grade 3 ELA/Reading Credit by Demonstrated Mastery for EOC courses are included in accountability reporting if student successfully completes program in entirety (Phase 1 and Phase 2 in same accountability year)

Details to Remember Per 115C-238.29F(l)
“A charter school shall ensure that the report card issued for it by the State Board of Education receives wide distribution to the local press or is otherwise provided to the public. A charter school shall ensure that the overall school performance score and grade earned by the charter school for the current and previous four school years is prominently displayed on the school Web site. If a charter school is awarded a grade of D or F, the charter school shall provide notice of the grade in writing to the parent or guardian of all students enrolled in that school.”

HB 230: Details to Remember
Testing Window Waiver: “For the school year only…” Exclusion of Growth: “If a school has met expected growth and inclusion of the school's growth score reduces the school's performance score and grade, a school may choose to use the school achievement score solely to calculate the performance score and grade.”

Test Development Information on Blueprints Technical Documentation
Alignment Study

Fall 2014 Released NCFE ITEMS
mmon-exams/released-items/ Items previously administered to students May not reflect all assessed standards or range of difficulty Includes correct answer, percent correct and standard/clarifying objective Includes list of standards/clarifying objectives addressed by the released items Note: MSL Spring 2013 Released Forms to be reviewed and revised as needed 117

Fall 2014 Released NCFE Items
Item Number Type Key Percent Correct Standard Example MC MC D 55% CCSS.ELA-Literacy.RL Example CR CR Rubric 47% CCSS.ELA-Literacy.WHST A Percent correct for multiple-choice (MC) item is the percentage of students who answered the item correctly. Percent correct for a constructed response (CR) item is the percentage of students who scored a 1 or 2. 118

Technical Documentation
EOG, EOC and NCFE test specifications to be updated to include blueprint information (number or percent of items at each standard) testing/generalinfo Technical reports will be posted to ting/technicalnotes 119

NC Alignment Study Content analysis of assessments (fall 2014)
EOG Grades 3-8 ELA/reading and math EOG Grades 5 & 8 science EOC English II, Math I, Biology Instructional alignment (survey) Early spring 2015 Recruit approximately 700 teachers Professional development Summer 2015 120

Testing Policy and Operations
Security Audit Administration protocols Security processes Assessment Briefs Banked Scores Understanding the Five Achievement Levels Beginning-of-Grade 3 ELA/Reading Test

Iris Garner, NAEP Coordinator Testing Policy and Operations
NAEP Update Iris Garner, NAEP Coordinator Testing Policy and Operations

NAEP 2008 Regional Training
NAEP Overview Known as "the Nation's Report Card" Only nationally representative and continuing assessment of what America's students know and can do in various subject areas Authorized by Congress Administered by the National Center for Education Statistics in the U.S. Department of Education NCDPI Accountability Services Division

NAEP Overview State-level NAEP: reading, mathematics, science, and writing Results: national and state level No individual scores for individual students or schools Results: are available for subgroups of the population (e.g., race/ethnicity, gender, free/reduced lunch eligibility, etc.) Reporting components: subject-matter achievement, instructional experiences, and school environment for populations of students NCDPI Accountability Services Division

NAEP Overview Since 1969, assessments have been conducted periodically in reading, mathematics, science, writing, U.S. history, civics, geography, and the arts. Starting in 1990, NAEP assessments have also been conducted to give results for participating states. Since 1990, North Carolina has participated in the state-level assessments of NAEP. NCDPI Accountability Services Division

State Board of Education Policy
NAEP 2008 Regional Training NAEP Overview State Board of Education Policy HSP-A-001 (16 NCAC 6D .0302): “To ensure adequate representation and generalizability of the data used to develop tests and to conduct evaluation studies, selected LEAs and schools, determined through stratified random samples, shall participate in field testing and other sample testing such as NAEP and other national or international assessments as designated by the department or the State Board of Education.” NCDPI Accountability Services Division 126

NAEP Overview "No Child Left Behind" legislation stipulates that states, districts, and schools within districts that receive Title I funds must participate in NAEP if selected. Reading and mathematics must be administered every two years in grades 4 and 8. Students and schools that are selected to participate are kept confidential. NCDPI Accountability Services Division 127

NAEP 2015 NAEP Assessment Window: Jan. 26 – March 6

Participation Requirements Curtis Sonneman, Education Consultant Accountability Services

Participation Requirements
Current Information Memo sent to all superintendents/charter school directors TNN of memo posted September 15, 2014 by Jaime Kelley (Use Revised Memo) Includes sample letter for Year 1 participation requirements Explains LEA/charter school requirements for missing participation in each of 3-years in cycle NCDPI finalizing plan for Year 2 monitoring

Participation Requirements Yearly Consequences Overview
The minimum participation rate for all subgroups is 95%; must have a subgroup of 30 students to be applied Year 1: Letter to parents with plan for improving participation Year 2: School labeled as “consistently low-participating school” and develops an intensive action plan Year 3: non-participating students (to meet 95%) counted as not-proficient

Year 1 Explanation Year 1 Requirements Within 30 days after accountability results are approved school must send letter to parents Includes plan of action to ensure full participation for all subgroups targeting those missing participation Within 30 days a copy of letter from each school, including plan of action, submitted to Regional Accountability Coordinator (RAC) RAC submits to NCDPI with in 35 days Upon receipt NCDPI will contact schools not in compliance

Year 1 Explanation Year 1 Timeline October 2, 2014, Final results approved by SBE November 3, 2014, Letters due to parents and RACs November 7, 2014, Letters due to NCDPI November 14, 2014, NCDPI begins sending notifications regarding non-compliance

Year 2 Explanation Year 2 Requirements Schools designated “consistently low-participating school” In School Report Card and documentation on Accountability website Schools create and implement an intensive intervention plan Plan submitted to LEA Testing Coordinator/RAC (charters) on or before November 1, 2014 LEA Testing Coordinators/RACs will verify submission with Accountability Services NCDPI implements state monitoring plan

Year 2 Explanation NCDPI Monitoring Plan Desk Monitoring Ensure that all schools have submitted a plan to LEA Testing Coordinator/RAC Randomly selected schools will be asked to submit plans to NCDPI Pre-site review Plans examined and schools identified for on-site review Additional documentation may be requested

Year 2 Monitoring Plan NCDPI Monitoring Plan (cont.) On-site review Schools selected will be visited by NCDPI staff Visit will occur no later than 1 month prior to testing Review/Discuss documentation submitted Strengths and weaknesses State Response and Targeted Assistance Provide recommendations and targeted assistance Letter summarizing visit mailed to LEA superintendent/charter director within 90 days of on-site visit

Participation Requirements Schools in both Year 1 and Year 2
Schools with subgroups in both years must complete both requirements Are not required to include information about Year 2 in Year 1 letter, but are encouraged to do so Are not required to include Year 1 subgroups in intensive intervention plan for Year 2, but are encouraged to do so

Year 3 Explanation Year 3 Requirements NCDPI counts non-participating students as not proficient Number added will be the number of students needed to raise participation to 95% These students will be added to the denominator of the AMO performance target Within 30 days after accountability results are approved school must send letter to parents Includes plan of action to ensure full participation for all subgroups targeting those missing participation

Year 3 Explanation Year 3 Requirements (cont.) Within 30 days a copy of letter from each school, including plan of action, submitted to Regional Accountability Coordinator (RAC) RAC submits to NCDPI with 35 days Upon receipt NCDPI will contact schools not in compliance

Ken Barbour George Stubblefield Analysis and Reporting
Participation Requirements Ken Barbour George Stubblefield Analysis and Reporting

Accountability Information Technology Team
Update on 20th day files for 2015 Setting up the New computers Moving Winscan to new computers Moving data to new computers When will we get Office? Who uses MYSQL in ReadyTools? Do you need training for Ready Tools Training other Questions on SSH Other questions 142

A new WinScan32 release for 2014-15
Accountability Information Technology Team A new WinScan32 release for CCRAA Grade 10 – PLAN Alternate October testing window No scores available at this time (January 2015?) Scanned locally Note that the CCRAA Grade 11 (March 2015) will be sent to TOPS for processing

Flexible Testing Waivers
Accountability Information Technology Team Flexible Testing Waivers EOC Modify the “TestDate” to differentiate from Summer NC Final Exams: use Spring 2014 tests but scan using files in the NCFE14Fall folder Can use grade conversions Need to keep separate from the Dec. NCFE tests which will be new and have different answer keys and need new grade conversions

Accountability Information Technology Team
For the EOC Flexible Testing Waivers and EOC Credit by Demonstrated Mastery Continue to download V14 ODF files to \Scan1314\ODF English II Delay in Scoring When the new EOC and NCFE tests are available in December (perhaps mid-November), the downloaded online files will be stored in \Scan1415\ODF and have a V15 file name extension Users might need to update the settings in READYTools

Every Student READY Questions?

Accountability Directors and Test Coordinators Fall Meeting

Similar presentations

Presentation on theme: "Accountability Directors and Test Coordinators Fall Meeting"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Accountability Directors and Test Coordinators Fall Meeting

Similar presentations

Presentation on theme: "Accountability Directors and Test Coordinators Fall Meeting"— Presentation transcript:

Similar presentations

About project

Feedback