Consortium for Educational Research and Evaluation– North Carolina The NC Race to the Top Evaluation: An Update CCRESA Board of Directors May 17, 2013 Jessica Anderson, SERVE Center, UNCG Julie Marks, CIPP, UNC-CH Trip Stallings, Friday Institute, NCSU

Consortium for Educational Research and Evaluation–North Carolina NC Race to the Top Evaluation: 2013 Forward NC RttT is designed to be a “game changer” A coordinated set of innovative activities and policy reforms designed to collectively improve the performances of students, teachers, leaders, and schools Evaluation contributes to NC RttT “changing the game” in two ways: 1.Program Evaluation: Provide formative information on the implementation of NC RttT initiatives Inform decisions to improve implementation 2.Policy Evaluation: Assess – from the perspective of students, teachers, leaders, and schools – the improvements that have occurred as a result of NC RttT initiatives collectively and individually: Includes LEA-level outcomes for some initiatives Inform decisions about sustainability and impacts 2 Ending in 2013

Consortium for Educational Research and Evaluation–North Carolina Focus of the Evaluation External evaluation provides objective analysis of the activities described in NC’s RttT grant proposal:  Implementation fidelity  Short-term outcomes  Collective/overall impact Even with the shift from formative to summative, however, this is still not an evaluation of specific teachers, leaders, or schools 3

Consortium for Educational Research and Evaluation–North Carolina NC RttT Initiatives: Evaluation Organization 1.Teacher and leader effectiveness Integration of value-added student achievement measures into educator evaluation system 2.Equitable supply and distribution of teachers and leaders Teach for America, NC Teacher Corps, Regional Leadership Academies, Teacher Induction Program, Virtual Public School, Incentives 3.Professional development All professional development activities in support of RttT initiatives, including: PD for standards and assessment, IIS, and data use; and PD delivery capacity-building efforts 4.Turnaround of LEAs and schools Low-achieving LEAs and schools; STEM schools 5.Local-level implementation and spending on RttT Cloud computing, allocation of RttT funds, cost savings 6.Overall impact of RttT on students, teachers, and school leaders Cross-Initiative: Omnibus survey of teachers and principals 4

Consortium for Educational Research and Evaluation–North Carolina Key Findings, ( Value-Added Models – Evaluation of nine commonly-used teacher value-added models suggests that four models (a three-level hierarchical linear model with one year of pretest scores, a three-level hierarchical linear model with two years of pretest scores, EVAAS, and a student fixed effects model) are sufficient for lower-stakes purposes. However, these models may not perform sufficiently well for high-stakes purposes. They typically do not fully account for the effects of differences in classroom composition (effects on outcomes for individual students that are related to whole-class composition, and potential differences in teacher effectiveness on the same student in different classroom configurations). 5

Consortium for Educational Research and Evaluation–North Carolina Key Findings, ( Regional Leadership Academies – RLAs currently serve high-need schools in 31 LEAs. Each RLA has a program-specific philosophy, curriculum focused on instruction and school improvement, and well-designed and integrated coursework and field work. Relative to traditional North Carolina MSA Programs, short-term outcomes suggest that RLAs appear to be much more deliberate, effective, and successful in developing and incorporating critical, research- based features into their programs. Most Cohort 1 participants interned in and were later hired by targeted (high-need) schools and LEAs. Local Strategic Staffing – 18 LEAs developed full strategic staffing plans, and 55 others developed partial plans. Highlights: emergence of 2 nd -generation strategic staffing plans; potential for sustainability; diversity in strategic staffing approaches; and opportunities for idea-sharing across LEAs. 6

Consortium for Educational Research and Evaluation–North Carolina 7 Key Findings, (

Consortium for Educational Research and Evaluation–North Carolina New Teacher Support Program – Most NC NTSP teachers in the first cohort reported feeling confident in their teaching ability after the Summer Institute, and early evidence from classroom observations suggests that they may be slightly more proficient as instructors, but that they also may have slightly less self-efficacy than comparison teachers. NCVPS First-Year Report, Pt. I – Courses are reaching students typically under-represented in STEM (females, minorities, etc.), and the Forensics course in particular exhibits multiple strengths. Technology tools, online resources, and project-based learning components are being integrated into courses with increasing frequency. Face-to-face and online teacher relationships are strong and constructive. 8 Key Findings, (

Consortium for Educational Research and Evaluation–North Carolina Statewide Professional Development – State-provided professional development in focused on four RttT priorities: (1) transition to new state standards; (2) implementation of formative and summative assessments; (3) use of data to support instruction; and (4) effective use of the North Carolina Educator Evaluation System. Data indicate that the overall quality of events was high. The State is clarifying state and local professional development needs, though providing adequate time and resources to meet these needs remains a challenge. Participants generally believed that these events helped them develop knowledge and skills related to these priorities. LEA-Level Professional Development – Overall, schools and districts appear to have taken initial steps toward crafting processes for providing professional development that supports the four RttT priorities listed above. 9 Key Findings, (

Consortium for Educational Research and Evaluation–North Carolina Distinguished Leadership in Practice – In Y1, DLP sessions were conducted for 194 principals (157 completed the program). Principal feedback and observation data indicate that face-to-face and online sessions were of high quality. Principal feedback and focus group data suggest that partici- pants developed specific leadership knowledge and skills. Online Professional Development – Approximately half of the state’s educators completed at least one online module by the end of the school year. Most participants agreed that webinars and modules were relevant to their profess- ional development needs, though some content was redundant and not always content- or grade-specific, and some modules did not meet NSDC standards. Most modules were completed independently and not in PLCs. 10 Key Findings, (

Consortium for Educational Research and Evaluation–North Carolina NC Teacher Corps & TFA – NCTC recruitment yielded 441 candidates, of whom 34 were selected (29 remain the in program; 22 have teaching jobs). TFA placed 157 corps members in , and 219 in —many in teaching “pods” of 3 or more. Ca. 90% of TFA corps members complete two-year commitments. LEA Expenditures – Total RttT funds allocated to LEAs averages to $36 per pupil per year. Based on analyses of DSWs, over the full four years of the grant, LEAs plan to use the largest proportion of their RttT funds (49%) for technology, followed by professional development (21%) and strategic staffing (15%). In terms of planned activities, LEAs plan to prioritize RttT funds for professional development (43%) followed by technology (24%) and strategic staffing (20%). 11 Key Findings, (

Consortium for Educational Research and Evaluation–North Carolina DST – In low-achieving districts, connections are weak or missing between and within central offices, schools, and classrooms. DST’s strategy at the district level is essentially the process of changing a disconnected district into a productively connected district. Early qualitative and quantitative evidence suggests that DST is making a measurable contribution to the improvement of student performance rates and student graduation rates in the schools it serves. DST PD – Data suggest that, overall, DST successfully constructed and delivered a high-quality professional development program to participants through its School Leaders Professional Development Series. 12 Key Findings, (

Consortium for Educational Research and Evaluation–North Carolina STEM Affinity Network: Second-Year Report – Anchor schools have improved instruction and implemented STEM features (such as project-based learning and partnering for improvement of student learning) internally. Structures for networking, PD, curriculum development, and partnerships are in place to support affinity schools, though some of these activities have been delayed. There is not yet universal buy-in among anchor school staff, but many report improved instruction and implementation of STEM strategies. Students in anchor schools enjoy personalized attention and exhibit high motivation, engagement, and a passion for learning. 13 Key Findings, (

Consortium for Educational Research and Evaluation–North Carolina Omnibus Teacher and Leader Survey Administered to probability sample of 365 schools across the state – a selection of schools that is reflective of the state as a whole Assesses “Instructional Climate”  23 dimensions on leadership and organizational conditions affecting instruction Baseline and Second Round Surveys completed – Fall 2011, Spring 2012 Next administration in progress now (launched February 2013) and again in 2014 (last administration) 14

Consortium for Educational Research and Evaluation–North Carolina School-Level Activities Reports: s/evaluationdb/index.html s/evaluationdb/index.html 15

Consortium for Educational Research and Evaluation–North Carolina 16

Consortium for Educational Research and Evaluation–North Carolina CERE-NC Reports:

Consortium for Educational Research and Evaluation–North Carolina 18

Consortium for Educational Research and Evaluation–North Carolina NC RttT Evaluation: Team Consortium for Educational Research and Evaluation-NC (CERE-NC): SERVE Center, Carolina Institute for Public Policy, and Friday Institute Steering Committee: Gary Henry, Terri Shelton, & Glenn Kleiman Principal Investigator: Gary Henry Management Committee: Julie Marks, Jessica Anderson, and Trip Stallings Team Leaders: Teacher and Leader Effectiveness – Heather Higgins Equitable Supply and Distribution of Teachers and Leaders – Trip Stallings Professional Development – Jeni Corn Turnaround of LEAs and Schools – Charles Thompson Local-Level Implementation and Spending – Nate Barrett Overall Impact – Gary Henry and Julie Marks Other Leadership Roles: LEA Coordinator – Lynn Amwake 336 State Liaison - Trip Stallings

Consortium for Educational Research and Evaluation–North Carolina Jess Anderson. SERVE Center, UNCG (336) Julie Marks, CIPP, UNCCH (919) Trip Stallings, Friday Institute, NCSU (919)