Consortium for Educational Research and Evaluation– North Carolina Evaluating North Carolinas Race to the Top Initiatives: An Overview Collaborative Conference for Student Achievement March 21, 2012 Jeni Corn, Friday Institute, NCSU Trip Stallings, Friday Institute, NCSU

Consortium for Educational Research and Evaluation–North Carolina NC Race to the Top Evaluation NC RttT is designed to be a game changer A coordinated set of innovative activities and policy reforms designed to collectively improve the performances of students, teachers, leaders, and schools Evaluation contributes to NC RttT changing the game in two ways: 1.Program Evaluation: Provide formative information on the implementation of NC RttT initiatives Inform decisions to improve implementation 2.Policy Evaluation: Assess – from the perspective of students, teachers, leaders, and schools – the improvements that have occurred as a result of NC RttT initiatives collectively and, to the extent possible, the contributions of specific individual initiatives Inform decisions about sustainability and impacts 2

Consortium for Educational Research and Evaluation–North Carolina Focus of the Evaluation External evaluation provides objective analysis of the activities described in NCs RttT grant proposal: Implementation fidelity Short-term outcomes Collective/overall impact It is not an evaluation of specific teachers, leaders, or schools 3

Consortium for Educational Research and Evaluation–North Carolina NC RttT Evaluation: Team Consortium for Educational Research and Evaluation-NC (CERE-NC): SERVE Center, Carolina Institute for Public Policy, and Friday Institute Steering Committee: Gary Henry, Terri Shelton, & Glenn Kleiman Principal Investigator: Gary Henry Management Committee: Trip Stallings, Jessica Anderson, and Julie Marks Team Leaders: Teacher and Leader Effectiveness – Rod Rose Equitable Supply and Distribution of Teachers and Leaders – Trip Stallings Professional Development – Jeni Corn Turnaround of LEAs and Schools – Charles Thompson Local-Level Implementation and Spending – Eric Houck Overall Impact – Gary Henry and Julie Marks Other Leadership Roles: LEA Coordinator - Alexa Edwards 336 State Liaison - Trip Stallings 919 4

Consortium for Educational Research and Evaluation–North Carolina NC Race to the Top Evaluation: Key Milestones in Year 1 Spring 2011: Evaluation designed and staffed, and baseline data collection initiated (e.g. interviews with RttT implementation leaders, Teacher & Principal Survey, local expenditures, field work in schools that have undergone turnaround) Summer 2011: Scopes of work for evaluation of initiatives finalized; Developed, piloted, & trained for instructional observation instruments; ongoing fieldwork Fall 2011: Completed draft reports and initial briefings for: Baseline analysis of School Turnaround, STEM, distribution of teacher quality, and PD, as well as plans for RLA cost-effectiveness; Began analysis of baseline Teacher & Principal Survey Winter : Conducted briefings on analysis of teacher value-added models; Finalized drafted reports ( and presented to SBE; began Y2 SOW activitieshttp://cerenc.org 5

Consortium for Educational Research and Evaluation–North Carolina NC RttT Initiatives: Evaluation Organization 1.Teacher and leader effectiveness Integration of value-added student achievement measures into educator evaluation system 2.Equitable supply and distribution of teachers and leaders Teach for America, NC Teacher Corps, Regional Leadership Academies, Teacher Induction Program, Virtual Public School, Incentives 3.Professional development All professional development activities in support of RttT initiatives, including: PD for standards and assessment, IIS, and data use; and PD delivery capacity-building efforts 4.Turnaround of LEAs and schools Low-achieving LEAs and schools; STEM schools 5.Local-level implementation and spending on RttT Cloud computing, allocation of RttT funds, cost savings 6.Overall impact of RttT on students, teachers, and school leaders 7.Omnibus survey of teachers and principals 6

Consortium for Educational Research and Evaluation–North Carolina 1. Teacher and Leader Effectiveness Purpose of this Evaluation Project To ensure quality, consistency, and fairness of new and ongoing teacher and principal evaluation processes through examination of validity and reliability across multiple observational perspectives To examine educators perspectives on new evaluation standards and the effect of these standards on educators practices 7

Consortium for Educational Research and Evaluation–North Carolina 1. Teacher and Leader Effectiveness Evaluation Questions Have valid and reliable measures of student growth been identified for inclusion in the teacher and administrator/principal evaluation process? Does the revised evaluation process allow for/make meaningful distinctions between teachers and administrators effective and ineffective performance? How do educators view the implementation/rollout of the evaluation process? Does the new evaluation process change educators attitudes? Does it change educators practices? Do performance incentives for teachers in low-performing schools have positive effects on student and teacher outcomes? 8

Consortium for Educational Research and Evaluation–North Carolina 1. Teacher and Leader Effectiveness Evaluation Approaches Data sources: existing scholarship; quantitative data (including longitudinal DPI data and survey data collected from principals, teachers, and students); and qualitative data (including teacher and principal interviews, focus group data, and observations of teachers). Key Deliverables and Estimated Delivery A. Teacher and Leader Effectiveness: Evaluation Preliminary Report: Reliability analysis of multiple VA models1/2012 Report: Preliminary evaluation of contractor's proposed approach to measuring educator impact on student achievement (based on existing data) 3/2013 Final Report: Evaluation of new EES elements and their implementation9/2014 B. Teacher and Leader Effectiveness: Incentives Report: School-level bonuses8/2013 Report: School-level and individual teacher bonuses6/2014 Final Report: Summative evaluation9/2014 9

Consortium for Educational Research and Evaluation–North Carolina Purpose of this Evaluation Project To provide both summative and formative information about RttT efforts to increase the overall supply and to ensure the equitable distribution of effective educators statewide Evaluation Strands Baseline Regional Leadership Academies NCTC & TFA Expansion Strategic Staffing New Teacher Induction NCVPS Blended STEM Courses Equitable Supply and Distribution of Teachers and Leaders

Consortium for Educational Research and Evaluation–North Carolina Overall Evaluation Questions What is the nature and quality of the experiences provided by each of the initiative programs? Are students affected by each of these programs better off than students in schools and districts not served by these programs? Are these initiatives cost-effective and sustainable? To what extent did the initiatives further the goal of having an effective teacher in every classroom and an effective principal in every school? Equitable Supply and Distribution of Teachers and Leaders

Consortium for Educational Research and Evaluation–North Carolina Evaluation Approaches: Not just.... Observations Focus Groups Surveys Interviews Artifact Review Quant Analyses Accounting Data Baseline X RLAsX NCTC/TFAX Strategic StaffingX InductionX NCVPSX Equitable Supply and Distribution of Teachers and Leaders

Consortium for Educational Research and Evaluation–North Carolina Evaluation Approaches: Also... Observations Focus Groups Surveys Interviews Artifact Review Quant Analyses Accounting Data Baseline X RLAsXXXXXXX NCTC/TFAXXXX XX Strategic Staffing X XXXX InductionXXX X NCVPSXXXXXXX Equitable Supply and Distribution of Teachers and Leaders

Consortium for Educational Research and Evaluation–North Carolina Key Deliverables and Estimated Delivery (I) A. Evaluation of overall changes in distribution of higher-quality & effective teachers & leaders Develop baseline estimates of the distribution of higher-quality teachers and school leaders, revised from initial RttT proposal estimates. 4/2012 Develop estimates of changes and trends in the distribution of higher- quality teachers and school leaders. 6/2014 B. Evaluation of Regional Leadership Academies Cost-effectiveness analyses2/2012 Final 2012 activity report2/2013 Final 2013 activity report2/2014 Final Report9/2014 C. Evaluation of TFA Expansion & NC Teacher Corps (NCTC) Report: Characteristics and placement of TFA and NCTC candidates12/2012 Interim Report: NCTC impact on teacher retention9/2013 Final Report: Impact, qualitative assessment, and policy recommendations9/ Equitable Supply and Distribution of Teachers and Leaders

Consortium for Educational Research and Evaluation–North Carolina Key Deliverables and Estimated Delivery (II) D. Evaluation of Induction Program for Novice Teachers Preliminary report/briefing on 1 st year implementation12/2012 Report: Impact and implementation of the 1 st year of the 2-year cycle 7/2013 Final Report: Program effectiveness, implementation, and sustainability10/2014 E. Evaluation of Strategic Staffing Efforts Report: Local SS plan and implementation review 9/2012 Report: State SS Y1 & Y2 review 9/2013 Final Report: Summative evaluation of local and state SS 9/2014 F. Evaluation of NCVPS Blended Courses Initial Report: Estimates of blended course impact on teachers, students 4/2013 Report: Qualitative assessment of Y1 and Y2 course offerings 9/2013 Final Report: Impact, qualitative assessment, and policy recommendations 9/ Equitable Supply and Distribution of Teachers and Leaders

Consortium for Educational Research and Evaluation–North Carolina 3. Professional Development Purpose of this Evaluation Project To conduct ongoing analysis of the delivery and quality of state- and local-level professional development, with the goal of analyzing the impact of the PDI on local capacity, teacher practices, and student achievement. We will examine longitudinal education data combined with data collected using a sample of schools approach. 16

Consortium for Educational Research and Evaluation–North Carolina 1. State Strategies: Did State implement and support proposed RttT PD efforts as planned? 2. Short-Term Outcomes: What were impacts on educators knowledge and practices? 3. Intermediate Outcomes: To what extent was the NC education workforce updated? 4. Student Performance To what extent were student gains asso- ciated with RttT PD? 17 Major Evaluation Questions 3. Professional Development

Consortium for Educational Research and Evaluation–North Carolina Evaluation Approaches Interviews and focus groups Observations PD, teaching practices, PLCs, local PD Sessions Research-based review of PD content and delivery, including reviews of the NCDPI Online Repository and eLearning Portal diagnostics Data reviews and analyses administrative data, PDI-specific data (PDI Participation Database data, survey data including PD Exit Surveys, leadership inventory, reflection), NCEES summary data, student data including EOG/EOC, graduation rates; LEA PD expenditure data Document reviews LEA PD Action Plans, other PDI-specific documents Key Deliverables and Estimated Delivery First Annual Report 1/2012 Annual Report: Status of PDI12/2012 Annual Report: Status of PDI12/2013 Final Report: Impact 9/ Professional Development

Consortium for Educational Research and Evaluation–North Carolina 4. Turnaround of LEAs and Schools Purpose of this Evaluation Project To understand the extent to which and the ways in which interventions by the District and School Transformation division (DST) improve outcomes for students in the states lowest-performing schools and districts To explore the fidelity of implementation of the STEM Schools initiative and examine its impacts on students, teachers, principals, schools, and school networks. 19

Consortium for Educational Research and Evaluation–North Carolina 4. Turnaround of LEAs and Schools Turnaround of LEAs and Schools Evaluation Questions What problems are identified in the low-performing schools and districts? What are the main intervention strategies that the District and School Transformation unit employs to improve low-performing schools? What are the intended mechanisms of improvement? How do the strategies work? Do the strategies and mechanisms play out as intended? What is the impact of the intervention strategies on intermediate outcomes as well as student achievement and graduation rates? 20

Consortium for Educational Research and Evaluation–North Carolina STEM Anchor Evaluation Questions To what extent have the network of STEM anchor and cluster schools been implemented as intended? What are the impacts of the network of STEM anchor and cluster schools on student and on school-level outcomes and how do these impacts compare with the impacts of other transformation models? Can the impacts on student performance be disaggregated by student and school characteristics? What mechanisms are put in place for the sustainability and scaling up of the model, or its most successful elements? Turnaround of LEAs and Schools

Consortium for Educational Research and Evaluation–North Carolina 4. Turnaround of LEAs and Schools Evaluation Approaches DST: 30+ site visits, multiple interviews (over the next three years with multiple people), observations, focus groups, surveys, artifact review, document analysis, and quantitative analyses. STEM: Predominantly qualitative analyses (observations of professional development, site visits to STEM schools, interviews with providers), with quantitative analyses (student and school staff surveys, administrative data). Key Deliverables and Estimated Delivery A. Evaluation of District and School Transformation RttT Work Preliminary Baseline Report12/2011 Report: Formative Assessments of the Efforts to Transform the Lowest-Performing Schools1/2013 Annual Report12/2013 Final Report: Quantitative and qualitative findings 9/2014 B. Evaluation of STEM Anchor School System Development Baseline Scan and Year 1 report12/2011 & 3/2012 Year 2 Report12/2012 Year 3 Report12/2013 Year 4 Report 9/

Consortium for Educational Research and Evaluation–North Carolina 5. Local-Level Implementation & Spending Purpose of this Evaluation Project To determine how Race to the Top funding is being allocated and used across districts and schools throughout NC 23

Consortium for Educational Research and Evaluation–North Carolina Cloud Questions To what extent does the Cloud reduce state & local expenditures for technology? To what extent does the Cloud provide reliable, secure, accessible, and efficient service? How satisfied are LEAs with the Cloud Computing infrastructure? Local Spending Questions How do local districts spend RttT funds? Are some local RttT spending patterns associated with higher student performance in schools and districts? Local Efficiencies and Savings Questions Do RttT funds alter costs incurred by the state and districts? Local-Level Implementation & Spending

Consortium for Educational Research and Evaluation–North Carolina Evaluation Approaches Mixed-method approach, combining document review, interviews, surveys, and quantitative analysis of administrative data Key Deliverables and Estimated Delivery Conduct baseline analysis of local education expenditures in preparation for cost analyses, which will specifically include the technology initiative 4/2012 Site reports on local RttT spending; survey findings; baseline productivity report4/2013 Follow-up to 2012 report; interval and summative productivity report9/ Local-Level Implementation & Spending

Consortium for Educational Research and Evaluation–North Carolina 6. Overall Impact Purpose of this Evaluation Project To provide estimations of the overall impact of RttT- funded initiatives To explore under what conditions and circumstances the initiatives collectively and in various combinations appeared to be most effective, and for whom To consider sustainability options beyond the life of the grant To track and compare the metrics/goals defined in the proposal 26

Consortium for Educational Research and Evaluation–North Carolina 6. Overall Impact Evaluation Questions Was each RttT initiative implemented as intended? What are the overall impacts of RttT on increasing student performance, such as achievement, engagement, attendance, graduation? Are the impacts of RttT on student performance larger in some schools/districts than others (for example, high-poverty or low- performing schools)? Are some RttT initiatives more effective in increasing student performance than others? How can the successful RttT initiatives be sustained after 2014? 27

Consortium for Educational Research and Evaluation–North Carolina 6. Overall Impact Evaluation Approaches The Overall Impact evaluation will consider selected quantitative and qualitative data and results from all initiative-level evaluations, as well as cross-initiative and all-inclusive data Key Deliverables and Estimated Delivery Overall cost and sustainability analysis of the RttT initiatives12/2013 Final Report: Synthesis and policy recommendations across initiatives 9/

Consortium for Educational Research and Evaluation–North Carolina 7. Omnibus Teacher and Leader Survey Administered to probability sample of 358 schools across the state – a selection of schools that is reflective of the state as a whole Assesses Instructional Climate 23 dimensions on leadership and organizational conditions affecting instruction Baseline Survey completed – Fall 2011 Next administration in progress now (Feb/Mar 2012) and annually throughout RttT Evaluation 29

Consortium for Educational Research and Evaluation–North Carolina Initial Findings ( DST: Evidence of success in providing high-quality assistance to most targeted low-achieving schools. Evidence of several undermining factors in the schools that made little or no progress. STEM Baseline: STEM schools serve greater proportions of lower-income, minority, and rural-based students. Student performance is higher in high-minority STEM (vs high-minority non-STEM) schools. STEM Formative: Most Y1 implementation activities were high- quality; however, delays in selecting schools for the network and in agreeing on curriculum development expectations hindered optimal implementation. VAM: Three VA models – a fixed-effects model, an EVAAS model, and a random effects model – outperformed all others; a final model – a different EVAAS model – was not testable. 30

Consortium for Educational Research and Evaluation–North Carolina Evaluation Results Spotlight: Professional Development Findings: Implementation and Support Cross-Division collaboration at NCDPI 2,212 attendees at 6 Summer Institutes 1,457 (66%) completed post-Institute survey; 83% rated Institute as valuable or very valuable. 31 Report pp

Consortium for Educational Research and Evaluation–North Carolina Recommendations: Implementation and Support 1.Encourage and strengthen cross- divisional work 2.Provide more pre-Institute information 3.Reorganize content sessions 4.Reconceptualize LEA planning sessions 5.Foster more collaboration across LEAs/charter school 32 Report pp Evaluation Results Spotlight: Professional Development

Consortium for Educational Research and Evaluation–North Carolina Recommendations: Implementation and Support (cont.) 6.Address concerns about ongoing, post- Institute support 7.Incorporate greater attention to technology 8.Build on strengths of the best resources 9.Review approaches to addressing diversity 10.Improve locations and logistics 33 Report pp Evaluation Results Spotlight: Professional Development

Consortium for Educational Research and Evaluation–North Carolina Findings: Teachers Baseline Data Elementary and Region 8 (Western Region) teachers most satisfied with prior PD TWCS items were very consistent with data from the PD dimensions of the RttT Omnibus Survey 34 Report pp Evaluation Results Spotlight: Professional Development

Consortium for Educational Research and Evaluation–North Carolina Findings: Principals Baseline Data Concerned about funding for PD Planning greater use of cost-effective online and blended approaches Using NCTEP to inform PD planning Before Institutes, about 50% of sample schools had received PD on one or more of: new standards, formative and summative assessments, and/or use of data to improve instruction 35 Report pp Evaluation Results Spotlight: Professional Development

Consortium for Educational Research and Evaluation–North Carolina Recommendations: Baseline Data 1.Extend PD activities already in place in many LEAs 2.Focus on developing coherent programs that address major PD needs 3.Differentiate PD for elementary, middle, and high school teachers 4.Harness growing interest in online approaches to PD, collaboration, mentoring, and resources. 5.Provide additional support for schools and districts rated low on PD on the TWCS and Omnibus Survey 36 Report p. 11 Evaluation Results Spotlight: Professional Development