1. EVALUATION & RESEARCH IN WAKE COUNTY (NC) PUBLIC SCHOOLS Presented by David Holdzkom, Assistant Superintendent, Evaulation & Research Wake County Public Schools

2. CONTEXT WCPSS serves about 134,000 students, making it the 19 th largest district in the U.S. and the largest district in North Carolina The district has 93 elementary schools, 28 middle schools, 22 high schools and 4 special schools. About 30% of students qualify for FR; 14% are SWD; and 7.4% are LEP.

3. CONTEXT (cont’d) Wake County includes 12 incorporated towns plus suburbs and rural communities. Since 1990, the county population has increased by 92%. Much of this in-migration has been from areas of the country that have different traditions of school district organization. Major employers in the county include IBM, GlaxoSmithKline, SAS, and Cisco 3 research universities within 25 miles

4. ORGANIZATION OF EVAL&RES DEPARTMENT Admin: Assistant Superintendent and 3.5 support staff; Testing: Sr. Director and 2 professionals plus 4 support staff; School Accountability: Sr. Director and 5 professionals plus 2 support staff; Program Accountability: Sr. Director and 4 professionals plus 3 support staff; Grants Admin: Sr. Director plus 2 support staff.

5. STUDENT INFORMATION RESOURCES NC WISE: A comprehensive data base designed by state to provide information (demographic, achievement data) for every student. Student Information Files: Maintained by WCPSS on a mainframe to provide supplementary data for every student. Testing files: Designed by the state to provide both achievement and growth data for all students in Grades 3-8 plus selected high school course takers. Local test files: Designed by WCPSS adds achievement data for students in K-2 as well as some additional information for students in Grades 3-5.

6. THE EFFECTIVENESS INDEX Designed in WCPSS E&R, this analysis tool results from regression analyses for individual students that control for FR, concentration of FR in school, special education status, special education service method, etc. Provided for each student, with reports aggregated at the class/course and school levels.

7. KINDS OF RESEARCH Policy Evaluation: Healthy Schools, Magnet Schools, Student Assignment Program Evaluation: Implementation of PBS, AVID, Standards-based grading, IRIS, etc. School Status Evaluation: Accountability measures of state and NCLB. Knowledge Creation: Effectiveness Studies; IRIS

8. Program Evaluation Standards Utility Standards: Study serves the info needs of the audience. Feasibility Standards: Study is realistic, diplomatic, and economical. Propriety Standards: Study is legal and ethical. Accuracy Standards: Study reveals and conveys technically adequate info.

9. POLICY EVALUATION Healthy Schools: Values for 100 variables (both input and outcome) are calculated annually. This information is used by the Leadership Team to ensure that schools remain “healthy”. Magnet Schools: Measure the degree to which Board goals for these schools are met. Student Assignment: How well do different schools serve sub-groups of students?

10. Magnet Program Review (2007) Review examined degree that board’s goals for magnets were realized: 1. Utilization of schools; 2. Use of choice to promote diverse student populations, reduce concentration of poverty; 3. Expand educational opportunity; 4. Promote program innovations.

11. Magnet Review (cont’d) Examined student assignment data and student demographic for indications of race, FRL-status, assignment to schools. Compared all schools with respect to concentration of FRL Conducted focus group interview/survey of principals to identify innovative programs. Examined differences of achievement between magnet/base populations, compared to district.

12. PROGRAM EVALUATION Schools seek to implement nationally- developed programs (PBS, AVID, Reading Recovery) to meet specific needs. Evaluation focuses on fidelity of implementation as well as change in student outcomes. Locally developed programs (Standards- based grading, PES, etc.) try to focus more on logic modeling and outcomes.

13. Evaluation of SES A Title I school in year 2 of school improvement was required to offer Supplemental Education Services. 252 students were provided tutoring service at a cost of $303,000. Interviews were conducted with teachers and parents; student achievement data were examined.

14. SCHOOL STATUS EVALUATION Accountability measures of state and NCLB. Calculation of absolute measures of achievement (i.e., percentage of students at/above grade level) and calculation of growth. Calculation of student residuals/effectiveness index.

15. Knowledge Creation Effectiveness Studies: Using Effectiveness index to identify higher/lower performing teachers of 5 high school subjects, studied classroom practice; this method was adapted for elementary & middle schools IRIS: An experimental program designed to provide reading intervention early. Professional Learning Communities: Surveys intended to measure the quality/degree of implementation of PLCs in all schools.

16. Project IRIS As part of an OCR consent decree, WCPSS agreed to analyze process of referrals to special education services and to design data- driven interventions to improve reading instruction. Intensive Reading Intervention Study (IRIS) was developed to respond The logic: too many poor readers are referred to student support teams and ultimately to special education. If we can improve reading early, referrals will decrease.

17. Project IRIS (cont’d) IRIS involved random assignment of 169 students to Control or Treatment groups. Treatment lasted for 45 minutes 4 times a week for ten weeks. Students were tested on a variety of measures before treatment, after treatment and end of year. Controls had “mirror” assessments.

18. Project IRIS (cont’d) At the end of the study, treatment group had more positive outcomes, but these did not reach statistical significance. Three years later, students from both groups were identified and achievement data analyzed. Again, no significant differences were found.

19. RANDOMIZED TRIALS Only 1 study (IRIS) has used randomization. Teachers, principals, parents object to randomization: “students aren’t guinea pigs”. How does intervention support/detract from the core mission of the school? Currently pursuing a randomized trial project with Duke University related to PLCs.

20. FUTURE DIRECTIONS/ISSUES Designing training in data use for principals & teachers and experimenting with delivery methods/formats Increased emphasis on logic models and fidelity of implementation Assisting schools to design accountability schemes that include but are not limited to test scores

21. FUTURE DIRECTIONS (CONT’D) Enhancing teachers’ ability to develop formative assessment plans/items Implementing the evaluation recommendations of the Curriculum Management Audit conducted in 2007