1. DATA ANALYTICS TO SUPPORT K-12 EDUCATION: 25 YEARS OF RESEARCH AND NATIONWIDE IMPLEMENTATION October 20, 2014 Robert H. Meyer, Research Professor, WCER and the La Follette School of Public Affairs, Director of Value-Added Research Center

2. Plan for Today  Mission of Value-Added Research Center  Origins and National Scale Up  Project Example: How should Wisconsin evaluate principals in 2015?

3. Mission  Develop and implement data analytics and policy tools that meet the twin standards of:  Usefulness in the real world of children and educators, and teaching and learning  Scientific rigor  Embedded research: conduct research informed by the needs of district and state partners  Embody the Wisconsin Idea

4. Milwaukee Origins  Milwaukee Embedded Research (1998)  Architects/engineers: Bill Clune & Norm Webb  Leadership: Andy Porter  Funding: The Joyce Foundation, Warren Chapman & John Luczak  MPS: Deb Lindsey  VAM Production (2001) WISCONSIN

5. Expansion in Milwaukee  Milwaukee Embedded Researcher (2005)  Gary Cook, Brad Carl  A Comprehensive Evaluation of Reading Instructional Practices and Policies (2005)  Evaluation of the NCLB Supplemental Educational Services Program (2006)  Evaluation of Charter Schools and High Schools (2008)  The Integrated Resource Information System, IES (2008)  Milwaukee Area Public and Private Schools (2009)  Performance Management and the Integrated Resource Information System (2010)  Milwaukee IDEAS Project (2012) WISCONSIN

6. Wisconsin  Design of Prototype Wisconsin State Value-Added System (1989)  Wisconsin SAGE Evaluation (2001)  Madison (2008)  Racine(2009)  WI State-wide VAM (2009)  Production of value-added estimates  Consortia building with regional meetings & advisory council  Educator Effectiveness (2011)  Principal Evaluation WISCONSIN

7. Nationwide Partner Districts and States  Chicago (2006)  Department of Education: Teacher Incentive Fund (TIF) (2006, 2010, and 2012)  New York City (2009)  Minnesota, North Dakota & South Dakota: Teacher Education Institutions and Districts (2009)  Hillsborough County/Tampa (2010)  Atlanta (2010)  Los Angeles (2010)  Tulsa (2010)  Illinois (2012)  Oklahoma Gear Up (2012)

8. Minneapolis Milwaukee Racine Chicago Madison Tulsa Atlanta New York City Los Angeles Hillsborough County NORTH DAKOTA SOUTH DAKOTA MINNESOTA WISCONSIN ILLINOIS Research & Data Analytics Projects Collier County NEW YORK CALIFORNIA OKLAHOMA MICHIGAN

9. Principal Evaluation: The Wisconsin Policy Context  Educator Effectiveness Design Team (policy makers & stakeholders) developed specific metrics and weights for teacher and principal evaluation  Separate scores for professional practice and student outcomes  For Principals with Value-Added:  SLOs: 1.8 pts (45% of score)  School Value-Added (VA): 1.8 pts (45%)  District Choice: (5%)  Schoolwide VA Reading or graduation rate (5%)  Principals without VA: SLOs are 90%

10. Criteria: Technical and Consequential Validity  Am educator effectiveness metric should be evaluated both with respect to:  Technical validity: the accuracy and fairness of the ratings  Consequential validity: the effects of the system on the behavior of the principals, and possibly teachers, students, and parents  The two criteria reinforce each other in many instances. Example:  Need to design an evaluation metric so that it does not underestimate the performance of principals in high poverty schools or in “turn-around” schools (a technical validity concern) because this would create a disincentive for effective principals to accept assignments in such schools (a consequential validity concern).

11. Why Value-Added? Growth in Student Achievement is the Goal  School-level and teacher-level value-added models provide appropriate metrics for measuring the contributions of teachers and schools to student achievement  The models control for differences across classrooms and schools that are external to classrooms and schools, such as prior achievement, poverty, other factors  What is the principals contribution to VA?

12. Challenge: Principal Impact is Not Immediate  Existing research suggests that it may take several years for the full effects of a principal’s effectiveness to kick in and be detected  For both continuing and new teachers  School value-added during the first few years of a principal’s assignment may be misleading  School value-added may be severely biased downward for highly effective principals employed in turn-around schools  This is undesirable from the standpoints of both technical and consequential validity.

13. How Many New Principals?

14. Options for Measuring Principal Performance  School VA  Inaccurate and unfair to principals assigned to turn- around schools  Change in VA  Inaccurate and unfair to principals assigned to high- performing school  Principal value-added: Partial adjustment model  Potentially accurate and fair to all principals  Sustainability of high-performance is valued  Improvement from low to high VA is valued

15. Adjustment from Medium Starting Point

16. Adjustment from Low Starting Point

17. Adjustment from High Starting Point

18. Model of Principal/School VA  Impact of new principal is gradual, with adjustment rate  True principal productivity =  Principal/school VA (for principal k at time t) =  Principal/school VA in year prior to start of new principal =

19. Principal Value-Added (PVA)  Solve the equation for principal productivity and indicate that all parameters are estimates (denoted by the ^ symbol).  As time (t) increases, the principal rating approaches principal/school VA

20. Principal Value-Added (PVA)  Principal value-added allows for an adjustment transition from one principal to the next  The speed of the adjustment process is captured simply by a “rate of adjustment” parameter  An adjustment rate equal to 0.25, for example, implies that in a single year 25% of the gap between past year and eventual (true) performance is closed within a single year  This rating metric has the desirable feature of classifying the following as excellent performance: increases in school value-added over time and sustainability of high value- added over time (even with no increase).

21. A Wrinkle: Is There a Disincentive to Transfer to New Assignment?  High and low productivity principals will typically receive mid-level ratings during early years in new assignment  Yes, disincentive to transfer for high productivity principals  No, positive incentive to transfer for low productivity principals  Implication: Incorporate performance at prior schools. Bonus: Using past and current information will substantially improve reliability

22. Summary  Principal VA controls for school VA in the period prior to the assignment of a new principal and thus allows for an adjustment transition from one principal to the next.  The indicators serve different purposes:  School value-added measures the contribution to schools to growth in student achievement, controlling for factors external to schools (but including the contributions of past and current principals)  The PPR is designed to measure the related, but different, dimension of the performance of a particular person – the principal.

23. Summary, continued  PVA and school value-added differ only during the early transition years from one principal to the next  The PVA is relatively noisy during the first year or two of a new principal assignment due to the need to control for VA productivity under the prior principal  The precision of PVA and school VA are both improved by measuring performance over many subjects, grades, and years

24. Summary, continued  Precision is further much improved by combining performance from previous assignments, although this is not an option for new principals  This approach has the advantage of neutralizing the incentive for a:  High-performing principal to stay at their current school (to retain the high PVA)  Low-performing principal to leave their current school (to escape the low PVA)

25. Frontier: Channels of Principal Impact  Changes in the productivity of continuing teachers and staff  Via being an instructional leader, implementing effective professional development, etc.)  Changes in the productivity of new hires (versus prior staff)  A function of principal effectiveness in recruiting and retaining effective teachers)  Useful to provide separate measures of school-level value-added for continuing and new teachers (pending availability of student-teacher links)

26. Resources

27. Components of Value-Added

28. Average & Principal/School VA  Average School VA  Principal/School VA