1. We would like to thank the Institute for Education Sciences for their support and funding. Please direct any questions to info@michiganconsortium.org 735 S. State Street | Ann Arbor, MI 48109 Principal Investigators: Brian Jacob & Susan Dynarski University of Michigan Barbara Schneider & Ken Frank Michigan State University Thomas Howell Center for Educational Performance & Information Joseph Martineau Michigan Department of Education

2. What is MCER?  Michigan Consortium for Educational Research  IES-funded collaboration between:  University of Michigan  Michigan State University  Michigan Department of Education  Michigan Center for Educational Performance and Information.

3. Objectives of MCER  Engage stakeholders and education experts in research for the benefit of public education in Michigan  Provide research-based evidence to policymakers in Michigan  Inform national policy initiatives for improving education

4. Inaugural Research Questions  What is the effect of the Michigan Merit Curriculum on…  course-taking patterns, student achievement  high school graduation, postsecondary attendance  What was the effect of the Michigan Promise Scholarship on…  college entry, college choice, college completion  Do these effects vary by school or student characteristics?

5. MCER Data Goal Detailed Student-Level Longitudinal Dataset Attainment: HS graduation, GED, college degrees K12: test scores, enrollment, demographics, GED, ACT scores Postsecondary: NSC, MI public college transcripts

6. Michigan Merit Curriculum Study  As of 2011, all graduating MI HS students must pass 16 rigorous courses (e.g., Algebra II, Biology, Chemistry, Physics) and complete end- of-course exams to measure content mastery  The evaluation will compare student outcomes before & after the MMC  Standardized test scores, HS graduation  College entry, choice & completion  Random sample of 100+ schools: assess fidelity of implementation  End-of-course exams  Student transcripts  Interviews

7. Michigan Promise Study  Intervention  $4,000 college scholarship for students with qualifying score on Michigan Merit Exam  Students must maintain 2.5 college GPA  Methods  Regression discontinuity: compare outcomes just above/below MME threshold  College enrollment/choice/persistence (NSC)  Scholarship receipt (Treasury)

8. Emerging Research Questions Has mandatory ACT (now part of MME) improved college attendance and choice? What is the “value-added” of individual schools, once we control for student characteristics and initial achievement? Are Michigan’s charter schools raising student achievement?

9. Example study The Impact of the Threat of School Sanctions: A Regression Discontinuity Study of Being on a Probationary List Guan K. Saw, I-Chien Chen, Barbara L. Schneider, Kenneth A. Frank Michigan State University

10. This study analyzes the effect of the first stage of school sanctions in Michigan, being on a probationary list. We pay attention to its possible impact on student achievement in high-stakes and low-stakes subjects at school level. Purpose of Study

11. Background School sanctions, increasingly used as instruments of education policy, have been the focus of debate at federal and state levels. The goal of sanctions is to incentivize schools that fail to meet academic standards to improve their students’ educational performance. How does it work to make school change?

12. Two possible explanations 1. Probations serve as a Social Stigma – Stigmatizing or labeling is a potent tool for guiding individuals to conform to social norms. – For failing schools, stigmatization becomes a motivating factor to make change (Figlio & Rouse, 2006; Ladd & Glennie, 2001; Mintrop, 2004; Sim, 2007, 2009). – “Being on a probationary list” can be seen as a social stigma, which may have a “labeling effect”.

13. 2. Effects of Sanction Threats – Not only imposing sanctions, but also threatening sanctions can change an individual’s behavior (Lacy & Niou, 2004). – In economic sanctions literature, some argue that sanctions threatened are often more effective than those that are deployed (Drezner, 1999; Drury & Li, 2006; Lacy & Emerson, 2004; Smith, 1996). – In education, the effects of sanction threats on low-performing schools are mixed (Chiang, 2009; Figlio & Rouse, 2006; Springer, 2008; Winters et al., 2010).

14. Crowding-out hypothesis – There is a growing concern that test-based accountability may cause schools to shift inputs from low-stakes subjects (Corbett & Wilson, 1991; Kohn, 1999; Nichols & Berliner, 2007; Whitford & Jones, 2000). – This crowding-out hypothesis was supported by some qualitative evidence (Au, 2007; Groves, 2002; King & Mathers, 1997; Murillo & Flores, 2002). – In contrast, some quantitative studies report that high-stakes testing policies led to significant gains in low-stakes subjects (Jacob, 2005; Winters, Trivitt, & Greene, 2010).

15. Michigan Context: PLA and Watch List Since 2009, Michigan Department of Education (MDE) has annually published a list of the lowest performing 5% schools, named the Persistently Lowest Achieving list (PLA list). The PLA list is established by certain criteria: (a) 2-year average percent proficiency in math and reading; (b) 4-year slope of percent proficiency in math and reading; (c) whether a school made Adequate Yearly Progress (AYP) status over the past two years; and (d) whether a school had a 4-year graduation rate below 60% for three years in a row.

16. The PLA list schools have to make significant gains in student achievement within a short time to get off the list. If not, further sanctions may be imposed including turnaround, restart, and closure of schools. With labeling and sanction threat effects, we hypothesize that the PLA list schools tend to positively affect student achievement.

17. In addition to the PLA list (bottom 5%), MDE also publishes a “Watch list” of schools in the lowest quintile (6-20%), which were identified as being in danger of falling under the 5% mark. This does not affect the PLA ranking but provides an alert to these schools to keep them out of the PLA category. Without a strong threat of further sanctions, we expect that the labeling effect of being on the watch list is relatively limited. Michigan Context: Watch List

18. Data Sources Our longitudinal school-level data constructed with: (1) Michigan Educational Assessment System (MEAS); (2) Common Core of Data (CCD). We only focus on a sample of regular high schools (only 333 schools ranked by state-wide achievement base percentile ranking).

19. Measures Treatment 1: Being on 2008-09 PLA list (<5%) Treatment 2: Being on 2008-09 Watch list (6-20%) Forcing variable: State-wide achievement base percentile ranking Outcomes: % of students met proficiency level in (a) high-stakes subjects: math, reading, writing, (b) low-stakes subjects: science, social studies Covariates: % of free/reduced lunch students, % of black students, school size, and pupil teacher ratio.

20. Analytic Method We employ a regression discontinuity design (RDD) method. This is a quasi-experimental design, in which treatment status depends on whether an observed covariate exceeds a fixed threshold (Lee & Card, 2008; Shadish, Campbell, & Cook, 2002). In our case, the fixed threshold is the cutoff (5% or 20%) of percentile ranking for being on the PLA or watch lists.

21. RD Analysis of PLA List RD Analysis of Watch List 05 020535 10100 PLA list Watch list Percentile ranking

23. RDD with covariates Given the imbalance of covariates between treatment and control groups, we include the covariates in the RD models, which can (Imbens & Lemieux, 2008) : (1) reduce small sample bias; (2) improve precision if covariates correlated with potential outcomes (as in analyses of randomized experiments)

24. Findings: PLA List Figure 1a. Percentage of Students Met Proficiency Level in 2011, by Tier 2 Percentile Rank in 2008-09 Mathematics Reading Writing Science Social Studies Cutoff = 5% <5% (PLA list) = 19 schools 5-10% (Control group) = 19 schools

25. Effects of Being on the PLA List (2008-2009)

26. RDD analyses show a positive “list” effect on all subjects 2010-2011. In models with presence of covariates, only the positive “list” effect on writing achieves statistical significance. No negative effect on low-stakes subjects of science and social studies was observed.

27. Findings: Watch List Figure 1b. Percentage of Students Met Proficiency Level in 2011, by Tier 2 Percentile Rank in 2008-09 Cutoff = 20% 5-20% (Watch list) = 54 schools 20-35% (Control group) = 51 schools Mathematics Reading Writing Science Social Studies

28. Effects of Being on the Watch List (2008-2009)

29. We found no effect of being on the watch list for all subjects in 2010-2011. This finding holds across estimation models using different bandwidths (5%, 10%, and 15% below and above cutoff).

30. Robustness Test We created a pseudo PLA list using the previous school year data (2007-08), before the state policy mandating assignment of low-performing schools to a probationary list had been enacted. Then, we compare the results of 2008-09 PLA list effects to 2007-08 pseudo PLA list effects. We expect that there would be no effect of being on the pseudo PLA list since these schools did not receive a labeling treatment or real threat of sanctions.

31. Findings: Pseudo PLA List Figure 1c. Percentage of Students Met Proficiency Level in 2010, by Pseudo Tier 2 Percentile Rank in 2007-08 Cutoff = 5% <5% (Pseudo PLA list) = 18 schools 5-10% (Control group) = 17 schools Mathematics Reading Writing

32. Effects of Being on the Pseudo PLA List (2007-08)

33. Results from the 2007-08 pseudo list analysis indicate no “list” effect on all subjects in 2009-2010. This finding further testifies the robustness of the effects of 2008-2009 PLA list.

34. Conclusion Being on PLA list as a social stigma combined with possible following incremental sanctions may spur school improvement. Without a real and strong threat of further sanctions, just being labeled by the watch list does not stimulate school performance. Crowding-out hypothesis was not supported by our data.

35. Limitation and Discussion Limitations: (1) given three years of data, we only can claim a short term effect of PLA list. (2) the limited variables in the data set provide no information to uncover real changes being implemented in the schools. Puzzles: (1) Are the PLA effects stronger for writing? Why? (2) What organizational processes occurred in the schools placed on the PLA list?

36. …take home message… Within the school sanctioning context: The l abeling effect is limited. The threat of sanctions does motivate low-ranked probationary schools to make changes. Crowding-out effects may not occur but spillover effects may be present.

37. “On the List” Example of Evaluation of Institution State implemented Involves state data as well as school composition (common core) Apply to Michigan Merit Curriculum – Graduate students – Data – Collaborative faculty

38. We would like to thank the Institute for Education Sciences for their support and funding. Please direct any questions to info@michiganconsortium.org 735 S. State Street | Ann Arbor, MI 48109 Principal Investigators: Brian Jacob & Susan Dynarski University of Michigan Barbara Schneider & Ken Frank Michigan State University Thomas Howell Center for Educational Performance & Information Joseph Martineau Michigan Department of Education