1. Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam Gamoran, Ellen Goldring and Keke Liu.

2. Research Questions Has attending a magnet school caused an increase in mathematics achievement? How large is the influence of peers on mathematics achievement? How much of the magnet school effect remains after controlling for the influence of peers?

3. Study Setting Middle Schools in a Large Southern District 1 selective academic magnet 4 non-selective magnets 5 student cohorts 6 years: 1998-99 through 2003-04 Grades 5 & 6

4. Admissions Lotteries Oversubscribed magnets conduct lotteries Students may enter multiple lotteries Students who are not outright winners are placed on wait lists Wait-listed students accepted until the first week of school

5. Non-lottery Admissions Sibling preferences Promotion from a feeder school Geographic priority zone These students are not included in the study sample (though they do enter the calculation of peer characteristics).

6. Research Design Lotteries assign students randomly to school type and to peers (magnet school peers vs. non-magnet peers). Randomized design circumvents biases arising from self-selection of schools and peers.

7. Limitations of Design Results may not generalize beyond lottery participants. Effects are relative (magnet schools vs. mix of non-magnet schools attended by lottery losers).

8. Lottery Participation AcademicComposite Non-Academic Applicants 2315 2594 Outright Winners 883 1450 Delayed Winners 223 756 Losers,This Lottery 1209 388 Losers, All Lotteries 539 199

9. Grade 5 Enrollments AcademicComposite Non-Academic This Magnet 758 1061 Other Magnets 287 339 Non-Magnets 834 846 Left system or Not Tested 436 346

10. Substantial non-compliance, especially among winners of non-academic lotteries, attenuates estimated treatment & peer effects based on comparison of winners and losers. Remedy: Use lottery outcomes as instruments to predict probability of attending magnet school, outcomes interacted with peer variables at magnet & zoned schools as instruments for peer characteristics. Potential Pitfalls

11. High rates of attrition from district can introduce systematic differences between treatment and control groups. Remedies: Control for student characteristics (race, income, ESL, special ed, gender, prior achievement). Analyze attrition patterns for evidence of differences between winners and losers.

12. Participating in multiple lotteries increases chances of winning. “Multiple participants” may differ in ways related to achievement. Remedy: Control for the combination of lotteries each student entered. Winners are compared to losers who entered the same combination.

13. Lotteries randomly assign students to magnet school peers or peers in their neighborhood (zoned) school, but lotteries do not determine the characteristics of the latter—residential decisions do. Remedy: Control for characteristics of the peers in the zoned school.

14. Peer Characteristics Percentages black, low income (free & reduced-price lunch program), special ed, ESL, female Absenteeism rate Disciplinary incidents (rate per student) Intra-year mobility Prior achievement in math and reading

15. Model (Summary) Two treatment variables (academic magnet, composite non-academic magnet) Variation in peers resulting from lottery outcomes Other controls (student characteristics, peers at the zoned school, lottery participation indicators, year by grade effects)

16. Findings When model does not include peer characteristics - Academic magnet, + 18% in grade 5, drops to +10% in grade 6 (% of normal year growth) - Non-academic magnet, no grade 5 effect, +54% in grade 6

17. When models include peer characteristics - Reducing percent black from 75% to 25% increases scores by 60% of normal year growth. - Effect of percent low income is about half that large. - Other peer characteristics have no statistically significant effect.

18. - Controlling for either percent black or percent low income, the effect of the academic magnet disappears. - The large 6 th grade effect in the non- academic magnets remains substantially undiminished.

19. Checking Alternative Interpretations Are peers a proxy for heterogeneous response to treatment? Check: Interact magnet treatment indicators with all observed student characteristics. Finding: Peer effects are undiminished.

20. Are peers a proxy for teacher quality? Check: Control for teacher quality by including teacher fixed effects. Finding: Peer effects are undiminished.

21. Attrition, Academic Magnet Lottery Participants Left System After Grade:WinnersLosers 4 13% 21% 5 8% 14% 6 9% 11% 7 6% 9%

22. Attrition, Composite Non-Academic Magnet Lottery Participants Left System After Grade:WinnersLosers 4 8% 12% 5 9% 6 10% 4% 7 12% 16%

23. Potential Attrition Biases Lottery losers are more likely to leave the system than winners. Losers are also more likely to leave when they can afford private schooling. These tend to be higher-achieving students. Result: Losers who remain in the system have lower achievement than winners who remain.

24. Unfavorable peers at zoned school make losers more likely to leave system. Effect greatest among those who can afford private schooling. Result: Quality of peers positively correlated with losers’ achievement. Estimated peer effects appear too strong.

25. Checking Attrition Bias Are rates of attrition correlated with variables that predict individual achievement (race, income, prior achievement)? Yes, but not differently for winners and losers.

26. Conclusions For at least some students in some places, magnet schools have a positive effect on academic achievement. There are very strong peer effects on middle school achievement. Do not appear to operate through behaviors readily quantified with administrative data (attendance, disruptions, mobility).