1. Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel Theory and Research Conference The Pennsylvania State University University Park, PA, May 17, 2015 The research reported here was supported by a grant from the WT Grant Foundation entitled “Learning from Variation In Program Effects: Methods, Tools, and Insights from Multi-site Trials.”

2. Outline Counter-factual account of causation The “drug-trial paradigm” for causal inference An alternative paradigm for social interventions Heterogeneous agents Social interactions among participants Curriular reform in chicago Conventional RDD Incorporating Agents and Social Interactions Identification: School-specific IV Conclusions

3. Counter-factual account of Causation In statistics (Neyman, Rubin, Rosenbaum) In economics (Haavelmo, Roy, Heckman)

4. Drug trial paradigm for causation Y (1) : Outcome if the patient receives Z = 1 (the “new drug”) Y (0) : Outcome if the patient receives Z = 0 (the “standard treatment”) Y (1) – Y (0) : Patient-specific causal effect E (Y (1) – Y (0) ) =  : Average causal effect

5. Stable Unit Treatment Value Assumption (Rubin, 1986) Each patient has two potential outcomes Implies – Only one “version” of each treatment – No “interference between units” Implies the doctor and the other patients have no effect on the potential outcomes

6. Formally…

7. Failure of SUTVA in Education Teachers enact instruction in classrooms –Multiple “versions of the treatment” Treatment assignment of one’s peers affects one’s own potential outcomes –EG Grade Retention –Hong and Raudenbush, Educational Evaluation and Policy Analysis, 2005 –Hong and Raudenbush, Journal of the American Statistical Association, 2006

8. Group-Randomized Trials Potential outcome Thus, each child has only two potential outcomes – if we have “intact classrooms” – if we have “no interference between classrooms”

9. Limitations of cluster randomized trial Mechanisms operate within clusters * Example: 4Rs teachers vary in response classroom interactions spill over We may have interference between clusters * Example: community policing

10. Alternative Paradigm for social interventions Treatment setting (Hong, 2004): A unique local environment for each treatment composed of * a set of agents who may implement an intervention and * a set of participants who may receive it Each participant possesses a single potential outcome within each possible treatment setting Causal effects are comparisons between these potential outcomes

11. Example: Community Policing (Verbitsky-Shavitz and Raudenbush, 2012) Let Z j =1 if Neighborhood j gets community policing Let Z j =0 if not Under SUTVA

12. “All or none” 1 1 1 1 1 0 0 0 0 0

13. Do it only in high-crime areas: effect on low-crime areas 1, HC 0, LC 1, HC 0, HC 0, LC 0, HC 0, LC

14. Results Having community policing was especially good if your surrounding neighbors had it Not having community policing was especially bad if your neighbors had it *** So targetting only high crime areas may fail***

15. Application: Double-dose Algebra Nomi and Raudenbush (2015) Requires 9 th -graders to take Double-dose Algebra if they scored below 50 percentile on 8 th -grade math test 12,000 students in 60 Chicago high schools

16. Double-dose Algebra enrollment rate by math percentile scores (city wide) Enrollment Rates ITBS percentile scores

17. Conventional Mediation Model (T, M,Y model) Cut off (T) Double-Dose Algebra (M) Algebra Learning (Y) γ = average effect of cutoff on taking Double Dose δ=average effect of taking Double Dose on Y for compliers (“CACE” effect) Assume no direct effect of T on Y (exclusion restriction) β= γδ (“ITT” effect) So δ= β/ γ Nomi, T., & Allensworth, E. (2009) γ δ

18. Conventional Model is Founded on SUTVA

19. Results of Conventional Analysis Large average impact of Cut on taking DD (ITT effect on M) Modest average impact of Cut on Y (ITT effect on Y) Modest CACE (Average Impact of M on compliers)

20. ITT effect on Y

21. But the policy changed classroom composition!!

22. Classroom average skill levels by math percentile scores Pre-policy (2001-02 and 2002-03 cohorts) Post-policy (2003-04 and 2004-05 cohorts)

23. Implementation varied across schools in--- Complying with the policy Inducing classroom segregation

24. Exclusion Restriction Revised T-M-C-Y model Cut off (T) Double-Dose Algebra (M) Algebra score (Y) Classroom Peer skill (C)

25. Identification Problem We have one equation, two unknowns: Strategy is school-specific

26. A simple two-level model At level 1 At level 2

28. Derivation of assumptions using potential outcomes

29. ParameterEstimateSE ITT impact on M 0.720.03 ITT impact on C-0.280.03 ITT impact on Y 0.070.03 CACE of M 0.200.05 CACE of C 0.220.09

31. 5. Conclusions on DD The reform Increased instructional time Changed class composition Median skill kids Gained a lot if not tracked into low-skill classes Gained little if they were

32. Conclusions on Causal Inference Conventional causal paradigm: * a single potential outcome per participant under each treatment Alternative paradigm: * a single potential outcome per participant in each treatment setting RDD as a means-tested program Potentially large policy implications of causal paradigm