1. Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the W.T. Grant Foundation to the University of Chicago entitled “ Building Capacity for Evaluating Group-Level Interventions. ” Thanks to Takako Nomi for her collaboration on these ideas.

2. Outline 1.Pervasiveness of Multi-site trials Non-compliance 2.Potential outcomes framework Observed data Potential outcomes Causal effects 3. Instrumental variables in a single-site study Under homogeneity of impact Under heterogeneity of impact Complier-Average Causal Effect (CACE) 4. Instrumental variables in multi-site studies –Estimating the Average CACE –Estimating the Variation in CACE

3. 1. Non-compliance in Multi-Site Trials Examples * National Head Start Evaluation (US Dept of HHS, 2010) * Moving to Opportunity (Sonbanmatsu, Kling, Duncan, Brooks Gunn, 2006) * School-based lottery studies ( Abdulkadiroglu, Angrist, Dynarski, Kane, and Pathak, 2009 ). * Tennessee STAR (Finn and Achilles, 1990) * Double-Dose Algebra (Nomi and Allensworth, 2009) * Small Schools of Choice (MDRC)

4. 2. Observed Data (Single Site) Observed variables

5. 2. Potential Outcomes and Causal Effects

6. 2. Average Causal Effects

8. What happens if impacts are heterogeneous?

9. Single site, heterogeneous treatment effects

10. Assume away “Compliance-Effect Covariance”??

11. Alternative Approach for binary M “Complier Average Causal Effect” (CACE) or “Local Average Treatment Effect” (Bloom, 1984; Angrist, Imbens, and Rubin, 1996)

12. Principal Stratification (Frangakis and Rubin, 2002) Stratum M(1)M(0)Ф=M(1)-M(0)Y(M(1))-Y(M(0))Fraction of pop Average Effect Compliers 101Y(1)-Y(0) π compliers δ compliers Always- takers 110Y(1)-Y(1)=0 π always 0 Never- takers 000Y(0)-Y(0)=0 π never 0 Defiers 01Y(0)-Y(1)00

13. Complier-average causal effect

14. In Sum We can estimate the Population-Average Effect of Participating if we assume Cov( Φ, Δ)=0 We can estimate CACE if we assume Pr( Φ ≥0)=1 The latter is a much weaker assumption

15. 2. Causal Effects in Multi-site Trials

17. Multiple Sites: Causal Effects

18. Combine 2 ITT analyses Step 1: Estimate the Impact of Treatment Assignment on the Outcome Results

19. Step 2: Estimate the Impact of Treatment Assignment on Program Participation Results

20. Step 3: Combine Results: mean

21. Step 3: Combine Results: variance

22. In sum True Values Our estimates