1. Causal Inference for Time-varying Instructional Treatments Stephen W. Raudenbush University of Chicago Joint Work with Guanglei Hong The research reported here was supported by a grant from the Spencer Foundation entitled “Improving Research on Instruction: Models Designs, and Analytic Methods;” and a grant from the W.T. Grant Foundation entitled “Building Capacity for Evaluating Group-Level Interventions.” See Hong and Raudenbush, 2008, Journal of Educational and Behavioral Statistics

2. 2 Outline 1. Cumulative effects of sequences of instruction * The changing social structure of instruction * Potential outcomes and causal effects 2. Statistical inference * Under sequential randomization * Given time-varying confounding 3. Estimating and applying IPTW 4. Illustrative results

3. 3 1. Instructional Regimes Explicit –Connor, Morrison, Fishman, Schatschneider, and Underwood, Science (2007) –Borman, Slavin, Cheung, Chamberlain, Madden, and Chambers Educational Evaluation and Policy Analysis (2005) –Bryk and Kerbow Implicit

4. 4 2. Cumulative Effects of Sequences of Regimes Changing Social Structure of Instruction

5. 5 Special Crossed and Nested Structure ChildSchool 1School 2 … Teacher 1Teacher 2 …Teacher 1Teacher 2 … 1XX 2XX... NXx

6. 6 Figure 3 Potential Outcomes in a 2-year Study of Binary Treatments, Z 1 and Z 2

7. 7 Potential Outcomes Year-0 Outcome Year-1 Outcome Year-2 Outcome

8. 8 Causal Effects of Time-Varying Treatments

9. 9 Figure 4 Causal Effects of Z 1, Z 2 in a Randomized 2-year Study U1U1 U2U2 Y1Y1 Y2Y2 Z1Z1 Z2Z2 Y0Y0 U 1, Y 0 indep. of Z 1 ; U 1, U 2, Y 0, Y 1, Z 1 indep. of Z 2

10. 10 3. Hierarchical Model for Observed Data Observed Year-0 outcome Observed Year-1 outcome growth yr1 treat yr 1 teacher Observed Year-2 Outcome growth yr1 treat in yr2 yr2 treat synergy yr 2 teacher

11. 11 Hierarchical Model (continued)

12. 12 Mixed Model

13. 13 Figure 4 Causal Effects of Z 1, Z 2 in a Randomized 2-year Study U1U1 U2U2 Y1Y1 Y2Y2 Z1Z1 Z2Z2 Y0Y0 U 1, Y 0 indep. of Z 1 ; U 1, U 2, Y 0, Y 1, Z 1 indep. of Z 2

14. 14 Statistical Inference given time- varying confounding

15. 15 Figure 5 Causal Effects of Z 1, Z 2 in a Non-Randomized 2-year Study Assuming Strongly Ignorable Treatment Assignment U1U1 U2U2 Y0Y0 Y1Y1 Y2Y2 Z1Z1 Z2Z2 X1X1 X2X2 U 1 indep. of Z 1 | X 1, Y 0 U 1, U 2 indep. of Z 2 | X 1, X 2, Y 0, Y 1, Z 1

16. 16 Strategies for Adjustment Covariance adjustment will fail Propensity score stratification will fail Inverse probability of treatment weighting holds promise –Robins, Hernan, and Brumback Epidemiology (2000) –Hong and Raudenbush, Journal of Educational and Behavioral Statistics (2008)

17. 17 4. Estimating and Applying IPTW Time 0 Time 1 Time 2

18. 18 5. Illustrative Example

19. 19 Longitudinal Evaluation of School Change and Performance (LESCP), 1997-1999 U.S. Department of Education Planning and Evaluation Service Sample - Longitudinal cohort of students: Grades 3 - 5 - 4,216 students, 72.3% eligible for free lunch - 190 classrooms  3 years - 67 Title I schools Outcome - Stanford Achievement Test 9 Close-ended Math

20. 20 Table 1 Sample Response Pattern

21. 21 Table 2 Propensity Model Results

22. 22 Table 3 Treatment Effect Estimation Results from Weighted Multi-level Growth Model

23. 23

24. 24 Table 4 Stability Analysis for Grade 5 Treatment Effect Estimation

25. 25 Table 5 Sensitivity Analysis for Grade 5 Treatment Effect Estimation

26. 26 Table 5 Sensitivity Analysis for Grade 5 Treatment Effect Estimation (Continue)