Regression Discontinuity Design Case Study : National Evaluation of Early Reading First Peter Z. Schochet Decision Information Resources, Inc.

2 Overview ERF program ERF program Evaluation research questions Evaluation research questions Regression discontinuity design Regression discontinuity design Conclusions Conclusions

3 ERF Program Part of the No Child Left Behind Act Part of the No Child Left Behind Act 3-year ERF grants provided to a collaboration of preschools 3-year ERF grants provided to a collaboration of preschools –Funding focus is on low-income children Goal: Enhance the language, cognitive, and early reading skills of preschool children Goal: Enhance the language, cognitive, and early reading skills of preschool children

4 ERF Funds Are Intended to: Provide professional development for teachers Provide professional development for teachers Create high-quality and print-rich environments Create high-quality and print-rich environments Promote the use of scientifically proven literacy methods and instructional materials Promote the use of scientifically proven literacy methods and instructional materials Identify preschool children at risk for reading failure Identify preschool children at risk for reading failure

5 Study Research Questions What are the impacts of ERF on : What are the impacts of ERF on : –Children’s language and literacy? –Quality of language and literacy instruction, practice, and materials?

6 KEY DESIGN FEATURES

7 Study Focus Is on FY 2003 ERF Grant Applicants 700 sites submitted pre-applications 700 sites submitted pre-applications 126 invited to submit full applications 126 invited to submit full applications

8 Random Assignment Was Not Possible ED required that funds be awarded based on rankings of applications ED required that funds be awarded based on rankings of applications –Applications were “scored”  30 sites were funded with scores  74  96 unfunded sites with scores < 74 –Scoring criteria were set a priori –Favorable conditions for a RD design

9 The Ideal ERF RD Design: Compare “73s” to “75s”: Almost Experimental Cutoff Mean for 75s Mean for 73s

10 But There Are Not Enough 73s and 75s: Need to Include Other Sites Score Number of Sites 42 to to to 7321 Cutoff Value=74 74 to to 9512 UNFUNDED FUNDED

11 Sampling All funded sites agreed to participate All funded sites agreed to participate Sorted unfunded sites by their scores Sorted unfunded sites by their scores –Sites with largest scores contacted first –64 sites contacted –37 agreed to participate Obtained lists of classrooms in sites Obtained lists of classrooms in sites –Sampled 3 classrooms per site –Selected 9 four-year olds per classroom 94 percent parental consent rates 94 percent parental consent rates

12 The RD Method Visually Estimated Regression Lines Unfunded Funded Impact Cutoff

13 Key Identifying Assumption There must be a continuous relationship between the outcome measure and the application score There must be a continuous relationship between the outcome measure and the application score

14 Differences-in-Means Estimates Could Be Biased Funded Spurious Impact = Simple Means Unfunded

15 RD Designs Require Larger Samples Than Experimental Designs Controlling for the application score reduces power Controlling for the application score reduces power Design effects are 3 to 4 Design effects are 3 to 4

16 The Correct Functional Form Specification Is Crucial for Obtaining Unbiased Estimates Assume Linearity When the True Relationship is Nonlinear Spurious Impact

17 There Also Has to Be a Clear Functional Form Relationship % of Classrooms in Site That Engage in an Activity

18 Basic Regression (HLM) Model Y = β 0 + β 1 *T + β 2 *f(SCORE-74) + u Y = β 0 + β 1 *T + β 2 *f(SCORE-74) + u –Y = Outcome measure ( child or teacher level) –T = 1 if funded site, 0 if unfunded –f(SCORE) = Function of application score –β 1 = Impact estimate –u = Error term accounting for site and classroom-level clustering classroom-level clustering

19 Selecting f(SCORE) Graph Y on SCORE Graph Y on SCORE Add SCORE 2, SCORE 3, and T*SCORE interaction terms and test for significance Add SCORE 2, SCORE 3, and T*SCORE interaction terms and test for significance Use nonparametric methods Use nonparametric methods Specification tests Specification tests –Impacts = 0 using baseline data –Impacts = 0 at “artificial” cutoff values –Adding covariates should not change impacts

20 Interpretation of Impact Estimates In the “73-75” model, results pertain only to sites around the cutoff value In the “73-75” model, results pertain only to sites around the cutoff value Results “generalize” more broadly using the parametric approach Results “generalize” more broadly using the parametric approach –Does this stretch the results too far? But not using the nonparametric approach But not using the nonparametric approach

21 Conclusions RD designs can produce rigorous impact estimates under the right conditions: RD designs can produce rigorous impact estimates under the right conditions: –Need exogenous “scores” –Scores and outcomes must have a smooth relationship that can be credibly modeled But there are limitations to the RD approach: But there are limitations to the RD approach: –Need larger samples than an experimental design –Generalizability –Nonresponse a problem in unfunded sites

22 EXTRA SLIDES

23 Data: Fall 2004 and Spring 2005 (Spring Response Rates) Child assessments (97%) Child assessments (97%) Teacher behavioral ratings (96%) Teacher behavioral ratings (96%) Teacher/classroom observations (79%) Teacher/classroom observations (79%) Parent surveys (69%) Parent surveys (69%) Teacher surveys (91%) Teacher surveys (91%) Center director interviews (88%) Center director interviews (88%)

24 Child Assessment Instruments

25 Observations and Surveys