1 National Reading First Impact Study: Critique in the Context of Oregon Reading First Oregon Reading First Center May 13, 2008 Scott K. Baker, Ph.D. Hank Fien, Ph.D.

2 Overview of Webinar Brief summary of the National Reading First Impact Study (i.e., National Impact Study) Summary of Oregon Reading First National Impact Study: Issues and Considerations Differences between Oregon Reading First and the National Impact Study Final Considerations

3 National Impact Study Brief Summary 125 Reading First schools 123 comparison schools Impact Questions *Impact on student reading achievement *Impact on classroom instruction Relation between implementation and achievement Findings No overall impact on student reading achievement More time on reading instruction in Reading First schools

4 Oregon Reading First Three-Year Impact Available in three-year technical report ( Cohort A: Improvement over time Year 1 to Year 2 to Year 3 Cohort A and Cohort B 3 years of implementation compared to 1 year of implementation

5 Cohort A Improvement Over Time Percent of children at grade level and meeting benchmark goals Evidence of impact is higher rates for year 3 vs. year 2 vs. year 1 Percent of children at high risk for reading difficulties Evidence of impact is lower rates for year 3 vs. year 2 vs. year 1

6 Comparability in Kindergarten at Beginning of Year

7 Percent Reaching Benchmark Goals on DIBELS

8 Percent Reaching Grade Level on High Stakes Measures

9 Percent at High Risk on DIBELS

10 Percent At High Risk on High Stakes Measures

11 Effect Sizes for Large Scale Reading Interventions (Borman et al., 2003; Borman & D’Agostino, 1996, 2001) Large Scale CSRD Interventions

12 Cohort A Effect Sizes (Year 3 – Year 1) Oregon Reading First DIBELS High Stakes Measure

13 Cohort A and Cohort B Percent of children at grade level and meeting benchmark goals Evidence of impact is higher rates for more years of implementation (Cohort A) Percent of children at high risk for reading difficulties Evidence of impact is lower rates for more years of implementation (Cohort A)

14 Comparability in Kindergarten at Beginning of Implementation Year 1

15 Performance on DIBELS After Year 1 of Implementation

16 Performance Y3 (Cohort A) and Y1 (Cohort B) on DIBELS

17 Percent at High Risk Y3 (Cohort A) and Y1 (Cohort B) on DIBELS

18 Performance on High Stakes Measure After Y1 of Implementation

19 Performance Y3 (Cohort A) and Y1 (Cohort B) on High Stakes Measure

20 Percent at High Risk Y3 (Cohort A) and Y1 (Cohort B) on High Stakes Measures

21 Effect Sizes (Cohort A Year 3 – Cohort B Year 1) Oregon Reading First DIBELS High Stakes Measure

22 Summary of Oregon Reading First Impact Evidence of increased achievement over years (Cohort A) Evidence of improvement for longer implementation duration (three years vs. one year) Improvement in performance on multiple reading achievement measures Improvement in increasing the percent of children reaching grade level and benchmark goals Improvement in reducing the percent of children at high risk for reading difficulties

23 National Impact Study Issues and Considerations Sample Instruction in comparison schools Time devoted to reading instruction Findings in two types of Reading First schools Interim Report vs. Final Report

24 National Impact Study Study Sample Sample was 125 Reading First schools 123 comparison schools There are 5,880 total Reading First schools National Impact Study sample was not a random sample Some important differences between ALL Reading First schools and sample schools Lower % of Hispanics Higher % of African Americans Higher % of large and mid-size cities Higher % of small-sized schools

25 National Impact Study Instruction in comparison schools Instruction in comparison schools may have been highly similar to Reading First Little evidence in report that instruction was different from Reading First No information was provided on: Use of core programs and other materials Use of coaches Use of reading data for decision making This information may be included in the Final Report

26 National Impact Study Instruction in comparison schools Medford School District in Oregon participated in National Impact Study Medford used Reading First as model for other schools in district Including ways to fund non-Reading First schools In the National Impact Study it is not clear whether Medford example was typical or not Other districts in Oregon were typical of Medford Evidence that many other districts throughout the country used Reading First as model for non- Reading First schools

27 National Impact Study Time Devoted to Reading Instruction The amount of daily reading instruction in Reading First sample schools did not meet Reading First requirements In both Reading First and comparison schools less than 60 minutes per day was on reading instruction Reading First requires a minimum of 90 minutes of reading instruction per day

28 National Impact Study Findings on Reading Achievement Overall, there were no differences between Reading First and comparison schools on reading comprehension in grades 1, 2, or 3 However, in states that received late Reading First awards ( ), Reading First schools had higher outcomes in grades 1 and 2 It is not clear what differences between late award sites and early award sites ( ) may have contributed to this finding This should be a key focus in the Final Report

29 National Impact Study Overall Considerations Issues related to the study sample, instruction in comparison schools, time devoted to instruction, and differences in outcomes based on award years are NOT trivial These issues -- and others -- should be the basis of serious attention in the Final Report Other considerations -- e.g., the relation between implementation quality and student outcomes

30 Oregon Reading First and the National Impact Study The Schoolwide Beginning Reading Model has been implemented in all Oregon Reading First schools Emphasis on implementation fidelity is essential to reading outcomes The National Impact Study has not addressed the association between fidelity and outcomes This issue should be a major focus in the Final Report

31 Oregon Reading First and the National Impact Study In Oregon Reading First: minutes of daily reading instruction is provided In National Impact Study: 59 minutes of daily reading instruction was provided Approximately 49 minutes of daily reading instruction was provided in comparison schools Minutes of reading instruction may explain why outcomes appear to be higher in Oregon Reading First 59 minutes of reading instruction raises additional concerns about overall fidelity of implementation

32 Oregon Reading First and the National Impact Study In Oregon Reading First, there have been systematic increases in reading outcomes on a variety of measures (including the SAT-10) across four years of implementation The pattern of findings presented in the three-year report continued in year four (preliminary analysis) In the National Impact Study, increase in reading achievement was not statistically significant across two years of implementation

33 Final Considerations The National Impact Study is important and requires serious study The design is high quality and complex The authors appear to be doing all they can to present the findings clearly and objectively However, there are many important issues to consider in relation to the findings

34 Final Considerations The findings of the National Impact Study are not consistent with findings of Oregon Reading First The National Impact Study presented an Interim Report The Final Report is expected early in 2009 The Interim Report did not address the relationship between fidelity and outcomes This fundamental consideration should be a major focus in the Final Report

35 Final Considerations Because of the requirements of Reading First every reading first school (all 5,880) can present high-quality data regarding the progress being made toward the two primary Reading First Goals: Increasing the percentage of children reading at grade level Decreasing the percentage of children at high risk for reading difficulties

36 Final Considerations How we are doing in Oregon? Percent at Grade LevelPercent At High Risk Every Reading First state and every Reading First school can present this type of information about impact

37 Effect Sizes for Large Scale Longitudinal Interventions (Borman et al., 2003)