1. Dr. Bonnie J. Faddis & Dr. Margaret Beam RMC Research Fidelity of Implementation and Program Impact

2. 2 PPS Striving Readers Partners Grantee: Portland Public Schools Professional Developer: University of Kansas, Center for Research on Learning Evaluator: RMC Research

3. 3 Targeted Intervention Overview Goals: – To increase reading achievement – To increase reading motivation Intervention features: – Xtreme Reading curriculum – Small class size (15 students/teacher) – Ongoing professional development

4. 4 Eligibility for Intervention Students enrolled in Grades 7–10 Students reading 2 or more years below grade level on the Oregon State Assessment Test (OSAT) Students without OSAT scores who are reading 2 or more years below grade level on the GRADE test

5. 5 Intervention Characteristics 9 schools – 5 middle schools – 4 high schools 18 targeted intervention teachers (2 per school) Year 1 9 schools –1 middle school –2 K–8 schools –1 6–9 girls’ school –1 8 th grade academy –4 high schools 18 targeted intervention teachers Year 2

6. 6 Intervention Characteristics Year 3 10 schools –2 middle schools –3 K-8 schools –1 6-10 girls’ school –4 high schools 14 targeted intervention teachers

7. 7 Intervention Staffing Plan 25-30 students per grade level assigned to intervention class at each school Intervention class paired with language arts or social studies class in high school Intervention class integrated with LA/SS block in middle school 2 teachers co-taught intervention class and LA/SS class (Years 1-2)

8. 8 Counterfactual Xtreme Reading taught during LA/SS block Control group students received LA/SS instruction with higher student- teacher ratios Middle Schools Xtreme Reading was an elective course Control group students enrolled in a range of other electives across content areas High Schools

9. 9 Xtreme Reading Model

10. 10 8 Stages of Instruction Describe rationale and strategy steps Model strategy through think-aloud Verbally rehearse strategy steps Guided reading practice Paired reading practice with fluency tests Independent practice w/comprehension check Apply strategy during oral reading with teacher Integrate strategy with text from other classes

11. 11 Implementation Results Professional Development Classroom Implementation Teacher Buy-In Factors that Facilitated Implementation Barriers to Implementation

12. 12 Professional Development Participation Group session attendance > 75% –Year 1: 40% of middle school and 25% of high school Xtreme teachers –Year 2: 89% of middle school and 75% of high school Xtreme teachers –Year 3: 100% of middle school and 71% of high school Xtreme teachers Added in Year 2 & 3: –Monthly meetings of Xtreme teachers

13. 13 Classroom Implementation School Average Fidelity Year 1Year 2Year 3 Middle Schools Alpha92%89%79%* Beta96%90%92% Gamma72%84%93% Delta93%91%94%* Epsilon94%88%— Phi — —95% Zeta — —80%* High Schools Kappa83%63%92% Lambda84%58%82% Sigma82%78%74%* Theta45%—*80%*

14. 14 Teacher Buy-In Professional Development & Support* Perceived Program Effectiveness** MiddleHighMiddleHigh Year 14.23.64.13.6 Year 24.23.24.03.7 Year 34.44.04.64.0 *Rating Scale: 1 = Strongly disagree; 5 = Strongly agree **Rating Scale: 1 = Not at all helpful; 5 = Very helpful

15. 15 Factors that Facilitated Implementation Xtreme teachers liked the curriculum Students liked the books Teachers liked the professional development Curriculum pacing schedules improved 80% of Year 2 teachers had experience 62% of Year 3 teachers had experience

16. 16 Barriers to Implementation Organization of Xtreme materials was confusing Teacher skills with low achievers Changes in middle school configurations reduced target population Small school organization in 2 high schools created scheduling problems High school counselors resistant

17. 17 Impact Evaluation Questions –Was the Striving Readers Xtreme intervention effective in improving students’ reading achievement? –Were the effects of the Striving Readers Xtreme intervention similar for middle schools and high schools?

18. 18 Impact Evaluation Random Assignment Eligible students were randomly assigned to treatment or control (stratified by school and grade level) Analytic Design Intent-to-treat statistical model using 2-level HLM (students clustered within schools)

19. 19 Impact Evaluation Outcome Measure Spring GRADE test normal curve equivalent (NCE) scores Covariates –Fall GRADE pretest normal curve equivalent (NCE) scores –Cohort –Ethnicity (Black, Hispanic) –English language proficiency –Grade level

20. 20 Statistical Power 2-level HLM cluster randomized trial N = 1,273 Power = 80% Alpha =.05 Minimum detectable effect (MDE) =.27

21. 21 Random Assignment

22. 22 Student Sample Grade LevelN 7347 8313 9342 10271 Total1,273

23. 23 Student Characteristics Characteristic Treatment Percent Control Percent Gender Male53 Female47 Ethnicity White25 American Indian22 Hispanic3031 African American2829 Asian1512 Special Education Services25 ELL Services2725 Note. Treatment total N = 599; Control total N = 674.

24. 24 Impact Results Estimated Impact Groupn Impact (  ) S.E. Effect Size p- value Overall8492.580.79.15.002 Middle School 4434.840.90.29.000 High School4060.161.31.01.901

25. 25 Impact Results Additional Finding Significant school level variability in treatment effects in both the overall analysis and in the middle school analysis

26. 26 Effects of Implementation Question To what extent do teacher level variables explain school level variability in treatment effects?

27. 27 Analytic Design Statistical model using 2-level HLM (students clustered within schools) Teacher level variables aggregated up to school level Separate models by implementation year

28. 28 Level 2 Variables Percent fidelity of implementation Percent of professional development training attended Years of teaching experience

29. 29 Results Fidelity of implementation significantly contributed to school level variability in treatment effects in Years 1 and 3, but not in Year 2

30. 30 Summary Program was more effective in middle than in high schools Variability in student outcomes partially explained by fidelity of classroom implementation Teaching experience and amount of professional development not related to student outcomes