1. DYS StARR Project March 2010 SREE Presentation, Washington DC  Ohio Department of Youth Services Striving Readers Grant Funded by IES Project Evaluation By Ohio State University  Issues on Design and Impact from a Randomized Control Trials Study  William E. Loadman, PI  Raeal Moore  Jing Zhu  Richard Lomax

2. Local Context  Incarcerated youth ages 12-21  7 High Schools (6 Male 1 Female)  Highly restrictive environments  Students with limited reading ability –2/3 below grade level  Average length of stay 10.5 months  Students with concomitant problems  No consistency on entry and exit of students

3. Mobility Issues  ODYS release and return –16% of Ineligible youth left and returned –8% of Read 180 youth left and returned –7.5% Traditional youth left and returned  Inter School movement –24% Ineligibles moved across schools at least once –27% of Read 180 and Traditional youth moved across schools at least once –Ineligible youth moved as many as 5 times and Read 180 and Traditional youth as many as three times  At any point in time there are approximately 1400 - 1700 youth across all DYS schools

4. Targeted Intervention Logic Model Program Inputs/Activities Classroom Practices: Intermediate Outcomes Short-TermLong Term Student Outcomes Leadership Training provided for Principals, Literacy Coaches, and District Staff [Initial and follow-up; 5 hours total in Y1] R180 Training for Teachers/Teacher Aides [initial 2 days and Semi-annual follow-up; 15 hours total in Y1. Two 5 hour PD sessions (10 hrs total) in Y2] Scholastic Software for classroom including supplemental materials. [including Classroom supplies/materials] Scholastic R180 Software Adapts instruction for Students needs A Maximum of 15 students are scheduled for 2-45 minute periods consisting of the 5 components of Read180: whole group, individualized learning, computer activities, small group (each 20 min), wrap up (10 min) Targeted Strategies Used include: -Sequencing -Identifying main idea/detail -Summarizing -Identifying cause/effect -Making Inferences Positive impact upon return to home/community Use of strategies to increase comprehension/ vocabulary Reduce recidivism Increased literacy, fluency, comprehension and confidence. Reading at Grade Level Increased engagement in educational tasks. Students are assessed using R-skills and Scholastic Reading Inventory Literacy Coach on classroom practices Increased student self-efficacy

5. Implementation Data Collection  Daily implementation logs  Weekly observation data (OSU)  Quarterly feedback from Scholastic observations  Professional development attendance –Literacy coaches –Read 180 teachers/aides –All teachers –Principals

6. Targeted Intervention Implementation Results Facility PD Attendance Year 1 PD Attendance Year 2 Instruction Year 1 Instruction Year 2 1ModerateHighNeeds Improv.Moderate 2 HighModerateNeeds Improv. 3High ModerateHigh 4 5 Moderate 7High Moderate 8High ModerateHigh TotalHigh ModerateHigh

7. Targeted Intervention Implementation Results  OSU Observation Results –Read 180 implemented on Model –Variation across schools –Youth interested and engaged –Differentiated instruction observed –Youth disrupted class on 60% of early visits, but youth not removed from classroom –When either aide or teacher not in classroom, instruction very difficult

8.  Scholastic observations –Reported as on model/protocol –Better able to meet 90 minute model in year two compared with year one –Encouraged use of more strategies to help keep youth engaged (year two) –Encouraged use of SAM reports to better inform instruction (year two) Targeted Intervention Implementation Results

9. Impact Design Targeted Intervention  Random assignment of student to condition (100% reading below grade level, 6 month stay, no GED/diploma) –Condition 1 Read 180 –Condition 2 Traditional English class  Project start-up baseline testing –CAT Reading and Math –SRI Reading Lexile scores  End of each term assessments on SRI (4 per year)

10. Targeted Intervention Sample: Classrooms/Teachers  8-12 students per classroom across facilities and quarters  One-on-one interaction was greater in Read180 relative to Traditional classes  One Read 180 teacher and aide per building/classroom  All teachers certified in English/Language Arts

11. Targeted Intervention Sample: Students

13. Targeted Intervention Analyses  Treatment of the Treated (TTT) –This study defines TTT based on being present in class for at least 5 weeks in each of 2 or more quarters –Estimated 5 HLM analyses using the number of quarters of actual treatment received –The base model (grand mean centered) included the following variables (white, age, Math CAT, Read CAT, disability, grade level, institution and mobility) –The dependent variable was the SRI score –Variables with p values of.08 or less were retained in each final model.

14. TTT Descriptive Results  Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, and the Overall including All Youth

15. TTT Descriptive Results Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, and the Overall for Youth with Less Than Two Quarters of Treatment

16. TTT Descriptive Results Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, and the Overall for Youth with At Least Two Quarters of Treatment OverallRead 180Comparison ncol %n n Slope > 040562.40%24872.09%15751.48% Slope  0 24437.60%9627.91%14848.52% Total649100%344100%305100%

17. TTT Descriptive Results Overall Traditional Read 180

18. TTT HLM Results Level 1: Level 2: Age is continuous; grand mean centered Math CAT is continuous; grand mean centered Read CAT is continuous; grand mean centered Disability Status is dichotomized (0= not disabled; 1 = disabled) Grade level is continuous and ranges from 8-12; grand mean centered  Youth (n = 542) who received between two and eight quarters of treatment.

19. TTT HLM Results Fixed EffectEstimateSEt-ratiop-value Effect Size Intercept α0α0 782.910.25376.36<.0001-- Age α1α1 15.92945.46622.910.00370.02 MathCAT α2α2 10.44143.88512.690.00740.01 ReadCAT α3α3 37.69653.707410.17<.00010.18 Disability α4α4 -28.331914.8666-1.910.05720.01 Grade Level α5α5 17.30267.24632.390.01730.01 ReadCAT*Time β1β1 3.4581.15053.010.00280.02 TRTGroup*Time β2β2 22.3423.48996.4<.00010.12 Random Effectb0b0 b1b1 b0b0 18440 * b1b1 -846.832181.18 *  18820 * Estimates for the Fixed & Random Effects Using 2- 8 Quarters of Treatment

20. Regression Adjusted Means 2006-2007 (Year 1) 2 Quarters or more3 Quarters or more READ 180 ComparisonREAD 180 Comparison Baseline Mean 784.34773.27783.88773.36 Estimated Post-test Mean 856.95827.95879.99820.12 Difference score +72.61+54.68+96.11+46.76

21. Regression Adjusted Means 2006-2008 (Year 1 & 2) 2 Quarters or more 3 Quarters or more 4 Quarters or more READ 180 Compar ison READ 180 Comparis on READ 180 Comparis on Baseline Mean 766.98753.64764.71757.11756.17759.64 Estimated Post-test Mean 813.02768.34823.28766.79842.02757.46 Difference score +46.04+14.7+58.57+9.68+85.85-2.18

22. Summary of Issues  Staff turnover  Release dates  Time in treatment  Student attendance  Student mortality  Local building level perturbations  Student cooperation  Variability of SRI measure  Psychometric properties of SRI  Use of CAT as an additional outcome measure

23. Summary of Methodological Issues  Tracking student entry, movement, exit and return  Power  SRI measure quality/variability  Missing data  Losing data  Maintaining student confidentiality  Reporting issues  Use of data

24. Summary of Issues Cont’d  Student disruptive behavior in class  Student’s placed in isolation  Student’s removed from class  Student receipt of treatment  Student assessment and cooperation  Teacher cooperation  Student release date

25. Summary of Findings  Read 180 being implemented on model and according to protocol  Randomized control trials able to verify causal relationship  Approximately 72% of Read 180 students demonstrate a positive slope of increasing reading scores over time, compared with about 51% for the traditional group  Read 180 adds about 22 Lexile points per quarter of treatment over and above the traditional group performance  Students in Read 180 significantly out perform students assigned to the traditional group  Effect sizes vary according to model, but most are trivial  Most Read 180 students gain, but are still not reading at grade level

26. Future Research  Student perception of Read 180 content and structure  Impact analysis with CAT as an outcome measure.  Sub-analysis determining the influence of disability status on program intervention impacts.  Analysis of the influence of program on recidivism.  Sub-analysis of reason for incarceration, reading improvement, and recidivism.