Evaluating a Literacy Curriculum for Adolescents: Results from Three Sites of the First Year of Striving Readers Eastern Evaluation Research Society Conference April 14, 2008 Galloway, New Jersey

Striving Readers—Overview  Goals of the Striving Readers program  Raise student achievement in middle and high schools by improving the literacy skills of struggling adolescent readers  Help build a strong, scientific research base around specific strategies that improve adolescent literacy skills.  Eight sites around the country  Three sites discussed in this presentation  5-Year grant period (2006–2011), assuming continuation funding

READ 180 Classroom Model

READ 180 Logic Model

The Memphis Striving Readers Project: Year 1 READ 180 Findings Deb Coffey Research for Better Schools

Motivation behind MSRP Memphis is one of the cities with the highest educational need in the U.S.

Motivation behind MSRP MCS middle schools compared with schools nationwide

Motivation behind MSRP  Memphis City Schools is 21st largest K–12 district in US (>116,000 students)  Over 95% of MCS’ 196 schools are Title I schools  71% of MCS students qualify for free or reduced- price lunch  87% of MCS students are African American; 9% are white; 4% are “other”  71.5% of students in grades 6–8 scored below the 50th percentile on the Reading/Language Arts portion of the Tenn. state assessment (TCAP)

MSRP Overview  Targeted intervention: READ 180  Focus of this presentation  Participants (students) randomly selected from pool of eligible students, i.e., struggling readers  Schoolwide intervention: Memphis Content Literacy Academy  Four schools (of eight) randomly selected in matched- pairs design  Teachers participate in intensive professional development program

Overall MSRP Goals — to determine 1. The effects of MCLA on core subject teachers’ knowledge and use of scientifically based reading strategies/methods 2. The separate and combined effects of MCLA and READ 180 on students’ reading achievement levels, especially students who are identified as struggling readers 3. The separate and combined effects of MCLA and READ 180 on students’ achievement in core subjects, especially students who are identified as struggling readers

Study Design  Evaluate student outcomes using an experimental design based on randomly assigning eligible students to treatment and control conditions within participating schools  Student outcomes include reading achievement (ITBS) and state assessment (TCAP) results in core content areas

Analytic Approach  Cross-sectional ITT analyses of reading and core content area achievement  Two-level HLM using spring ITBS and TCAP scores as a function of student and school variables

Students Enrolled in READ 180

Experiences of READ 180 students and control condition students

Variables included in impact analysis  Independent  READ 180 Participation  Dependent  Spring 2007 ITBS:Total Reading Comprehension Vocabulary  Spring 2007 TCAP:Reading/LA Mathematics Science Social Studies

Covariates included in impact analysis  Fall 06 ITBS: Total Reading Comprehension Vocabulary  Fall 06 TCAP:Reading/LA Mathematics Science Social Studies  Free or Reduced Price Lunch  Grade  Gender  African-American / Hispanic  English Language Learner  Percentage Female  Percentage African- American  Percentage Special Ed  Percentage FRL  Percentage ELL  School Enrollment

READ 180 Impacts on Students (Y1)

Conclusions  No significant Year One student impact  Late startup  (Most) students will receive two years of intervention  Planned Future Analyses:  Exploratory analyses of relationships between amount of READ 180 instruction and effects on student outcomes

Implementation: Changes  First day of teacher training divided into two groups (novices and experienced)  Follow-up teacher training was 2 days (instead of 1)  Year One training for administrators and implementation support were not provided  Classroom substitution of “boom boxes” for personal CD players

Implementation: Cross-Site Variations  Differential school-level participation in professional development  Special education students not assigned READ 180 at some schools (legal concerns)  Adherence to READ 180 model

Implementation: Barriers  Equipment delays  DOE prohibition on evaluator sharing implementation findings with MCS  Teacher contracts preventing MCS from requiring attendance of meetings held after hours

Secondary/Exploratory Analyses  What did students actually experience?  Was class “on model”?  Were students present?  Sources of data  Classroom observations  6 annually: 3 by RBS, 2 by MCS, 1 by developer  Data produced inside READ 180 program  District records  Teacher surveys (RBS)  Student surveys (MCS—element of READ 180

Secondary/Exploratory Analyses: Possible inclusion in addl. HLM  Did classes follow the overall READ 180 model?  Number of students in class  Length of class  “Rotations”  Use of READ 180 materials  How much READ 180 did students experience?  Student absences  Student engagement  Time on task  READ 180 quizzes

Secondary/Exploratory Analyses: Background information  What was level of teacher professional development for READ 180?  Did teachers attend PD sessions?  What was teachers’ level of satisfaction with PD?  How do students describe their experiences with READ 180?  Did teachers report systemic challenges that prevented adherence to READ 180 model?

Clarifying Questions?

END

READ 180 Training Responsibilities Provided by Developer  2 full days of teacher and coach PD  7 two-hour network meetings (with PD) for teachers & coaches  PD for principals & technology coordinator  On-site technical support, as needed  Online “Red” Course (teachers & coaches) Provided by MCS  Classroom support, as requested by teachers

Implementation Roles & Responsibilities MCS  Purchase equipment & supplies  Hire/assign READ 180 teachers  Arrange schedule (90 minute)  Roster randomly assigned students Developer  Technical implementation check (within 4 weeks)  One year unlimited (limited) support  Review MCS data & provide recommendations