Three ‘R’s for Evaluating the Memphis Striving Readers Project: Relationships, Real-World Challenges, and RCT Design Jill Feldman, RBS Director of Evaluation Kelly Feighan, RBS Senior Researcher

Memphis Striving Readers Five-year US Department of Education grant Whole-school and targeted READ 180 ™ interventions aimed at improving middle school students’ achievement in reading and core content areas Research for Better Schools (RBS), evaluator

Targeted Intervention Eight urban, high-poverty middle schools (6-8 th grades) Nineteen READ 180 ™ teachers, 48 sections Eligibility: no prior participation, two or more grade levels behind in reading, bottom quartile on Tennessee Comprehensive Assessment Program (TCAP)

Evaluation Design Random assignment of eligible students: –Treatment group participates in READ 180 ™ –Control group in “traditional” ELA class Baseline and follow-up assessments: Iowa Test of Basic Skills (ITBS), Scholastic Reading Inventory (SRI), and TCAP

Relationships with the Research Team School District: Office of Research Evaluation and Assessment, principals, teachers, coaches, and parents Issues: Buy-in, compliance with randomization, and READ 180 ™ implementation and staff training READ 180 ™ publisher

Real-World Challenges High student mobility (attrition) Parental consent (IRB) Exclusion of ELL and special education students (sample bias)

Real-World Challenges Balancing local knowledge of individual students’ needs with identification of “eligible pool” Understanding specific school contexts Inconsistent availability of supplemental reading instruction Delayed startup due to slow delivery of equipment

Maintaining RCT Design Two waves of random assignment within first 20 days of school Ongoing negotiations with school personnel Construction of school profiles Monthly data “chats” between district and external evaluators

Preliminary Findings Students eligible for READ180 ™ : N = 2,277 No differences in race, gender, ethnicity, or poverty level between conditions Higher percentage of ELLs in the control group (87 of 1,337 students, or 6.5%) than in READ 180 ™ group (35 of 940 students, or 3.7%) Higher percentage of special education eighth graders in READ 180 ™ group (28.2%) than in control group (20.9%)

Baseline Findings No differences in TCAP scores (N=2,277): –Control (N=1,337): M = 471.9; SD = 29.0 –READ 180 ™ (N=940): M = 473.5; SD = 26.3 No differences in ITBS scores (N=2,063): –Control (N=1,199): M = 187.9; SD = 16.2 –READ 180 ™ (N=864): M = 189.3; SD = 16.3

Attrition

Planned Analyses Analyses: Impact of READ 180 ™ participation on student reading scale ITBS score Control Variables: school average TCAP and ITBS performance, enrollment, students’ previous ITBS score, gender, disability status, and poverty status Treatment effects: difference between average student scale ITBS score for those in READ 180 ™ and those in control group

Next Steps… ITBS assessment: May 2007 TCAP assessment: April 2007 Analysis of SRI scores for READ 180 ™ students: Summer 2007