MSRP Year 1 (Preliminary) Impact Research for Better Schools RMC Corporation

Overview: Study Design and Analytic Approach Study Design READ 180: Evaluate student outcomes using an experimental design based on randomly assigning eligible students to treatment and control conditions within participating schools Student outcomes include reading achievement (ITBS) and state assessment (TCAP) results in core content areas Study Design MCLA: Evaluate teacher and student outcomes using an experimental design based on randomly assigning schools within matched pairs to treatment and control conditions Teacher outcomes include preparedness and frequency of literacy strategy use Analytic Approach READ 180: Cross-sectional ITT analyses of reading and core content area achievement Two-level models using spring ITBS and TCAP scores as a function of student and school variables Analytic Approach MCLA: Two-level models using spring indices as a function of teacher and school variables Two-level models using spring ITBS and TCAP scores as a function of student and school variables

MCLA: Random Assignment of Schools

Demographic Characteristics of Year 1 MCLA Student Sample

Baseline Comparisons of Students in MCLA Treatment and Control Schools

READ 180: Enrolled Students

Statistical Model Used to Estimate READ 180 Impacts At the student level,

Statistical Model Used to Estimate READ 180 Impacts

Variables Included in READ 180 Impact Analytic Models (Year One): Dependent and Independent

Variables Included in READ 180 Impact Analytic Models (Year One): Covariates

Analytic Decisions Missing Data –students missing pretest score(s) deleted from impact analysis on relevant measure(s) Covariates –include all student- and school-level covariates in the model –run the model –eliminate the school covariate with the lowest significance level (highest p-value) not less than 0.2 – repeat steps 2 and 3 until the remaining covariates had p-values less than 0.2 – repeat steps 2-4 for the student covariates

READ 180 Impacts on Students (Year One)

Selected Characteristics of the Year 1 Teacher Sample for MCLA Impact Analyses

Statistical Model Used to Estimate MCLA Teacher Impacts

All Variables Included in MCLA Impact Analytical Models for Year 1

Analytic Decisions Missing Data –teachers missing pretest score deleted from impact analysis on measure Covariates –include all teacher and school-level covariates in the model –run the model –eliminate the school covariate with the lowest significance level (highest p-value) not less than 0.2 – repeat steps 2 and 3 until the remaining covariates had p-values less than 0.2 – repeat steps 2-4 for the teacher covariates

Comparison of Teachers in MCLA Treatment and Control Schools on Year-End Indices for Preparedness and Frequency of Use

Regression Models Employed to Test Year 1 MCLA Impact on Student Achievement

Analytic Decisions Missing Data –students missing pretest score(s) deleted from impact analysis on relevant measure(s) Covariates –include all student- and school-level covariates in the model –run the model –eliminate the school covariate with the lowest significance level (highest p-value) not less than 0.2 – repeat steps 2 and 3 until the remaining covariates had p-values less than 0.2 – repeat steps 2-4 for the student covariates

MCLA Impacts on Students (Year One)

(Brief) Conclusions & Discussion READ 180: No significant Year One student impact Late startup (Most) students will receive two years of intervention Planned Future Analyses: Three-level analyses planned to examine whether teacher characteristics exert a moderating effect on student outcomes Exploratory analyses of relationships between amount of READ 180 instruction and effects on student outcomes

(Brief) Conclusions & Discussion MCLA: Significant (moderate) impact on teachers’ frequency and preparedness to use MCLA strategies No significant impact on students’ achievement in reading or core content areas Discuss: –Subjectivity of measure (“Hawthorne Effect”) –Teacher findings support program logic model –Explore relationship between impact and participation in PD