1. CREP Center for Research in Educational Policy SES Student Achievement Methods/Results: Multiple Years and States Steven M. Ross Allison Potter The University of Memphis Center for Research in Educational Policy

2. Supplemental Educational Services (SES) Required under No Child Left Behind (NLCB) for Title I Schools that have not made Adequate Yearly Progress (AYP) for three consecutive years. Low-income students from identified Title I schools are eligible to receive free tutoring services. Students are prioritized by greatest academic need if district funds are limited.

3. Potential service providers apply to serve students and may be approved by the State Department of Education. Service Providers Providers contract with Local Educational Agencies (LEAs) to provide tutoring services to students. Providers are paid for their services - an amount not to exceed the Title I per pupil allotment.

4. Effectiveness (Student Achievement) Service Delivery and Compliance Customer Satisfaction Provider Survey District Coordinator Survey Principal/Liaison Survey Teacher Survey Parent Survey Additional Tests State Tests Figure 1. Components of a Comprehensive SES/Evaluation Modeling Plan Overall Provider Assessment

5. A. Benchmark Comparison Rating = ++ (Low to Moderate rigor) Percentage of SES students by provider attaining “proficiency” on state assessment Evaluation Designs: Student Achievement

6. A. Benchmark Comparison Evaluation Designs: Student Achievement Upgrades Percentage of SES in all performance categories (“Below Basic”, “Basic”, etc.) Comparison of performance relative to prior year and to state norms Comparison to a “control” sample

7. A. Benchmark Comparison Evaluation Designs: Student Achievement Advantages Inexpensive and less demanding Easily understood by practitioners and public Linked directly to NCLB accountability Disadvantages Doesn’t control for student characteristics Doesn’t control for schools Uses broad achievement indices

8. B. Multiple Linear Regression Design Rating = +++ (Moderate rigor) Compares actual gains to predicted gains for students enrolled in SES, using district data to control for student variables (e.g., income, ethnicity, gender, ELL, special education status, etc.). Evaluation Designs: Student Achievement

9. B. Multiple Linear Regression Design Evaluation Designs: Student Achievement Advantages More costly than Benchmark, but relatively economical Student characteristics are statistically controlled Disadvantages Doesn’t control for school effects Less understandable to practitioners and public Effect sizes may be less stable than for Model C.

10. C. Matched Samples Design Rating = ++++ (High Moderate to Strong rigor) Match and compare SES students to similar students attending same school (or, if not feasible, similar school) Use multiple matches if possible Evaluation Designs: Student Achievement

11. C. Matched Samples Design Evaluation Designs: Student Achievement Advantages Some control over school effects Easily understood by practitioners and public Highest potential rigor of all designs Disadvantages More costly and time consuming Within-school matches may be difficult to achieve

12. Tennessee Results 2003-2004 Analysis Model:Multiple Regressions Predictors:Prior achievement, district, poverty, gender, special ed status State-wide Aggregate Results: Out of 17 Reading analyses, 0 were significant Out of 14 Math analyses, 0 were significant

13. Tennessee Results 2004-2005 A. Analysis Model:Multiple Regressions Predictors:Prior achievement, district, poverty, gender, special ed status State-wide Aggregate Results: Out of 25 Reading analyses, 0 were significant Out of 23 Math analyses, 0 were significant

14. Tennessee Results 2004-2005 B. Analysis Model: Control Variable:Prior achievement and teacher State-wide Aggregate Results: Out of 9 Reading analyses, 0 were significant Out of 12 Math analyses, 0 were significant Multiple Regression using SAS EVAAS

15. Tennessee Results 2004-2005 C. Analysis Model: Control Variable:Predicted achievement and teacher State-wide Aggregate Results: Out of 9 Reading analyses, 0 were significant Out of 12 Math analyses, 0 were significant Matched Student Pairs using SAS EVAAS

16. Tennessee Results 2005-2006 A. Analysis Model: Control Variable:Prior achievement and teacher State-wide Aggregate Results: Out of 10 Reading analyses, 1 was significant & negative Out of 10 Math analyses, 0 were significant Multiple Regression using SAS EVAAS

17. Tennessee Results 2005-2006 B. Analysis Model: Control Variable:Predicted achievement and teacher State-wide Aggregate Results: Out of 10 Reading analyses, 1 was significant & negative Out of 10 Math analyses, 1 was significant & negative Matched student pairs using SAS EVAAS

18. Louisiana Results 2004-2005 Analysis Model: Controls (covariates): Prior achievement, district, ethnicity, and gender State-wide Aggregate Results: Out of 7 Reading analyses, 0 were significant Out of 10 Math analyses, 1 was significant & positive Matched Samples

19. Virginia Results 2005-2006 Analysis Model: Controls (covariates): Poverty and school State-wide Aggregate Results: Out of 13 Reading analyses, 5 were significant & negative Out of 12 Math analyses, 1 was significant & positive 4 were significant & negative Benchmark

20. SUMMARY OF ACHIEVEMENT EFFECTS BY STATE AND SUBJECTS State and Subject Analyses Significantly Positive Significantly Negative Directionally Positive fff% Tennessee Reading800250% Math810151% Louisiana Reading70086% Math1010100% Virginia Reading130523% Math121533% TOTAL2032 (0.9%)10 (5%)51%