1. Evaluating SES Providers Steven M. Ross Allison Potter Center for Research in Educational Policy The University of Memphis http://www.memphis.edu/crep

2. Supplemental Educational Services (SES) Required under No Child Left Behind (NLCB) for Title I Schools that have not made Adequate Yearly Progress (AYP) for three consecutive years. Low-income students from identified Title I schools are eligible to receive free tutoring services. Students are prioritized by greatest academic need if district funds are limited.

3. Potential service providers apply to serve students and may be approved by the State Department of Education. Service Providers Providers contract with Local Educational Agencies (LEAs) to provide tutoring services to students. Providers are paid for their services - an amount not to exceed the Title I per pupil allotment.

4. Effectiveness: Increased student achievement in reading/language arts or mathematics. Customer satisfaction: Positive perceptions by parents of SES students. Service delivery and compliance: Positive perceptions by principals, teachers, LEA staff, etc. Determining Evaluation Measures

5. Effectiveness (Student Achievement) Service Delivery and Compliance Customer Satisfaction Provider Survey District Coordinator Survey Principal/Liaison Survey Teacher Survey Parent Survey Additional Tests State Tests Figure 1. Components of a Comprehensive SES/Evaluation Modeling Plan Overall Provider Assessment

6. 1. Student-level test scores from state-mandated assessments Effectiveness Measures Considerations: availability only for certain grades (e.g., 3-higher)? Lack of pretest scores prevents gains from being determined

7. 2. Supplementary individualized assessments in reading/language arts or math Effectiveness Measures Considerations: Without pretest scores and comparison students, SES gain cannot be determined Validity may be suspect if assessments not administered by trained independent testers

8. 3.Provider-developed assessments in reading/language arts or math Effectiveness Measures Considerations: Test results may not be valid or suitable for state’s evaluation purpose Tests may favor provider’s strategies

9. 1. Parent and family perceptions Customer Satisfaction Measures Considerations: Parent respondents may not be representative of the population served by provider Sample sizes will vary due to provider size Comparisons limited due to parent familiarity with only one provider

10. 2. Student perceptions Customer Satisfaction Measures Considerations: Young students may have difficulty judging quality of services and communicating impressions Time consuming and may require parent permission to obtain

11. 1. Records of services provided, student attendance rates, and costs Service Delivery and Compliance Measures Considerations: States may obtain data from a variety of sources, including providers, teachers, principals, and district staff Corroborating data from multiple sources can increase accuracy of evaluation conclusions

12. 2. Feedback from SES customers Service Delivery and Compliance Measures Considerations: First-hand impressions or observations may be lacking Translation may be needed to reach parents who do not speak English Obtaining representative samples may be difficult

13. 3. Feedback from district staff Service Delivery and Compliance Measures Considerations: Districts may lack firsthand impressions or observations of tutoring services Some districts may also be SES providers

14. 4. Feedback from school staff Service Delivery and Compliance Measures Considerations: Teachers may also be SES instructors or lack first- hand impressions of providers Teachers may need to provide information on multiple providers, which may be confusing and time consuming Identifying teachers to solicit responses may be difficult

15. A. Benchmark Comparison Rating = ++ (Low to Moderate rigor) Percentage of SES students by provider attaining “proficiency” on state assessment Evaluation Designs: Student Achievement

16. A. Benchmark Comparison Evaluation Designs: Student Achievement Upgrades Percentage of SES in all performance categories (“Below Basic”, “Basic”, etc.) Comparison of performance relative to prior year and to state norms Comparison to a “control” sample

17. A. Benchmark Comparison Evaluation Designs: Student Achievement Advantages Inexpensive and less demanding Easily understood by practitioners and public Linked directly to NCLB accountability Disadvantages Doesn’t control for student characteristics Doesn’t control for schools Uses broad achievement indices

18. B. Multiple Linear Regression Design Rating = +++ (Moderate rigor) Compares actual gains to predicted gains for students enrolled in SES, using district data to control for student variables (e.g., income, ethnicity, gender, ELL, special education status, etc.). Evaluation Designs: Student Achievement

19. B. Multiple Linear Regression Design Evaluation Designs: Student Achievement Advantages More costly than Benchmark, but relatively economical Student characteristics are statistically controlled Disadvantages Doesn’t control for school effects Less understandable to practitioners and public Effect sizes may be less stable than for Model C.

20. C. Matched Samples Design Rating = ++++ (High Moderate to Strong rigor) Match and compare SES students to similar students attending same school (or, if not feasible, similar school) Use multiple matches if possible Evaluation Designs: Student Achievement

21. C. Matched Samples Design Evaluation Designs: Student Achievement Advantages Some control over school effects Easily understood by practitioners and public Highest potential rigor of all designs Disadvantages More costly and time consuming Within-school matches may be difficult to achieve

22. D. Combination (Hybrid) Design Evaluation Designs: Student Achievement Uses a mixture of three main designs to meet special data situations within the State State level analysis may be benchmark for most districts and matched samples for largest district(s) Accommodates different student-level data and statistical staff resources

23. Surveys for LEAs, principals/site coordinators, teachers, parents, and providers. Data Collection Tools Common core set of questions from all groups to permit triangulation. Open-ended question, “Additional comments”

24. Rubric of Overall Evaluation of Provider Effectiveness Outcome Insufficient Information Below Standards Marginal Quality AcceptableAbove Standards 1. Student Achievement There is insufficient information available to determine student achievement outcomes. Students have not shown gains related to tutoring received from service providers. About half of the students have made some gain related to tutoring received from service providers. There has been some gain for the majority (over 60%) of students related to tutoring received from service providers. The effect size for students in the provider’s program is in the top one-third of all the effect sizes demonstrated by providers meeting standards for student achievement. 2. CommunicationThere is insufficient information available to determine communication outcomes. Provider has not communicated with the principals, teachers, and parents of students served. There has been limited communication throughout the year between the provider and at least two of the following: principals, teachers, and parents. There has been some regular communication throughout the year between the provider and the principals, teachers, and parents of students served. There is an ongoing and sustained system of communication between the provider and the school- level educators as well as parents of students served. 3. Instructional Plans There is insufficient information available to determine instructional plans of the provider. Provider does not plan instruction explicitly geared to student needs or to reinforce their regular academic program. Provider is in the planning stages of gearing instruction to student needs, and reinforcing the regular academic program. Provider has made some attempt with the majority of students to plan instruction explicitly geared to student needs and to reinforce the regular academic program. Provider instructional plans are explicitly geared to the needs of most or all students and reinforce the regular academic program. 4. Local and State Standards There is insufficient information available to determine alignment with local and state standards. None of the instructional plans used by the provider are aligned with local and state academic standards for students. Provider is in the process of aligning instructional plans with local and state academic standards for students. Some of the instructional plans used by the provider are presently aligned with local and state academic standards for students. Most or all of the instructional plans are presently aligned with local and state academic standards for students. 5. Special Ed/ELL students There is insufficient information available to determine special ed/ELL student outcomes. Provider does not offer accommodations for addressing the needs of special ed or ELL students. Provider has made limited accommodations for addressing the needs of special ed and ELL students. Provider has made some accommodations for addressing the needs of special ed and ELL students. Provider offers appropriate services, if needed, to special education and ELL students. 6. Provider Overall There is insufficient information available to determine provider overall outcomes. There is overall dissatisfaction with the provider at the district and school levels. There is more dissatisfaction than satisfaction with the provider at the district and school levels. There are mixed but mostly positive reactions about the provider at the school and district levels. There is overall satisfaction with the provider at the district and school levels.

25. Decision Tree for SES Providers Probation I

26. CONCLUSION Each state should begin its SES evaluation planning process by identifying a)the specific questions that its SES evaluation needs to answer, and b)the resources that can be allocated reasonably to support further evaluation planning, data collection, analysis, reporting, and dissemination.

27. CONCLUSION Work through the hierarchy of evaluation designs presented here and select the design that allows the highest level of rigor. States may wish to engage third-party evaluation experts in helping to plan and conduct these evaluations.