1. Evaluation of Hawaii’s Supplemental Educational Services Program DRAFT Evaluation Plan Presentation to Service Providers and Complex Areas April 2006 Office for Evaluation and Needs Assessment Social Science Research Institute University of Hawaii

2. Evaluation Team Dr. Judith Inazu, P.I., Associate Director, Social Science Research InstituteDr. Judith Inazu, P.I., Associate Director, Social Science Research Institute Dr. Shuqiang Zhang, Statistician, College of EducationDr. Shuqiang Zhang, Statistician, College of Education Dr. Daniel Anderson, Consultant, Planning and Evaluation, Inc.Dr. Daniel Anderson, Consultant, Planning and Evaluation, Inc. Dr. Aiko Oda, Consultant, Planning and Evaluation, Inc.Dr. Aiko Oda, Consultant, Planning and Evaluation, Inc. Ms. Nancy Marker, Educational Specialist, Social Science Research InstituteMs. Nancy Marker, Educational Specialist, Social Science Research Institute Dr. Patty Reiss, Lecturer, College of EducationDr. Patty Reiss, Lecturer, College of Education Ms. Julie Holmes, Graduate Assistant, Social Science Research InstituteMs. Julie Holmes, Graduate Assistant, Social Science Research Institute

3. Goal of SES To increase the academic achievement of economically disadvantaged students in low- performing schools by providing additional academic instruction outside of the regular school day.

4. Purpose of Evaluation To determine the effectiveness of service providers in increasing students’ academic achievement. Providers must: ► Increase students’ achievement for 2 consecutive years, and ► Provide services consistent with applicable federal, state, and local health, safety, and civil rights requirements.

5. Guiding Questions 1.Effectiveness Did the provider increase student achievement in reading and/or mathematics? 2.Customer Satisfaction Are parents, schools, and complex areas involved in SES satisfied with the service provider? 3.Service Delivery Did the provider comply with applicable federal and state laws and regulations, and contractual procedures and requirements associated with the delivery of SES?

6. Interim and Final Evaluation Plans Interim Evaluation Plan Year 1 (Fall 2005 - Summer 2006) Year 2 (Fall 2006 - Summer 2007) Year 3 (Fall 2007 - Summer 2008) Final Evaluation Plan Year 4 (Fall 2008 - Summer 2009)

7. Measuring Academic Achievement In Year 1 of the Evaluation, academic achievement will be the sole criterion for evaluating service providers

8. Measuring Academic Achievement Analysis to be conducted separately for each grade by subject matter, and for each provider. Analysis to be conducted separately for each grade by subject matter, and for each provider. Example: For 4 th graders who received tutoring in reading by College Connections.Example: For 4 th graders who received tutoring in reading by College Connections.

9. Minimum Data Set for Analyses Data on all SES-eligible students. Arrayed by grade level, subject matter, and service provider. Student ID Enrollment in SES Prior HSA Score 2006 HSA Score 001Yes 002Yes 003Yes 004Yes 005Yes 006No 007No 008No 009No 010No

10. Analyses Analyses of Covariance (ANCOVA) with prior HSA score as the covariate, SES status (yes/no) as the binary independent variable, and 2006 HSA score as the dependent variable. The ANCOVA asks the question, “If you hold the pre-test scores constant (since all students begin at different levels of performance), is there a significant difference in post-test scores between students who received tutoring and students who did not?”

11. Analyses for Students Without Pre-test Scores Conduct t-test analyses (difference between mean scores) Compare 2006 HSA test scores between students who received tutoring and those who did not.Compare 2006 HSA test scores between students who received tutoring and those who did not. Again, analyses conducted separately by grade level, subject matter, and for each service provider.Again, analyses conducted separately by grade level, subject matter, and for each service provider.

12. Measuring Customer Satisfaction Pilot tested with selected samples in Years 1-3: ParentsParents TeachersTeachers Complex Area AdministratorsComplex Area Administrators PrincipalsPrincipals Students (To-be-determined)Students (To-be-determined)

13. Measuring Customer Satisfaction Pilot testing to begin in May 2006 Paper and pencil questionnaires with a select sample of parents, teachers, complex area administrators, and principals.Paper and pencil questionnaires with a select sample of parents, teachers, complex area administrators, and principals. Translation requirements for parents.Translation requirements for parents. Internet survey for teachers, school administrators, and complex areas.Internet survey for teachers, school administrators, and complex areas.

14. Sample Questions: Parental Satisfaction 1. The school staff was qualified and supportive in helping me get tutoring for my child. oYes oNo 2. I would recommend this tutor to other parents. oYes oNo 3. My child's school work improved because of the tutoring. oYes oNo 4. My child found tutoring was a positive experience. oYes oNo 5. The tutors were good at reporting to me about my child’s progress. oYes oNo 6. Overall how would you rate this tutor? oStrong oAverage oWeak

15. Measuring Service Delivery Pilot testing to begin in May 2006 Site visits and observations of tutoring sessionsSite visits and observations of tutoring sessions Interview(s) with contact person(s) at the siteInterview(s) with contact person(s) at the site Document review during site visitDocument review during site visit Self-administered compliance checklistSelf-administered compliance checklist

16. Instructional Observation Clear academic expectations are set and articulatedClear academic expectations are set and articulated Instruction is on task without interruption Instruction is on task without interruption Criterion material is covered with instructor’s directionCriterion material is covered with instructor’s direction Instructional time is adequate for material yet flexibleInstructional time is adequate for material yet flexible Learning modalities are active, variable and appropriateLearning modalities are active, variable and appropriate Instructional pacing is appropriate for student’s interest and abilityInstructional pacing is appropriate for student’s interest and ability Student progress is frequently assessedStudent progress is frequently assessed Evidence of mutual respect, positive interaction and feedbackEvidence of mutual respect, positive interaction and feedback Evidence of the instructor’s enthusiasm and interestEvidence of the instructor’s enthusiasm and interest

17. Instructional Delivery Student-teacher ratio Computer, lectures, etc.

18. Compliance Checklist (Examples) SES Provider Assurances Use research-based strategies designed to improve academic achievementUse research-based strategies designed to improve academic achievement Offer instruction consistent with state academic content and achievement standardsOffer instruction consistent with state academic content and achievement standards Meet federal, state and local health, safety and civil rights lawMeet federal, state and local health, safety and civil rights law Provide services that are secular, neutral, and non- ideologicalProvide services that are secular, neutral, and non- ideological Provide information on student’s progress, as agreed upon with the LEAProvide information on student’s progress, as agreed upon with the LEA Remain financially viableRemain financially viable Abide by Family Educational Rights and Privacy Act of 1975Abide by Family Educational Rights and Privacy Act of 1975

19. What We’ll Need From You Complex Areas Written permission to be on campus to conduct site observationsWritten permission to be on campus to conduct site observations Names and addresses of parentsNames and addresses of parents Completion of satisfaction surveyCompletion of satisfaction survey

20. What We’ll Need From You Service Providers Assistance in scheduling site visitsAssistance in scheduling site visits Set aside time for an interviewSet aside time for an interview Documents available for review (e.g., attendance logs, student records, pre/post test reports)Documents available for review (e.g., attendance logs, student records, pre/post test reports) Completion of compliance checklistCompletion of compliance checklist

21. No Vendor Left Behind (NVLB)