1. The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community Schools

2. Agenda www.communityschools.org 2 Research summary Community Schools Evaluation Toolkit Discussion Work time: work on your evaluations and network

3. Research Summary www.communityschools.org 3 Are community schools effective at improving outcomes for students, families, and the community? Collected 153 studies of community school and community school-like initiatives – CS models: lead-agency, community agency/CBO, university- assisted, and staff-initiated – Scale: boutique, local, state, national No studies in peer-reviewed journals

4. Characterizing the Literature www.communityschools.org 4 Mix of internal/external evaluators Great variation in study design – Process/implementation studies (e.g., number of students served) – Outcome studies (e.g., improved achievement) Number of outcomes depends on the unique strategies of each community school – found at least 30 outcomes – Tied to theory of action Selected 22 quasi-experimental studies (comparison group, interrupted time series, controls)

5. Achievement www.communityschools.org 5 The most rigorous studies indicate that community schools are improving student achievement for some students. High-implementing CIS schools scored +6% than comparison non-CIS schools in percent proficient in grade 8 math (p<.01) and +5.1% in grade 8 reading (p<.05) Mixed results out of Chicago for school-level data – Student-level demonstrates that OST participation matters More students who participated in Children’s Aid Society (CAS) after-school programs demonstrated a steady increase from 2004 to 2007 in their math performance levels as measured by the state assessment compared to students who did not attend CAS activities (p<.05)

6. Attendance www.communityschools.org 6 The most rigorous studies indicate that community schools are improving attendance. CIS: High-implementing elementary schools had higher attendance levels than comparison group (+0.2%, p<.05). So did high-implementing high schools (+0.3%, p<.01) CAS: students who participated in CAS after-school programs for 3-4 years had better attendance than students with lower or no participation (p<.05) SF Beacons: participants who attended 30+ days of Beacon after-school had 3.9% less total days of unexcused absences than those who participated less than 30 days (p<.001)

7. Graduation Rate www.communityschools.org 7 While a focus of many community schools, there is little evidence of improving the graduation rate. CIS: high-implementing community schools had a significantly better graduation rate than comparison schools (+4.8 percent, p<.01) – Compared to other large-scale dropout programs, high- implementing CIS schools had the highest effect size for graduation rate (ES=.31) and second-highest for dropout rate (ES=.36) CPS: schools that have been in CSI longer have better rates of ninth-grade students “on-track” to graduate than comparison group (p<.05)

8. Behavior www.communityschools.org 8 While a focus of many community schools, there is little evidence of improving student behavior. CPS schools had significantly less disciplinary incidents than their matched comparison group from 2002 to 2006 CAS teachers reported that they saw greater improvement in students getting along with others for students with higher rates of participation in CAS than comparison students during the 2006-2007 school year (significance not reported)

9. Other Outcomes www.communityschools.org 9 Overall, few studies measure, evidence is weak, for the following outcomes:  Parental engagement  Relationships with adults  Student engagement  Impact on instruction

10. Future Directions for Evaluating Community Schools www.communityschools.org 10 Utilize high-quality study designs that establish causality  Why? Funding. Have to demonstrate effectiveness and grant-makers are increasingly aware of study design (e.g., i3) Evaluate the CS strategy, not just programs  Community schools are more than after-school programming  Is the strategy or particular programming causing changes in outcomes? Use appropriate unit of analysis (school, student, or both) Measure:  level of implementation: fidelity to design matters (e.g., CIS)  longevity  intensity of the intervention (after-school participation, receiving health services, … Evaluate outcomes for families and community; also student health Develop standard measures: e.g., how do you measure student engagement? Assess effectiveness of particular CS components to modify and strengthen the design (e.g., do CS coordinators matter?)

11. Community Schools Evaluation Toolkit www.communityschools.org 11 Rationale for Results Framework John W. Gardner Center Community School Evaluators Coalition for Community Schools JP Morgan Chase

12. What is in the Toolkit? www.communityschools.org 12 35 Page step-by-step manual Coalition for Community Schools Logic Model Indicators for 5 short-term results Indicators for 4 long-term results Descriptions of school, city, county, and state data that is available to sites 45 surveys –for sites that are ready to collect additional data Examples from real sites

13. Goals of the Toolkit www.communityschools.org 13 Introduce the Coalition for Community Schools (CCS) Results Based Logic Model Support sites to use existing data sources and collect additional data (if needed) Support sites to identify areas of success and areas in need of improvement Provide evaluation planning tools

14. 4 Parts to the Toolkit: www.communityschools.org 14 1. Before you Start: Begin with the End in Mind 2. Get Ready: Prepare to Evaluate 3. Get Set: Designing the Evaluation 4. Go!: The Evaluation Process

15. www.communityschools.org 15

16. Discussion www.communityschools.org 16 Describe your experience evaluating CS. What challenges have you experienced? Successes? What are your evaluation needs?

17. Work Time www.communityschools.org 17 Work on your evaluations and network

18. How to Get the Toolkit www.communityschools.org 18 Coalition for Community Schools Web site: www.communityschools.orgwww.communityschools.org Questions? Comments? shahs@iel.org

19. Contact information www.communityschools.org 19 Reuben Jacobson  reubenjacobson@gmail.com reubenjacobson@gmail.com Shital C. Shah  shahs@iel.org shahs@iel.org