1. A Collaborative Mixed-Method Evaluation of a Multi-Site Student Behavior, School Culture, and Climate Program

2. Tailoring our Evaluation Practices to Leverage Available Resources

3. Outline What is City Year? Behavior, Culture, and Climate Initiative (BCCI) BCCI Study – Data and methods – Study Challenges Multi-Participant Approach – Benefits – Challenges – Strategies

4. WHAT IS CITY YEAR? City Year is an education-focused nonprofit organization that partners with public schools to keep students in school and on track to succeed.

5. 25 locations 242 schools 2,600+ corps members

6. Whole School, Whole Child A poor Attendance B disruptive Behavior C Course failure in math or English

7. Full time 1-on-1 Small groups After school

8. 9 9 Cities 31 31 Schools in 2011-2012 Pilot Behavior, Culture, and Climate Initiative 8 Supporting student social emotional learning (SEL) skills is strongly connected to work in providing academic supports. Elements of BCCI include: In-class/homeroom behavior support Targeted support: 50 Acts Program Integration of City Year culture Additional training and support for corps members 2011-2012: third year of pilot program and evaluation

9. Behavior, Culture, and Climate Initiative Third-Year Study 9

10. BCCI Study Purpose To strengthen implementation and replicate this work across the City Year network. Help the organization understand and share some early outcomes. Guiding evaluation questions centered on outcomes, implementation, replication and evaluation practices.

11. The Study Evaluation Team External Evaluator: – Brett Consulting Group Internal Staff, members from the: – City Year Headquarters Evaluation Team, and – Headquarters Program Design Team

12. Data and Methods: Specific to the BCCI student prosocial behaviors rich case observations interviews/focus groups student suspensions, detentions, office referrals

13. City Year centrally collects network-wide survey and intervention tracking data Relevant network data was used in combination with information collected for the BCCI pilot Internal database tracking student-level interventions Online, network-wide teacher, principal/ school liaison, and corps member surveys

14. How We Used Evaluation Findings External Engage champions. Inform funders. Attract new funders. Internal Refine program design, metrics, and quality of implementation. Early knowledge provided a head start on planning for the following school year.

15. Challenges to This Study Variations in program implementations Different ways of measuring same construct by different organizations in the study Data availability, including timeliness Imperfect match of measures with goals of program

16. Multi-Participant Approach to Evaluation

17. Benefits of Multi-Participant Approach Better use of available resources Cost and time savings. Increased capacity for access to schools and program staff. Increased capacity for data collection, analysis, and report writing. Ability to draw upon multiple perspectives and backgrounds. Further internal understanding of program Real time information on measurement and outcomes to inform planning for the following school year.

18. Possible Challenges to the Multi-Participant Approach buy-in by participants objectivity by internal staff openness to approach by external evaluator agreement about where to put energy and resources Lack of… coordinating schedules (esp. for observation) multiple report writers Logistics

19. Strategies When Working with an External Evaluator in a Multi-Participant Evaluation Choose an external evaluator who… embraces and has experience with this approach Assure that the external evaluator is familiar with… the program model all existing instrumentation/ data collection systems Use the external evaluator to… design at least one key part of the study be involved with overall design

20. General Strategies for Multi-Participant Approaches clearly define purposes of the study clarify each person’s role reinforce expectations, importance, timelines use point people across multiple sites Communication have one person oversee and coordinate report writing allow time for the unexpected Logistics

21. General Strategies for Multi-Participant Approaches (2) develop efficient systems to decrease burden on staff and others develop written protocols for data collection train staff and others in data collection protocols Working with Staff emphasize recording of specific behaviors double up on observations share perceptions across observers ask for evidence to back up observations Conducting Observations

22. Thank You! Contact Us: – Belle Brett: bellebrett@comcast.net – City Year Evaluation: evalteam@cityyear.org