Cathy Burack and Alan Melchior The Center for Youth and Communities The Heller School for Social Policy and Management, Brandeis University Your Program Evaluation: Making it Work While You Do AmeriCorps Program Kickoff Conference Massachusetts Service Alliance July 23, 2015

Center for Youth and Communities, Brandeis University2 Purpose of Today’s Session To provide a brief introduction to evaluation design and some of what’s involved in planning an evaluation. To assist you in thinking about your organization’s capacity to undertake an evaluation. To provide an opportunity to develop a work plan for your organization’s evaluation.

Center for Youth and Communities, Brandeis University3 Our Assumptions Evaluation is an organizational management tool that is used to both prove and improve programs. Non-profits are increasingly being asked by funders and other stakeholders to use evidence-based approaches to demonstrate that they are making a difference. Organizations frequently have more data than they currently use, and not enough capacity to use it effectively. Fortunately, organizations can learn to assess their current capacity to use data and evaluation and can create plans to guide their evaluation efforts. Evaluations need to be part of organizational planning, and plans need to be customized to address each organization’s interest, needs, and resources.

Center for Youth and Communities, Brandeis University4 Why Do Evaluation Why Do Evaluation? Accountability - key stakeholders require it. Proving program impact - making the case that what you do makes a difference. Program improvement - to see what’s working, using data for program management and improvement. Marketing/Fundraising - to generate support for your program. Knowledge development - to generate information that will improve the quality of practice in the field.

Center for Youth and Communities, Brandeis University5 “Wouldn’t it be nice if just once they’d give us money, skip all the falderal, and say ‘go crazy.’”

Center for Youth and Communities, Brandeis University6 Different Types of Evaluation Answer Different Kinds of Questions Process/Implementation EvaluationOutcomes Evaluation Describing ContextDocumenting Activity/Outputs Understanding ProcessAssessing Outcomes Assessing Impact  Who is my target group?  What are the needs of my target group or community?  What personal or collective strengths or assets exist within the target community?  How many participants?  What are characteristics of participants?  How many hours of service?  What types of activities?  What are the Costs/Funding for the program?  Was the program implemented as expected?  What parts of the program worked well, and what parts need to be strengthened?  Did the program staff have the resources and training they needed?  Did the program’s design and activities match the program’s goal?  Did the expected changes in participant attitudes and behavior take place?  Were there changes in attitudes, instructional strategies, etc. among teachers/ leaders during the program?  What contributions did the program make to the community?  What difference has the program made (relative to no program or a different program)?  Would the observed changes have taken place in the absence of the program?

Center for Youth and Communities, Brandeis University7 Different Kinds of Evaluations Require Different Methods/Resources Preliminary EvidenceModerate Evidence Strong Evidence MeasuresOutcomes Impacts – with Reservations Impact/Causality SubjectsParticipants only Well-Matched Comparison Group Control Group (generally randomly assigned) SitesOne or more Multiple/”Typical” or Representative Multiple/ Representative Analysis Basic: Comparisons (t- Test, Chi Square, etc.) Complex: PSM, Multivariate Analysis, Time Series, etc. Complex: Multivariate Analysis, Time Series, etc. Cost$$$$$$ Org resources Staff with evaluation experience or outside evaluator Outside help needed Audiences Your staff and board, supporters, some funders Foundations, some federal/state agencies, some policy makers, researchers Federal/State funders, policy makes, researchers Pros Provides basic outcome data (is change happening?) for program mgmt, reporting, marketing Demonstrates impact – accepted by many audiences Considered “Gold Standard” for demonstrating impact Cons Can only assume impact – doesn’t control for outside factors More expensive, only partially controls for outside factors; not accepted by all funders Expensive. Requires major commitment. Can negatively impact program operations.

Center for Youth and Communities, Brandeis University8 Making it real, almost… Mini Case

Center for Youth and Communities, Brandeis University9 Planning your Evaluation – What you’ll need to get started Have you already identified some outcomes or questions you want to measure or answer (e.g., in a logic model, performance measures, board meeting action item)? What data do you already collect? What staff members will be engaged in this effort? Will you need outside help (e.g., someone at a site)? Will you need an external evaluator? What is your program’s calendar? What are evaluation reporting dates?

Center for Youth and Communities, Brandeis University10 A reminder: your logic model process asked: What is the problem to be addressed or goal to be achieved? Who will be served through your initiative? What are your assumptions about how and why your initiative will work? What strategies/activities will you use? What outcomes do you expect? How will you measure success? What are the long-term impacts you hope to achieve?

Center for Youth and Communities, Brandeis University11 Think about your own organization What data do you collect, have on hand, and how are you using it? What are you staff/organizational resources? Where are there gaps, what else will you need?

Center for Youth and Communities, Brandeis University12 Developing the plan Design the evaluation to meet your needs – there is no one “right” approach. Involve your stakeholders in the process. Make effective use of the resources (people and information) that you have on hand. Make evaluation a living, useful process – a “want to” instead of a “have to.” Start with the questions, not the instrument

Center for Youth and Communities, Brandeis University13 What Kinds of Information Will I Need to Answer My Questions What Kinds of Information Will I Need to Answer My Questions? Use your logic model or theory of change to identify priorities What do we most want to measure? What do we need to know about now vs. later? Match the data to the questions What kinds of information “fits”? Create a flexible and responsive design The program should drive the evaluation, not the other way around! Think about what you already collect Collect and analyze data from multiple perspectives Don’t rely on just one type of source! Keep the available resources in mind

Center for Youth and Communities, Brandeis University14 Work plans Your evaluations should be aligned with your organization’s overall work plan. What happens next?

Center for Youth and Communities, Brandeis University15 The Center for Youth and Communities, Brandeis University Research, policy, program assistance center located at The Heller School for Social Policy and Management, Brandeis University. Major focus on youth, education, community engagement and development. Contact Information: Cathy Burack Alan Melchior