Introduction to Program Evaluation Lindsay Shepard, MSW, MSc University of Utah 3 July 2013

Agenda Review Introduction to Evaluation Types of Program Evaluation Context Evaluation Implementation Evaluation Outcome Evaluation Introduction to Logic Models

Why Evaluate? “Evaluation should be conducted not only to demonstrate that a project worked, but also to improve the way it works” (Kellogg Foundation, Evaluation Handbook, p. 4)

How to Evaluate: “Effective evaluation is not an ‘event’ that occurs at the end of a project, but is an ongoing process which helps decision makers better understand”: The project Its impact on participants, partner agencies, and the community Its relationship with internal/external factors Ways to improve (Kellogg Foundation, Evaluation Handbook, p. 3)

Types of program Evaluation

Context Evaluation Examines how the project functions within the economic, social, and political environment of its community and project setting

Context Evaluation Can answers questions such as: What are the community’s or clients’ needs? What are the community’s or clients’ assets? What are the community’s or clients’ resources? What are the goals/objectives of the planned services? What resources does the program have? How might needed resources be attained? How do the project plans fit or not fit with community priorities? What data can be collected and what is the plan for evaluation and dissemination?

Implementation Evaluation Helps with the planning, setting up, and carrying out of a project, as well as the documentation of the evolution of a project

Implementation Evaluation Can answers questions such as: What project activities are linked with client/community outcomes? How well is the program sticking to its model? How well are practitioners sticking to the manual? What contextual factors inhibited or promoted the implementation and success of this program? What feedback do staff and participants have about the successes/failures of the project?

Outcome Evaluation Assesses the short- and long-term results of the project

Outcome Evaluation Can answer questions such as: Is the intervention effective? Who is the intervention effective for? What are the long-term impacts?

Developing an Outcome Evaluation 1. Who are you going to serve? 2. What outcomes are you trying to achieve for your target population? 3. How will you measure whether you’ve achieved these outcomes? 4. What data will you collect and how will you collect it? 5. How will you use the results? 6. What are your performance targets?

Which Type? “Evaluation plans should include all three components. How much each component is emphasized, however, depends on the phase of project development, the purpose of the evaluation, and the questions you are attempting to address” (Kellogg Foundation, Evaluation Handbook, p. 20)

Introduction to Program Logic Models A logic model: “is a picture of how your program works” “links outcomes (both short- and long-term) with program activities” and theory “provides a roadmap of your program, highlighting how it is expected to work, what activities need to come before others, and how desired outcomes are achieved” Kellogg Evaluation Handbook, p. 3

Logic Model Definition “A logic model is a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan and the changes or results you hope to achieve” (Kellogg Foundation, Logic Model Handbook, p. 1)

When to Use a Logic Model? In Program Design and Planning, a logic model serves as a planning tool to develop program strategy and to engage key stakeholders. In Program Implementation, a logic model forms the core for a focused management plan. For Program Evaluation and Strategic Reporting, a logic model presents progress toward goals.

Types of Logic Models Theory Model – illustrate how and why you think your program will work Activities Model – links the various planned activities together to describe what a program intends to do in great detail Outcomes Model – attempts to connect the program resources and activities with their outcomes