Presentation on theme: "Evaluating Your Programs: Measurement and Outcomes Presented by: Nina Gottlieb, Ph.D. NMG Evaluation."— Presentation transcript:
Evaluating Your Programs: Measurement and Outcomes Presented by: Nina Gottlieb, Ph.D. NMG Evaluation
Why evaluate your program? Program evaluation is a valuable tool for organizations seeking to strengthen the quality of their programs and improve outcomes for those they serve. Program evaluation answers questions about a program’s effectiveness, and the data collected can be used to improve program services, document best practices, highlight program outcomes, and inform funders.
What is an Evaluation? Collecting analyzing utlizing Program evaluation is simply a systematic method for information to answer questions about a program
Overall Benefits of Evaluation A well-done evaluation can find out what works, what doesn’t work and why. It provides continuous feedback to inform program improvement. Program evaluation can highlight a program’s effectiveness to the community, funders, and potential funders.
Types of Evaluations Process Evaluations examine the extent to which a program is operating as intended and/or the quality of the implementation. Outcome Evaluations investigate whether changes occur for program participants and the extent to which these can be attributed to the program activities.
More Benefits of Process & OutcomesEvaluations ✦ A program evaluation can provide critical information to improve program services and activities, inform future training, and align program goals with realistic implementation strategies. ✦ Evaluation gives program developers and staff objective information on which to base decision- making about program services.
Why you need to gather deeper information about your program
Can you do an outcomes evaluation without a process evaluation? ✦ While programs may want to jump right into finding out how effective they are, this evaluation strategy can easily backfire.
The Real Value of Process Evaluations No matter how simple you may think a program is, its implementation in the real world is likely to be somewhat different from its original vision. Looking more closely at program implementation can give you valuable information about how the program is actually working, whether the intended target population is being reached, and the major challenges and successful implementation strategies.
The Value of Process Evaluations: Case Study of New York Road Runners Foundation Mighty Milers Program ✦ What you may think of as a simple program may not be so simple in action. ‘Mighty Milers’ was described as a simple running program. ✦ There were substantial differences in where students ran (outside, in a school gym, around a classroom, in the lunchroom, on the roof, at a local park), how long they ran (anywhere from 5 minutes to 45 minutes), how their mileage was tracked (by students themselves, by teachers, estimated as a group, etc.) and a host of other issues.
The Value of Process Evaluations: Case Study of New York Road Runners Foundation Mighty Milers Program Real world Implementation After conducting a process evaluation that uncovered the range of implementation models of this “simple” program, the NYRRF realized how many different variations of the program existed and how far the models had strayed from their original vision. Targeted training NYRRF subsequently revamped their training materials to better prepare people to implement the Mighty Milers program under the many different circumstances that existed in the schools with which they worked. Technical assistance The organization also reorganized their staff to provide more “hands- on” assistance to schools to successfully implement the program.
Other Advantages of Process Evaluations Examining the details of program implementation is also important in linking outcomes with implementation strategies and models. Without evaluating how a program is actually working in the ‘real world’ it is impossible to identify the most effective strategies for your program or to fine-tune the program to have the greatest impact.
Identifying Program Outcomes Before conducting a program evaluation, the most critical step is to identify the particular outcomes the program hopes to achieve. These outcomes are specific to each program and should be a natural extension of the program’s goals, objectives, and activities or services. Do not worry too much about what you think a particular funder might want to know; your outcomes must flow from your program.
Steps in Identifying Program Outcomes #1: Reflect on the program’s mission and purpose and what impacts you want to have on the population you serve. In essence, why are you doing what you’re doing? Typically, outcomes are related to changes in attitudes, knowledge, behavior, skills, or quality of life. #2: Choose the outcomes you want to examine. You may not have the resources to examine all of the outcomes at one time, so you may want to prioritize them.
Things to keep in mind Outcomes are big picture concepts. For example, “to improve the quality of life of older adults” or “to increase the number of students attending college” or “to increase the financial literacy of young adults.” Outcomes Once you have one or more outcomes for your program, you need to figure out how to measure those outcomes. These measurements are known as indicators. Indicators
Let’s practice developing outcomes! Why do you think the the program was developed? What possible results might come from it for school-age children? (There might be several). Use the example of the “Mighty Milers” program offered by the NY Road Runners Foundation to develop some possible outcomes. Things to guide your thinking:
Let’s practice developing outcomes! What needs to be measured will vary based an organization's specific mission, and the specific clientele served. For your program to be considered successful, what kind of change should be visible over time? What types of results suggest that your program is doing what it aims to do?
Next Step: How do we develop indicators and measurement tools? OutcomeIndicator Measurement Tools Increased student achievement in math Change in percentage of students performing proficiently in math Percentages of students performing at levels 3 or 4 on NYS Math Assessment in 2013-14 and 2014-15
Steps in Identifying Indicators For each outcome, identify what observable measures or indicators will tell you that you are achieving that outcome. There might be several indicators for each outcome. Work backwards to make sure that for each indicator, your program is actually providing services that make it likely that the indicator will show improvement or change.
Next Step: How do we develop indicators and measurement tools? Outcome What will indicate whether or not we have achieved this outcome? Indicators must be clear, measurable, understandable, relevant, time-specific, and valid
Let’s practice developing indicators For each outcome you developed with your table, create indicators. Talk about how you would know that you achieved that outcome. What would be possible indications of success for each outcome? Develop at least 3 indicators. These could all be for one outcome, or one for each outcome.
Next Step: How do we measure indicators? ❖ The tools by which we measure indicators will depend on factors including: The outcome itself and how it can be measured The population being served (for example, can they complete surveys? Can you obtain health information about them?) How many people are being served? (You do not want to conduct interviews with hundreds of people. Nor do you want to survey 10 people). Ultimately, you want to ask yourself: How will you know if the outcomes/goals were achieved?
Common Measurement Tools Pre-Post surveys Retrospective Post-surveys Surveys Appropriate for assessing information gained over time Tests of Knowledge Identification of pertinent data available Analysis of relationships between/among different types of data Existing Data
Common Measurement Tools Perceptual data for small/special populations In-depth information from select individuals Interviews Perceptual, in-depth data from medium- sized representative groups Focus Groups Provides a record of time spent on tasks Logs
Practicing identifying measurements/methods For each indicator you developed at your table, discuss what types of measurements/ methodologies would be appropriate. Generate at least one way to measure each indicator.
Tying it all together: Process and Outcomes Evaluations Process Evaluations……tell you about how your program is really being implemented Outcomes Evaluations……tell you about the impacts of your program and the results achieved Together, you can link program results with the ways in which your program is implemented to produce a stronger program and better results
❖ Conducting the process evaluation allowed us to examine the program’s outcomes with various aspects of the program’s implementation. ❖ For example, we found that having the opportunity to accumulate significant mileage over the course of the school year was associated with the most positive feelings about the program as well as enjoyment of getting exercise and feeling that the running program made school more fun. Linking Process Findings with an Outcomes Evaluation: Case Study of the NYRRF Mighty Milers Evaluation
Yes, but…… it is always better to have someone else do it Can you evaluate yourself?
How can you obtain an outside evaluation with limited funds? There are foundations and agencies willing to fund evaluations Get funding Work with local graduate programs to get students to help with the evaluation, from survey development to data collection Get creative Hire an outside evaluator to consult on the evaluation, even if you can’t afford a full objective evaluation Target your funds
If you must do a self-evaluation…… Do not spread your organization too thin. Decide what outcomes are most important to your program. Focus on a few outcomes When you use data that is easily accessible, you can use your resources to link data together in meaningful ways. Use already existing data Put yourself in the place of a potential funder. What would you want to see as evidence of effectiveness? Try to look at your program with an outsider’s view
QUESTIONS? If you want more specific advice, information, resources: email: firstname.lastname@example.org Call: 914-293-7511