1. Program Evaluation for Nonprofit Professionals Unit 2: Creating an Evaluation Plan

2. Your evaluation plan is your road map http://farm8.staticflickr.com/7213/7278697432_a6137fd1a8_c.jpg

3. Resources: what you put in Activities: What you will do Outputs: Immediate tangible results Outcomes: The changes you anticipate Logic Model Activities: What you will do Outputs: Immediate tangible results Data Collection Method, Effort & Schedule Outcomes: The changes you anticipate Indicators: What those changes will look like Data Collection Method, Effort & Schedule Process Evaluation Plan Outcomes Evaluation Plan Adapted from Innovation Network www.innonet.org The logic model informs your plans

4. Two Part Evaluation Plan Process Evaluation Plan What did we do? How well did we do it? Outcomes Evaluation Plan What difference did it make?

5. Process Evaluation Planning Activities (from logic model) Outputs (from logic model) Data Collection (How will you get the info?) Schedule (When will you get the Info?) Level of Effort (low, med, high) Output: Question: Output: Question: Output: Question: Output: Question: Output: Question: Adapted from Innovation Network www.innonet.org

6. Process Evaluation Planning Activities (from logic model) Outputs (from logic model) Data Collection (How will you get the info?) Schedule (When will you get the Info?) Level of Effort (low, med, high) Conduct dairy worker safety education sessions at 55 participating dairies. Outputs: # of sessions conducted # of workers that attended sessions Questions: Was session attendance what we anticipated? Were workers satisfied with the information presented? Attendance records Session feedback form After each session Low Adapted from Innovation Network www.innonet.org

7. Outcomes Evaluation Plan Resources: what you put in Activities: What you will do Outputs: Immediate tangible results Outcomes: The changes you anticipate Indicators: What those changes will look like Data Collection Method, Effort & Schedule Logic Model Outcomes Evaluation Plan

8. Outcomes Evaluation Planning Step 1: Identify your priority outcomes Could be: Outcomes at the core of your mission Outcomes you were specifically funded for Outcomes vital to decision-making in the future

9. Outcomes Evaluation Planning Step 2: Draft indicators for each priority outcome. Indicators are the evidence an outcome has occurred. They answer the questions: What will it look like when it happens? How will you know change has occurred? What does success look like?

10. Outcome Indicator Example Outcome Increase dairy worker’s knowledge of occupational risks. Indicators Dairy workers demonstrate increased knowledge in training pre/post test scores. 75% of dairy workers say “yes” when asked if their knowledge has increased in participant focus groups.

11. Outcomes Evaluation Planning Guidelines for Creating Indicators: 1.Make them specific. 2.Make them measureable. 3.They should directly relate back to the outcome. 4.No more than three. Collecting data on more than thee indicators per outcome is unrealistic.

12. Weak v. Strong Indicators Example Outcome: Improve worker health and safety practices Weak Workers report feeling safer on the job in participant focus groups. Strong Workers report using one or more practices taught in the training within 3 months during focus groups. Top 3 most important improvements are observed in monitoring visits. Dairy producers report making at least one change aimed at improving worker health and safety (within 6 months).

13. Example Outcome: Improved dairy worker health and safety practices Indicator(s)Data Collection Method ScheduleEffort - Decrease in documented workplace hazards within 6 months Document review Y1 Y2 Y3 Low - Workers report using one or more practices aimed at improving health and safety during focus groups Worker focus groups Y1 Y3 High - Top 3 most important improvements are observed during monitoring visits. Observation checklist Y2 Y3 Med - Dairy producers report making at least one change in practices aimed at improved health & safety within 6 mo. Dairy producer focus groups Y1 Y3 High

14. Real World Consideration: Time When do you need this information? What is realistic given your grant cycle or other time limitations? What amount of time is realistic for your staff to dedicate? http://www.flickr.com/photos/kim_carpenter_nj/5462877477/sizes/z/in/photostream/

15. Real World Consideration: Staffing Who will be executing this evaluation plan, current staff or an external consultant? Does your staff have the knowledge necessary? Can your staff get help when necessary? http://www.flickr.com/photos/53558245@N02/4978362207/sizes/z/in/photostream/

16. Real World Consideration: Budget What evaluation efforts are realistic given your budget? What parts of the evaluation plan can be dropped or changed if the budget is reduced unexpectedly? http://www.flickr.com/photos/epsos/5394616925/sizes/z/in/photostream/

17. Real World Consideration: Context What kind of environment will you be conducting this evaluation in? What challenges do you anticipate? What will help you along?

18. Tips for Planning Your Evaluation Keep good notes on the thinking behind the plan, you will likely revisit this in the future Put all the ideas out there, then establish priorities based on what is realistic What is feasible given staff capacity? What parts of this plan may be the most challenging? Which parts depend on factors outside your control? Where can cuts be made if the budget changes? Include others in the planning process Program staff doing the work Those who will be responsible for collecting the data Executive team member or Board member

19. Give it a try! Review the additional information in the readings section Draft your evaluation plans (add them to the logic model template, use additional templates provided or simply write them out on a sheet of paper) Bring your questions to the discussion board!