1. Program Evaluation How to Effectively Evaluate Your Program Carol Pilcher Department of Entomology Iowa State University

2. Logic Model University of Wisconsin-Extension

3. Purpose of Logic Model Visual display to connect inputs, outputs and outcomes Helps people get on the same page Describes a program and why it works Roadmap to help you know where you are going

4. Logic Model InputsOutputsOutcomes-Impact SituationSituation AssumptionsExternal Factors PrioritiesPriorities What We Invest What We Do Who We Reach What the Short Term Results Are What the Medium Term Results Are What the Ultimate Impact Is Adapted From University of Wisconsin-Extension

5. Situation What is the problem or issue? –Background information, grant proposal Why is this a problem? –Background information, grant proposal Who has a stake in the problem? –Identify stakeholders

6. Priorities Determine the focus of your program –Establish priorities based on: Resources Experience Situation –Objectives and Timeline, grant proposal

7. Logic Model InputsOutputsOutcomes-Impact SITUATIONSITUATION AssumptionsExternal Factors PRIORITIESPRIORITIES What We Invest What We Do Who We Reach What the Short Term Results Are What the Medium Term Results Are What the Ultimate Impact Is

8. Inputs What resources will you invest? –Staff Number of staff and hours available –Time Hours reported –Money Allocated funds

10. Logic Model InputsOutputsOutcomes-Impact SITUATIONSITUATION AssumptionsExternal Factors PRIORITIESPRIORITIES What We Invest What We Do Who We Reach What the Short Term Results Are What the Medium Term Results Are What the Ultimate Impact Is

11. Outputs-Activities What will you do? –Conduct workshops, meetings –Deliver services –Develop products, resources

12. Outputs-Activities Examples Thirty-nine food safety programs were held including hand- washing demonstrations and food safety during the RAGBRAI intra-state cycling trek Starting with a pilot of 4 districts during the 2001-2002 school year, the School Integrated Pest Management Program has trained more than 30 districts statewide.

13. Output-Participation Who will you reach? –Participants –Clients –Agencies –Decision makers –Customers

14. Output-Participation Examples Attendance at Crop Advantage Meetings 2002933 2003720 20041735 5,982 individuals were involved in parent education programs

15. Output-Reactions Reactions about course materials Reactions about program

16. Output-Reactions Example Of the participants that completed the survey (n=120), 98% indicated that the information presented was useful for their farm operation Strongly Agree UndecidedDisagreeStrongly Disagree The information presented today was useful for my farm operation. ___

17. Logic Model InputsOutputsOutcomes-Impact SITUATIONSITUATION AssumptionsExternal Factors PRIORITIESPRIORITIES What We Invest What We Do Who We Reach What the Short Term Results Are What the Medium Term Results Are What the Ultimate Impact Is

18. Outcomes-Impacts Short Term Results Amount of learning –How much was learned during the activity? –KASA Knowledge, Attitudes, Skills, Awareness

19. Outcomes-Impacts KASA Example Of those who returned evaluation surveys, 78% reported moderate to significantly improved understanding of the factors affecting groundwater and surface water contamination of pesticides. Please circle the best response for the following questions Low Level 1 234High Level 5 Prior to todays program indicate your level of knowledge about: Understanding the factors affecting groundwater and surface water contamination of pesticides 12345 After todays program indicate your level of knowledge about: Understanding the factors affecting groundwater and surface water contamination of pesticides 12345

20. Outcomes-Impacts Medium Term Results Changes in actions –Behavioral changes –Changes in practices –Changes in decision making

21. Example Year One As a result of todays training, will you use a drift-reduction nozzle in your sprayer to assist in reducing drift – Adopted Prior To Training, – Plan to Adopt, – Choose Not to Adopt, – Does Not Apply Year Two As a result of last years program, do you use a drift-reduction nozzle in your sprayer to assist in reducing drift – I Adopted – I Did Not Adopt – Does Not Apply Outcomes-Impacts Change in Actions

22. Examples As a result of the program, 50% of the participants said they would use drift reduction nozzles in their sprayers to reduce drift. This post-training evaluation examined if participants had indeed successfully implemented new pesticide safety activities. According to the respondents, 85% of the participants said that they now use drift reduction nozzles in their sprayers. Ninety-eight percent of the food safety program participants reported adopting one of more of the recommended food handling practices.

23. Outcomes-Impacts Ultimate Impacts Changes in conditions –Social/Health –Environment –Economic

24. Outcomes-Impacts Changes in economic conditions Example Approximately, 4.3 million acres were managed with the IPM practices of improved disease resistant varieties, use of safe and effective fungicides as recommended by a forecasting model, and use of the wheat midge risk map and midge scouting. A total of $105.7 million was gained by producers using these IPM practices in 2003.

25. Outcomes-Impacts Ultimate Impacts IPM Focus Areas Production Agriculture Residential and Public Areas Natural Resources and Recreational Environments IPM Impact Areas Environmental Impacts (Reduce environmental risk) Health Impacts (Reduce risks to health) Economic Impacts (Cost/Benefit)

26. Logic Model InputsOutputsOutcomes-Impact SITUATIONSITUATION AssumptionsExternal Factors PRIORITIESPRIORITIES What We Invest What We Do Who We Reach What the Short Term Results Are What the Medium Term Results Are What the Ultimate Impact Is

27. Evaluation Answers the so what question. What difference does your program make for the participants? Determines the worth of a program.