Program Evaluation How to Effectively Evaluate Your Program Carol Pilcher Department of Entomology Iowa State University

Logic Model University of Wisconsin-Extension

Purpose of Logic Model Visual display to connect inputs, outputs and outcomes Helps people get on the same page Describes a program and why it works Roadmap to help you know where you are going

Logic Model InputsOutputsOutcomes-Impact SituationSituation AssumptionsExternal Factors PrioritiesPriorities What We Invest What We Do Who We Reach What the Short Term Results Are What the Medium Term Results Are What the Ultimate Impact Is Adapted From University of Wisconsin-Extension

Situation What is the problem or issue? –Background information, grant proposal Why is this a problem? –Background information, grant proposal Who has a stake in the problem? –Identify stakeholders

Priorities Determine the focus of your program –Establish priorities based on: Resources Experience Situation –Objectives and Timeline, grant proposal

Logic Model InputsOutputsOutcomes-Impact SITUATIONSITUATION AssumptionsExternal Factors PRIORITIESPRIORITIES What We Invest What We Do Who We Reach What the Short Term Results Are What the Medium Term Results Are What the Ultimate Impact Is

Inputs What resources will you invest? –Staff Number of staff and hours available –Time Hours reported –Money Allocated funds

Logic Model InputsOutputsOutcomes-Impact SITUATIONSITUATION AssumptionsExternal Factors PRIORITIESPRIORITIES What We Invest What We Do Who We Reach What the Short Term Results Are What the Medium Term Results Are What the Ultimate Impact Is

Outputs-Activities What will you do? –Conduct workshops, meetings –Deliver services –Develop products, resources

Outputs-Activities Examples Thirty-nine food safety programs were held including hand- washing demonstrations and food safety during the RAGBRAI intra-state cycling trek Starting with a pilot of 4 districts during the school year, the School Integrated Pest Management Program has trained more than 30 districts statewide.

Output-Participation Who will you reach? –Participants –Clients –Agencies –Decision makers –Customers

Output-Participation Examples Attendance at Crop Advantage Meetings ,982 individuals were involved in parent education programs

Output-Reactions Reactions about course materials Reactions about program

Output-Reactions Example Of the participants that completed the survey (n=120), 98% indicated that the information presented was useful for their farm operation Strongly Agree UndecidedDisagreeStrongly Disagree The information presented today was useful for my farm operation. ___

Logic Model InputsOutputsOutcomes-Impact SITUATIONSITUATION AssumptionsExternal Factors PRIORITIESPRIORITIES What We Invest What We Do Who We Reach What the Short Term Results Are What the Medium Term Results Are What the Ultimate Impact Is

Outcomes-Impacts Short Term Results Amount of learning –How much was learned during the activity? –KASA Knowledge, Attitudes, Skills, Awareness

Outcomes-Impacts KASA Example Of those who returned evaluation surveys, 78% reported moderate to significantly improved understanding of the factors affecting groundwater and surface water contamination of pesticides. Please circle the best response for the following questions Low Level 1 234High Level 5 Prior to todays program indicate your level of knowledge about: Understanding the factors affecting groundwater and surface water contamination of pesticides After todays program indicate your level of knowledge about: Understanding the factors affecting groundwater and surface water contamination of pesticides 12345

Outcomes-Impacts Medium Term Results Changes in actions –Behavioral changes –Changes in practices –Changes in decision making

Example Year One As a result of todays training, will you use a drift-reduction nozzle in your sprayer to assist in reducing drift – Adopted Prior To Training, – Plan to Adopt, – Choose Not to Adopt, – Does Not Apply Year Two As a result of last years program, do you use a drift-reduction nozzle in your sprayer to assist in reducing drift – I Adopted – I Did Not Adopt – Does Not Apply Outcomes-Impacts Change in Actions

Examples As a result of the program, 50% of the participants said they would use drift reduction nozzles in their sprayers to reduce drift. This post-training evaluation examined if participants had indeed successfully implemented new pesticide safety activities. According to the respondents, 85% of the participants said that they now use drift reduction nozzles in their sprayers. Ninety-eight percent of the food safety program participants reported adopting one of more of the recommended food handling practices.

Outcomes-Impacts Ultimate Impacts Changes in conditions –Social/Health –Environment –Economic

Outcomes-Impacts Changes in economic conditions Example Approximately, 4.3 million acres were managed with the IPM practices of improved disease resistant varieties, use of safe and effective fungicides as recommended by a forecasting model, and use of the wheat midge risk map and midge scouting. A total of $105.7 million was gained by producers using these IPM practices in 2003.

Outcomes-Impacts Ultimate Impacts IPM Focus Areas Production Agriculture Residential and Public Areas Natural Resources and Recreational Environments IPM Impact Areas Environmental Impacts (Reduce environmental risk) Health Impacts (Reduce risks to health) Economic Impacts (Cost/Benefit)

Logic Model InputsOutputsOutcomes-Impact SITUATIONSITUATION AssumptionsExternal Factors PRIORITIESPRIORITIES What We Invest What We Do Who We Reach What the Short Term Results Are What the Medium Term Results Are What the Ultimate Impact Is

Evaluation Answers the so what question. What difference does your program make for the participants? Determines the worth of a program.