Results Based Program: Outcomes & Impacts Your project was selected because it was determined to have the potential to make significant impact on sustainable strawberry production. NSSI – a results based program; the grant selection, reporting, and management The Goal: To increase the probability that funded programs will improve the skills, practices and technology of the strawberry growers and industry that participate. Key components (Logic Model based): Outputs Outcomes Impacts Evaluation
Beginning with the “end” in mind Projects should Begin with the End in Mind first Determine first what the outcome is that the participants will achieve – or you want them to achieve. Design the program (outputs) to lead the participants to those results/outcomes. “How will you get there?” Evaluation – how will you measure the change?
Logic Model: Beginning With the End in Mind OUTCOMES (Identify Measurable Technology/ Practice Results) OUTPUTS (Lead project participants to the desired results) INPUTS (Resources that will strengthen participant achievement of results) INPUTS OUTPUTS OUTCOMES Logic Model Implementing Forward From: Western Risk Management Education Center, http://westrme.wsu.edu/award-management/http://westrme.wsu.edu/award-management/
Logic Model as a Tool to Construct Realistic Outcomes First: Identify the situation(s) that your project will help to address (pest management, input reduction, food safety etc.) and consider if what you are proposing will be well received by the target audience you have identified (Producer/Industry Demand).Then: 1.Outcomes 1.Outcomes – Identify the changes in knowledge, actions, or conditions/system you would like program participants to realize 2.Outputs 2.Outputs – Identify activities (workshops, on-farm trials, demonstrations, etc.) that will support and lead to participant outcomes 3.Inputs 3.Inputs – Resources that go into your program in the way of project team, partners, collaborators, who provide expertise, research, money etc.
An Outcome Defined …. Progressive Change in: Knowledge* Actions, Attitudes* Condition, System, Environment *Useful Tool: Search Bloom’s Taxonomy of Action Verbs Or See: http://www.clemson.edu/assessment/assessmentpractices/referencematerials/documents/Blooms%20Taxonomy%20Action%20Verbs.pdf
Outcomes Defined…. Changes in knowledge… When project participants Learn or become aware of new fundamental or applied knowledge which could include: 1.Practices and methods 2.Technology 3.Improved skills such as decision making, evaluating
Outcomes Defined…. Change in Actions… When project participants Act upon what they have learned such as: 1.Implementing new practices, methods or technology 2.Changing or modifying a practice 3.Adoption of improved skills 4.Direct application of knowledge gained
Outcomes Defined…. Change in Conditions, System, Environment… A condition is changed because of industry/growers wide-spread actions 1.Increased number of strawberry producers in a state/region leading to increased production and availability. 2.Increased local/regional strawberry production or season extension to increase availability to consumers. 3.Adoption of practices that leads to a reduction in greenhouse gas emissions. Etc.
Outcomes Will Depend Upon: -Knowledge of your target audience -Why industry/producers will choose to participate -Action and learning processes (project steps) that help industry/producers achieve long term outcomes -Program resources including the team, collaborators and expert presenters sufficient to lead participants to the desired outcomes.
Producer Results or Outcomes an example of progressive steps… ↓ Understand or have improved understanding of the issue, the practice/technology/methods or the need to take action. ↓ Decide to take an action. - Producer makes a commitment to write a business or marketing plan, set up a record keeping system, or initiate other steps that will lead to adoption ↓ Develop –customize a system or plan ↓ Implement or take an action that improves farm or business i.e. – implementing a food safety plan, using a new technology, etc.
In Summary… Outcomes Measure of success for the participant Ability to change/incorporate/implement/adopt practices, technology or skill, etc. as a result of participation in your project. Achievable and measureable within a reasonable timeframe Sequential Identifies changes in condition or practice you seek for the participant Specifies the degree of change required for participant success Stated in a measurable way knowing methods you will use to verify changes in practices, condition, etc.
NSSI Reporting: Outputs “List your outputs in relation to the objectives of the proposal. Include activities such as presentations (professional, academic, outreach, stakeholders, consumers, etc.), publications (professional, academic, outreach, etc.), workshops, websites, blogs, etc. Include target audience and number of people.” What activities did you conduct to move your participants towards project outcomes?
NSSI Reporting: Outcomes “List the outcomes of your project, i.e. what change in knowledge, attitude or actions did your outputs accomplish in the short to intermediate time frame. Include who was reached, how many were reached and how outcomes were measured.” What did your participants/target audience do with the project outputs? Increase knowledge? Initiate change in thought processes? Change behaviors/practices? How did you measure this?
NSSI Reporting: Impact “List the actual or anticipated impact of your project, i.e. the big-picture effect of your project on the industry/growers/region etc. (changes in condition, environment, society) over the longer-term. Include how impact were or will be measured.” True impacts are harder to measure and may not be evident until late in the project or after the project has been completed.
Other NSSI Reporting Questions Summary of your project Objectives of your project Media contacts Internet media Problems delays or changes in plans Next accomplishments Attachments/Uploads
Underperformance Performance will be evaluated based upon your timely and thorough reporting Underperforming projects risk termination. Performance metrics include; Timely activities as proposed Accomplishments towards your project goal Timely and appropriate use of funds Termination Unsatisfactory project performance will receive notice Notice will include corrective actions and reporting Project have one reporting period to correct Reminder: Project Deadline is June 30, 2014
NSSI Reporting and Submitted Financials Project Progress Reports should be in alignment with your Project Financial Reports. The progress, outputs, and outcomes of your project will be compared to what funding has been used and its purpose.
Evaluation How will you measure the effect of your outputs to ensure they are creating suitable/appropriate participant change or any change at all? Protocols Methodologies Data collection
Measuring Outcomes Begin with the end in mind Begin with determining: The results that will be accomplished How they will be evaluated Design evaluation when developing project Key tip For each proposed outcome you should be able to write an evaluation question Do this when you develop your funding proposal Validates that proposed results are actual results Helps design the evaluation from the beginning
NIFA Metrics for Measuring Outcomes Test your methods Will your evaluation tool(s) clearly demonstrate progress participants are making towards achieving the outcomes? -Changes in Knowledge? -Changes in Attitude? -Changes in Behavior/Approach? -Changes in Condition or the System?
How can you evaluate your outcomes? 1.Identify measurable expected outcomes. 2.Define criteria for success. 3.Develop evaluation tools to measure results. – End-of-training evaluations – Post-training evaluations 4.Evaluation distribution.
Defining Criteria for Success Consider What will you actually be measuring? What is the standard for success? How many participants need to achieve ‘success’ in order to be successful? When should participants achieve results?
Evaluation Mechanics Suitable Question Formats – Multiple Choice – Yes/No – Rating scales – Open ended Simple for respondent to understand Easy to analyze for project teams with less evaluation experience Variety of Methods: Pre/Post Tests Audience Response Systems (interactive polling Clickers or apps) Follow-up Evaluations (3 Months – Year) Mail or Internet Survey Telephone Interview Face-to-face interviews (in-person survey) Dot survey Informal Interview
Evaluation: Pre and Post Test Pre-test Purpose Benchmark info on knowledge levels Information on participant background, past experience, attitudes and needs for program managers and instructors Post-test Purpose Measures Knowledge Gain Attitude Change Actions participants expected to take as a result of participating in workshop/demonstration/project.
Combining Pre/Post into Post “Please rate your knowledge PRIOR to attending today’s workshop” “Please rate your knowledge AFTER attending this workshop” “After attending today’s workshop, answer the following questions: Please rate to what extent you have the knowledge, tools and resources needed to … I better understand…” Likert Scale or Similar Method
Follow-Up Evaluation Purpose: To see what actions participants have taken as a result of participating in your project and how these actions have led to benefits for their farm/business.
Evaluation Best Practices Measuring results/outcomes successfully: Choose methods that balance best practices with available resources Combination of direct and indirect measures Direct measures (assignments, action plans etc.) yield stronger evidence of change in knowledge and beneficial practices, but are more time intensive and may take more expertise to evaluate. Indirect measures can give instant feedback but may not provide real evidence of knowledge gained. Qualitative measures (like phone interviews or focus groups) that may be more resource intensive could be accomplished through a representative sample.
Evaluation Best Practices Measuring results/outcomes successfully: Direct Interactions with Producers can increase the chance that participants will implement beneficial practices/technology etc. Good follow-up can lead to collegial relationships that may bring results beyond the initial results/outcomes that were initially projected (Unexpected results). Good program design allows for learning and application to take place in linked stages within a timeframe that encourages the implementation/adoption of longer term results/outcomes. When writing the evaluation questions, think about: How the information gathered will be used. The way you ask a question will influence how participants respond. When you ask a question will influence how participants respond.
Results/Outcome Resources The Rensselaerville Institute: http://whatworks.org/displaycommon.cfm?an=1&subarticlenbr=266 “Outcome Funding: A New Approach to Targeted Grant making” Based on the premise that grant makers are really investors in human gain, Outcome Funding addresses the key questions that the funder as investor should ask: What am I buying? What is the probability I will get it? Outcome Funding begins with an abrupt and timely challenge to the standard wisdom of the Request for Proposal process. The authors make the point that very little of the traditional proposal content addresses results, favoring explanations of process instead. This book is not only persuasive on the need for change; it is clear and practical on how to achieve it. Developing a Basic Logic Model For Your Program: http://www.publichealth.arizona.edu/chwtoolkit/pdfs/logicmod/chapter2.pdf I have become convinced that it makes a considerable difference if you do the outcomes before planning the activities. I definitely advocate doing the outcomes first! I find that people come up with much more effective activities when they do. Use the motto, “plan backward, implement forward.” Beverly Anderson Parsons, WKKF Cluster Evaluator http://www.uwex.edu/ces/pdande/evaluation/evallogicmodelworksheets.html Ties to the Logic Model. http://extension.oregonstate.edu/sites/default/files/for-employees/employee-resources/soars- modules/logic_model/index.htm\ Measuring Impacts for Sustainable Agriculture Programming http://agsci.oregonstate.edu/sites/default/files/understanding_program_development_and_evaluation.pdf
Citations Information in these slides was borrowed and adapted from Western Risk Management Education Center http://westrme.wsu.edu/award-management/ How Will Your Verify? – Measuring Producer Results Extension Risk Management Education http://www.extensionrme.org/ Oregon State University, Writing Powerful Impacts http://agsci.oregonstate.edu/research/writingimpacts
National Strawberry Sustainability Initiative Reporting Project Outcomes email@example.com