1. Simple Surveys: From Program Objectives to Impact Statements in 90 Minutes 2016 Extension Conference Jeff Buckley & Jennifer Cantwell January 13, 2016

2. Topics we will cover….. General Evaluation Overview: – Why and what do we evaluate? – Outcomes → Constructs → Questions Survey Templates Data Tabulation Templates Impact Statement Templates Create you own!

3. You will leave with…. Free, adaptable resources! www.georgia4h.org/evaluationresources/

4. Topics we will cover….. Why evaluate? Proper Usage of data (IRB) Survey Templates Data Tabulation Templates Impact Statements Tools for creating Surveys The overly-busy 4-H Agent: Scene 1: Why evaluate?

5. Why do you need to evaluate? Program Impact: Did kids learn? Program Improvement Professional Development & Accountability – Improve your impact statements and other reports. – Gather data for posters, workshop proposals, and awards

6. Topics we will cover….. Why evaluate? Proper Usage of data (IRB) Survey Templates Data Tabulation Templates Impact Statements Tools for creating Surveys The overly-busy 4-H Agent: Scene 2: Don’t I need permission from IRB?

7. Institutional Review Board (IRB) Research oversight committee charged with ensuring that human subjects research is conducted in compliance with the applicable federal, state, and institutional policies and procedures.

8. When is IRB-approval needed? Program Evaluation vs. Research

9. Topics we will cover….. Why evaluate? Proper Usage of data (IRB) Survey Templates Data Tabulation Templates Impact Statements Tools for creating Surveys The overly-busy 4-H Agent: Scene 3: Okay, well that’s great and all, but I don’t know anything about evaluation!

10. The Process – Keep it simple! Determine Program Objectives Develop Survey Questions Administer Survey Analyze Data Create Impact Documents Share / Use Impact Data

11. Types of Outcomes Short-term – Change in Knowledge, Skills, Attitude – Behavioral Intent Intermediate – Change in Behavior Long-term – Improved Condition

12. Today we will focus on: Short-term Outcomes – Increase in Knowledge or Skill – Demonstration of Knowledge or Skill – Change in Behavioral Intention As a result of intervention, I plan to….

13. Outcomes What are some of the outcomes you hope youth or adults will experience from participation in your programs?

14. Measurable Outcomes Measurable outcomes will translate into more effective questions. Resources – Handout “Verbs for Writing Measurable Objectives” – Packet – Building Capacity in Evaluating Outcomes

15. Constructs More than one question to measure the same outcome or concept. Ask the same question in a different way Ask multiple questions to show they gained knowledge (ex. Circle the 4 main functions of government)

16. Examples – Measurable Outcomes & Constructs Program - Teach youth to make healthier food choices and to increase activity. 2 Short-term Outcomes – Youth are able to identify healthy food choices – Youth plan to make more healthy choices

17. Identify the intended outcomes 1.I know more about what foods are healthy. 2.I learned new ideas for being active. 3.I can identify healthy food options. 4.I have learned new things I can do to be healthier. 5.I plan to exercise more. 6.Decreasing screen time is a healthy choice.

18. Intended Outcomes: Answers 1.I know more about what foods are healthy. 2.I learned new ideas for being active. 3.I can identify healthy food options. 4.I have learned new things I can do to be healthier. 5.I plan to exercise more. 6.Decreasing screen time is a healthy choice.

19. Let’s go back-wards – What are the two outcomes? 1. The food in the cafeteria was fresh. 2. The dining area was clean. 3. There were a variety of fruits and vegetables. 4. Beverage selections included healthy options. 5. Vegetables were not over-cooked. 6. The servers were pleasant.

20. What are the two Outcomes? 1.The cafeteria will provide a pleasant dining experience. 2.The cafeteria will provide healthy options for diners.

21. Topics we will cover….. Why evaluate? Proper Usage of data (IRB) Survey Templates Data Tabulation Templates Impact Statements Tools for creating Surveys The overly-busy 4-H Agent: Scene 4: Okay, now I get it. I don’t know where to start!

22. Things to consider What do you want to know? Does a survey already exit to measure what you want to know? Who is your audience? Adults? Youth? How much time do you have to administer survey? Paper surveys or online? How will you share the results?

23. General Types of Surveys  Quantitative Data (Purse Strings)  Qualitative Data (Heart Strings)  Pretest and Posttest  Posttest Only  Retrospective Post then Pre

24. Topics we will cover….. Why evaluate? Proper Usage of data (IRB) Survey Templates Data Tabulation Templates Impact Statements Tools for creating Surveys The overly-busy 4-H Agent: Scene 5: Okay, well, I’m so busy! What kinds of questions should I ask? What should it look like? I wish there was a template I could use to get started…..

25. Our Survey Tools  Survey Templates  Likert Style - Pretest and/or Posttest  Likert Style - Retrospective Post then Pre  Combination Knowledge & Likert  Kid-friendly format  Program-specific Surveys  Ready to use with specified curricula  All surveys include open-ended questions  Most include Data Tabulation Spreadsheets

26. Simple Surveys: Likert Style

27. Simple Survey Templates – Likert Style Standard Likert Questionnaire – Can be used as pretest/posttest or posttest only – Data Tabulation Spreadsheet & Directions Retrospective Post-Then-Pre – Administered after intervention

28. Points to Consider when Writing Likert Style Questions/Statements Consist of a statement and a rating scale Need to conform to one rating scale One data point per question Is what you’re measuring important to the intended audience of the evaluation? Good for older youth and adults

29. Simple Survey Templates – Likert Style Adaptable Data Tabulation Spreadsheets with directions Good for trainings, adults, older youth

30. Combination Knowledge and Likert Style

31. Quiz-style / Program or Topic Based Include quiz-type questions (matching, definitions, true/false, etc.) Can also include Likert Style questions Pre and posttest or posttest only Survey Answer Key & Data Tabulation Spreadsheet Spreadsheet can include pre-made outcome statements

32. Existing Surveys CCRPI: Career Awareness – Marketing, Finance, Government/Public Administration, Information Technology States of Matter (Paulding County) Manners Worth Millions Bullying (Cook County) Farm to Fork Less Stress on the Test

33. Existing Surveys continued Project Achievement* – Cloverleaf, Junior/Senior Coming Soon: – Inherited traits – Barrier Islands – Mikey the Microbe – Plant ID *These are best suited for traditional demonstrations.

34. Topics we will cover….. Why evaluate? Proper Usage of data (IRB) Survey Templates Data Tabulation Templates Impact Statements Tools for creating Surveys The overly-busy 4-H Agent: Scene 6: I think I can write some questions and administer these surveys. Now what do I do with all these papers?

35. Tabulation Spreadsheets

36. Topics we will cover….. Why evaluate? Proper Usage of data (IRB) Survey Templates Data Tabulation Templates Impact Statements Tools for creating Surveys The overly-busy 4-H Agent: Scene 7: Great! I love these simple tools! What should I do with all this great data?

37. Impact Statements Translate the numbers from your Data Tabulation Spreadsheet into Impact Statements. Spreadsheets for existing surveys include impact statements. Let’s look at some examples!

38. Impact Report Templates

39. Topics we will cover….. Why evaluate? Proper Usage of data (IRB) Survey Templates Data Tabulation Templates Impact Statements Tools for creating Surveys The overly-busy 4-H Agent: Final. Climactic Meeting with funders…

40. QUESTIONS??

41. To Access Templates, go to… www.georgia4h.org/evaluationresources www.georgia4h.org/evaluationresources For more information contact: University of Georgia, State 4-H Office Jeff Buckley, jbuckley@uga.edujbuckley@uga.edu Jennifer Cantwell, jecantw@uga.edujecantw@uga.edu