1. 1 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Surveys

2. 2 Survey A structured way to collect standardized information from individuals using a questionnaire. – Surveys may be conducted once; at repeated intervals, or concurrently with multiple samples – They may be used to collect information from a few or many

3. 3 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation http://learningstore.uwex.edu/pdf/G3658-10.PDF This set of slides draws on the booklet seen at left and other resources. We will cover: Types of surveys Pros and cons of surveys Steps in conducting a survey Response rate Cover letter

4. 4 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Checking in… What do you think? Answer YES or NO to each of the following 1.A survey is always appropriate 2.Surveys are one of the most popular ways to collect information 3.An email or online survey is better than the old mail or telephone surveys 4.Careful planning is necessary 5.Advance notice to potential respondents helps increase response rate 6.A low response rate increases the likelihood of biased results Check your answers

5. 5 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Check your answers 1.A survey is always appropriate - NO 2.Surveys are one of the most popular ways to collect information - YES 3.An email or online survey is better than the old mail or telephone surveys – NO, not necessarily 4.Careful planning is necessary - YES 5.Advance notice to potential respondents helps increase response rate - YES 6.A low response rate increases the likelihood of biased results - YES

6. 6 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Surveys are used when… You want to collect information from individuals (vs. a group or collective) You want standardized information from everyone Potential respondents can read and write You want information from many people Privacy is important or independent opinions and responses are needed You have resources to send, track, analyze and interpret the questionnaires

7. 7 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Pros and cons of survey PROS Way to collect information from many people; dispersed people Person can remain anonymous Provides standardized information across respondents Allows easy tabulation CONS Results can be easily biased Can miss important information – questions and answer choices are predetermined Requires literacy skills

8. 8 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Will using a survey be culturally appropriate? Things to consider: Literacy level Tradition of reading, writing Setting Not best choice for people with oral tradition Translation (demands more than just literal translation) How cultural traits affect response How to sequence the questions Pretesting the questionnaire when it may be viewed as intrusive Computer access and use if an electronic survey

9. 9 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Types of surveys 1.Hand-out 2.Mail 3.Telephone 4.Face-to-face 5.Email 6.Web survey – Online survey 7.Mixed mode: uses two or more of above Recommendation: use a mix of modes to ensure that everyone can and does respond

10. 10 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Which type of survey should I choose? It depends… upon –What you want to know – how complex or sensitive the information is –Who the respondents are – their characteristics and which type of survey will be most appropriate –Your time line, and –Available resources

11. 11 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation If youve determined that a survey is the best and most appropriate way to collect information, then take some time to plan your survey.

12. 12 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Planning a survey: aspects to think about 1.Determine who should be involved in conducting the survey - - engage them 2.Define what information you will collect –What do you want to know? –How will use the information? 3.Identify the respondents –Determine sampling strategy, if a sample is to be used 4.Select how the survey will be distributed: telephone, mail, hand-out, email, web-based 5.Think about data analysis – what will the end product/final report include (keep the end in mind!)

13. 13 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Survey planning continued… 6.Develop the questionnaire 7.Pilot test the questionnaire and other materials 8.Develop a communication strategy to garner support for the survey 9.Consider budget, timeline, and management process –What resources are available?

14. 14 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Implementing your survey For tips and examples for getting your survey done, see pages 9-15 in the Survey booklet See the Questionnaire Design section of this web site for help with developing your survey questionnaire. http://learningstore.uwex.edu/pdf/G3658-10.PDF

15. 15 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation A note on anonymity and confidentiality in surveys Anonymous means that NO ONE can identify who provided the information –This may be difficult to assure if there is a need to follow-up with non-respondents or when the survey is administered online (internet or intranet)…so, dont promise anonymity! Confidentiality means that you are able to identify the person but you guarantee that the information will not be identified with the person –This applies to all aspects of data collection, analysis and reporting –When reporting and communicating, ensure that no names or other identifying information is used

16. 16 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Response rate The proportion of people who respond: Example: If you distribute 50 questionnaires and you get 25 questionnaires back, your response rate is 50%. # that answered = response rate # you contacted

17. 17 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Response rate High response rate promotes confidence in results. Lower response rate increases the likelihood of biased results. See pages 15-17 in the survey booklet

18. 18 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation The higher, the better. Anything under 60% is a warning. Why is a high return important? Its the only way to know if your results are representative. There is no standard response rate

19. 19 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation If your response is low, address it! Determine how people who responded are different from those who didnt respond. Describe your results in terms of who did respond. Dont imply that the results apply to anyone other than those who responded.

20. 20 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation KEYS getting a high response rate The survey topic is of interest to the respondents (called saliency) Personalized request and communications related to the survey KISS: Keep It Short and Simple Follow-up Trust, respect, like the sponsor

21. 21 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Ways to increase response rate Generate positive publicity for your survey. Appeal to peoples helping tendencies – ask them to help. Make the topic salient - seem important –Ensure that respondents see the value of participating. –Point out personal connection to the topic Tailor, personalize communications Make the questionnaire interesting-short and easy to complete AND easy to return

22. 22 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation How to increase response rate, cont. Provide incentives Show positive regard; Say thank you Indicate that opportunities to respond are limited For mail survey, provide 1st class postage/return postage. Over sample Use a combination of survey modes – telephone plus mail Make (multiple) follow-up contacts – by mail, email, telephone, in person…

23. 23 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation If response rate is low… Use language that is suggestive rather than decisive. For example: The data suggests vs. The data prove; It appears vs. It shows Dont generalize findings to the entire group. Clearly describe who responded, i.e., who the data represents.

24. 24 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Cover Letters

25. 25 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation A good cover letter includes… The purpose of the survey and its importance Survey sponsor use letterhead if a mail survey Why the respondent was selected to participate How the information will be used Assurance of anonymity or confidentiality - Human Subjects Protection Instructions for returning the survey Date - when to respond by How results will be made available Your contact information

26. 26 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Cover letters Tips Personalize the letter in salutation or signature –Connect with the respondent Hand-sign the letter Express appreciation for their participation Keep it short – not wordy For mail survey, include pre-addressed, stamped return envelope

27. 27 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Cover letters – Pilot-test Remember to pilot -test your cover letter just as you pilot -Test your questionnaire! May be a colleague or collaborator or other stakeholder will check your cover letter for you. Is the cover letter likely to motivate the person to respond?

28. 28 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Checking back in… Spend a few minutes reflecting on what we covered in these slides 1.What, if anything, did you learn that you didnt know before? 2.What is one thing you will do when you develop a survey?

29. 29 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Resources Dillman, D., Smyth, J., Christian, LM. 2009. Internet, Mail and Mixed-Mode Surveys: The tailored design method. John Wiley and Sons. Fink, A. 2003. The Survey Kit. Sage Publications Scheuren, F. What is a survey. American Statistical Association. Free download at http://www.whatisasurvey.info/

30. 30 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Creative research systems. The survey system, at http://www.surveysystem.com/sdesign.htm University of WI-Extension, Program Development and Evaluation –Collecting Evaluation Data: Surveys –Questionnaire Design: Asking questions with a purpose –Sampling