Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU 515-4184

Similar presentations


Presentation on theme: "1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU 515-4184"— Presentation transcript:

1 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU

2 2 Overview We’ll cover What is a survey? Sampling Questionnaire development and design Analysis, interpretation, and reporting Using the results

3 3 What is a Survey? A scientific social research method that involves:  Selecting a random sample of people to answer some questions  Designing & administering a standardized questionnaire to get information about the research questions  Analyzing the results to provide descriptions about the people in the sample and find relationships between different responses  Generalizing the results to the population from which the sample was drawn

4 4 Basic survey formats  Telephone interview  Face-to-face interview  Computer assisted interview  Self-administered “paper & pencil” survey (scannable or not)  Self-administered Web survey  Self-administered survey

5 5 Why do a survey? Really bad answer: Why not? Anyone can do a survey. Slightly less bad answer: My boss told me to. Better answer: I want to systematically evaluate the extent to which my program is meeting its goals and objectives in order to improve our programs through pro-active, well-informed decision making.* *Adapted from CUPR FAQ “What is Assessment?” (http://www.ncsu.edu/provost/academic_programs/ FAQ/UAPRFAQwhatisassment.htm)http://www.ncsu.edu/provost/academic_programs/ FAQ/UAPRFAQwhatis

6 6 But why do a survey?  To collect information not available from other sources  To have comparable information  To be able to generalize results to the population in which you’re interested

7 7 Before you plan to do a your own survey… Think about alternative sources Information from non-survey sources Information from existing survey data

8 8 Still think you need to do a survey? Who is going to Identify the goals and objectives Develop the research design Design the questionnaire Identify the population and select the sample Pre-test the survey Administer the survey Analyze and interpret the results Write up the results Present the findings

9 9 Still think you need to do a survey?  How much TIME do you have ?  Who is going to PAY for it all ?  Who is going to USE the results ?

10 10 So you still want to do a survey … Helpful hints!  Familiarize yourself with survey research ‘best practices’ and code of ethics (see  (Get approval from your Institutional Review Board)  Form an advisory group w/ relevant constituents  Organize a survey team

11 11 So you still want to do a survey … More Helpful Hints! Develop a detailed overall research design What? Why? When? How? Who?

12 12 What is the purpose of the survey? Explicitly state your objectives Example: Enhance employee well-being

13 13 What is the purpose of the survey? Identify factors related to objectives Examples of factors related to “well-being”: Pay and benefits Physical work environment Work relationships Etc.

14 14 What is the purpose of the survey? Make an exhaustive list of everything you want to be able to measure related to your objective. Examples of dimensions of “work relationships”: Communication of necessary information Feedback on performance Opportunities to provide input in decision-making Etc.

15 15 Who do you need to survey? Identify your population(s) Population = The theoretically specified aggregation of all units about which information is collected. Examples for “work relationship” populations? Supervisors Employees Unit members Other?

16 16 Who do you need to survey? Survey population can influence Data collection methods you use Questions you ask Analyses you do Information you gather Assumptions you can make Recommendations you can make Action you can take Etc.

17 17 Who do you need to survey? All possible populations? (which technically would be a “census”) If you Must keep everyone happy Have unlimited time Have unlimited resources

18 18 Who do you need to survey? Everyone in a specific population? (which would still be a census) If you Have a relatively small population Have reason to expect a relatively low response rate Want to have enough respondents to do sub- group analyses for small groups

19 19 Who do you need to survey? A sample of the population? If you Understand sampling procedures Can convince others that it’s okay to sample Want/need to limit time in the field Want to enhance ability for quality control over entire process Want to keep costs down

20 20 Select the sample Create a “sampling frame” A list of elements from which the sample will be selected (i.e. members of the population) Examples? Company personnel data file Unit directory Etc.

21 21 Select the sample Probability sampling methods Simple random sample List elements in the sampling frame and use random number chart or computer program to select elements Systematic sampling List elements in the sampling frame and take every Nth element, based on how many needed in sample  Warning: Be attentive to any inherent patterns in how the sampling frame is ordered. Stratified sampling Draw appropriate proportion of elements from homogenous groups interested in

22 22 Select the sample Sample size? Depends on: Cost Anticipated response rate Number of responses needed per subgroup interested in Level of precision want (“margin of error”) See Appendix A for more information on sampling.

23 23 Select the sample Sample Warnings ! BAD SAMPLE = BAD DATA  Survey results are only representative of the sampling frame from which the sample was selected.  Sample survey results are generalizable only if respondents are randomly selected.

24 24 Select the sample A Warning About Sampling Error! It’s NOT the ONLY source of survey error!!!

25 25 Select the sample A Warning About Response Rates! A high response rate does NOT mean good data if those who responded are ‘biased’ in some way

26 26 Select the sample A Warning about Web Surveys! If you can not control who accesses and responds to a web survey the results are NOT generalizable to the population.

27 27 Design the questionnaire REMEMBER YOUR OBJECTIVES!  What do you want to learn?  How will you use the information?

28 28 Design the questionnaire Don’t reinvent the wheel!! Develop (or borrow*) questions/items to measure your specific outcomes *In general it is perfectly acceptable to borrow questions from other surveys, but always get permission and/or check for copyright restrictions. See Appendix B for examples of places to find questions.

29 29 Design the questionnaire To have confidence in your survey results your questions must be  Reliable Questions consistently convey the same meaning to all people in the population being surveyed.  Valid Questions measure what they are intended to measure

30 30 Design the questionnaire Warning! Bad Question = Bad Data The way a question is worded and the response options offered determine the nature of the data received.

31 31 Design the questionnaire Open-end questions  Pros Stimulate free thought Solicit suggestions Clarify positions Richer information  Cons Respondent burden Incomplete, irrelevant, uninterpretable responses Coding, analyzing, reporting

32 32 Design the questionnaire Suggestion for open-end questions! Limit the number of open-end questions BUT always include at least one for additional comments.

33 33 Design the questionnaire “Forced-choice” (closed-ended) questions  Pros Less demanding on respondent Determine levels of intensity, frequency of participation, etc. Use to form scales Easy to analyze and make comparisons  Cons Limits richness of data

34 34 Good Questions… 1) Are clear and use simple language 2) Are concise 3) Are specific 4) Are possible to answer 5) Don’t overly tax the respondent’s memory 6) Are not overly sensitive 7) Are relevant to the respondent 8) Do not use double negatives 9) Avoid biased terms 10) Have only 1 part (not “double-barrel”)

35 35 Good Questions… Most importantly… Can be connected to your objectives and Provide usable information

36 36 Good Questions… Good Questions: Use simple language... My supervisor gives me clear feedback on my job performance. Is better than : My supervisor provides lucid feedback when transmitting her thoughts regarding the feats I have accomplished in the preceding year.

37 37 Good Questions… Good Questions: Are concise... How interested would you be in a flexible work schedule in which you could work at home one day a week? Is better than: How interested would you be in a flexible work schedule that would still require you to work 40 hours per week, but on one weekday per week you could not physically come in to the office but work from home?

38 38 Good Questions… Good Questions: Are specific... In what year did you first start working in a permanent, full-time position at NC State University? Is better than: When did you start working here?

39 39 Good Questions… Good Questions: Are possible for the respondent to answer... NO: The overhead money I bring in on my grants has a direct impact on the pay raises provided by the state legislature.

40 40 Good Questions… Good Questions: Do not overly tax the respondent’s memory... NO: How many of your colleagues said hello to you within two weeks of you starting to work at NC State?

41 41 Good Questions… Good Questions: Are not overly sensitive… NO: How often do you take time out to pray during the course of a normal work day?

42 42 Good Questions… Good Questions: Are relevant... NO: My supervisor should get a better haircut.

43 43 Good Questions… Good Questions: Do not use double negatives... The lack of flexibility in my work hours makes it difficult for me to spend the amount of time I would like with my family. Is better than: The inflexibility in my work hours does not make it difficult for me to spend the amount of time I would like with my family.

44 44 Good Questions… Good Questions: Do not use biased terms or suggestions... To what extent do you agree or disagree that all employees need to be in the office during regular business hours? Is better than: Don’t you agree with the Chancellor when he says that all employees need to be in the office during regular business hours?

45 45 Good Questions… Good Questions: Do not ask two questions in one (“double- barrel”)... My supervisor is knowledgeable about performance review policies. My supervisor takes the time to make sure I understand performance review policies. Is better than: My supervisor is knowledgeable about performance review policies and takes the time to make sure I understand them as well.

46 46 Response Options Make sure your response options:  Reflect the concepts you are trying to measure  Fit with the question wording  Get at level of precision/intensity needed  Allow for comparisons between items  Use “undecided/don’t know’ sparingly  Are used consistently  Use balanced scales  Are mutually exclusive  Are exhaustive

47 47 Response Options Most importantly  Relate back to your objectives and  Correspond to how want to use the results

48 48 Response Options Must use  Balanced scales Example: NOT balanced... Q. How would your rate your supervisor’s leadership skills? Outstanding Excellent Very good Good

49 49 Response Options  Balanced scales continued Example: are balanced... Q. How would your rate your supervisor’s leadership skills? Excellent Good Fair Poor

50 50 Response Options Must be:  Mutually exclusive can select only one appropriate answer Example: NOT mutually exclusive... Q. On most days, what time do you leave your house for work? 6:00 AM or earlier 7:30-8:00 AM 6:00-6:30 AM 8:00-8:30 AM 6:30-7:00 AM 8:30-9:00 AM 7:00-7:30 AM 9:00 AM or later

51 51 Response Options  Mutually exclusive continued Example: IS mutually exclusive... Q. On most days, what time do you leave your house for work? Before 6:00 AM 7:30-7:59 AM 6:00-6:29 AM 8:00-8:29 AM 6:30-6:59 AM 8:30-8:59 AM 7:00-7:29 AM 9:00 AM or later

52 52 Response Options Must be:  Exhaustive all possible answers are listed (including e.g., “other, “ “don’t know,” etc.) Example: NOT exhaustive Q. On most days, what time do you leave your house for work? 6:00-6:29 AM 7:30-7:59 AM 6:30-6:59 AM 8:00-8:29 AM 7:00-7:29 AM 8:30-8:59 AM

53 53 Response Options  Exhaustive continued Example: IS exhaustive Q. On most days, what time do you leave your house for work? Before 6:00 AM 7:30-7:59 AM 6:00-6:29 AM 8:00-8:29 AM 6:30-6:59 AM 8:30-8:59 AM 7:00-7:29 AM 9:00 AM or later

54 54 Response Options Some common scales:  Excellent, Good, Fair, Poor  Strongly agree, Agree, Disagree, Strongly disagree  Very useful, Somewhat useful, Not very useful, Not at all useful  Definitely, Probably, Probably not, Definitely Not  All of the time, Most of the time, Some of the time, Seldom, Never  Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree

55 55 Questionnaire format  As short as possible  Spread out & uncluttered  Attractive & professional looking  Broken into logical sections  Have a good flow  Uncomplicated & easy to follow  Include simple instructions for answering questions  Should have clear skip patterns for contingency questions

56 56 Questionnaire format Question order  Start with interesting, easy, non-threatening questions Remember – ‘Question order effects’ Earlier questions provide information and context that influences responses to later questions

57 57 Questionnaire Suggestion! Before you start collecting data use the questionnaire to  Prepare an analysis plan  Outline the final report

58 58 Pre-Test A pre-test is VERY VERY important! Test for: Question clarity Questionnaire format Variance in responses Etc.

59 59 Data collection Stages (in a perfect world…) 1. Preliminary announcement 2. Cover letter, questionnaire w/ return envelope and/or survey URL 3. Reminder (with copy of questionnaire and return envelope and/or survey URL) 4. 2 nd reminder (with contact info to request copy of questionnaire and/or URL) 5. Thank you card/page

60 60 Cover or “welcome” letter Include with any self-administered survey (On letterhead) “Signed” by most important person possible Briefly explain purpose of survey how results will be used why respondent was selected why participation is important (how to access questionnaire)

61 61 Cover or “welcome” letter Continued Emphasize confidentiality (if applicable) (Describe/explain incentives) (Follow IRB guidelines) Provide contact information THANK respondent for participating

62 62 Cover or “welcome” letter Warnings about Announcements  Be sure to follow any mass policies  Think about spam filters: Subject line text? ‘From’ field?  Try to have identifiable name/title in ‘from’ field  Keep track of undelivered/unopened s

63 63 A Word about Web Surveys…. LOTS of software to choose from Consider: Extent of involvement/control over administration Flexibility of survey design Population/sampling options Data collection capabilities non-response follow-ups Data management and analysis options Reporting capabilities Cost

64 64 A Word about Web Surveys…. Warning! Even if using web survey software you still need the skills to design and administer a good, useful survey and to analyze the results!!!

65 65 Clean, analyze, & interpret the data Get input from your advisory committee Remember your objectives Remember your report outline Remember your audience  Keep it simple…  You can always do more analyses…

66 66 Interpreting the results Everything is relative! Suggestions for what to look for: Do responses/ratings meet pre-defined goals? (e.g., % giving a positive rating) How do responses/ratings for individual items compare to each other? What factors/characteristics are related to ratings? How do responses/ratings from one group of respondents compare to another? (How have ratings changed over time?)

67 67 Prepare written report(s)  Get input from your advisory committee  Think about the intended audience What do they need to know? How they are going to use the report? How they like to get information…

68 68 Prepare written report(s) Generally include  What you did (methods)  Why you did it  What you learned  How it can be used

69 69 Using the results Take action: Some suggestions…  Spread the word!!  Institute policies/changes to address problem areas identified by ratings that were lower than predefined set goals.  Gear policies towards specific groups as needed.  Do follow-up research

70 70 Want to learn more about surveys? American Association of Public Opinion Research, “Best Practices for Survey and Public Opinion Research.” Babbie, Earl (2002), The Basics of Social Research 2nd ed. Wadsworth Thomson Learning: CA. Dillman, D.A. (2000), Mail and Internet Surveys 2nd ed. Wiley: NY. Fink, Arlene and Jacqueline Kosecoff (1998), How to Conduct Surveys. Sage: CA. Fowler, Floyd J. (1993), Survey Research Methods. Sage: CA. Fowler, Floyd J. (1995), Improving Survey Questions. Sage: CA. National Council on Public Polls, “20 Questions a Journalist Should Ask About Poll Results.” Sudman, Seymour and Bradburn, Norman (1982), A Practical Guide to Questionnaire Design. Jossey-Bass: San Francisco. Tourangeau, R., and Smith, T.W. (1996), “Asking Senstive Questions: The Impact of Data Collection Mode, Question Format, and Question Context, “ Public Opinion Quarterly, 60:

71 71 Appendix A Response Rate # surveys completed and returned net sample size* *net sample size = sample size - undeliverable surveys

72 72 Appendix A Margin of Error Based on # people in the sample (or population) # people who respond the response distribution e.g., 75% said yes, 25% said no how sure you want to be of you data (“confidence interval”) e.g., a 95% confidence interval means that you can be sure that 95 out of 100 times the responses to a given question would be within some calculable percentage points of the actual number you got.

73 73 Appendix A Margin of Error Examples (at 95% confidence interval) Population Size # Respondents Margin of Error E.G. If 50% of the 7 respondents (from your population of 10) said “yes,” you can be sure that 95 out of 100 times between 38.9% and 61.1% of those in your population would say “yes.”

74 74 Appendix B Survey Question ‘Banks’  Eagleton Poll Archives  The Roper Center (fee based)  The Odum Institute  The Gallup Poll  ICPSR


Download ppt "1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU 515-4184"

Similar presentations


Ads by Google