Presentation on theme: "SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research"— Presentation transcript:
1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and AnalysisNCSU
2 Overview We’ll cover What is a survey? Sampling Questionnaire development and designAnalysis, interpretation, and reportingUsing the resultsEmphasis throughout =Most useful data - - -1) keep focused on OBJECTIVES2) REDUCING ERROR
3 What is a Survey? A scientific social research method that involves: Selecting a random sample of people to answer some questionsDesigning & administering a standardized questionnaire to get information about the research questionsAnalyzing the results to provide descriptions about the people in the sample and find relationships between different responsesGeneralizing the results to the population from which the sample was drawnQUESTION: WHO’S INVOLVED IN SURVEYS (EITHER COMMERCIAL OR HOMEGROWN)?
4 Basic survey formats Telephone interview Face-to-face interview Computer assisted interviewSelf-administered “paper & pencil” survey (scannable or not)Self-administered Web surveySelf-administered surveyFormat determines much of research design (e.g., sampling, questionnaire design, administration, data analysis, reporting, etc.)Each has implications for cost, timing, etc.Today’s talk applicable to most methods…
5 Why do a survey? Really bad answer: Slightly less bad answer: Why not? Anyone can do a survey.Slightly less bad answer:My boss told me to.Better answer:I want to systematically evaluate the extent to which my program is meeting its goals and objectives in order to improve our programs through pro-active, well-informed decision making.**Adapted from CUPR FAQ “What is Assessment?” ( FAQ/UAPRFAQwhatisassment.htm)
6 But why do a survey?To collect information not available from other sourcesTo have comparable informationTo be able to generalize results to the population in which you’re interestedInfo not available = Attitudes and opinions, Values, Experiences and behaviors, Expectations, Needs,Demographics, Descriptive informationComparable info = standardized instrument allows for comparisons between groups, between surveys, benchmarking, etcGeneralize = w/ random sample it’s not just people speaking up
7 Before you plan to do a your own survey… Think about alternative sourcesInformation from non-survey sourcesInformation from existing survey dataNon-Survey sources = At col/univ IR office has access to lots of data - - maybe don’t need ‘opinion’ infoExisting surveys = Other surveys done on similar topic. May not be perfect but is close enough
8 Still think you need to do a survey? Who is going toIdentify the goals and objectivesDevelop the research designDesign the questionnaireIdentify the population and select the samplePre-test the surveyAdminister the surveyAnalyze and interpret the resultsWrite up the resultsPresent the findingsTakes a lot of skilled help - -
9 Still think you need to do a survey? How much TIME do you have ?Who is going to PAY for it all ?Who is going to USE the results ?Time: Common mistake to underestimate amount of time. Time up front in planning stage is especially important.(story about staff survey - - want to do survey NOW)2) Money: Surveys are expensive. Lots of labor costs, but also, in some cases, material costs (e.g., stamps, envelopes, scannable forms, etc.)3) Usefulness: Waste of time if not going to be used…
10 So you still want to do a survey … Helpful hints!Familiarize yourself with survey research ‘best practices’ and code of ethics (see(Get approval from your Institutional Review Board)Form an advisory group w/ relevant constituentsOrganize a survey teamAdvisory group:Keep it manageable in sizeGet regular input on objectives, population, data analysis plan, reporting expectations, uses of results, etc.Survey teamInclude all those working on any part of the survey projectGet input from all team members at each step along the way
11 So you still want to do a survey … More Helpful Hints!Develop a detailed overall research designWhat?Why?When?How?Who?Each decision has implications for other steps!
12 What is the purpose of the survey? Explicitly state your objectivesExample:Enhance employee well-beingOkay to include multiple objectives
13 What is the purpose of the survey? Identify factors related to objectivesExamples of factors related to “well-being”:Pay and benefitsPhysical work environmentWork relationshipsEtc.ThroughLiterature reviewLocal empirical evidence (IR data, surveys, etc.)Anecdotal evidence
14 What is the purpose of the survey? Make an exhaustive list of everything you want to be able to measure related to your objective.Examples of dimensions of “work relationships”:Communication of necessary informationFeedback on performanceOpportunities to provide input in decision-makingEtc.
15 Who do you need to survey? Identify your population(s)Population = The theoretically specified aggregation of all units about which information is collected.Examples for “work relationship” populations?SupervisorsEmployeesUnit membersOther?
16 Who do you need to survey? Survey population can influenceData collection methods you useQuestions you askAnalyses you doInformation you gatherAssumptions you can makeRecommendations you can makeAction you can takeEtc.
17 Who do you need to survey? All possible populations?(which technically would be a “census”)If youMust keep everyone happyHave unlimited timeHave unlimited resources
18 Who do you need to survey? Everyone in a specific population?(which would still be a census)If youHave a relatively small populationHave reason to expect a relatively low response rateWant to have enough respondents to do sub-group analyses for small groups
19 Who do you need to survey? A sample of the population?If youUnderstand sampling proceduresCan convince others that it’s okay to sampleWant/need to limit time in the fieldWant to enhance ability for quality control over entire processWant to keep costs down
20 Select the sample Create a “sampling frame” A list of elements from which the sample will be selected (i.e. members of the population)Examples?Company personnel data fileUnit directoryEtc.
21 Select the sample Probability sampling methods Simple random sample List elements in the sampling frame and use random number chart or computer program to select elementsSystematic samplingList elements in the sampling frame and take every Nth element, based on how many needed in sampleWarning: Be attentive to any inherent patterns in how the sampling frame is ordered.Stratified samplingDraw appropriate proportion of elements from homogenous groups interested in
22 Select the sample Sample size? Depends on: Cost Anticipated response rateNumber of responses needed per subgroup interested inLevel of precision want (“margin of error”)See Appendix A for more information on sampling.
23 Select the sample Sample Warnings! BAD SAMPLE = BAD DATA Survey results are only representative of the sampling frame from which the sample was selected.Sample survey results are generalizable only if respondents are randomly selected.Problem = sampling frame is rarely perfect…Out of dateIncompleteEtc.“Probability sample” = Everyone in sampling frame has an equal chance of being selected.
24 A Warning About Sampling Error! It’s NOT the ONLY source of Select the sampleA Warning About Sampling Error!It’s NOT the ONLY source ofsurvey error!!!Ie. Margin of error….
25 A Warning About Response Rates! are ‘biased’ in some way Select the sampleA Warning About Response Rates!A high response ratedoes NOT mean good dataif those who respondedare ‘biased’ in some way
26 Select the sample A Warning about Web Surveys! If you can not control whoaccesses and responds to a web surveythe results are NOTgeneralizable to the population.
27 Design the questionnaire REMEMBER YOUR OBJECTIVES!What do you want to learn?How will you use the information?
28 Design the questionnaire Don’t reinvent the wheel!!Develop (or borrow*) questions/items to measure your specific outcomes*In general it is perfectly acceptable to borrow questions from other surveys, but always get permission and/or check for copyright restrictions. See Appendix B for examples of places to find questions.
29 Design the questionnaire To have confidence in your survey results your questions must beReliableQuestions consistently convey the same meaning to all people in the population being surveyed.ValidQuestions measure what they are intended to measure
30 Design the questionnaire Warning!Bad Question = Bad DataThe way a question is wordedand the response options offereddetermine the nature of the data received.
32 Design the questionnaire Suggestion foropen-end questions!Limit the number of open-end questionsBUT always includeat least onefor additional comments.
33 Design the questionnaire “Forced-choice” (closed-ended) questionsProsLess demanding on respondentDetermine levels of intensity, frequency of participation, etc.Use to form scalesEasy to analyze and make comparisonsConsLimits richness of data
34 Good Questions… Are clear and use simple language Are concise Are specificAre possible to answerDon’t overly tax the respondent’s memoryAre not overly sensitiveAre relevant to the respondentDo not use double negativesAvoid biased termsHave only 1 part (not “double-barrel”)
35 Good Questions… Most importantly… Can be connected to your objectives andProvide usable information
36 Good Questions… Good Questions: Use simple language... My supervisor gives me clear feedback on my job performance.Is better than :My supervisor provides lucid feedback when transmitting her thoughts regarding the feats I have accomplished in the preceding year.
37 Good Questions… Good Questions: Are concise... How interested would you be in a flexible work schedule in which you could work at home one day a week?Is better than:How interested would you be in a flexible work schedule that would still require you to work 40 hours per week, but on one weekday per week you could not physically come in to the office but work from home?
38 Good Questions… Good Questions: Are specific... In what year did you first start working in a permanent, full-time position at NC State University?Is better than:When did you start working here?
39 Good Questions… Good Questions: Are possible for the respondent to answer...NO:The overhead money I bring in on my grants has a direct impact on the pay raises provided by the state legislature.
40 Good Questions… Good Questions: Do not overly tax the respondent’s memory...NO:How many of your colleagues said hello to you within two weeks of you starting to work at NC State?
41 Good Questions… Good Questions: Are not overly sensitive… NO: How often do you take time out to pray during the course of a normal work day?
42 Good Questions… Good Questions: Are relevant... NO: My supervisor should get a better haircut.
43 Good Questions… Good Questions: Do not use double negatives... The lack of flexibility in my work hours makes it difficult for me to spend the amount of time I would like with my family.Is better than:The inflexibility in my work hours does not make it difficult for me to spend the amount of time I would like with my family.
44 Good Questions… Good Questions: Do not use biased terms or suggestions...To what extent do you agree or disagree that all employees need to be in the office during regular business hours?Is better than:Don’t you agree with the Chancellor when he says that all employees need to be in the office during regular business hours?
45 Good Questions… Good Questions: Do not ask two questions in one (“double- barrel”)...My supervisor is knowledgeable about performance review policies.My supervisor takes the time to make sure I understand performance review policies.Is better than:My supervisor is knowledgeable about performance review policies and takes the time to make sure I understand them as well.
46 Response Options Make sure your response options: Reflect the concepts you are trying to measureFit with the question wordingGet at level of precision/intensity neededAllow for comparisons between itemsUse “undecided/don’t know’ sparinglyAre used consistentlyUse balanced scalesAre mutually exclusiveAre exhaustive
47 Response Options Most importantly Relate back to your objectives and Correspond to how want to use the results
48 Response Options Balanced scales Must use Example: NOT balanced... Q. How would your rate your supervisor’s leadership skills?OutstandingExcellentVery goodGood
49 Response Options Balanced scales continued Example: are balanced... Q. How would your rate your supervisor’s leadership skills?ExcellentGoodFairPoor
50 Response Options Mutually exclusive Must be: can select only one appropriate answerExample: NOT mutually exclusive...Q. On most days, what time do you leave your house for work?6:00 AM or earlier :30-8:00 AM6:00-6:30 AM :00-8:30 AM6:30-7:00 AM :30-9:00 AM7:00-7:30 AM :00 AM or later
51 Response Options Mutually exclusive continued Example: IS mutually exclusive...Q. On most days, what time do you leave your house for work?Before 6:00 AM :30-7:59 AM6:00-6:29 AM :00-8:29 AM6:30-6:59 AM :30-8:59 AM7:00-7:29 AM :00 AM or later
52 Response Options Exhaustive Must be: all possible answers are listed (including e.g., “other, “ “don’t know,” etc.)Example: NOT exhaustiveQ. On most days, what time do you leave your house for work?6:00-6:29 AM :30-7:59 AM6:30-6:59 AM :00-8:29 AM7:00-7:29 AM :30-8:59 AM
53 Response Options Exhaustive continued Example: IS exhaustive Q. On most days, what time do you leave your house for work?Before 6:00 AM :30-7:59 AM6:00-6:29 AM :00-8:29 AM6:30-6:59 AM :30-8:59 AM7:00-7:29 AM :00 AM or later
54 Response Options Some common scales: Excellent, Good, Fair, Poor Strongly agree, Agree, Disagree, Strongly disagreeVery useful, Somewhat useful, Not very useful, Not at all usefulDefinitely, Probably, Probably not, Definitely NotAll of the time, Most of the time, Some of the time, Seldom, NeverStrongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree
55 Questionnaire format As short as possible Spread out & uncluttered Attractive & professional lookingBroken into logical sectionsHave a good flowUncomplicated & easy to followInclude simple instructions for answering questionsShould have clear skip patterns for contingency questionsLength – keep objectives in mind. Don’t ask for info won’t use. Don’t ask for info can get elsewhere.Uncluttered – better to have more pages than cluttered pagesAttractive – makes a huge differenceLogical sections/flow – easier for respondentEasy to follow – have clear skip patterns (use arrows, shading, etc)
56 Questionnaire format Question order Remember – Start with interesting, easy, non-threatening questionsRemember –‘Question order effects’Earlier questions provide information and context that influences responses to later questions
57 Prepare an analysis plan QuestionnaireSuggestion!Before you start collecting datause the questionnaire toPrepare an analysis planOutline the final report
58 Pre-Test A pre-test is VERY VERY important! Test for: Question clarity Questionnaire formatVariance in responsesEtc.In a perfect world, pre-test your sample design, data collection methods, data processing, and data analysis if possible in addition to questionnaire and cover letter.Use respondents similar to those who will be in survey sampleQuestion clarity: failure to answer? multiple answers? “other” answers? qualified answers? understanding of content?Questionnaire format: instructions? flow? layout? Variance in responses (do most respondents give the same answer to a question?)
59 Data collection Stages (in a perfect world…) Preliminary announcement Cover letter, questionnaire w/ return envelope and/or survey URLReminder (with copy of questionnaire and return envelope and/or survey URL)2nd reminder (with contact info to request copy of questionnaire and/or URL)Thank you card/page
60 Cover or “welcome” letter Include with any self-administered survey(On letterhead)“Signed” by most important person possibleBriefly explainpurpose of surveyhow results will be usedwhy respondent was selectedwhy participation is important(how to access questionnaire)Goal = create trust and influence respondent’s expectations for rewards and costsInvoke exchange relationship
61 Cover or “welcome” letter ContinuedEmphasize confidentiality (if applicable)(Describe/explain incentives)(Follow IRB guidelines)Provide contact informationTHANK respondent for participatingIncentives are good!Confidentiality is criticalConfidentiality is NOT the same as anonymous
62 Cover or “welcome” letter Warnings aboutAnnouncementsBe sure to follow any mass policiesThink about spam filters:Subject line text?‘From’ field?Try to have identifiable name/title in ‘from’ fieldKeep track of undelivered/unopened sIncentives are good!Confidentiality is criticalConfidentiality is NOT the same as anonymous
63 A Word about Web Surveys…. LOTS of software to choose fromConsider:Extent of involvement/control over administrationFlexibility of survey designPopulation/sampling optionsData collection capabilities non-response follow-upsData management and analysis optionsReporting capabilitiesCostCommon mistake to underestimate amount of time. Time up front in planning stage is especially important.(story about staff survey - - want to do survey NOW)2) Surveys are expensive. Lots of labor costs, but also, in some cases, material costs (e.g., stamps, envelopes, scannable forms, etc.)
64 A Word about Web Surveys…. Warning!Even if using web survey software you still need the skills to design and administer a good, useful survey and to analyze the results!!!Common mistake to underestimate amount of time. Time up front in planning stage is especially important.(story about staff survey - - want to do survey NOW)2) Surveys are expensive. Lots of labor costs, but also, in some cases, material costs (e.g., stamps, envelopes, scannable forms, etc.)
65 Clean, analyze, & interpret the data Get input from your advisory committeeRemember your objectivesRemember your report outlineRemember your audienceKeep it simple…You can always do more analyses…
66 Interpreting the results Everything is relative!Suggestions for what to look for:Do responses/ratings meet pre-defined goals?(e.g., % giving a positive rating)How do responses/ratings for individual items compare to each other?What factors/characteristics are related to ratings?How do responses/ratings from one group of respondents compare to another?(How have ratings changed over time?)
67 Prepare written report(s) Get input from your advisory committeeThink about the intended audienceWhat do they need to know?How they are going to use the report?How they like to get information…How like to get info: Detailed tables? Simple descriptive report? Top line bullets?
68 Prepare written report(s) Generally includeWhat you did (methods)Why you did itWhat you learnedHow it can be used
69 Using the results Take action: Some suggestions… Spread the word!! Institute policies/changes to address problem areas identified by ratings that were lower than predefined set goals.Gear policies towards specific groups as needed.Do follow-up researchDo follow-up research(survey, focus group, interviews…)With a population not included in the surveyWith a different sample of the same population (trend survey)With the same respondents (panel survey)
70 Want to learn more about surveys? American Association of Public Opinion Research, “Best Practices for Survey and Public Opinion Research.”Babbie, Earl (2002), The Basics of Social Research 2nd ed. Wadsworth Thomson Learning: CA.Dillman, D.A. (2000), Mail and Internet Surveys 2nd ed. Wiley: NY.Fink, Arlene and Jacqueline Kosecoff (1998), How to Conduct Surveys. Sage: CA.Fowler, Floyd J. (1993), Survey Research Methods. Sage: CA.Fowler, Floyd J. (1995), Improving Survey Questions. Sage: CA.National Council on Public Polls, “20 Questions a Journalist Should Ask About Poll Results.”Sudman, Seymour and Bradburn, Norman (1982), A Practical Guide to Questionnaire Design. Jossey-Bass: San Francisco.Tourangeau, R., and Smith, T.W. (1996), “Asking Senstive Questions: The Impact of Data Collection Mode, Question Format, and Question Context, “ Public Opinion Quarterly, 60:
71 Appendix A Response Rate # surveys completed and returned net sample size**net sample size = sample size - undeliverable surveys
72 Appendix A Margin of Error Based on # people in the sample (or population)# people who respondthe response distributione.g., 75% said yes, 25% said nohow sure you want to be of you data (“confidence interval”)e.g., a 95% confidence interval means that you can be sure that 95 out of 100 times the responses to a given question would be within some calculable percentage points of the actual number you got.
73 Appendix A Margin of Error Examples (at 95% confidence interval) Population Size # Respondents Margin of ErrorE.G. If 50% of the 7 respondents (from your population of 10) said “yes,” you can be sure that 95 out of 100 times between 38.9% and 61.1% of those in your population would say “yes.”
74 Appendix B Survey Question ‘Banks’ Eagleton Poll Archives The Roper Center (fee based)The Odum InstituteThe Gallup PollICPSR