1. Tips for creating good surveys
Alan Howard
Academic Computing Services
The University of Vermont
August 2007

2. You might be a statistician if:
No one wants your job
You are right 95% of the time
You found accounting too exciting
You never have to say you are certain

3. Types of surveys – kinds of information
Most surveys attempt to obtain one or more kinds of information:
Knowledge – what people know
Behavior – what people do
Attitude – what people say they want
Opinion/Belief – what people think is true
Attribute – what people are/have

4. Four stages of conducting a survey
Stage one – Planning
Define survey purpose/mission
Outline research objectives/statistical methods
Define population
Determine sample size
Select sampling technique
Select survey method
Determine timetable
Establish budget

5. Four stages of conducting a survey
Stage two – Questionnaire design
Determine survey length
Design and format questions
Design response choices
Design questionnaire layout
Pre-test questionnaire
Redesign as appropriate based on pre-test results

6. Four stages of conducting a survey
Stage three – Data collection
Pre-notify those in sample to expect survey
Use incentives if appropriate
Code data for easier data entry
Decide on “best” data entry method
Use CATI (computer-aided telephone interview) or CAPI (computer-aided personal interview) software if available
Follow-up with non-respondents

7. Four stages of conducting a survey
Stage four – Data analysis and reporting
Use unique ID variable
Perform simple descriptive analyses to catch data entry errors
Use statistical methods appropriate for the type of data being collected and analyzed

8. Steps involved in obtaining a sample
Identify population of interest
Group
Geographic area
Time period
Obtain sampling frame
Accurate
Complete and current
No duplicates
No pattern
Determine sample size
Select sampling technique

9. For large populations (most commonly used equation):
Determine sample size
For large populations (most commonly used equation):
n = Z2pq / e2 ,
where Z = a/2 percentage point of standard normal
p = estimated proportion of desired attribute in the population
q = 1 – p
e = desired level of precision (margin of error)
Example:
We want 95% confidence that p = 0.5 with ± 5% precision.
n = (1.96)2 * 0.5 * 0.5 / = 385

10. Determine sample size
Proportion of 0.5 is almost always used because it gives the most conservative estimate of sample size:
Assume p = 0.25; use equation from previous slide:
n = Z2pq / e2
n = (1.96)2 * 0.25 * 0.75 / = 289
Finite population correction:
n’ = n / (1 + ((n-1)/N)), where n = sample size from above and N = population size.
Table on next slide uses:
n = N / (1 + N(e2)), where N = population size.

11. Determine sample size Table of sample sizes
Source: Document PEOD6, Agricultural Education and Communication Department, Florida Cooperative Extension Service, Institute of Food and Agricultural Sciences, University of Florida, 1992.

12. Sample size calculator:
Determine sample size
Sample size calculator:
Handy utilities:
Russ Lenth’s Power and Sample Size Page

13. Select sampling technique
Non-probability samples
Convenience
Purposive
Quota
Snowball
Probability samples – allow generalization to population of interest
Simple random sampling
Systematic sampling
Stratified sampling

14. Tangent 1: A famous photograph from 20th century politics

15. Tangent 1: A famous photograph from 20th century politics
Myth
Telephone polling over-sampled wealthier Republicans who favored Dewey
Widely perpetuated, even on Wikipedia

16. Tangent 1: A famous photograph from 20th century politics
Truth
Sampling methods were flawed: used quota sampling
There was a large proportion of “undecided” voters who made up their minds late
No polling was done 2 – 3 weeks before election
Source: The Pre-Election Polls of 1948, Mosteller, Frederick, et al., Social Science Research Council, New York, 1949

17. Internet (e-mail or web-based) Face-to-face (in person interview)
Select survey method
Mail
Telephone
Internet ( or web-based)
Face-to-face (in person interview)
Hand-out

18. Advantages Mail Surveys Low cost Can have wide geographic distribution
Can use visual aids
Allows respondent more time to complete (and can consult personal records)
Questions may be of moderate complexity
May get more truthful responses to sensitive topics

19. Disadvantages Mail Surveys Questionnaire cannot be complex
No control over order of questions
Often poor response to open-ended questions
No control over response situation
Little control over quality of recorded responses

20. Advantages Telephone Surveys Low/medium cost
Can have wide geographic distribution
Questionnaire may be complex
Questionnaire may be relatively long
Can establish rapport with respondent
Good control over order of questions
Good response rate
Low response bias
High quality of recorded responses

21. Disadvantages Telephone Surveys Questions cannot be complex
Use of visual aids not possible
Difficult for respondent to consult personal records
Responses to sensitive topics may be poor
Difficult to get information regarding non-response

22. Advantages Internet Surveys Very low cost
Can have wide geographic distribution
Can use visual aids
Allows respondent more time to complete (and can consult personal records)
Questionnaire may be of moderate complexity
Questions may be of moderate complexity
May get more truthful responses to sensitive topics

23. Disadvantages Internet Surveys Sampling frame bias can be high
Response bias can be high
Response rate can be poor
No control over response situation
Little control over quality of recorded responses

24. Advantages Face-to-face Surveys Questionnaire can be long and complex
Questions can be complex
Good control of question order
Can use visual aids
Allows respondent more time to complete (and can consult personal records)
Can develop rapport with respondent
Good response rate
Good control of response situation
Good control of quality of recorded responses

25. Disadvantages Face-to-face Surveys High cost
Geographic distribution usually limited
Data collection period usually very long
Responses to sensitive topics may be poor

26. Questionnaire design Determine survey length
Shorter surveys usually have better response rate
Decide which information is absolutely essential
Balance length and information desired
Don’t ask questions that “would be nice to know”
Don’t ask questions to convey information
Minimize the need to skip over questions or branch to another section

27. Design and format questions
KISS (keep it simple, …)
Questions should be clear and concise
Don’t make assumptions about respondents
Don’t ask a question everyone will likely answer the same way
Avoid asking for responses that are too precise
Avoid “double-barreled” questions
Avoid double negatives

28. Design and format questions
Avoid jargon, acronyms, abbreviations, etc.
Avoid injecting your personal biases into questions
Avoid “leading” questions
Avoid “loaded” questions
Be specific (avoid words like “few”, “usually”, “older people”, “often”)
Avoid starting questions with responses
Avoid antagonizing or embarrassing respondents

29. Tangent 2: What is wrong with each of these questions?
How many children do you have?
Making an assumption that respondent has children
The instructor of this course was well-prepared and thorough.
Double-barreled statement
How many hours of TV did you watch over the last 30 days?
Asking for too precise a response
This course was not appropriate for students who are not Art majors.
Double negative

30. Tangent 2: What is wrong with each of these questions?
Have you ever taken illegal drugs?
May antagonize or embarrass respondent
Congress has too many partisan extremists and too few who are willing to compromise.
Double-barreled, “loaded” and not specific (how many is too many or too few?)
Do you always, often, sometimes, or never eat out at McDonalds?
Question starts off with list of responses
Those convicted of sexually molesting children should not be school teachers.
Not likely to get anyone to disagree

31. Tangent 2: What is wrong with each of these questions?
Are you aware that Federal and state funding for these programs has not kept pace with inflation and increased expenses?
Leading question designed to convey information rather than obtain information
Do you think the President did a good job handling the current crisis?
Not specific (“good” can be defined differently by different people) and “current crisis” is somewhat loaded and not specific
The wind farm project, if approved, will be an environmental abomination and destroy the view of the mountain ridge line.
Statement is biased, double-barreled, “leading” and “loaded”

32. Design response choices
Closed-ended responses
Single selection from list of possible responses (gender, Likert scale, “choose one”)
Multiple selection from list of possible responses (“choose all that apply”)
Lists should be exhaustive and mutually exclusive
Lists should be organized logically to avoid confusion
If you include “Other” as a choice, also include space to “Please specify”
Open-ended responses (unstructured clear text comments, opinions)
Easiest to ask, hardest to analyze
Use to probe for more information or to solicit for comments, suggestions, etc.

33. Design response choices
Topics for discussion:
Should you provide a “neutral” response for Likert scale questions?
Should you provide a “Don’t know” response choice?
Should you provide a “Not applicable” response choice?

34. Design response choices
Related to “measurement level” of data
Binary response (dead/alive, yes/no)
Categorical response (race, nationality)
Ordered categorical response (Likert scale, income level, education level)
Continuous scale (age, height, weight)
Multiple responses (“choose all that apply”) are binary responses to each choice

35. Tangent 3: What is wrong with these response choices?
How many acres of forest do you own?
1000 or more acres
999 to 500 acres
499 to 100 acres
99 to 10 acres
9 to 1 acres
0 acres
Not logically arranged

36. Tangent 3: What is wrong with these response choices?
Please indicate your political party affiliation:
Democratic
Republican
List is neither exhaustive nor does it include an “Other” category.

37. Tangent 3: What is wrong with these response choices?
Please indicate your religious affiliation:
Christian
Jewish
Islamic
Catholic
None
Other (please specify) ________________
List is not mutually exclusive

38. Tangent 3: What is wrong with these response choices?
How much RAM is installed on your computer’s motherboard?
256 MB or less
More than 256 MB but less than 512 MB
More than 512 MB but less than 1 GB
More than 1 GB but less than 2 GB
More than 2 GB
Don’t know
Question and responses use acronyms and jargon

39. Tangent 3: What is wrong with these response choices?
Little difference between “Frequently” and “Very often”. “Always” would be a better choice for 4.

40. Design questionnaire layout
Introduction
State purpose
Explain how information will be used
Assure confidentiality or anonymity
First questions
Non-controversial
Easy to answer
Pertinent
Arrange questions logically in groups
Keep question and answer or list of answers all on same page

41. Design questionnaire layout
Keep same codes for similar responses throughout
Use no more than two easy-to-read fonts
Use color, if possible, but no more than two
Use plenty of white space
Make it easy to distinguish question from answer by using bolding, italics, etc.

42. Design questionnaire layout
Put demographic questions at end
For open-ended questions, don’t supply lines, just white space
Use transitional statements to help flow from one section or group of questions to another
Include instructions on how to answer at the beginning of each group
For telephone surveys, script out the entire dialogue, from beginning to end

43. Pre-test questionnaire
Also known as “pilot study”
Use 20 to 50 respondents in pre-test sample
Pre-test sample should have similar characteristics as population
Ask pre-test sample to be critical and point out problems or confusing questions
May need to do a second pre-test if first one identifies a lot of problems

44. Pre-test questionnaire
Pre-testing can help identify problems:
Long, complicated questions
Duplicate or redundant questions
Too many questions left unanswered
Confusing or cluttered layout
Phone or personal interviewers have trouble reading questions or recording responses

45. Pre-test questionnaire
Pre-testing can help identify problems:
Survey takes longer than anticipated
Not enough white space for open-ended questions
Little or no variation in responses to a question
Too many “Don’t know” responses
Too many “Other” responses
Confusing skip patterns or branching

46. Pre-test questionnaire
Pre-testing can help estimate:
Response rate
Time needed between follow-up contacts
Differences in responses among subgroups
Whether an open-ended question can be changed to closed-ended

47. Data collection Decide how to code data for ease of entry
For paper-based surveys only (usually)
Use numeric codes
Use similar coding for similar kinds of data
CATI, CAPI, and web-based surveys capture data for you

48. Data collection
Incorporate coding into the response:
Example:

49. Data collection What to use for data entry
Necessary decision for paper-based surveys only (usually)
Word processor or text editor– OK but not structured
Spreadsheet – good because of its row-column structure
Database – usually overkill for small to medium sized data sets
OMR scanner (bubble sheets) – no transcribing of data necessary
Enter directly into statistical package like SPSS

50. Data collection Structure of data file
Rectangular (matrix) structure, like a spreadsheet
Rows represent cases, columns represent variables
One case represents one subject, survey, etc.
One variable represents a measurement on each subject, one survey question, etc.

51. Miscellaneous topics Reliability Validity Minimizing sources of error
Missing data

52. Reliability
“Repeatability” – does the survey instrument give similar results in similar situations?
Test-retest reliability
Equivalence
Internal consistency (homogeneity)
Inter-rater or intra-rater reliability

53. Validity
“Accuracy” – does the survey instrument measure what it is supposed to be measuring?
Face validity
Content validity
Criterion validity
Construct validity

54. Minimizing sources of error
Use trained interviewers for phone or face-to-face surveys
Over-sample to ensure an adequate sample size in the end
Ensure confidentiality or anonymity
Send reminders to mail respondents
Repeat calls to potential phone survey respondents
Use incentives
Ensure the reading level of the survey is appropriate for your sample

55. Missing data Caused by Non-respondents Skipped or missed questions
Refusal to answer some questions
Confusing questionnaire design
Inadequate response choices
Did not provide a “Don’t know”
Did not provide a “Not applicable”

56. A biologist, a physicist, and a statistician…
…are riding on a train through Vermont.
They pass a herd of cows, one of which
appears to be completely white. "Oh look,
there are white cows in Vermont," says the
biologist.

57. A biologist, a physicist, and a statistician…
"You mean," says the physicist with
an air of superiority, "there is at
least one white cow in Vermont."

58. A biologist, a physicist, and a statistician…
"No," says the statistician, "there
is at least one cow in Vermont
that's white on at least one side!"

59. References
Czaja, Ronald and Johnny Blair, Designing Surveys, A Guide to Decisions and Procedures (2nd ed.), Pine Forge Press, Thousand Oaks, CA, 2005.
Dillman, D. A., Mail and Telephone Surveys, The Total Design Method, John Wiley & Sons, New York, 1978.
Fink, Arlene, The Survey Handbook (2nd ed.), Sage Publications, Thousand Oaks, CA, 2003.
Kosecoff, Jacqueline and Arlene Fink, Evaluation Basics, A Practitioner’s Manual, Sage Publications, Beverly Hills, CA, 1982.
SPSS Inc., SPSS Survey Tips, SPSS Inc., Chicago, 2004.

60. References
Sudman, Seymour and Norman M. Bradburn, Asking Questions, Jossey-Bass, San Francisco, 1982.
Taylor-Powell, Ellen, Questionnaire Design, Asking Questions with a Purpose, Cooperative Extension of the University of Wisconsin-Extension, Madison, 1996.
Taylor-Powell, Ellen and Carol Hermann, Collecting Evaluation Data: Surveys, Cooperative Extension of the University of Wisconsin-Extension, Madison, 2000.
Thomas, Susan J., Designing Surveys That Work! A Step-by-Step Guide, Corwin Press, Thousand Oaks, CA, 1999.