1. HOW TO WRITE GOOD SURVEY QUESTIONS
Amy Moffat, PhD

2. Characteristics of a Survey
Questionnaires
Large number of respondents (if small number, better to do interviews)
Either use probability or nonprobability sampling
Systematic procedures are used to ask predetermined questions and record answers
Answers are numerically coded and analyzed with aid of statistical software
Probability sample = randomized
Nonprobability => can NOT infer about the larger population

3. How to Ask Survey Questions
CLARITY
Questions should mean the same thing to every respondent
6th grade reading level
Short questions are best
CONSISTENCY
Keep directions the same, or as similar as possible
Keep answers to positive/negative questions in same question (unless there is a good reason to mix it up)
Cluster similar types of items together

4. How to Ask Survey Questions
OPEN-ENDED AND CLOSE-ENDED QUESTIONS
Close-ended questions ensure uniformity, can be easier for the subject to answer, and can be easier to analyze
Open-ended questions allow for more subtle responses and perhaps more precise measurement

5. ISSUE: Double-barreled
Avoid double-barreled questions, which are questions that “force” two sentiments to coexist
Example: Please indicate how much you agree or disagree with each of the following statements about the childcare program.
Incorrect:
I feel welcomed by staff and other youth at the center
Correction:
I feel welcomed by staff at the center
I feel welcomed by other youth at the center

6. ISSUE: Biased / Leading questions
Avoid biased questions
Social desirability is a common problem
Avoid associating an attitude or a position with a prestigious person or agency
Include both positive and negative choices in the question
Be aware of “loaded” terms
Be aware of “acquiescence” – which is when a respondent “gives up” and provides similar responses to similarly formatted question

7. ISSUE: Biased / Leading questions
Incorrect:
Community organizing is hard. Do leadership trainings help you feel prepared for community organizing?
Much more prepared Somewhat more prepared
Slightly more prepared Not more prepared
Correction:
The leadership trainings prepare me for community organizing.
Strongly agree Agree Disagree Strongly disagree

8. ISSUE: Double negatives
Avoid questions with double negatives
It could be a negative statement containing two negative elements (didn't say nothing ).
Or it could be a positive statement in which two negative elements are used to produce the positive force, usually for some particular rhetorical effect (there is not nothing to worry about! ).
Incorrect
Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?
Very possible Possible Impossible Very impossible
Correct
Do you doubt that the Holocaust actually happened or not?

9. ISSUE: Accessible Information
Questions in a survey need to be answerable
Avoid questions respondents may not know the answer to
One option is to use a filter question
SKIP patterns are very helpful
Computerized surveys and questionnaires can be programmed so that respondents never see that questions have been skipped based on earlier responses.

10. Example #1
More people have attended the movie, Gone with the Wind, than any other motion picture produced this century. Have you seen this movie?
Yes No
Problem?
Biased / Leading
Solution?
Have you seen the movie Gone with the Wind?

11. Example #2
In your opinion, how would you rate the speed and accuracy of your work?
Excellent Good Fair Poor
Problem?
Double-barreled
Solution?
In your opinion, how would you rate the speed of your work?
In your opinion, how would you rate the accuracy of your work?

12. Example #3 Did you first hear about the bombing: Problem? Solution?
______ from a friend or relative ______ from a newspaper ______ from the television or radio or other electronic media
______ from your spouse
______ at work
Problem?
Categories are not mutually exclusive
Solution?
____ from a friend
____ from a relative ____ from a media source

13. Example #4
How do you feel about the following statement? We should not reduce military spending.
Strongly agree Agree Disagree Strongly disagree
Problem?
Double Negative
Solution?
How do you feel about the following statement? We should reduce military spending.
Strongly agree Agree Disagree Strongly disagree

14. Example #5
Higher than single-family density is acceptable in order to make housing affordable.
Strongly Agree Agree Undecided Disagree Strongly Disagree
Problem?
Ambiguous
Solution?
In order to make housing affordable, is it acceptable to build housing whose buildings hold more than one family?

15. Example #6
People grow up in all different types of families. What type of family did you grow up in?
______Mom as single parent ______Dad as a single parent ______Both Mom and Dad
Problem?
Categories are not exhaustive
Solution?
People grow up in all types of families. What type of family did you grow up in?
____two biological parents ____one biological parent & a step parent ____ mom as a single parent ____ dad as a single parent ____ with a relative other than my biological parents ____ in foster care ____ with adoptive parent(s)

16. Critique Real Life Example