1. Questionnaire Design & Formatting in Qualtrics Day 2
Nathan R. Jones, PhD
Senior Project Director
University of Wisconsin Survey Center

2. Schedule Day 1 - Overview Day 2 - Questionnaire development
Day Data collection

3. Principal parts of a survey
Study Design
Sample Design
Mode of administration
Instrument development
Pretesting, testing, IRB
Data collection
Data preparation - weighting, processing, delivery
Documentation
Analysis
Budget

4. A Survey From a Process Perspective
(Groves et al. 2004)

5. Measurement: Terms and definitions
Conceptualization
Defining, formulating, and clarifying concepts by
specifying dimensions
selecting indicators
Operationalization
Exact procedures used to obtain a measurement of a construct

6. How do we measure “health care use”? Specifying dimensions:
Example
How do we measure “health care use”?
Specifying dimensions:

7. How do we measure “health care use”? Specifying dimensions:
Example
How do we measure “health care use”?
Specifying dimensions:
Who qualifies as a health care provider?
What reasons for use should be included?
(e.g., only major problems or all)?
Include self only or all family members?
Include measure of frequency?
What is the reference period?

8. Operationalization of a construct
Example
Operationalization of a construct
During the past 12 months, how many times have you seen a doctor or other health care professional about your own health at a doctor’s office, a clinic, or some other place?
(Do not include times you were hospitalized overnight, visits to hospital emergency rooms, home visits, or telephone calls)

9. Evaluating survey questions
Reliability
Repeated measurements with the same measurement instrument gives the same value
Validity
Instrument measures construct it is supposed to measure

10. Improving standardization through question design
Goal of standardization is to reduce measurement errors
In interviewer-administered surveys, interviewers must be able to…
read questions exactly as worded
record answers completely and accurately
provide neutral clarifications of survey concepts
In self-administered surveys, respondents must be able to…
record their answers completely and accurately
understand the meaning of words and phrases as investigators intended

11. Instrument complexity
Complexity of the instrument
Are there complicated skip patterns?
Will there be more than one version of the instrument or more than one instrument?
Will you need to know about answers to prior questions?
Will household members need to be “rostered?”
Are there grids, for example, that ask a series of questions about each person in the household?

12. Writing questions Good question must…
yield responses that are reliable and valid
promote easy and accurate cognitive processing
be understood by all respondents in the same way
be understood by all respondents as intended by the question writer
Respondents must be willing and able to answer
Interviewers must be able to read the question accurately and consistently

13. Problematic Questions: Agree/Disagree
How strongly do you disagree or agree with the following statements:
Strongly disagree, disagree, agree, or strongly agree? SA A D SD
I am very likely to go to college
I am very likely to go to medical school
I am NOT very likely to do my residency at UW

14. Changing Agree/Disagree to construct specific question
Example
Changing Agree/Disagree to construct specific question
How strongly do you disagree or agree with the following statement:
- I am satisfied with this program [SA] [A] [D] [SD]

15. - I am satisfied with this program. [SA] [A] [D] [SD]
Example
How strongly do you disagree or agree with the following statement:
- I am satisfied with this program [SA] [A] [D] [SD]
How satisfied are you with this program?
Extremely satisfied
Very satisfied
Somewhat satisfied
Neutral
Somewhat unsatisfied
Very unsatisfied
Extremely unsatisfied

16. Example: Response dimension mismatch
I would be treated respectfully if I went to the Prefecture Hospital for a medical problem.
Is that statement:
Very untrue
Somewhat untrue
Both true and false
Somewhat true
Very true

17. Example: Clarifying the response dimension
How often are you treated respectfully when you visit the Prefecture Hospital for a health problem?
Never
Sometimes
Most of the time
All the time

18. Example: Clarifying the reference period
In the last 3 months, how often were you treated respectfully when you visited the Prefecture Hospital for a health problem?
Never
Sometimes
Most of the time
All the time

19. Self-administered questionnaires
Appealing cover and graphics
Block question text
Overhang question numbers to left
1) Question
Response 1
Response 2
Distinguish questions from instructions
Keep it clean
No parentheses, slashes, or abbreviations in question text
Use conventions consistently
No extraneous type
Leave white space
No more than four variations in type or font
Include contact information

20. Avoid analytic vocabulary
Analytic vocabulary: language used by the analyst such as technical terms and academic jargon.
Analytic Vocabulary:
- occupants of this household
Revision:
people who live here
- your responses to this questionnaire
your answers
- work-related employment
issues
job concerns

21. Example
Problematic Question:
How strong is your fertility motivation: Not at all, a little, somewhat, quite, very, or extremely?

22. Example
Problematic Question:
How strong is your fertility motivation: Not at all, a little, somewhat, quite, very, or extremely?
Revision:
How sure are you that you want another child: not at all sure, a little sure, somewhat sure, quite sure, very sure, or extremely sure?

23. Avoid negative constructions
A negative construction asks the respondent to consider the negation of a target object or poses a question with a double-negative.
Problematic Question:
Do you agree or disagree that children should not have to take care of their parents when the parents become old?

24. Avoid negative constructions
A negative construction asks the respondent to consider the negation of a target object or poses a question with a double-negative.
Problematic Question:
Do you agree or disagree that children should not have to take care of their parents when the parents become old?
Revision:
Do you agree or disagree that children should take care of their parents when the parents become old?

25. Avoid leading words or phrases
Leading, biased, or emotionally loaded words or phrases provide a point of view or orientation toward an object.
Problematic Questions:
Do you favor or oppose President Jones’s humanitarian policy in Somalia?
Do you favor or oppose President Jones’s harmful policy in Somalia?

26. Avoid leading words or phrases
Leading, biased, or emotionally loaded words provide a point of view or orientation toward an object.
Problematic Questions:
Do you favor or oppose President Jones’s humanitarian policy in Somalia?
Do you favor or oppose President Jones’s harmful policy in Somalia?
Revision:
Do you favor or oppose President Jones’s policy in Somalia?

27. Effects of small question wording changes
Do you think the United States should forbid speeches against democracy? (n=1475)
Yes (forbid): 21%
No (no forbid): 79
Do you think the United States should allow speeches against democracy? (n=1375)
No (not allow): 48%
Yes (allow): 52%
Difference: 48-21=27%; X2=223.2, p<.001
Source: Schuman and Presser 1981

28. Avoid compound or complex question structures
A compound or complex question includes two or more clauses.
Problematic Question:
In the last 12 months, did you ever reduce the size of your meals or skip meals because there wasn’t enough money for food?

29. Avoid compound or complex question structures
A compound or complex question includes two or more clauses.
Problematic Question:
In the last 12 months, did you ever reduce the size of your meals or skip meals because there wasn’t enough money for food?
Revision:
In the last 12 months, did you ever reduce the size of your meals because there wasn’t enough money for food?
In the last 12 months, did you ever skip meals because there wasn’t enough money for food?

30. Exhaustive and mutually exclusive
Exhaustive categories allow all respondents to be classified
Mutually exclusive categories do not overlap so each respondent can be classified in only one category
Problematic Question:
How did you first learn about the earthquake in Turkey?
RADIO
TELEVISION
SOMEONE AT WORK
WHILE AT HOME
WHILE TRAVELING TO WORK

31. Exhaustive and mutually exclusive
From which one of these sources did you first learn about the earthquake in Turkey?
RADIO
TELEVISION
ANOTHER PERSON
Where were you when you first heard about it?
AT WORK
AT HOME
TRAVELING TO WORK
SOMEWHERE ELSE

32. Exercise: Question 1 Problematic Question:
When did you last see a doctor?
MONTH _____ YEAR _____

33. Exercise: Question 1 Problematic Question:
When did you last see a doctor?
MONTH _____ YEAR _____
Revision:
In what month and year was your most recent visit to a medical doctor about your own health?
MONTH _____ YEAR _____

34. Exercise: Question 2 Problematic Question:
Would you say you are physically more active, less active, or about as active as other persons?

35. Exercise: Question 2 Problematic Question:
Would you say you are physically more active, less active, or about as active as other persons?
Revision:
Thinking about your physical activity during the last 30 days, would you say you are more active, less active, or about as active as other persons your age?

36. Exercise: Question 3 Problematic Question:
How often do you use a seatbelt when you drive a car? Always, nearly always, sometimes, never?
Always
Nearly always
Sometimes
Never

37. Exercise: Question 3 Revision:
1A. During the past 30 days, that is since (DATE), did you drive
a car, van, or truck?
YES NO
1B. Considering the times that you drove during the past 30 days,
how often did you use a seat belt? Would you say...
Never
Less than half the times you drove
About half the times you drove
More than half the times you drove
Every time you drove

38. Exercise: Question 4 Problematic Question:
When you were growing up, did you live with anyone in your household who smoked? Y/N

39. Exercise: Question 5 Problematic Question:
When you were growing up, did you live with anyone in your household who smoked?
Revision:
When you were growing up, that is during your first 16 years, which of the following people lived in your household and smoked?
Father YES NO
Mother YES NO
Anyone else YES NO

40. Pay attention during testing
Flow of instrument
Time
Debugging

41. General recommendations
What are some strategies for effectively wording questions?
Ask about what you need to know (related to your research question)
Ask questions that are easy to answer (low cognitive burden)
Make sure answers match the question (response dimension)
Ask about one item/concept/thing at a time (avoid double barreled)
What types of questions make a questionnaire useful when analyzing the data, especially when using statistics?
START with an analysis plan
Are you going to compare your data with another study?
How will you use every question?
Think about response categories
- Continuous, categorical, open-ended, check one, check all that apply?

42. General recommendations
How to construct questions and responses to be as unbiased as possible?
Avoid leading statements
Test question and response category ordering
Borrow from validated instruments
How do I make it long enough but not too long?
Test, test, test
Adjust incentives
How do I take foreign languages into account when writing a questionnaire?
Translate, then independently back-translate

43. Questions?

44. Senior Project Director
Thank you!
Nathan R. Jones
Senior Project Director
University of Wisconsin Survey Center