STAT 472 Survey Design Constructing the Questionnaire

 A good questionnaire forms an integrated whole.  The researcher weaves questions together so they flow smoothly.  He or She includes Introductory Remarks, Instructions for clarification and measure each variable with one or more survey questions

 There are two keys principles: - Avoid confusion (Keep it clear and simple) - Keep the respondent’s perspective in mind  Good survey questions give the researcher Valid and reliable measures Help respondents feel that they understand the question and that their answers are meaningful Do not mesh with a respondent's view point

Question writing is more of an art than a science. It takes skill, practice, patience, and creativity  A survey researcher must exercise extra care if the respondents are come from different life situations  Researchers face a dilemma All respondents hear exactly the same questions, equally clear, relevant, meaningful and exact wording If respondents have diverse back grounds and frames of reference, the exact same wording may not have the same meaning. Yet, tailoring question wording to each respondent makes comparisons almost impossible

(1) Avoid jargon, slang, and abbreviations: (unless a specialized population is being surveyed) Jargons and technical terms come in many forms: Plumbers talk about snakes Slang: is a kind of jargon Skiers about a hotdog Abbreviations: for example NATO means North Atlantic Treaty Organization North African Tea Office National Auto Tourist Organization Native Alaskan Trade Orbit

(2) Avoid ambiguity, confusion and vagueness Ambiguity and vagueness plague most question writers For example: What is your income? Could mean, weekly, monthly, or annual; family or personal; before tax or after; for this year or last year; from salary or from all sources "Do you jog regularly? Yes / No”  Use of indefinite word regularly or response categories “hinges on the meaning of the word regularly. It may define as every day, others as once a week” Good Question: “To reduce respondent confusion and get more information” Do you jog "about once a day”, “a few times a week”, “once a week”?

The question wording is an art that may improve with practice, patience, and pilot testing Here are three survey questions written by experienced professional researchers. They revised the original wording after a pilot test discover that 15% of respondents asked for clarification or gave inadequate answers (e.g., don't know)

Q1. Do you exercise or play sports regularly? Original question Problem: What counts as exercise? Response to question: 48% saying “YES” % asking for clarification: 5% Q1. Do you do any sports or hobbies, physical activities, or exercise, including walking, on a regular basis? Revised question Response to question: 60% saying “YES” % asking for clarification: 0%

Q2. What is the average number of days each week you have butter? Original question Problem: Does margarine count as butter ? Response to question: 33% saying “none” % asking for clarification: 18% Q2. The next question is just about butter-not including margarine. How many days a week do you have butter? Revised question Response to question: 55% saying “none” % asking for clarification: 13%

Q3. [Following question on eggs] What is the number of servings in a typical day? Original question Problem : How many eggs is a serving? What is a typical day? Response to question: 80% saying “one” % asking for clarification: 33% Q3. On days when you eat eggs, how many eggs do you usually have? Revised question Response to question: 33% saying “one” % asking for clarification: 0%

(3) Avoid emotional language and prestige bias murderousthreaten freedompeace-loving Example ( emotional ): What do you think about a policy to pay murderous terrorists who threaten to steal the freedom of peace-loving people? Is full of emotional words (murderous, freedoms, steal, and peace) Problem: respondents may react to the emotionally laden words rather than to the issue

Example 1 (Prestige bias) Most doctors say that cigarette smoke causes lung disease for those near a smoker, do you agree? Problem: tends to provoke "yes" answers because people trust doctors Example 2 (Prestige bias) "Do you support the president's policy on Zimbobutu? P roblem: tends to provokes "yes" answers because people trust president

(4) Avoid double-barreled questions Double-barreled questions consists of two or more questions joined together. Example : How do you evaluate the work of the Jordanian parliament and prime minister ? The two variables are: Jordanian parliament : prime minister

wrong The wrong way is to ask one question: How do you evaluate the work of the Jordanian parliament and prime minister ? correct The correct way is to ask two questions: 1- How do you evaluate the work of the Jordanian parliament ? 2- How do you evaluate the work of the prime minister ?

Example: Tell me whether you like summer and winter in Irbid or not? correct The correct way is to ask two questions: 1- Do you like summer in Irbid? 2- Do you like winter in Irbid?

(5) Avoid leading (Bias) Questions Make respondents feel that all responses are legitimate. Do not let them become aware of an answer that the researcher wants. A leading question is one that leads the respondent to choose one response over another by its wording. Example: The government should force you to pay higher taxes. No one likes to be forced, and no one likes higher taxes. Alternative: The government should increase taxes, or the government needs to increase taxes.

Example: Don’t you think that suffering terminal cancer patients should be allowed to be released from their pain? Researchers should never try to make one response option look more suitable than another. Alternative: The suffering terminal cancer patients should be allowed to be released from their pain, do you agree? Example: Wouldn’t you like to receive our free brochure? Example: You don't smoke, do you?

(6) Avoid asking questions that are beyond respondents capabilities. People have cognitive limitations, especially when it comes to memory of past events Example: (asking adult) How do you feel about your brother when you were 6 years old? is probably useless It is pointless to ask people about things that are not natural ways for them to think. Example: How many gallons of gasoline did you buy for your car last year? "Yet, respondents may be able to answer a question about gasoline purchases for a typical week, which the researcher can multiply by 52 to estimate annual purchases

Principle of good questions (7) Avoid false premises (assumption/ hypothetical ) Since the assumption is not correct, the conclusion drawn may be in error. For example, false premise: If the streets are wet, it has rained recently. (premise) The streets are wet. (premise) Therefore it has rained recently. (conclusion)

Example: The post office is open too many hours. Do you want it to open four hours later or close four hours earlier each day? Do not begin a question with a premise with which respondents may not agree, then ask choice regarding it. leaves those who either oppose the premise or oppose both alternatives without a meaningful choice Respondents who disagree with the premise will frustrated and not know how to answer

A better question explicitly asks the respondent to assume a premise is true, then asks for a preference A better question: Assuming the post office has to cut back its operating hours, which you find more convenient, operating four hours later or closing four hours earlier? Answers to a hypothetical situation are not very reliable, but being explicit will reduce frustration Example: When did you stop beating your kids? A better question: Have you ever slapped, punched, or hit your kids?

(8) Avoid asking about future intentions ( hypothetical ) Avoid asking people about what they might do under hypothetical circumstances far in the future. Responses are poor predictors of behavior removed far from their current situation or far in the future Example: Suppose a new grocery store opened down the road, Would you shop at it? Respondents answer specific, concrete questions that relate to their experiences It is better to ask about current or recent attitude and behavior

What do you do if you get 10 millions JD a day? Imagine you are married and have a 16 year old son who wants to give up his apprenticeship in order to become a professional footballer. Would you support him or would you advise him to complete his apprenticeship?

(9) Avoid double negative Double negative is ordinary language are grammatically incorrect and confusing. Examples : I am not got no job? logically means that the respondent does have a job, but the second negatives used in this way for emphasis Do you agree or not with changing your study plan or not?

Tell me whether you like or dislike having you exam online or not? Do you agree or not with changing your study plan or not? They arise when respondents are asked to agree or disagree with a statement. For example, respondents who disagree with the statement, "Students should not be required to take a comprehensive exam to graduate“ are logically stating a double negative because they disagree with not doing something.

(10) Avoid overlapping (unbalanced response categories) Make response categories or choices mutually exclusive and exhaustive and balanced. Mutually exclusive means that response categories do not overlap Numerical ranges( e.g., 5-10, 10-20,..etc) should be as ( 5- 9, 10-19, …etc)

Exhaustive means that every respondent has a choice a place to go "Are you working or unemployed?" leaves out respondents who are not working but do not consider themselves unemployed (e.g., full-time home makers, people on vacation, students, people with disabilities, retired people, etc.) A case of unbalanced choices is the question What kind of job is the mayor doing: outstanding, excellent, very good, or satisfactory? Asking respondents to rate whether a mayor is highly, somewhat, or not very honesty isn't the same as asking them to rate the mayor's level of dishonesty

Which of the five candidates running for mayor do you favor: Eugene Oswego or one of the others? Do you think the mayor is: very honest, somewhat honest, neither honest nor dishonest, somewhat dishonest, or very dishonest? Did you find the service at our hotel to be, Outstanding, Excellent, Superior, or good? Please rate the service at our hotel: outstanding, very good,inadequate, or poor.

Q. What is the composition of you household? 1. One single adult 2. Two adults 3. Two adults and one child 4. Two adults with two children 5. Two adults with three children 6. One adult with one child The categories are not exhaustive since there are several variations of adults and children possible Q. What is the composition of you household? 1. Number of adults ….. 2. Number of children ….