2 Problems that careful questionnaire design can alleviate 4/5/2017Problems that careful questionnaire design can alleviateConsistency effect – respondents attempt to make their later answers fit the earlier onesFatigue effect – questions toward the end of the survey are not carefully answeredRedundancy effect – all questions in the same style are answered in the same way
3 Question types There are two major types of questions 4/5/2017Question typesThere are two major types of questionsClosed-ended questionsOpen-ended questions
4 Closed-ended questions 4/5/2017Closed-ended questionsThe respondent is provided with a limited number of response optionsEach response option has a numerical value or code it represents
5 Closed-ended questions 4/5/2017Closed-ended questionsExamples:Are you:18-34 years of age (1)35-54 years of age (2)55+ years of age (3)
6 Closed-ended questions 4/5/2017Closed-ended questionsIf you were about to purchase a digital TV, which source of information, if any, would you consult first?1. Magazines2. Advertising3. Salesperson4. Product brochure5. Friends6. Other7. Would not consult information source
7 Closed-ended questions 4/5/2017Closed-ended questionsPlease say whether youStrongly agree (5),agree somewhat (4),neither agree nor disagree (3),disagree somewhat (2)or strongly disagree (1)with the following statement:“People who spend over 400 dollars on a cell phone are vain.”
8 4/5/2017Open-ended questionsThe question is standardized, but the respondent answers in her own words.The interviewer usually ‘probes’ at least once for expansion and/or specificity after the initial response.Example:“What features would the ideal science fiction movie have?”
9 Closed-ended v. open-ended 4/5/2017Closed-ended v. open-endedClosed-ended questions take less time to completeClosed-ended questions take less time to enter into the databaseClosed-ended questions allow the respondent to interpret her position in relation to the question
10 Closed-ended v. open-ended 4/5/2017Closed-ended v. open-endedOpen-ended questions allow the respondent to elaborate and to make connections you may not have anticipatedAnswers to open-ended questions must be ‘coded’ in order to be interpreted in a quantitative mannerRespondents often prefer open-ended questions and are bored and frustrated by too many closed-ended questions
11 Closed-ended v. open-ended 4/5/2017Closed-ended v. open-endedBecause they take longer and are more difficult to deal with, open-ended questions should be used sparinglyMix open-ended questions through the survey instrument, breaking it up and adding interest
12 Question wording Concrete questions are best 4/5/2017Question wordingConcrete questions are besta. Are you happy with the service at Radio Shack?b. Better: Were you served within 5 minutes the last time you went to Radio Shack?
13 Question wording Avoid abbreviations 4/5/2017Question wordingAvoid abbreviationsIn your view, does USC provide a liberal arts education worth its yearly tuition?In your view, does the University of South Charleston provide a liberal arts education worth its yearly tuition?
14 Question wording Avoid slang and colloquial expressions 4/5/2017Question wordingAvoid slang and colloquial expressionsShould president Obama take his economic advisers behind the woodshed?Avoid jargon and technical termsShould a summative evaluation of Head Start be commissioned by the U.S. government?Better: Should the U.S. government commission a history of Head Start to review its activities and accomplishments?
15 Question wording Avoid biasing words and phrases 4/5/2017Question wordingAvoid biasing words and phrasesThe left-wing student group protesting U.S. presence in Sumatra claims that American presence there is unwarranted. Do you agree??An exception: If necessary, use loaded questions to prompt response, but be careful, respondents may be offendedParents get really angry at their children sometimes. In the past week, have you been really angry at your son?
16 Question wording Avoid double-barreled questions 4/5/2017Question wordingAvoid double-barreled questionsDo you think the University of Kentucky should subsidize student tuition and staff benefits?Would you say that the decision to increase President Lee Todd’s salary by 67% will be a boon to the University of Kentucky or don’t you care?You could think it would be a boon but not care
17 Question wording Avoid negative wording 4/5/2017Question wordingAvoid negative wordingDouble negatives are especially bad“Please indicate whether you strongly agree, agree, neither agree nor disagree, disagree or strongly disagree with the following statement:The United Nations should not have more authority to intervene in a nation’s military affairs.Either emphasize NOT when asking the question or else reword the question in a positive manner
18 Proper scale construction 4/5/2017Proper scale constructionResponse options on scales should be1) Mutually exclusive2) Equivalent3) Exhaustive
19 4/5/2017Mutual exclusivityA given respondent or item should not fit into more than one categoryExample:Are you:a) Single; b) Married; c) Divorced; d) Widowed; e) Separated, or f) Living in a couple relationship but not marriedIn one interview I heard, a 70+ year old woman said “All of the above”
20 4/5/2017Category equivalencyCategories in a scale must differ on a single dimensionExample: Would you say that the service in the shoe department is:a) fastb) slowc) polited) knowledgeable
21 4/5/2017ExhaustivenessAll respondents or items must be classifiable in one of the categoriesExample: Does your Herald-Leader arrive on your doorstep by 6 AM:a) Alwaysb) Seldomc) NeverWhat about “usually”?
22 Scales used to measure knowledge, attitudes, beliefs 4/5/2017Scales used to measure knowledge, attitudes, beliefsDeveloping measures of knowledge, attitudes, and/or beliefs is a challenging task.While anyone can come up with a measure, it is hard to develop a truly valid measure of KAB’s
23 Main types of attitude measures Semantic differentialsLikert scalesThurstone scales
25 Likert scales Example: The Employment Self Esteem Scale INSTRUCTIONS: Please rate how strongly you agree or disagree with each of the following statements by placing a check mark in the appropriate box.(Strongly Disagree/Somewhat Disagree/ Somewhat Agree/Strongly Agree)
26 I feel good about my work on the job. On the whole, I get along well with others at work.I am proud of my ability to cope with difficulties at work.When I feel uncomfortable at work, I know how to handle it.I can tell that other people at work are glad to have me there.I know I'll be able to cope with work for as long as I want.I am proud of my relationship with my supervisor at work.I am confident that I can handle my job without constant assistance.I feel like I make a useful contribution at work.I can tell that my coworkers respect me.
27 Guttman scaleStatements are ordered so that they represent increasing agreement with or acceptance of one position on a dimensionIf respondent accepts one statement higher on the scale, he should also accept those below it on the scale
28 A type of Guttman scale: Bogardus social distance scale I am willing to permit immigrants to live in my countryI am willing to permit immigrants to live in my communityI am willing to permit immigrants to live in my neighborhoodI am willing to permit immigrants to live next door to meI am willing to have immigrants for friendsI am willing to have an immigrant marry my son/daughterAgreement with item 3 implies agreement with items 1 and 2, and so on
29 Thurstone scalesThurstone scales include a number of statements that have been developed to represent their position or magnitude on some dimensionThe average of the scores of the statements agreed with by the respondent provides her placement on the scale
30 People with AIDS are like my parents. Because AIDS is preventable, we should focus our resources on prevention instead of curing.People with AIDS deserve what they got.Aids affects us all.People with AIDS should be treated just like everybody else.AIDS will never happen to me.It's easy to get AIDS.AIDS doesn't have a preference, anyone can get it.AIDS is a disease that anyone can get if they are not careful.If you have AIDS, you can still lead a normal life.AIDS is good because it helps control the population.I can't get AIDS if I'm in a monogamous relationship.
31 Response option considerations “Don’t know” optionResearchers debate whether the interviewer should explicitly ask whether the respondent doesn’t know an answerFear that the DK option gives the respondent an easy out without thinking about the questionIf an explicit “Don’t Know” option is provided, you’ll get more DKs but your conclusions may actually be more valid
32 “Neutral” optionShould scales have a middle option (odd-numbered) or no neutral point (even-numbered)?Even-numbered scales force respondents to take a stand
33 4/5/2017IndexesIndexes combine scores from multiple items to generate a single score for each respondent on some variable of interestThurstone scales are indexesSome variables are complex, needing several items to provide a valid measureThe items could be organized in a single dimension (unidimensional) or in several dimensions (multidimensional)
34 Unidimensional indexes 4/5/2017Unidimensional indexesSummated indexmeasured by a set of indicators that can be added together to derive a single, overall scoreItems are related among themselves (correlated), (if you score high on one item you should score high on the other ones)Make sure the items are not too highly correlated
35 Multidimensional indexes 4/5/2017Multidimensional indexesIf the construct you are measuring has more than one significant subconcept, you probably would gain by using a multidimensional indexExample -- communicative credibility is composed of three subconcepts:AuthoritativenessTrustworthinessDynamismTo measure communicative credibility you would need to ask questions related to each of the subconcepts
36 So: Questionnaire design is the most important part of survey analysis 4/5/2017So:Questionnaire design is the most important part of survey analysisDeveloping valid measures of ‘black box’ phenomena is especially difficultWriting good questions is probably the most important part of questionnaire designA number of common problems in question wording exist
37 4/5/2017Closed-ended questions are simpler and faster but open-ended questions allow respondents to provide more personalized information and are more interesting for themA number of forms of attitude/belief measures have been developed over the yearsCommonly used types are Likert scales, semantic differentials, Thurstone scales, Guttman scales
38 Using multiple items to measure a single construct and then combining them increases measurement validity