2 Types of Questions Open-ended Closed-ended high validity, low manipulative qualityClosed-endedlow validity, high manipulative quality
3 Open-endedAn open-ended question is one in which you do not provide any standard answers to choose from.How old are you? ______ years.What do you like best about your job?
4 Closed-endedA closed-ended question is one in which you provide the response categories, and the respondent just chooses one:What do you like best about your job? (a) The people (b) The diversity of skills you need to do it (c) The pay and/or benefits (d) Other: ______________________________
5 Dichotomous Questions Dichotomous Question: a question that has two possible responsesCould beYes/NoTrue/FalseAgree/Disagree
6 Questions based on Level of Measurement Use a nominal question to measure a variableAssign a number next to each response that has no meaning; simply a placeholder.Use an ordinal question to measure a variableRank order preferencesMore than 5 – 10 items is difficultDoes not measure intensity
7 Interval Level Attempt to measure on an interval level Likert response scale: ask an opinion question on a 1-to-5, 1-to-7, etc. bipolar scaleBipolar: has a neutral point and scale ends are at opposite positions of the opinionSemantic differential: an object is assessed by the respondent on a set of bipolar adjective pairsGuttman scale: respondent checks each item with which they agree; constructed as cumulative, so if you agree to one, you probably agree to all of the ones above it in the list
8 Filter/Contingency Questions To determine if a respondent is ‘qualified’ to answer questions, might need a filter or contingency question (also known as knowledge)Limit # of jumpsIf only two levels, use graphic to jumpIf you can't fit the response to a filter on a single page, it's probably best to be send them to a page, rather than a question #
9 How many steps in the response scale? Statistical reliability of the data increases sharply with the number of scale steps up to about 7 stepsAfter 7, it increases slowly, leveling off around 11After 20, it decreases sharply
10 Should there be a middle category? Does it make sense to offer it?Should not be used as the “don’t know or no opinion” option.The middle option is usually placed between the positive and negative responses.Sometimes it’s last in an interview.
11 Direct Magnitude Scaling Method of obtaining ratio-scaled dataIdea is to give respondents an anchor point, and then ask them to answer questions relative to thatExample:Suppose you are interested in the severity of crimes.Begin by assigning a number to one crime and then have respondents assign numbers to the others based upon a ratio.
12 Filtering "Don't Know" Standard format Quasi filter Full filter No "don't know" option is presented to the respondent, but is recorded if the respondent volunteers it.Quasi filterA "don't know" option is included among the possible responses.Full filterFirst the respondent is asked if they have an opinion. If yes, the question is asked.
13 Question PlacementIt's a good idea to put difficult, embarrassing or threatening questions towards the endMore likely to answer.If they get mad and quit, at least you've gotten most of your questions asked!Put related questions together to avoid giving the impression of lack of meticulousnessWatch out for questions that influence the answers to other questions.
14 Wording of Questions Direction of Statements Always and never Language Response biasSocially desirableAlways and neverAvoid thisBetter to phrase as ‘most’, ‘infrequently’LanguageReflect educational level and reading abilityNeed for various languages
15 Frequency and Quantity Consider both frequency and quantityConsider number of timesConsider duration of times
16 Mutually Exclusive and Exhaustive Mutually exclusive: not possible to select more than one category/valueExhaustive: providing all possible categories/values
17 Forced Choice Choose between 2 choices Might not be relevant Other choices exist (or at least possible)Lesser of two evils
18 Recalling Behavior Can be difficult to remember Ask questions that can be answeredChoose time frames that are reasonablePilot test for time frame issues
19 Response Bias Exaggerating the truth Socially desirable answers Consider using ‘trap’ questionsPossibly fictional choice
20 Sensitive Items More comfortable answering in categories Minimize missing dataMight loose statistical power
22 Evaluating Questions Pre-testing Cognitive interviewing Behavior codingPeer reviewPeer review has shown to be the best method but it’s the least used.
23 Validity and Reliability Questions Evaluative strategies:Analysis of data to evaluate the strength of predictable relationships among answers and with other characteristics of respondents.Comparisons of data from alternatively worded questions asked of comparable samples.Comparison of answers against records.Measuring the consistency of answers of the same respondents at two points in time.
24 Coding the Questionnaire Create a codebook: reference guide for the data setCode: assigning a value to a response categoryOften numeric codePre-coding makes it easierContent analysis on open-ended itemsYes/No often coded as present or not (0 or 1)
25 Missing Responses Why blank? To code or not Missed them Refusal to answerDidn’t feel it appliedDidn’t know the answerTo code or notAnalyze the differenceIf know why, might consider
26 Piloting the Questionnaire Test it on yourselfPossibly other expertsTest on people similar to sampleDon’t reuse (some exceptions)Discuss the survey with individualsDuring completion or After
27 Finding Respondents Best Methods of Selection Even with a good survey, poorly chosen sample leads to poor results