1. Surveys Alfred Kobsa University of California, Irvine

2. When to chose a survey Scope of problem very clear Questions can be clearly formulated at concrete level Complex/holistic answers not needed. Checking pre- formulated answers is sufficient (plus occasionally one or two free-text sentences). Mailing lists with current addresses available Specific location not required Answers from many users needed Answers needed quickly Broad geographic distribution of respondents Anonymity required

3. Problems to consider Non-representative recipient population ( ☛ compare demographics of respondents with demographics of names in mailing list and normalize answers accordingly) Low response rate ( ☛ pre-announcement, personalized cover letter, reward, short survey, few open-ended questions, self-addressed envelope w/ stamp, multi-mode reminders) Self-selection bias of respondents based on topic ( ☛ hide your study goal among more general questions) Honesty ( ☛ check consistency, discard data, blacklist) Self-incrimination ( ☛ emphasize “harmlessness”, impunity, anonymity) Social desirability bias ( ☛ phrase questions carefully, use several questions for same concept, go for facts and not habits, attitudes or intentions) Self-image ( ☛ see above)

4. Steps Identify the objectives of the study (in writing) Select the type of questionnaire (closed questions, open questions, paper or web based) Decide how you will analyze the data (analysis software, statistical tests if any) Brainstorm questions Formulate questions (mind double-barreled, double negatives, leading or loaded questions, and questions with self-image and social desirability bias) Reduce question set Test and revise your questions Decide on rewards for respondents (if any)

5. Response format Closed-ended questions Single-choice questions vs. multiple-choice (plus room for “other”) scalar value Likert* scale: [1 … n] or [-n -n+1…. n-1 n] ( typically n = 5, 6 or 7) with mid-point or without with verbal “anchors” for first and last value**, or for all/some values Ranking scales (e.g., sorting by priority) Open-ended questions Should be kept to a minimum and optional since they may cause users not to fill in or abandon a questionnaire if users answer them, they only provide brief answers answers are hard to evaluate *) pronounced like “ike” in the US, but “lickert” in Europe, and by Mr Likert himself and his son **) First and last value anchors should be “extreme”