Structured Interviews and Instrument Design Part I Lecture 5 Info 271B

“Continuum” of Interview Situations Unstructured Structured Open-ended interviews Interview Guides, still open-ended Identical Instrument (questions, etc) given to all participants

What is a Survey? To survey: Research Surveys “act of looking or seeing, observing” Research Surveys Qualitative interviews Focus groups Specific, systematic, quantitative data-collection instruments

Example Surveys US Census General Social Survey Online Surveys

Structured Interview Formats Face-to-face, self-administered (email, mail, etc), phone surveys Typically either cross-sectional or longitudinal Often a balance between (1) standardization (2) ability to reach sample (3) available resources (4) type of questions you are asking

Standardization of Wording and Format in an Interview Instrument Must have clear wording Must have clear formatting Must take into account the sample population that will actually take/use the survey instrument Culture, language, interpretive ambiguity Must have a logic for each question Must have a logic for the question responses (if provided) Must have a logic for the sequencing of questions

Types of Questions (review) Dichotomous Categorical; only 2 primary responses are possible (True/False, Yes/No) Nominal Categorical; numbers only used as placeholders for responses. Ordinal Categorical; numbers imply an underlying ordered structure Interval Semantic differential Likert-style questions

Question Logic: Purposeful Questions Avoid asking unnecessary questions Tendency to put in the ‘kitchen sink’ but other concerns (Considering participant fatigue, comprehension, time and other costs are all part of the survey design process) Example: Survey on computer usage Socio-demographic questions (age, gender)? Risk behavior questions (drug use, etc)? Does the response type allow you to answer the intended question? Examples: Closed or Open-ended questions? Meaningful and interpretable responses?

Question Logic: Closed versus Open Questions (Q1a) What do you spend most of your time doing on the Internet? (Q1b) Which of the following things do you do most of the time on the Internet? (a) Email (b) Browse the web (c) Play games (d) Other

Question Logic: Redundancy Usually we try to avoid redundant questions Exception 1: Sometimes it is a good idea to ask two questions that tap same concept (first as a scale, then as a categorical decision) Example: First ask respondent to rate 1-10 how much they agree with several statements. Next, ask respondents to rank the statements by how much they agree with them (i.e., 5 statements, rank them 1-5). Exception 2: For complex concepts, we may have to ask several similar questions to ‘get’ at the underlying concepts. “Most people are generally trustworthy” (agree/disagree scale) “In general, I can trust others” (agree/disagree scale)

Question Response Logic: Scales versus Categories As a basic rule, try to get as much information as you can without missing the intended concept. Example: Age Are you 18 or older (yes/no)? How old are you (in years)? _________ With general scales and Likert-scales, consider having no “middle” category (neutral, no opinion). (More on Scales and Scaling on Thursday)

Question Response Logic: Categorical Responses Responses must be mutually exclusive Examples (bad): What is the best college basketball team? 1) UNC-Chapel Hill 2) Duke 3) Any team that is not Duke. Example (bad): Where do you live? Berkeley, San Fran, Bay Area, Other Responses must be exhaustive Example (bad): What operating system (OS) do you use? 1) Windows 2) OSX Use ‘Don’t Know’, ‘Other’, ‘Not Applicable’ when absolutely necessary. Consider whether you really want respondents to indicate that they don’t know, or if you want to force a response if possible.

Mutual Exclusivity and Exhaustive Responses How do you perceive communication between your department and other departments in the university? [Check one] (1) There is much communication (2) There is sufficient communication (3) There is little communication (4) There is no communication (5) No basis for perception ‘little’ ‘much’ and ‘sufficient’ are completely different issues.

Question Sequence Static order for questions needs to have some rationale/logic Grouping similar items together coherent instrument or to keep topics clear in survey Scattering similar items throughout survey perhaps to hide an underlying concept that you want to measure without priming the respondent Personal demographic questions often work best at end of survey (response rate and completion) Randomization for all respondents if order effect is a concern.

Clear Wording / Leading Questions Examples: “What ISP do you use? “What company or service provider do you use to get access to the Internet, if applicable?” Leading Questions “Don’t you think we should support our troops in Iraq?” “How strongly do you agree or disagree with the following question: ‘We should support our troops in Iraq’” Key issues: Unambiguous Clear vocabulary that isnt condescending Respondents must know enough to be able to respoond

Clear Formatting, Logic and Branching Not all questions apply to everyone Example: “How much do you spend on gas heat each month?” Branching is a possible solution “Do you have gas heat? If yes, go to next question. If not, skip to question #3. Condense when possible to avoid unnecessary branching. “How much do you spend on gas heat each month? (write a value, or check ‘not applicable’) Purpose for all questions! Be especially careful with contingent questions and filters

‘Unpacked’ Questions Some people believe that the internet reduces face-to-face interaction and may cause ‘internet addiction’. Do you agree or disagree with these people? (1) Agree (2) Disagree (3) Do Not Know (4) Refuse to Answer