Survey Design: Designing Questionnaires Ronda L. Cochran, MPH Office of Prevention Research and Evaluation Division of Healthcare Quality Promotion Centers for Disease Control and Prevention

Why Survey? - To know or not to know… that is the question To find out the characteristics, behavior, or opinions of a particular population (i.e., healthcare personnel in long term care centers) To find out the characteristics, behavior, or opinions of a particular population (i.e., healthcare personnel in long term care centers) To measure the degree to which an individual exhibits the characteristic(s) of interest (i.e., use of alcohol-based handrub) To measure the degree to which an individual exhibits the characteristic(s) of interest (i.e., use of alcohol-based handrub)

A Simple Example Do you eat candy? Do you eat candy? How often do you eat candy? How often do you eat candy? □Yes □No □Never…□Rarely…□Sometimes…□Often…□All the time

Survey Research Most frequent mode of observation/measurement Most frequent mode of observation/measurement Exploratory, descriptive, and explanatory Exploratory, descriptive, and explanatory Unit of analysis usually individuals (respondents) Unit of analysis usually individuals (respondents) Used most often for: Used most often for: describing a population too large to observe (practices/behaviors) directly describing a population too large to observe (practices/behaviors) directly measuring attitudes measuring attitudes Common terms: Survey, Poll, Evaluation, Assessment, Questionnaire… Common terms: Survey, Poll, Evaluation, Assessment, Questionnaire…

Modes of Survey Administration Face-to-Face Face-to-Face Computer-assisted Computer-assisted Telephone Telephone Interactive Voice Response Surveys Interactive Voice Response Surveys Mail Mail Internet Internet

Types of Survey Research Self Administered Questionnaires: Self Administered Questionnaires: Respondents complete the questionnaire themselves (e.g., mail survey) Respondents complete the questionnaire themselves (e.g., mail survey) Interview: Interview: researchers ask the questions and record respondents answers (e.g., face-to-face interview and telephone interviews) researchers ask the questions and record respondents answers (e.g., face-to-face interview and telephone interviews)

Self-Administered Questionnaire Mail Survey: Mail Survey: Cover letter of explanation (why is this important to complete???) Cover letter of explanation (why is this important to complete???) Self-addressed stamped return envelope Self-addressed stamped return envelope But… Why not return? But… Why not return?

Self-Administered Questionnaire Response Rate: Response Rate: how many responded how many mailed Representativeness of the sample Representativeness of the sample Higher response rate, less response bias Higher response rate, less response bias Depends on target audience Depends on target audience How to increase??? How to increase??? Follow-up mailings Follow-up mailings Non respondents Non respondents All respondents All respondents

Interview Surveys Usually have higher response rate than mail surveys Usually have higher response rate than mail surveys Less “do not knows” or blanks Less “do not knows” or blanks Less items misunderstood Less items misunderstood Additional observations possible (e.g., ability to speak/understand English, general reactions) Additional observations possible (e.g., ability to speak/understand English, general reactions)

Interview Surveys Guidelines for Interviewing Guidelines for Interviewing Appearance Appearance Attitude/Demeanor Attitude/Demeanor Familiar with survey Familiar with survey Follow wording exactly Follow wording exactly Record responses exactly Record responses exactly Probe (request elaboration) appropriately Probe (request elaboration) appropriately Trained and supervised Trained and supervised

Survey Research and Technology CATI - computer assisted telephone interviewing CATI - computer assisted telephone interviewing CAPI - computer assisted personal interviewing CAPI - computer assisted personal interviewing CASI - computer assisted self-interviewing CASI - computer assisted self-interviewing CSAQ - computerized self-administered questionnaire CSAQ - computerized self-administered questionnaire TDE - touchtone data entry TDE - touchtone data entry VR - voice recognition VR - voice recognition

Define the purpose Write rules (e.g., for administration) Determine survey format Write items Identify respondents Write objectives General Steps in Survey Development

Constructing a Survey First & foremost, decide what information you need and why you need it First & foremost, decide what information you need and why you need it Define the purpose of the survey – be specific (i.e., assess extent to which occupational exposure management policies and practices are available and consistent with CDC guidelines ) Define the purpose of the survey – be specific (i.e., assess extent to which occupational exposure management policies and practices are available and consistent with CDC guidelines ) Think in terms of results – prepare hypothetical tables – what you put in… is what you get out… Think in terms of results – prepare hypothetical tables – what you put in… is what you get out…

Constructing a Survey: Determine Data to be Collected Demographics (e.g., age, gender, years in practice) Demographics (e.g., age, gender, years in practice) Consider the respondents: Consider the respondents: Who are they? Who are they? What are their expected levels of language and reading skills? What are their expected levels of language and reading skills? How familiar/aware are they with the topic ? How familiar/aware are they with the topic ? How will they be recruited? How will they be recruited? How much time will they have to complete the survey? How much time will they have to complete the survey?

Constructing a Survey: Determine Data to be Collected Variables important to testing hypothesis/research questions ( e.g., i ndependent variables, outcomes, time in study [if applicable]) – date of admission/discharge) Variables important to testing hypothesis/research questions ( e.g., i ndependent variables, outcomes, time in study [if applicable]) – date of admission/discharge) Research questions/objectives should guide the selection of items Research questions/objectives should guide the selection of items

Write theoretical definition Write items Identify format Identify sources of items Determine number of items Create blueprint Identify concept dimensions Steps in Item Writing

Item Sources….Generating Items Experts Experts Literature (review – can often find templates for item phrasing) Literature (review – can often find templates for item phrasing) Interviews (information from members of the targeted population) Interviews (information from members of the targeted population) Other Instruments (designed to measure the concept/area of study) Other Instruments (designed to measure the concept/area of study)

Types of Questions Open-ended: respondent provides his or her own answer to the question Open-ended: respondent provides his or her own answer to the question What is the most common problem associated with survey construction? What is the most common problem associated with survey construction? Close-ended: respondent is asked to select an answer from among a list provided (response choices) Close-ended: respondent is asked to select an answer from among a list provided (response choices) The most common problem associated with survey construction is: The most common problem associated with survey construction is: A. Inadequate instructions B. Use of vague or unfamiliar terms C. Overall format of the instrument D. All of the above

Responses Response choices/categories should be EXHAUSTIVE Response choices/categories should be EXHAUSTIVE Other (Please specify):________________ Other (Please specify):________________ Response choices/categories should be MUTUALLY EXCLUSIVE Response choices/categories should be MUTUALLY EXCLUSIVE How many times have you visited the doctor over the past 6 months? How many times have you visited the doctor over the past 6 months? A. 0-1 A. 0-1 B. 1-2 vs.B. 2-3 C. 2 or moreC. 4 or more

Question Development Writing Good Questions Writing Good Questions Clear and unambiguous Clear and unambiguous Spell out acronyms and define unusual terms Spell out acronyms and define unusual terms Use simple words (reading level) Use simple words (reading level) Give specific time frame/period (for recall) – i.e., “past 12 months” Give specific time frame/period (for recall) – i.e., “past 12 months” Relevant to the respondent (Do not know, no opinion, undecided, not applicable… however, use selectively) Relevant to the respondent (Do not know, no opinion, undecided, not applicable… however, use selectively) Short items are best Short items are best

Question Development Writing Bad Questions Writing Bad Questions Avoid double-barreled questions (questions with multiple parts) Avoid double-barreled questions (questions with multiple parts) Patient safety applies to patients and healthcare personnel. Patient safety applies to patients and healthcare personnel. Patient safety initiatives should be funded on a national level without any tort reform. Patient safety initiatives should be funded on a national level without any tort reform. Avoid negative items (negation leads to misinterpretation) Avoid negative items (negation leads to misinterpretation) Patient safety should not be a national priority. Patient safety should not be a national priority. Avoid biased items and terms (noting social desirability of questions and answers) Avoid biased items and terms (noting social desirability of questions and answers)

Questionnaire Construction Format - spread out, uncluttered, not squeezed on to one page, response choices should be clear Format - spread out, uncluttered, not squeezed on to one page, response choices should be clear Contingency questions: survey question intended for only some respondents, determined by their responses to some other question (skip patterns) – i.e., “Does your facility provide Hepatitis B vaccine to healthcare personnel? If NO, Go to #10” (skipping questions on Hepatitis B vaccine administration practices) Contingency questions: survey question intended for only some respondents, determined by their responses to some other question (skip patterns) – i.e., “Does your facility provide Hepatitis B vaccine to healthcare personnel? If NO, Go to #10” (skipping questions on Hepatitis B vaccine administration practices)

Questionnaire Construction Order and group items and answers to use space efficiently and effectively Order and group items and answers to use space efficiently and effectively Be careful of -- Response sets: respondents develop a pattern of responses (e.g., agreeing with all statements) -- Alternate statements/questions Be careful of -- Response sets: respondents develop a pattern of responses (e.g., agreeing with all statements) -- Alternate statements/questions INSTRUCTIONS!!! (and transitions too) INSTRUCTIONS!!! (and transitions too) Pretesting the questionnaire (pilot): writing rules for administration and scoring Pretesting the questionnaire (pilot): writing rules for administration and scoring

Survey Data Entry and Analysis Clear measurement rules for administration and scoring: Clear measurement rules for administration and scoring: Standardized - process applied the same way each time (e.g., duplicate data entry/verification) Standardized - process applied the same way each time (e.g., duplicate data entry/verification) Explicit - all users able to follow the same process Explicit - all users able to follow the same process Unambiguous - clear rules that cover all possibilities (e.g., missing data) Unambiguous - clear rules that cover all possibilities (e.g., missing data) Subject to evaluation - modify based on experience Subject to evaluation - modify based on experience

Measurement Rules For example… Survey Construction – one must consider: For example… Survey Construction – one must consider: How a correct response is scored? How a correct response is scored? How incorrect response is scored? How incorrect response is scored? What to do about skips? What to do about skips? What to do with >1 answer to an item? What to do with >1 answer to an item?

Cognitive design principles & formatting Simple design – reduce noise, include only the information you need Simple design – reduce noise, include only the information you need Use alternatives to grid lines – dot leaders and response boxes Use alternatives to grid lines – dot leaders and response boxes De-emphasize irrelevant info – use footers for copyright and other related info De-emphasize irrelevant info – use footers for copyright and other related info Use response boxes Use response boxes Emphasize selected terms – use judiciously and consider underlining Emphasize selected terms – use judiciously and consider underlining Avoid repeated instructions Avoid repeated instructions Use a consistent design throughout the questionnaire Use a consistent design throughout the questionnaire Order scale options consistently – use negative and positive item wording to control acquiescence Order scale options consistently – use negative and positive item wording to control acquiescence Place instructions in the context of the questions Place instructions in the context of the questions Group concepts Group concepts Create a natural reading flow (left to right) Create a natural reading flow (left to right) Positioning question screens and skip instructions – place to the right at the appropriate location, consider arrows Positioning question screens and skip instructions – place to the right at the appropriate location, consider arrows Use left justification Use left justification Use graphical guides to enhance tracking (upside down Triangle) Use graphical guides to enhance tracking (upside down Triangle) Use spacing to help respondent distinguish between sections Use spacing to help respondent distinguish between sections Enhance interest with a title page – may include graphics Enhance interest with a title page – may include graphics

What you put in… is what you get out… Be specific about what new information you need and why Be specific about what new information you need and why Choose the survey method that works best for you and the situation Choose the survey method that works best for you and the situation Determine sampling method(s) Determine sampling method(s) Write good questions that will provide useful, accurate information Write good questions that will provide useful, accurate information Design and test an instrument that is easy and interesting to answer Design and test an instrument that is easy and interesting to answer Put together the necessary mix of people, equipment, and supplies in the necessary time frame (logistics) Put together the necessary mix of people, equipment, and supplies in the necessary time frame (logistics) Analyze the data Analyze the data Share and present your results Share and present your results

Find the problem… Do you exercise regularly? Do you exercise regularly? □ Yes □ No Vague wording (regularly); what is considered exercise?

Find the problem… How many times have you been admitted to a hospital? How many times have you been admitted to a hospital?_____ Reference period not included

Find the problem… Do you exercise and eat a low-fat diet? Do you exercise and eat a low-fat diet? □ Yes □ No □ Yes □ No Double-barreled; what is low-fat?

Find the problem… What percentage of your weekly grocery bill is spent on dairy products? What percentage of your weekly grocery bill is spent on dairy products? _____ PERCENT Requires a calculation, what counts as dairy? Consider listing items and asking for a total.

‘The ability to ask the right question is more than half the battle of finding the answer.’ -Thomas J. Watson (1874– 1956, president of IBM) -Thomas J. Watson (1874– 1956, president of IBM) *The finding and conclusions in this presentation are those of the author and do not necessarily represent the official position of the Centers for Disease Control and Prevention.