Questionnaire Design

Question Design Is it an Art? Or a Science?

Something to think about “The reason that seemingly good advice, taken literally, may turn out to be bad advice is that questions are not written in the abstract. Writing questions for a particular questionnaire means constructing them for a particular population, a particular purpose, and placement next to another particular question.”

The challenge “It is the need to consider many competing things at once that makes it difficult to write questions for self-administered surveys” (Dillman, 2000)

Objectives in designing questionnaires To maximise response rates  Every potential respondent will be willing to answer To obtain accurate relevant information  Every potential respondent will interpret in the same way,  Be able to respond to accurately

Deciding what to ask Information we are primarily interested in (ie dependent variables) Information which might explain the dependent variables (ie independent variables) Other factors related to both dependent and independent factors which may distort the results and have to be adjusted for (ie confounding variables).

Question content Types of question content:  Behaviour (what people DO)  Beliefs (what people believe is TRUE or FALSE)  Attitudes (what people think is DESIRABLE)  Attributes (respondent’s CHARACTERISTICS)

Overview Question structure Wording of individual questions Criteria for assessing questions Questionnaire layout Pilot testing

Question structure 3 main types  Open-ended question  Closed-ended question with ordered response categories  Close-ended question with unordered response categories

Open-ended questions No answer choices are provided Used in exploratory research Disadvantage: inability to get adequate answers, time-consuming coding Example: “What was your reason for screening?” _________________________ (Reason for screening)

Example What is your current occupation? ______________________ Occupation What kind of work do you do? ______________________ Kind of work What is the name of your employer? _________________________ Name of employer

Closed questions Also called forced choice-format Easy and quick to fill in Minimise discrimination against the less literate / less articulate Easy to code, record, and analyse results quantitatively Easy to report results Disadvantage: can create false opinions due to insufficient range of alternatives

Closed questions - ordered response categories Likert scales e.g., FOBT is an effective test for a population- based CRC screening program:  Strongly agree  Agree  Neither agree or disagree/ Not sure  Disagree  Strongly disagree

Closed question - ordered response categories continued Differential scales e.g., “How would you rate the presentation?” Extremely interesting Extremely dull It may also be diagrammatic: Extremely interesting Extremely dull

Closed questions - ordered response categories cont Scalar options  Strongly agree to strongly disagree  Very favourable to very unfavourable  Excellent to poor  Extremely satisfied to extremely dissatisfied  High priority goal to low priority goal  A complete success to a complete failure  A scale of 1-7 (or 1-10 or 1-100) where 1 means lowest possible quality and 7 (or 10 or 100) means highest possible quality  A scale of -3 to +3 where -3 means completely lacks this characteristic and +3 means completely exhibits this characteristic

Closed questions – unordered response categories Choice of categories e.g., “What method of service delivery do you prefer for population-based colorectal cancer screening?”  A centrally coordinated program (eg, like the breast cancer screening program)  Through GPs (eg, like the cervical cancer screening program)  Program involving a centralised system, with secondary involvement of GPs for information, counselling and on- going management  Self-testing kits made available to the public directly (eg, through pharmacies)

Closed questions - unordered response categories cont Checklists “What qualities are important in a supervisor? Circle as many options as apply” Patience Intelligence Research experience Teaching ability Good looks Good personal hygiene Ranking “Please rank the importance of the following qualities in a supervisor from most to least important by placing a 1 next to the most important, a 2 next to the second most important and so forth” 2 Patience 5 Intelligence 4 Research experience 3 Teaching ability 1 Good looks 6 Good personal hygiene

Maybe a little too taxing?! “Please rank the importance of the following qualities in a supervisor from most to least important by placing a 1 next to the most important, a 2 next to the second most important and so forth” Patience Intelligence Research experience Teaching ability Good looks Good personal hygiene Sense of humour Friendly Green eyes Clean shaven Good stats skills Eccentricity Common sense Mentoring Good cook Qualitative skills Management skills Supportive Counselling skills Well-published Healthy diet Good teeth

Combinations Which of the following is your favourite type of physical activity? ⃞ Walking ⃞ Swimming ⃞ Cycling ⃞ Tennis ⃞ Other (Please specify:________________) Of the four types of physical activity listed below, which one do you most like? ⃞ Walking ⃞ Swimming ⃞ Cycling ⃞ Tennis

Wording of individual questions Is the language simple?  Avoid jargon and technical terms  Use simple words without sounding condescending  Use simple writing guides or a thesaurus

Wording of individual questions Can the question be shortened?  Reduces confusion and ambiguity Problem question: “Has it happened to you that over a long period of time, when you neither practised abstinence nor used birth control, you did not conceive?”

Wording of individual questions Is the question double-barrelled?  Ask for only one piece of information at a time Problem question: “Do you think Australians should eat less and exercise more?” Alternative: “Do you think Australians should eat less?” 1 Yes 2 No “Do you think Australians should exercise more? 1 Yes 2 No

Wording of individual questions Is the question leading?  Respondent should feel free to give answer without feeling they are giving a wrong answer or a disapproved-of response Problem question: “Do you agree with the majority of people that health service is failing” Alternative: “Do you agree or disagree that health service is failing” “Do you believe that health service is passing or failing?”

Wording of individual questions Does the questions use negatives?  Questions using ‘not’ can be difficult to understand especially when asking respondents if they agree or disagree Problem question: “Marijuana should not be decriminalised” 1 Agree 2 Disagree Alternative: “Marijuana should be legalised” 1 Agree 2 Disagree Or “Marijuana should remain illegal” 1 Agree 2 Disagree

Wording of individual questions Is the respondent likely to have the necessary knowledge? Problem question: “Do you agree with the Gut Foundation’s guidelines for bowel cancer screening”  Assumes respondent is aware of these guidelines  Provide a filter question to see if respondent is aware of these guidelines, and then ask the substantive question only if people answered ‘yes’ to the filter question  OR outline the guidelines in the questionnaire

Wording of individual questions Is the frame of reference for the question sufficiently clear? Problem question: How often do you visit your GP?  Establish within what time frame-> within the last year? The last month?  Provide alternatives-> eg ‘weekly’ through to ‘never’

Wording of individual questions Is the question unnecessarily detailed or objectionable?  Precise age or income can create problems  If you do not need precise information on these issues, provide categories

Wording of individual questions Is the question likely to produce a response set?  Acquiescent response set: people simply agree regardless of their true opinion  Social desirability response set: people provide answers that make themselves look good  Help respondents feel comfortable regardless of their answer  State both sides of attitude scales in question stems ie “do you agree or disagree”  Remove “agree/disagree” terminology eg “easier, the same, more difficult”

Criteria for assessing each survey question 1. Does the question require an answer?  In order for an inquiry to constitute a survey question, it must require an answer from each person to whom the question is asked  Introductory words “if” and “when” invites non- response

Examples If you exercised in the last week, did you do strength exercises as part of that exercise?  Yes  No When you exercise, which type of exercise do you most prefer?  Walking  Jogging  Swimming  Other

Criteria for assessing each survey question (cont) 2. To what extent do survey recipients already have an accurate, ready-made answer for the question?  Some questions are easier to get accurate, automatic responses eg age  Some questions don’t have a “ready-made” answer eg attitudes and beliefs  Vague questions, remote from people’s experience, produce inconsistent responses

Example “Cancer screening tests save lives” Do you strongly agree, somewhat agree, somewhat disagree, or strongly disagree with this statement.

Criteria for assessing each survey question (cont) 3. Can people accurately recall and report past behaviours?  Keep recall simple and related to recent events

Examples How many minutes did you exercise in the first week of June? How many minutes per week do you “usually” exercise? How many minutes have you spent exercising in the last 24 hours?

Criteria for assessing each survey question (cont) 4. Is the respondent willing to reveal the requested information?  Income, illegal activity, sexual issues

Criteria for assessing each survey question (cont) 5. Will the respondent feel motivated to answer each question?  Important issue in self-administered questionnaires  Consider how people will react to questions  Keep questions and procedures simple  Consider incentives, follow-up reminders, respondent-friendly questionnaire design

Criteria for assessing each survey question (cont) 6. Is the respondent’s understanding of response categories likely to be influenced by more than words?  Words are only part of the question stimulus (eg high to low categories, defining numerical scales)

Example How many hours to you study a day? Less than.5 hour.5 – 1 hour 1 – 1.5 hours 1.5 – 2 hours 2 – 2.5 hours More than 2.5 hours How many hours do you study a day Less than 2.5 hours 2.5 – 3 hours 3 – 3.5 hours 3.5 – 4 hours 4 – 4.5 hours More than 4.5 hours

Questionnaire layout – Instructions General instructions: introduction to purpose of questionnaire, assurance of confidentiality, how respondent was chosen, how and when to return questionnaire Section introductions: Provide brief introduction to each sub-section eg “Finally we would like to know a little about your background so we can see how different people feel about the topics about which you’ve answered questions” Question instructions: Indicate how many responses the respondent can tick (eg one only, as many as apply) ‘Go to’ instructions: Make use of these when using contingency questions

Questionnaire layout - Use of space Avoid clutter by: Print questions on one side of page only (easy to miss questions printed on back, area for respondents to write additional comments) Provide margin for coding Leave sufficient space for open-ended questions Is the font legible and appropriate for the age group?

Questionnaire layout - Order of questions Logical flow Build rapport Commence with questions the respondent will enjoy answering (salient, easily answered, factual, not demographic) Easy -> more difficult Concrete -> abstract Least sensitive -> sensitive Keep open-ended to minimum and if possible towards end Group questions into sections Use filter questions to increase relevance of questions-> use arrows and inset boxes to highlight follow-up questions When using positive and negative items to form a scale, mix the order to help avoid acquiescent response set Introduce a variety of question formats to create interest (don’t go overboard)

Questionnaire layout - Answering procedures Are the questions numbered on the left? Is the question and answer relating to a single item on the same page? Justify your typing on the answer side Matrix presentation can be used for Likert-style questions

Shirley’s top five – a data entry typists’ perspective 1. Number or code your questionnaire, question and response options 2. Request respondents circle code rather than the response 1 YesYes 2 NoNo 3. Code open ended responses before sending for data entry 4. Provide breaks in repetitive/lengthy question sets 5. Avoid complicated tables Contact Shirley Neill (Extension x1211) $30 an hour, 20% discount if Shirley has input into layout

Questionnaire length In general: SHORT + SIMPLE = HIGHER RESPONSE RATES But also consider: Method of administration Nature of the sample Topic under investigation

Method of application Mail? Telephone? ? Internet? Personal interviews? Computer?

Pilot testing 3 phases 1. Question development  Respondents are told questions are being developed and they are being asked to improve them  Ask how they would phrase question, what they had in mind when gave particular answer, whether there were unavailable alternative answers, alternative wordings  Because of intensive nature, only a limited number of questions can be tested

Pilot testing 3 phases 2. Questionnaire development  Administer complete questionnaire (usually longer than final questionnaire)  Evaluates individual items and questionnaire as whole  Analyses respondents’ answers and uses interviewer comments  Phase often undeclared

Pilot testing 3 phases 3. Polishing pilot test  Use information from phase 2 to revise questions and where necessary, shorten, reorder and finalise skip patterns  Attention to final layout

Pilot testing - Items Variation Meaning Redundancy Scalability Non-response Acquiescent response set

Pilot testing - Questionnaire Flow Question skips Timing Respondent interest and attention

Pilot testing - How to Who to pretest? How many to pretest? Who should do the interviews? Code responses Interviewer debriefing

References W. Leung “How to design a questionnaire”, tion/187.html tion/187.html D.A. de Vaus (1995) Surveys in Social Research (4 th ed) D.A. Dillman (2000) Mail and Internet Surveys: The Tailored Design Method (2 nd ed) Central Public Health Unit Network – Queensland Health (Feb, 2005) “Key guidelines for the development of questionnaires”