Presentation on theme: "Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,"— Presentation transcript:
Questionnaire Design Van Bennekom, Chapter 4
Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g., monitor service quality to avoid future problems Different managers might have different issues – Immediate recipients The direct target, truly reflect their concerns (otherwise, low response rate or wrong answers) Personal interview: interview representatives Focus groups: small group interviews Related literatures, documentations, white papers, reports, etc. – Downstream customers Might be the indirect cause, their influence to the immediate recipients – Follow your statement of purpose with clear definition of scopes and goals.
Identifying the questions to ask Personal Interviews – Defining a set of interview questions that helps elicit the survey purpose – Semi-structured questionnaire (creating your questions during the interview) – Questionnaire: add more questions, alter the sequences of questions, change wording,
Identifying the questions to ask Personal Interviews – How to take the notes Using tape recorder (please check the legal issues) Start with the statement of the interview The best notes are quotes from the interviewee. – How many interviewers should we interview? 1 to many – Who should conduct the interviews Someone with the interview training Person who will design the questionnaire, conduct the survey
Identifying the questions to ask Personal Interviews – How should the interview be sampled Purposive sample (most representative, hard and critical – Where should these interviews be conducted Convenient for interviewees Face-to-face is the best – How long should these interviews last? 30min-one hour – When should we stop When you stop hearing new things
Identifying the questions to ask Personal interview – Pros Deep in context Explore unexpected paths Paints the initial picture – Cons How structured a questionnaire? How many interviewers By phone or in person length
Identifying the questions to ask Focus group – To have a directed discussion with a small group of people – The dynamic interaction with groups can bring some issues or ideas that would have been missed or ignored during personal interview. – Time efficient – How many people should be in a focus group 7-15
Identifying the questions to ask Focus group – Who should be invited Purposive sample, attendee covers different fields – Who should not be invited Limit the number of observers – What is the role of the moderator? Help to bring out all sides Encourage members to discuss/present their ideas Control an overbearing opinion leader – What type of questionnaire should be used? Same as personal interview: semi-structured
Identifying the questions to ask Focus group – Where should the focus group be held and how should the discussion be captured. A comfortable meeting room Audio taping – How many focus groups should be conducted? Until you did not hear new information. – How long should it last Two hours, morning is better than afternoon, evening is also good Serving a light snack is always good.
Identifying the questions to ask Focus group – Pros Deep in context Explore unexpected paths Interaction among participants – Cons Planning & execution critical Good moderator critical Have one central theme How many to invite Whom to invite Time consuming textual analysis
Documentary Data Existing documents White papers Related literature
Analyzing data Doing content analysis of the textual notes – Boiling down the words into categories, supplemented with a few well-chosen statements – Identify patterns: group similar comments to reveal patterns – Some free-form comments can address the “why” for a question – Literal transcript of every word is not necessary, only capture key points and key focuses. – Connect words to interviewee’s background – A distillation process
Drafting the questionnaire It is an iterative process: Step 1: Organizing your essential findings from the interviews, group them into logic groups Step 2: Consider three basic types of questions (e.g., demographic, specific, overall) Step 3: Select question formats: scale type (limit the number of different scale types), open-end Step 4: Draft some questions Step 5: Take a break Step 6: Repeat steps 3-5. Step 7: seek other input (look for external reviewers to go through your questionnaire, ask domain experts Step 8: Pilot test your draft questionnaire with a representative sample (critical stage)
Elements of a questionnaire A questionnaire contains several components – An introduction – Instructions – Initial questions – Sections and their headings – Summary – A thank you
Elements of a questionnaire Introduction – Personally addressed to the respondent – To explain the purpose of the survey – Who should complete the survey – The expected time to complete the survey – Emphasize anonymity, if appropriate – What to do with the completed questionnaire – When to do the survey (now!) – Definition of certain terminology – Example: Van Bennekom, p75
Elements of a questionnaire Instructions – How to enter answers – Explain the scales – Example: Van Bennekom, p78
Elements of a questionnaire Initial questions – Pay attention to the initial questions (engage them, subject matter, motivation to move ahead, easy to answer) – Never open with demographic questions in the beginning
Elements of a questionnaire Section headings – Should be short – Group 50 questions into 3 section vs. put all 50 questions in one section.
Elements of a questionnaire Summary and Thank you
Sequencing questions From general to specific – Initial questions should be general Demographic questions – Do not ask questions which data you do not need Question interaction – Rating a person, rather than a system – Connection of different questions Rotating questions Branching: direct respondents to the right sections
Question formats Unstructured – Open-ended, free-form – Limited the number of open-ended questions (respondent burden) – Post them after certain structured questions about key theme, or ask them to clarify or expand on previous answers
Question formats Structured (see page 85) – multiple choice or categorical, – ordinal scales, – interval-rating scales, – ratio scales
Question formats Structured: multiple choice or categorical, – For frequency distribution (e.g., job title, company size) – Kinds: Multiple choice, multiple response Multiple choice, single response Binary choice Adjective checklist (page 90) – Design issues: Range of choices in response sets (include other category) Distinctions among choices Number of choices (max 6-8) Sequencing of choices
Question formats Structured: Ordinal scales (page 93) – Basic ordinal scale: age range, or salary range – Forced-ranking scale: ask respondent to rank some items – Paired comparison – Design issues Number of items Clarify of instructions on what the rank means
Question formats Structured: Interval-rating scales (page 95) – Such as: level of agreement – Verbal scales: scale in text – Numeric scales: scale in number – Likert-type scale: strongly disagree-disagree-neither agree nor disagree-agree-strongly agree – Many others (page ) – Design issues Advantages of grouping by type Which step first, developing scale or writing questions? Importance of anchor choice Scale direction (1 is best or 5 is best?) Interval equality Anchor balancing (extremely dissatisfied, vs. very satisfied) Even (1-7) vs. odd (1-6)-numbered scales Number of points on the scale: 10 points are too much. Presenting the scale low to high or high to low Use of multiple scale types creates confusion Avoiding response ruts Dispersion of results
Question formats Structured: Ratio scales (page 108) – Fractionation scale: open-ended scale, no upper limit, has better precision.
General guidelines Goal – Maximizing the information we receive – Minimizing response burden Do not ask meaningless questions Keep the instrument focused A survey is not a test (instruction should be clear) Be concerned about the visual and layout of your questionnaire Keep wording and grammar simple Be creative in question design
Pilot testing Don’s skip it It is not complicated It should be conducted as one-on-one or personal interview on purposive sampled targets The purposes are – To know whether questions are well designed, difficult to answer, or have proper response choices – Engage the respondent?, too long? Clarity of instructions, find awkward wording