2Identifying the questions to ask What are concerns and interests from yourUpstream shareholdersE.g., monitor service quality to avoid future problemsDifferent managers might have different issuesImmediate recipientsThe direct target, truly reflect their concerns (otherwise, low response rate or wrong answers)Personal interview: interview representativesFocus groups: small group interviewsRelated literatures, documentations, white papers, reports, etc.Downstream customersMight be the indirect cause, their influence to the immediate recipientsFollow your statement of purpose with clear definition of scopes and goals.
3Identifying the questions to ask Personal InterviewsDefining a set of interview questions that helps elicit the survey purposeSemi-structured questionnaire (creating your questions during the interview)Questionnaire: add more questions, alter the sequences of questions, change wording,
4Identifying the questions to ask Personal InterviewsHow to take the notesUsing tape recorder (please check the legal issues)Start with the statement of the interviewThe best notes are quotes from the interviewee.How many interviewers should we interview?1 to manyWho should conduct the interviewsSomeone with the interview trainingPerson who will design the questionnaire, conduct the survey
5Identifying the questions to ask Personal InterviewsHow should the interview be sampledPurposive sample (most representative, hard and criticalWhere should these interviews be conductedConvenient for intervieweesFace-to-face is the bestHow long should these interviews last?30min-one hourWhen should we stopWhen you stop hearing new things
6Identifying the questions to ask Personal interviewProsDeep in contextExplore unexpected pathsPaints the initial pictureConsHow structured a questionnaire?How many interviewersBy phone or in personlength
7Identifying the questions to ask Focus groupTo have a directed discussion with a small group of peopleThe dynamic interaction with groups can bring some issues or ideas that would have been missed or ignored during personal interview.Time efficientHow many people should be in a focus group7-15
8Identifying the questions to ask Focus groupWho should be invitedPurposive sample, attendee covers different fieldsWho should not be invitedLimit the number of observersWhat is the role of the moderator?Help to bring out all sidesEncourage members to discuss/present their ideasControl an overbearing opinion leaderWhat type of questionnaire should be used?Same as personal interview: semi-structured
9Identifying the questions to ask Focus groupWhere should the focus group be held and how should the discussion be captured.A comfortable meeting roomAudio tapingHow many focus groups should be conducted?Until you did not hear new information.How long should it lastTwo hours, morning is better than afternoon, evening is also goodServing a light snack is always good.
10Identifying the questions to ask Focus groupProsDeep in contextExplore unexpected pathsInteraction among participantsConsPlanning & execution criticalGood moderator criticalHave one central themeHow many to inviteWhom to inviteTime consuming textual analysis
11Documentary DataExisting documentsWhite papersRelated literature
12Analyzing data Doing content analysis of the textual notes Boiling down the words into categories, supplemented with a few well-chosen statementsIdentify patterns: group similar comments to reveal patternsSome free-form comments can address the “why” for a questionLiteral transcript of every word is not necessary, only capture key points and key focuses.Connect words to interviewee’s backgroundA distillation process
13Drafting the questionnaire It is an iterative process:Step 1: Organizing your essential findings from the interviews, group them into logic groupsStep 2: Consider three basic types of questions (e.g., demographic, specific, overall)Step 3: Select question formats: scale type (limit the number of different scale types), open-endStep 4: Draft some questionsStep 5: Take a breakStep 6: Repeat steps 3-5.Step 7: seek other input (look for external reviewers to go through your questionnaire, ask domain expertsStep 8: Pilot test your draft questionnaire with a representative sample (critical stage)
14Elements of a questionnaire A questionnaire contains several componentsAn introductionInstructionsInitial questionsSections and their headingsSummaryA thank you
15Elements of a questionnaire IntroductionPersonally addressed to the respondentTo explain the purpose of the surveyWho should complete the surveyThe expected time to complete the surveyEmphasize anonymity, if appropriateWhat to do with the completed questionnaireWhen to do the survey (now!)Definition of certain terminologyExample: Van Bennekom, p75
16Elements of a questionnaire InstructionsHow to enter answersExplain the scalesExample: Van Bennekom, p78
17Elements of a questionnaire Initial questionsPay attention to the initial questions (engage them, subject matter, motivation to move ahead, easy to answer)Never open with demographic questions in the beginning
18Elements of a questionnaire Section headingsShould be shortGroup 50 questions into 3 section vs. put all 50 questions in one section.
19Elements of a questionnaire Summary and Thank you
20Sequencing questions From general to specific Demographic questions Initial questions should be generalDemographic questionsDo not ask questions which data you do not needQuestion interactionRating a person, rather than a systemConnection of different questionsRotating questionsBranching: direct respondents to the right sections
21Question formats Unstructured Open-ended, free-form Limited the number of open-ended questions (respondent burden)Post them after certain structured questions about key theme, or ask them to clarify or expand on previous answers
22Question formats Structured (see page 85) multiple choice or categorical,ordinal scales,interval-rating scales,ratio scales
23Question formats Structured: multiple choice or categorical, For frequency distribution (e.g., job title, company size)Kinds:Multiple choice, multiple responseMultiple choice, single responseBinary choiceAdjective checklist (page 90)Design issues:Range of choices in response sets (include other category)Distinctions among choicesNumber of choices (max 6-8)Sequencing of choices
24Question formats Structured: Ordinal scales (page 93) Basic ordinal scale: age range, or salary rangeForced-ranking scale: ask respondent to rank some itemsPaired comparisonDesign issuesNumber of itemsClarify of instructions on what the rank means
25Question formats Structured: Interval-rating scales (page 95) Such as: level of agreementVerbal scales: scale in textNumeric scales: scale in numberLikert-type scale: strongly disagree-disagree-neither agree nor disagree-agree-strongly agreeMany others (page )Design issuesAdvantages of grouping by typeWhich step first, developing scale or writing questions?Importance of anchor choiceScale direction (1 is best or 5 is best?)Interval equalityAnchor balancing (extremely dissatisfied, vs. very satisfied)Even (1-7) vs. odd (1-6)-numbered scalesNumber of points on the scale: 10 points are too much.Presenting the scale low to high or high to lowUse of multiple scale types creates confusionAvoiding response rutsDispersion of results
26Question formats Structured: Ratio scales (page 108) Fractionation scale: open-ended scale, no upper limit, has better precision.
27General guidelines Goal Do not ask meaningless questions Maximizing the information we receiveMinimizing response burdenDo not ask meaningless questionsKeep the instrument focusedA survey is not a test (instruction should be clear)Be concerned about the visual and layout of your questionnaireKeep wording and grammar simpleBe creative in question design
28Pilot testing Don’s skip it It is not complicated It should be conducted as one-on-one or personal interview on purposive sampled targetsThe purposes areTo know whether questions are well designed, difficult to answer, or have proper response choicesEngage the respondent?, too long? Clarity of instructions, find awkward wording