CHAPTER 11 DESIGNING DATA COLLECTION FORMS

The Questionnaire Development Process Determine Survey Objectives and Constraints Determine Survey Objectives and Constraints Determine Data Collection Methods Determine Data Collection Methods Evaluate the Questionnaire and Layout Evaluate the Questionnaire and Layout Describe Questionnaire Flow and Layout Describe Questionnaire Flow and Layout Decide Question Wording Decide Question Wording Determine Question Response Format Determine Question Response Format Obtain Approval from all Relevant Parties Obtain Approval from all Relevant Parties Pretest and Revise Prepare Final Copy Implementation

Determine Objectives, Resources, and Constraints Survey (information) objectives should be spelled out as clearly and precisely as possible. If this step is completed in a thorough fashion, the rest of the process will follow more smoothly and efficiently.

Determine the Data Collection Method Chapter 9 discussed the variety of ways that survey data can be gathered, such as in-person, telephone, mail, internet or self- administration. Each method will have an impact on survey design.

Determine the Question Response Format Open-Ended Questions Open-ended questions are those in which the respondent can reply in his or her own words. Closed-Ended Questions A closed-ended question is one that requires the respondent to make a selection from a list of options. The primary advantage of closed-ended questions is simply the avoidance of many of the problems (such as subjectivity) of open- ended questions. 1

Determine the Question Response Format(cont.) Dichotomous Questions This simplest form of a closed-ended question is the dichotomous choice. Example of a dichotomous question: Would you rather live in a small town or a city? A. Small Town 1 B. City 2

Determine the Question Response Format(cont.) Multiple-Choice Questions Multiple-Choice Questions permit the respondent to select from several alternatives. Example of a multiple-choice question: Who do you think will win the World Series? A. New York Yankees B. St. Louis Cardinals C. Los Angeles Dodgers D. Atlanta Braves

Determine the Question Response Format (cont.) Scaled-Response Questions: Question format that permits the measurement of the “intensity” of a respondents’ answers. Example of a scaled-response question:. Now that you have read a sample of the Wall Street Journal, would you say that you would…(Check one): Definitely buy it Probably buy it Might or might not buy it Probably will not buy it Definitely will not buy it

Decide the Question Wording Four Guidelines in Selecting Wording The wording must be clear. Select words so as to avoid biasing the respondent. Consider the ability of the respondent to answer the question. Consider the willingness of the respondent to answer the question.

Establish Questionnaire Flow and Layout After the questions have been properly formulated, the next step is to sequence them and develop a layout for the questionnaire. The general guidelines concerning questionnaire flow are included in the next slide.

How a Questionnaire Should be Organized Use the screener questions to identify qualified respondents Use the screener questions to identify qualified respondents After obtaining a qualified respondent, begin with a question that obtains a respondent’s interest- Warm-up questions After obtaining a qualified respondent, begin with a question that obtains a respondent’s interest- Warm-up questions Ask general questions first. Use skip questions If it is necessary Ask general questions first. Use skip questions If it is necessary Position sensitive, threatening, and Classification or demographic questions at the end Position sensitive, threatening, and Classification or demographic questions at the end Insert “prompters” at strategic points Insert “prompters” at strategic points Ask questions that require “work” in the middle of the Questionnaire- Transition questions Ask questions that require “work” in the middle of the Questionnaire- Transition questions

Evaluate the Questionnaire Issues to Consider in Reviewing the Design of the Questionnaire: Is the Question Necessary? Is the Questionnaire Too Long? Will the Questions Provide the Desired Information to Accomplish the Research Objectives? Appearances of Mail and Self-Administered Questionnaires Avoid a Cluttered Look Allow Plenty of Space for Open-Ended Responses Consider Color-Coding the Questionnaires Instructions Printed Within The Questionnaire Should be in Capital Letters

Obtain Approval of All Relevant Parties At this point in the questionnaire design process, the first draft of the questionnaire has been completed. Approval for going forward with the questionnaire is the objective of this step.

Pretest and Revise When final managerial approval has been obtained, the questionnaire must be pre-tested. The primary purpose of a pretest is to make certain that the questionnaire gives the respondent clear, understandable questions that will evoke clear, understandable responses.

Prepare Final Copy Even the final copy phase does not allow the researcher to relax. Precise typing instructions, spacing, numbering, and pre-coding must be set up, monitored, and proofread.

Implementing the Survey Supervisor’s Instructions: Inform them of the nature of the study, start and completion dates, quotas, reporting times, equipment and facility requirements, sampling instructions, number of interviewers required, and validation procedures. Interviewer’s Instructions Cover many of the same points as supervisor’s instructions but are geared to the actual interview.

Implementing the Survey (cont.) Call Record Sheets Interviewers’ logs listing the number and results of a contact. Call record sheets are used to measure the efficiency of the interviewers. Visual Aids and Other Supplements Many studies use visual aids to facilitate the interviewing process.

Implementing the Survey (cont.) Field Management Companies Field management companies such as Qfact, On-Line Communications, and Direct Resource generally provide questionnaire formatting, screener writing, development of instructional and peripheral materials, shipping departments, field auditing, and all coordination of data collection, coding, and tab services required for the project.

Questionnaire Challenges The Difficulty of Finding a Qualified Respondent in a Central Location Telephone Interview 1. Failed Attempts - Busy - No Answer - Answering Machine - Business Number - Phone/Language Problem - Discontinued Line 2. Cooperation Problems - Respondent Not at Home - Refused to be Interviewed

Questionnaire Challenges The Difficulty of Finding a Qualified Respondent in a Central Location Telephone Interview 3. Screener Determines Respondent Not Eligible - Failed security test (works for marketing research firm, advertising agency, or the client) - Doesn’t use the product - Demographic disqualification (wrong gender, age, etc.) - Quota filled (survey has quota of 500 users of Tide and 500 users of other clothing washing powder. Interviewer already has 500 Tide users; the current respondent uses Tide. 4. Respondent Terminates During Interview 5. Completed interview

Questionnaire Design Checklist Step 1. Specify The Information Needed Step 2. Type of Interviewing Method Step 3. Individual Question Content Step 4. Overcome Inability and Unwillingness to Answer Step 5. Choose Question Structure Step 6. Choose Question Wording Step 7. Determine the Order of Questions Step 8. Form and Layout Step 9. Reproduce the Questionnaire Step 10. Pretest

Questionnaire Design Checklist Step 1.Specify The Information Needed: Ensure that the information obtained fully addresses all the components of the problem. Review components of the problem and the approach, particularly the research questions, hypotheses, and characteristics that influence the research design. Step 2.Individual Question Content: Is the question necessary? Are several questions needed instead of one to obtain the required information in an unambiguous manner? Do not use double-barreled questions Question should be focused on a single issue. Question should be brief. Question should be interpreted the same way by all responded. Question should use the respondent’s core vocabulary.

Step 3. Overcoming Inability and Unwillingness to Answer: Is the respondent informed? If respondents are not likely to be informed, filter questions that measure familiarity, product use, and past experience should be asked before questions about the topics themselves. Can the respondent remember? Avoid errors of omission, telescoping and creation. Questions which do not provide the respondent with cues can underestimate the actual occurrence of an event. Can the respondent articulate?

Step 3Overcoming Inability and Unwillingness to Answer (cont.): Minimize the effort required of the respondents. Is the context in which the questions are asked appropriate? Make the request for information seem legitimate. If the information is sensitive: Place sensitive topics at the end of the questionnaire. Preface the question with a statement that the behavior of interest is common. Ask the question using the third-person technique. Hide the question in a group of other questions which respondents are willing to answer. Provide response categories rather than asking for specific figures.

Step 4What Should Not be in Question Wording: Question should not assume criteria that are not obvious. Question should not be beyond the respondent’s ability or experiences. Question should not use a specific example to represent a general case. Question should not ask the respondent to recall specific when only generalist will be remembered. Question should not require the respondent to guess a generalization. Question should not ask for details that cannot be related. Question should not use words that overstate the condition. Question should not ambiguous wording. Question should not be double barreled. Question should not lead the respondent to particular answer. Question should not have “loaded” wording or phrasing.

Step 5.Choosing Question Structure: Open-ended questions are useful in exploratory research and as opening questions. Use structured questions whenever possible. In multiple-choice questions, the response alternatives should include the set of all possible choices and should be mutually exclusive. In a dichotomous question, if a substantial proportion of the respondents can be expected to be neutral, include a neutral alternative. Consider the use of the split ballot technique to reduce order bias in dichotomous and multiple-choice questions. If the response alternatives are numerous, consider using more than one question to reduce the information processing demands on the respondents.

Step 6. Choosing Question Wording: Define the issue in terms of who, what, when, where, why, and way (the six Ws). Use ordinary words. Words should match the vocabulary level of the respondents. Avoid ambiguous words: usually, normally, frequently, often, regularly, occasionally, sometimes, etc. Avoid leading questions that clue the respondent to what the answer should be. Avoid implicit alternatives that are not explicitly expressed in the options. Avoid implicit assumptions. Respondent should not have to make generalizations or compute estimates. Use positive and negative statements.

Step 7.Determine the Order of Questions: The opening questions should be interesting, simple, and non-threatening. Qualifying questions should serve as the opening questions. Basic information should be obtained first, followed by classification, and, finally, identification information. Difficult, sensitive, or complex questions should be placed late in the sequence. General questions should precede the specific questions. Questions should be asked in a logical order. Branching questions should be designed carefully to cover all possible contingencies. The question being branched should be placed as close as possible to the question causing the branching, and (2) the branching questions should be ordered so that the respondents cannot anticipate what additional information will be required.

Step 8. Form and Layout: Divide a questionnaire into several parts. Questions in each part should be numbered. The questionnaire should be pre-coded. The questionnaires themselves should be numbered serially. Step 9.Reproduction of the Questionnaire: The questionnaire should have a professional appearance. Booklet format should be used for long questionnaires. Each question should be reproduced on a single page (or double-page spread). Vertical response columns should be used. Grids are useful when there are a number of related questions which use the same set of response categories. The tendency to crowd questions to make the questionnaire look shorter should be avoided. Directions or instructions for individual questions should be placed as close to the questions as possible.

Step 10.Pre-testing:: Pre-testing should be done always. All aspects of the questionnaire should be tested, including question content, wording, sequence, form and layout, question difficulty, and instructions. The respondents in the pretest should be similar to those who will be included in the actual survey. Begin the pretest by using personal interviews. Pretest should also be conducted by mail or telephone if those methods are to be used in the actual survey. A variety of interviewers should be used for pretests. The pretest sample size is small, varying from 15 to 30 respondents for the initial testing. Use protocol analysis and debriefing to identify problems. After each significant revision of the questionnaire, another pretest should be conducted, using a different sample of respondents. The responses obtained from the pretest should be coded and analyzed.