4 U.S. Census Bureau: –…leading source of quality data…nations people and economy –Tries to minimize burden of data collection Burden = time, level of effort Reduce cognitive burden through visual design, structure of data request
5 Introduction & Background (contd) Paper does not focus on specific question wording and order issues. Paper presents preliminary guidelines in questionnaire design – formatting, layout, navigation, instructions, and data request phrasing and style.
7 Development of the Guidelines: Census Bureau Agency standards for various survey quality issues No standard for design of questionnaires Economic surveys moving toward consistent look and feel
8 Development of the Guidelines: ESMS Establishment Survey Methods Staff –Group of survey methodologists –In-house consultants to economic programs –Noticed differences across questionnaires –Survey programs work with ESMS on questionnaire design and pretesting –Catalyst towards consistency
9 Development of the Guidelines: Methods Questionnaire design principles from standard texts Special attention to literature on visual design Based on pretesting a variety of questionnaires with respondents
10 Development of the Guidelines: Considerations List of design elements to consider Used to assess tradeoffs within specific survey conditions Not a cookbook
15 Guideline A: Text styles These are favored by respondents: –Print item numbers in reverse-print bubbles, e.g.,,, –Print questions in bold black text. –Print instructions in plain text or italics. –Use an 8-point font or larger.
16 Guideline A: Text styles Answer spaces: –Open, non-delineated vs. delineated –Respondents do not seem to have a strong preference –Be consistent
17 Guideline A: Text styles Key codes / Punch codes: –De-emphasize processing codes for respondents –Print in a darker shade of background color –Example:
18 Guideline B: Reduce visual clutter. Use natural navigational paths. Reduce clutter on the page. Use navigational paths and layouts that are natural and readable for respondents.
20 Guideline B: Reduce visual clutter. Use natural navigational paths. Format in 1997 Economic Census confusing: –Lines were speedbumps to navigation –Not clear how respondent was to navigate two adjoined columns Format changed for 2002 –Single column of questions –Generally, a single column of response options
21 Guideline B: Reduce visual clutter. Use natural navigational paths. Two columns of questions –Occasionally done when questions: Are shorter Do not involve extensive instructions Do not ask for numerical information –Survey of Business Owners (SBO) Information collected is categorical or ordinal, not interval or ratio
22 Guideline B: Reduce visual clutter. Use natural navigational paths. Spread out the questions across more pages Use bulleted lists More open space = more user-friendly
23 Guideline C: Instructions Place instructions close to questions, or incorporate them into questions. Where possible, convert instructions into questions.
24 Guideline C: Instructions Instructions convey specifications, intent of question Respondents tend not to pay attention to instructions, or only look when they think they need them
25 Guideline C: Instructions 2 Goals for instructions: –Eliminate, or reduce, amount of instructions located separate from question –Place instructions/information where it is most needed
26 Guideline C: Instructions Convert instructions into questions so respondents attend to them –When content critical to correct interpretation of later questions –When it helps clarify/correct reported data Example (2004 Medical Expenditure Panel Survey):
27 Guideline D: Avoid sentence fragments and key words Phrase data requests as questions or imperative statements, not as sentence fragments or key words.
28 Guideline D: Avoid sentence fragments and key words Types of data requests –Question (question word, question mark) –Imperative statement (report, enter, add) –Sentence fragment (key words, no verb)
29 Guideline D: Avoid sentence fragments and key words Example, 2002 Economic Census: –Is this establishment physically located inside the legal boundaries of the city, town, village, etc.? Versus: –Type of municipality where this establishment is physically located
30 Guideline D: Avoid sentence fragments and key words Cognitive interview study: respondents prefer questions over sentence fragments Survey methodology grad students: questions more effective, imply respondent has to do something
31 Guideline E: Simple vs. complicated questions Ask additional, simple questions, rather than fewer, more complicated ones.
32 Guideline E: Simple vs. complicated questions Length and complexity of question affects how long it takes for respondent to understand it More complicated questions might also be more likely to be double-barreled May be easier for respondents to answer series of shorter, simpler questions
33 Guideline E: Simple vs. complicated questions Example: 2002 Industrial Research & Development Survey
35 Guideline F: Use matrices judiciously Use matrices judiciously, and consider the likely respondents background when deciding whether or not to use them.
36 Guideline G: Be consistent! Items should be numbered in the same way Bold and italicized print should always mean the same thing Navigation path should remain constant Data requests should be in the same form Variation can be confusing for respondents
38 Guideline A: Text Styles Work in progress among estab surveys at Census Bureau Economic Census automation required consistent design features –Plain text: questions –Italics: instructions, definitions, include/exclude lists –Bold: Separating items into categories, sums of added lines, emphasis within questions
39 Guideline A: Text Styles Survey of Business Owners: –Questions: plain –Instructions: bold and/or italics –Emphasis within questions: bold and/or italics
40 Guideline B: Reduce visual clutter. Use natural navigational paths. Survey of Residential Alterations and Repairs (SORAR) –Two columns to one column –Removed or lightened lines –Use of bulleted lists –More open space
41 Guideline B: Reduce visual clutter. Use natural navigational paths. Old SORAR
42 Guideline B: Reduce visual clutter. Use natural navigational paths. New SORAR
43 Guideline B: Reduce visual clutter. Use natural navigational paths. Two columns: Survey of Business Owners
44 Guideline B: Reduce visual clutter. Use natural navigational paths. BEA quarterly foreign direct investment (FDI) Complex navigational path Crowded text
45 BEA form revised Guideline B: Reduce visual clutter. Use natural navigational paths.
46 Guideline C: Instructions Economic census continues to use separate instruction sheets and booklets Two questionnaires moved many instructions to questionnaire: –2007 Commodity Flow Survey –BEA quarterly FDI form
55 Conclusion & Next Steps Guidelines are preliminary, based on principles of questionnaire design Further refinement is necessary before they can be applied to establishment surveys at Census Bureau
56 Conclusion & Next Steps Other agencies/organizations may develop guidelines that work better for their environment There is no cookbook for questionnaire design
57 Thank you! Rebecca L. Morrison 301-763-7595 Rebecca.L.Morrison@census.gov
58 Preliminary Guidelines A: Be aware of text styles, and how they are used for emphasis within a survey instrument. B: Reduce clutter on the page. Use navigational paths and layouts that are natural and readable for respondents. C: Place instructions close to questions, or incorporate them into questions. Where possible, convert instructions into questions.
59 Preliminary Guidelines D: Phrase data requests as questions or imperative statements, not as sentence fragments or key words. E: Ask additional, simple questions, rather than fewer, more complicated ones. F: Use matrices judiciously, and consider the likely respondents background when deciding whether or not to use them.