Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison

2 Outline Introduction & Background Development of the Guidelines Preliminary Guidelines Application of Guidelines Conclusion & Next Steps

Introduction & Background

4 U.S. Census Bureau: –…leading source of quality data…nations people and economy –Tries to minimize burden of data collection Burden = time, level of effort Reduce cognitive burden through visual design, structure of data request

5 Introduction & Background (contd) Paper does not focus on specific question wording and order issues. Paper presents preliminary guidelines in questionnaire design – formatting, layout, navigation, instructions, and data request phrasing and style.

6 Development of the Guidelines

7 Development of the Guidelines: Census Bureau Agency standards for various survey quality issues No standard for design of questionnaires Economic surveys moving toward consistent look and feel

8 Development of the Guidelines: ESMS Establishment Survey Methods Staff –Group of survey methodologists –In-house consultants to economic programs –Noticed differences across questionnaires –Survey programs work with ESMS on questionnaire design and pretesting –Catalyst towards consistency

9 Development of the Guidelines: Methods Questionnaire design principles from standard texts Special attention to literature on visual design Based on pretesting a variety of questionnaires with respondents

10 Development of the Guidelines: Considerations List of design elements to consider Used to assess tradeoffs within specific survey conditions Not a cookbook

Preliminary Guidelines

12 Guideline A: Text styles Be aware of text styles, and how they are used for emphasis within a survey instrument.

13 Guideline A: Example 2004 Annual Survey of Local Government Finances

14 Guideline A: Example

15 Guideline A: Text styles These are favored by respondents: –Print item numbers in reverse-print bubbles, e.g.,,, –Print questions in bold black text. –Print instructions in plain text or italics. –Use an 8-point font or larger.

16 Guideline A: Text styles Answer spaces: –Open, non-delineated vs. delineated –Respondents do not seem to have a strong preference –Be consistent

17 Guideline A: Text styles Key codes / Punch codes: –De-emphasize processing codes for respondents –Print in a darker shade of background color –Example:

18 Guideline B: Reduce visual clutter. Use natural navigational paths. Reduce clutter on the page. Use navigational paths and layouts that are natural and readable for respondents.

19 Guideline B: Reduce visual clutter. Use natural navigational paths.

20 Guideline B: Reduce visual clutter. Use natural navigational paths. Format in 1997 Economic Census confusing: –Lines were speedbumps to navigation –Not clear how respondent was to navigate two adjoined columns Format changed for 2002 –Single column of questions –Generally, a single column of response options

21 Guideline B: Reduce visual clutter. Use natural navigational paths. Two columns of questions –Occasionally done when questions: Are shorter Do not involve extensive instructions Do not ask for numerical information –Survey of Business Owners (SBO) Information collected is categorical or ordinal, not interval or ratio

22 Guideline B: Reduce visual clutter. Use natural navigational paths. Spread out the questions across more pages Use bulleted lists More open space = more user-friendly

23 Guideline C: Instructions Place instructions close to questions, or incorporate them into questions. Where possible, convert instructions into questions.

24 Guideline C: Instructions Instructions convey specifications, intent of question Respondents tend not to pay attention to instructions, or only look when they think they need them

25 Guideline C: Instructions 2 Goals for instructions: –Eliminate, or reduce, amount of instructions located separate from question –Place instructions/information where it is most needed

26 Guideline C: Instructions Convert instructions into questions so respondents attend to them –When content critical to correct interpretation of later questions –When it helps clarify/correct reported data Example (2004 Medical Expenditure Panel Survey):

27 Guideline D: Avoid sentence fragments and key words Phrase data requests as questions or imperative statements, not as sentence fragments or key words.

28 Guideline D: Avoid sentence fragments and key words Types of data requests –Question (question word, question mark) –Imperative statement (report, enter, add) –Sentence fragment (key words, no verb)

29 Guideline D: Avoid sentence fragments and key words Example, 2002 Economic Census: –Is this establishment physically located inside the legal boundaries of the city, town, village, etc.? Versus: –Type of municipality where this establishment is physically located

30 Guideline D: Avoid sentence fragments and key words Cognitive interview study: respondents prefer questions over sentence fragments Survey methodology grad students: questions more effective, imply respondent has to do something

31 Guideline E: Simple vs. complicated questions Ask additional, simple questions, rather than fewer, more complicated ones.

32 Guideline E: Simple vs. complicated questions Length and complexity of question affects how long it takes for respondent to understand it More complicated questions might also be more likely to be double-barreled May be easier for respondents to answer series of shorter, simpler questions

33 Guideline E: Simple vs. complicated questions Example: 2002 Industrial Research & Development Survey

34 Guideline E: Simple vs. complicated questions 2006 R&D Survey

35 Guideline F: Use matrices judiciously Use matrices judiciously, and consider the likely respondents background when deciding whether or not to use them.

36 Guideline G: Be consistent! Items should be numbered in the same way Bold and italicized print should always mean the same thing Navigation path should remain constant Data requests should be in the same form Variation can be confusing for respondents

37 Application of Guidelines

38 Guideline A: Text Styles Work in progress among estab surveys at Census Bureau Economic Census automation required consistent design features –Plain text: questions –Italics: instructions, definitions, include/exclude lists –Bold: Separating items into categories, sums of added lines, emphasis within questions

39 Guideline A: Text Styles Survey of Business Owners: –Questions: plain –Instructions: bold and/or italics –Emphasis within questions: bold and/or italics

40 Guideline B: Reduce visual clutter. Use natural navigational paths. Survey of Residential Alterations and Repairs (SORAR) –Two columns to one column –Removed or lightened lines –Use of bulleted lists –More open space

41 Guideline B: Reduce visual clutter. Use natural navigational paths. Old SORAR

42 Guideline B: Reduce visual clutter. Use natural navigational paths. New SORAR

43 Guideline B: Reduce visual clutter. Use natural navigational paths. Two columns: Survey of Business Owners

44 Guideline B: Reduce visual clutter. Use natural navigational paths. BEA quarterly foreign direct investment (FDI) Complex navigational path Crowded text

45 BEA form revised Guideline B: Reduce visual clutter. Use natural navigational paths.

46 Guideline C: Instructions Economic census continues to use separate instruction sheets and booklets Two questionnaires moved many instructions to questionnaire: –2007 Commodity Flow Survey –BEA quarterly FDI form

47 Guideline C: Instructions

48 Guideline D: Avoid sentence fragments and key words Mixed success in application across surveys

49 Guideline D: Avoid sentence fragments and key words Commodity Flow Survey uses mix of questions and imperative statements

50 Guideline E: Simple vs. complicated questions Example: 2002 Survey of Business Owners

51 Guideline E: Simple vs. complicated questions Example: 2007 Survey of Business Owners (still in cognitive pretesting)

52 Guideline F: Use matrices judiciously Example: BEA quarterly FDI

53 Guideline F: Use matrices judiciously

54 Conclusion & Next Steps

55 Conclusion & Next Steps Guidelines are preliminary, based on principles of questionnaire design Further refinement is necessary before they can be applied to establishment surveys at Census Bureau

56 Conclusion & Next Steps Other agencies/organizations may develop guidelines that work better for their environment There is no cookbook for questionnaire design

57 Thank you! Rebecca L. Morrison

58 Preliminary Guidelines A: Be aware of text styles, and how they are used for emphasis within a survey instrument. B: Reduce clutter on the page. Use navigational paths and layouts that are natural and readable for respondents. C: Place instructions close to questions, or incorporate them into questions. Where possible, convert instructions into questions.

59 Preliminary Guidelines D: Phrase data requests as questions or imperative statements, not as sentence fragments or key words. E: Ask additional, simple questions, rather than fewer, more complicated ones. F: Use matrices judiciously, and consider the likely respondents background when deciding whether or not to use them.