1

2 Issues in the Design and Testing of Business Survey Questionnaires: Diane K. Willimack U.S. Census Bureau Economic Census The International Conference on Reshaping Official Statistics What We Know Now that We Didn’t Know Then – and What We Still Don’t Know

3 Issues in the Design and Testing of Business Survey Questionnaires: Diane K. Willimack U.S. Census Bureau Economic Census The International Conference on Reshaping Official Statistics Lessons from 10 Years of Pretesting Business Surveys at the U.S. Census Bureau

4 Outline  Background –U.S. Census Bureau economic programs –Characteristics of economic surveys Response process  Pretesting procedures and methodology  Issues and recommendations

5 Economic Survey Programs at the U.S. Census Bureau  More than 100 economic survey programs plus economic census  Authority – mandatory vs. voluntary  Data collection modes

6  Survey organization’s perspective –Population, data, and survey design  Respondent’s perspective –Response process Characteristics of Economic Surveys

7 The Response Process in Economic Surveys Cognitive response model PLUS Organizational processes

8 Cognitive Response Model (Tourangeau, 1984) 1. Comprehension 2. Retrieval 3. Judgment 4. Communication Survey

9 Response Process Model for Establishment Surveys (Sudman et al., 2000 ICES-2) Organizational Aspects 4. Comprehension 5. Retrieval 6. Judgment 7. Communication Business Survey

10 Response Process Model for Establishment Surveys Encoding in Memory / Record Formation Selection / Identification of Respondent(s) Assessment of Priorities (Motivation) 4. Comprehension 5. Retrieval 6. Judgment 7. Communication Release of the Data Business Survey from Memory and / or Records from Memory and / or Records

11 Pretesting Goals  Identify potential response error, e.g. --- –Question clarity –Contents of records –Response strategy  Suggest improvements to survey questions

12  Dedicated staff  Consultant – client relationship with survey programs –More than 25 separate programs plus the economic census –Variety of survey types and topics  Electronic mode –Usability testing Economic Survey Pretesting at the U.S. Census Bureau

13  Collaborative approach  Voluntary participation by respondents  Site visits  Post-testing follow-through  Multiple rounds – test, revise, re-test Pretesting Procedures

14 Pretesting Methodology  Qualitative research  Traditional cognitive interviewing methods  Adaptations –Hypothetical questions / probes –Funnel approach: general  specific –Directive questions / probes  In-depth unstructured interviews

15 Six Unresolved or Persistent Issues for Economic Surveys  Consequences  Methodology  Shortcomings  Recommendations

16 Issue #1: Business survey response is labor-intensive, burdensome & costly  Consequences –Respondents unwilling or unable to complete a draft questionnaire prior to or during the pretest interview. –Researchers unable to use some traditional cognitive methods with draft questionnaires

17  Pretesting methodology –Hypothetical probes  Shortcoming –Hypothetical report  Actual behavior Issue #1: Business survey response is labor-intensive, burdensome & costly

18  Recommendations –Respondent debriefings during field period Actual behavior –“Pilot” test new questionnaires in the field Conduct respondent debriefings Evaluate data quality Issue #1: Business survey response is labor-intensive, burdensome & costly

19 Issue #2: “Omnibus” surveys require multiple respondents or data sources  Consequences –No single “right” respondent –Hindrances to gathering data Knowledge Authority Motivation –Poor data quality

20  Pretesting methodology –Detailed probes about data sources and communication practices  Shortcoming –Unable to meet with or interview downstream informants Issue #2: “Omnibus” surveys require multiple respondents or data sources

21  Recommendations –New methodologies needed to research --- Data locations Company policies and practices for exchanging data internally Strategies for internal company communication –Develop data collection procedures that encourage, facilitate and support respondents’ data-gathering Issue #2: “Omnibus” surveys require multiple respondents or data sources

22 Issue #3: Mismatch between survey concepts and company records  Consequences –Response error –Response burden

23  Pretesting methodology –Directive questions  Shortcoming –Cognitive interviews devolve into a hybrid of cognitive probes and exploratory questions. Issue #3: Mismatch between survey concepts and company records

24  Recommendations –Involve methodologists in content development –Conduct in-depth exploratory interviews to identify the “business model” for the concept of interest Use directive questions Issue #3: Mismatch between survey concepts and company records

25 Issue #4: Questions request technical or financial data  Consequences –Methodologists  subject area experts –Methodologists  financial accountants –Cognitive interviews may not capture discrepancies between respondents’ answers and the intent of the question

26  Pretesting methodology –Collaboration between methodologists and experts in subject matter or accounting  Shortcomings –Different approaches Methodologists: how the respondent answers questions without help Subject area / accounting experts: instruct respondents on how to report correctly Issue #4: Questions request technical or financial data

27  Recommendations –Improved collaborative methods Experts aid methodologists in developing the interview protocol Methodologists report respondent behaviors; experts assess adequacy of response Experts participate in interviews as observers Issue #4: Questions request technical or financial data

28 Issue #5: Many / most economic surveys are self-administered  Consequences –No interviewer –Heavy reliance on questionnaire to convey technical requirements of the data request –Need for detailed instructions Business Survey

29  Pretesting Methodology –Probes on wording, language, visual design and placement of instructions  Shortcoming –Questionnaire design cannot fix every problem Issue #5: Many / most economic surveys are self-administered

30  Recommendation –Encourage new paradigms in --- Questionnaire development –Top-down and bottom-up –Identify collectable data with identifiable shortcomings Use of data –Consider context and utility of reported data –Consider shortcomings in data analysis and interpretation Issue #5: Many / most economic surveys are self-administered

31 Issue #6: Heavy reliance on cognitive interviewing to identify and correct questionnaire problems  Consequence –Improper or unrealistic expectations –Difficult to evaluate effectiveness

32  Pretesting Methodology –Multiple iterations of design and testing – test, revise, re-test  Shortcomings –Cognitive testing is --- Qualitative methodology Somewhat artificial Issue #6: Heavy reliance on cognitive interviewing to identify and correct questionnaire problems

33  Recommendations –Use multiple complementary methods Helps gain confidence in results –Develop and monitor meaningful objective measurements of data quality Issue #6: Heavy reliance on cognitive interviewing to identify and correct questionnaire problems

34 Conclusion: 10 Years of Pretesting Business Surveys at the U.S. Census Bureau  Cognitive testing –Useful –Integrated into survey development

35 Conclusion: The Next 10 Years of Pretesting Business Surveys  Challenges –Multiple research methods –Quantitative evaluation of effectiveness –Organizational behavior and the survey response process

36  Thank you.  Questions?? Comments?? Diane K. Willimack U.S. Census Bureau