Presentation is loading. Please wait.

Presentation is loading. Please wait.

1. 2 The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates Diane K. Willimack U.S. Census Bureau Economic Census.

Similar presentations


Presentation on theme: "1. 2 The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates Diane K. Willimack U.S. Census Bureau Economic Census."— Presentation transcript:

1 1

2 2 The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates Diane K. Willimack U.S. Census Bureau Economic Census 10 11

3 3 Outline Motivation –Economic Census Conceptual framework for nonresponse in establishment surveys Data analysis, results, and conclusions –Number of pages –Economic conditions

4 4 U.S. Economic Census Establishment level data Collected every 5 years –Reference year ends in 2 or 7 –Collection year is the following year

5 5 U.S. Economic Census continued Self-administered –Paper mail-out/mail-back –2002: Electronic via downloadable software Tailored by industry –2002: 520 different questionnaire versions

6 6 Motivation Changes in format affecting the 2002 Economic Census –Paper size: Legal Letter –Page layout: 2 columns 1 column Number of pages increased Response rates decline??? Thank you for completing the 2002 Economic Census

7 7 Research Question 1)Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?

8 8 The Data 1992, 1997, and 2002 Economic Censuses –Check-in rates Receipts + 2 nd time Undeliverables Mail Counts –1 st week of June, 1993, 1998, 2003 questionnaire version Unit of analysis = questionnaire version n = 1459 questionnaire versions

9 9 Check-In Rates by Year, 1 st week of June of the Collection Year Census Year Number of Pages (Range) Mean Check-in Rates Across All Versions 19922 – 1477.34% 19972 – 1669.50% 20023 – 2765.19%

10 10 Industry-Level Summary Check-in Rate# PagesCorrelation* IndustryMean (resprate, #pp) Mining67.91%10.26-0.4683 Manufacturing68.71%9.79-0.4661 Retail70.09%5.14-0.5225 Utilities71.71%4.48-0.5168 Wholesale73.46%6.35-0.6577 Construction73.88%6.07-0.5784 Fin, Ins, Real Est75.01%5.33-0.2967 Services75.44%4.95-0.6239 Overall70.59%8.90-0.6061 * All correlations are statistically significant at =.05.

11 11 Industry-Level Summary Check-in Rate# PagesCorrelation* IndustryMean (resprate, #pp) Mining67.91%10.26-0.4683 Manufacturing68.71%9.79-0.4661 Retail70.09%5.14-0.5225 Utilities71.71%4.48-0.5168 Wholesale73.46%6.35-0.6577 Construction73.88%6.07-0.5784 Fin, Ins, Real Est75.01%5.33-0.2967 Services75.44%4.95-0.6239 Overall70.59%8.90-0.6061 * All correlations are statistically significant at =.05.

12 12 Industry-Level Summary Check-in Rate# PagesCorrelation* IndustryMean (resprate, #pp) Mining67.91%10.26-0.4683 Manufacturing68.71%9.79-0.4661 Retail70.09%5.14-0.5225 Utilities71.71%4.48-0.5168 Wholesale73.46%6.35-0.6577 Construction73.88%6.07-0.5784 Fin, Ins, Real Est75.01%5.33-0.2967 Services75.44%4.95-0.6239 Overall70.59%8.90-0.6061 * All correlations are statistically significant at =.05.

13 13 Industry-Level Summary Check-in Rate# PagesCorrelation* IndustryMean (resprate, #pp) Mining67.91%10.26-0.4683 Manufacturing68.71%9.79-0.4661 Retail70.09%5.14-0.5225 Utilities71.71%4.48-0.5168 Wholesale73.46%6.35-0.6577 Construction73.88%6.07-0.5784 Fin, Ins, Real Est75.01%5.33-0.2967 Services75.44%4.95-0.6239 Overall70.59%8.90-0.6061 * All correlations are statistically significant at =.05.

14 14 Research Question 1)Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?

15 15 A Conceptual Framework for Business Survey Participation Out of Survey Organization ControlUnder Survey Organization Control External Environment Survey Design The Business Response Burden Business Goals Survey Participation Decision Source: Willimack, D.K., Nichols, E., and Sudman, S., (2002), Understanding Unit and Item Nonresponse in Business Surveys, in Survey Nonresponse, Groves, et al. (eds.) New York: Wiley. The Respondent

16 16 Defining the Model Concept (External Environment; Characteristics of the Business; Characteristics of the Respondent; Survey Design) Measure Economic Conditions % GDP, % Employment Industrial Sector ---------------------- #pp for the form, availability of electronic mode Response Rate = f

17 17 The Model Concept (External Environment; Characteristics of the Business; Characteristics of the Respondent; Survey Design) Measure Economic Conditions % GDP, % Employment Industrial Sector ---------------------- #pp for the form, availability of electronic mode Response Rate = f

18 18 Main Effects Model Main Effects Model: What is the relationship, regardless of the census year? Response Rate = a + b (#pages) + [effects of economic variables] + g i (Sector*) + h (Mode) + [error term] * The omitted variable among the Sector dummies is Mining.

19 19 Care in Interpreting Results Behavioral model NOT predictive –Explanatory, NOT predictive –Estimated coefficients Relative magnitude Direction (+ or -) Significance

20 20 Main Effects Model Estimated Coefficient Significantly different from 0?p-value Intercept81.81Yes<.0001 # Pages-1.13Yes<.0001 Availability of Electronic Mode-6.91Yes<.0001

21 21 Research Question Does 2002 differ from previous censuses? H 0 : 1992 = 1997 = 2002 Pairwise comparisons

22 22 Interaction Effects Model: Interaction Effects Model: Does the effect of pages differ by census year? c 92 (#pp) 2 + d 92 (#pp) 3 = a + b 92 (#pp) + c 92 (#pp) 2 + d 92 (#pp) 3 c 97 (#pp) 2 + d 97 (#pp) 3 + b 97 (#pp) + c 97 (#pp) 2 + d 97 (#pp) 3 c 02 (#pp) 2 + d 02 (#pp) 3 + b 02 (#pp) + c 02 (#pp) 2 + d 02 (#pp) 3 + [effect of economic variables] + g i (Sector*) + h (Mode) + [error term] Response Rate * The omitted variable among the Sector dummies is Mining.

23 23 Testing H 0 : 1992 = 1997 = 2002 Does the effect of pages differ by census years? Testing H 0 : 1992 = 1997 = 2002 Reject H 0 F-stat p-value #pp H 0 : b 92 = b 97 = b 02 Yes 27.23 p<.0001 (#pp) 2 H 0 : c 92 = c 97 = c 02 Yes 18.28 p<.0001 (#pp) 3 H 0 : d 92 = d 97 = d 02 Yes 12.04 p<.0001

24 24 Interaction Effects Model: Interaction Effects Model: Does the effect of pages differ by census year? Page TermYear Estimated Coefficient Differ from 1997? Differ from 2002? #pp 1992-9.94 1997-11.92 2002-5.78 (#pp) 2 19921.14 19971.23 20020.33 (#pp) 3 1992-0.04 1997-0.04 2002-0.01

25 25 Interaction Effects Model: Interaction Effects Model: Does the effect of pages differ by census year? Page TermYear Estimated Coefficient Differ from 1997? Differ from 2002? #pp 1992-9.94p=.0028p<.0001 1997-11.92p<.0001 2002-5.78 (#pp) 2 19921.14 19971.23 20020.33 (#pp) 3 1992-0.04 1997-0.04 2002-0.01

26 26 Interaction Effects Model: Interaction Effects Model: Does the effect of pages differ by census year? Page TermYear Estimated Coefficient Differ from 1997? Differ from 2002? #pp 1992-9.94p=.0028p<.0001 1997-11.92p<.0001 2002-5.78 (#pp) 2 19921.14p=.5278p<.0001 19971.23p<.0001 20020.33 (#pp) 3 1992-0.04 1997-0.04 2002-0.01

27 27 Interaction Effects Model: Interaction Effects Model: Does the effect of pages differ by census year? Page TermYear Estimated Coefficient Differ from 1997? Differ from 2002? #pp 1992-9.94p=.0028p<.0001 1997-11.92p<.0001 2002-5.78 (#pp) 2 19921.14p=.5278p<.0001 19971.23p<.0001 20020.33 (#pp) 3 1992-0.04p=.7564p<.0001 1997-0.04p<.0001 2002-0.01

28 28 Number of Pages Response Rate Estimated Model 1992

29 29 Number of Pages Response Rate Estimated Model 1997

30 30 Number of Pages Response Rate Estimated Model 2002

31 31 Rates of Change in Response Rates per Added Page 1992: Rate of Change = -9.94 + 2.28(#pp) – 0.12(#pp) 2 1997: Rate of Change = -11.92 + 2.46(#pp) – 0.12(#pp) 2 2002: Rate of Change = -5.78 + 0. 66(#pp) – 0.03(#pp) 2

32 32 Research Questions 1)Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?

33 33 Research Question #1 Conclusion Negative effect –#pages response rates –Caution: Behavioral model vs. Controlled Experiment Cannot identify causality Cannot isolate factors

34 34 Research Question #1 Conclusion continued Reject H 0 : 1992 = 1997 = 2002 –2002 differed from the previous censuses (per results from pairwise comparisons) Negative effect of #pages on economic census response appears to have diminished over time

35 35 A Conceptual Framework for Business Survey Participation Out of Survey Organization ControlUnder Survey Organization Control External Environment Survey Design The Business Response Burden Business Goals Survey Participation Decision Source: Willimack, D.K., Nichols, E., and Sudman, S., (2002), Understanding Unit and Item Nonresponse in Business Surveys, in Survey Nonresponse, Groves, et al. (eds.) New York: Wiley. The Respondent

36 36 Household Survey Literature Better economic times refusal rates increased (in a household survey) –Harris-Kojetin and Tucker, 1999, Exploring the Relation of Economic and Political Conditions with Refusal Rates to a Government Survey, JOS, 15:167-184

37 37 Research Questions continued 1)Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses? 2)Did economic conditions affect economic census response rates?

38 38 The Model Concept (External Environment; Characteristics of the Business; Characteristics of the Respondent; Survey Design) Measure Economic Conditions % GDP, % Employment Industrial Sector ---------------------- #pp for the form, availability of electronic mode Response Rate = f

39 39 Main Effects Model Main Effects Model: What is the relationship, regardless of the census year? Response Rate = a + b (#pages) + e (% GDP) + f (% Employment) + g i (Sector*) + h (Mode) + [error term] * The omitted variable among the Sector dummies is Mining.

40 40 Main Effects Model Estimated Coefficient Significantly different from 0?p-value % GDP-0.12No0.4193 % Employment-1.11Yes<.0001

41 41 Interaction Effects Model: Interaction Effects Model: Do the effects of economic conditions differ by census year? = a + [effect of page variables] + e 92 (% GDP) + f 92 (% Employment) + e 97 (% GDP) + f 97 (% Employment) + e 97 (% GDP) + f 97 (% Employment) + e 02 (% GDP) + f 02 (% Employment) + e 02 (% GDP) + f 02 (% Employment) + g i (Sector*) + h (Mode) + [error term] Response Rate * The omitted variable among the Sector dummies is Mining.

42 42 Testing H 0 : 1992 = 1997 = 2002 Do the effects of economic conditions differ by census year? Testing H 0 : 1992 = 1997 = 2002 H 0 : 1992 = 1997 = 2002 Reject H 0 F-stat (p-value) % GDP No 0.65 (0.5209) % Employment No 1.25 (0.2861)

43 43 Research Questions 1)Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses? 2)Did economic conditions affect economic census response rates?

44 44 Research Question 3)Does the effect of number of pages on response rates vary with economic conditions?

45 45 Mixed Effects Models + j i (#pp) * (% GDP) + k i (#pp) 2 * (% GDP) + m i (#pp) * (% Emply) + n i (#pp) 2 * (% Emply) + Response Rate = a + b i (#pp) + c i (#pp) 2 + d i (#pp) 3 + e i (% GDP) + f i (% Emply) + g (Sector*) + h (Mode) * The omitted variable among the Sector dummies is Mining. For each census year (i = 1992, 1997, 2002):

46 46 Summary of Results Under the Mixed Effects Model non-significant Variables that become non-significant (with p-values) –(#pp) 2 for 1992 (0.1159) –(#pp) 3 for 2002 (0.3753)

47 47 Summary of Results Under the Mixed Effects Model continued % GDP (0.0282) (#pp) * (% GDP) (0.0103) (#pp) 2 * (% GDP) (0.0087) % Emply (0.0322) (#pp) * (% Emply ) (0.0117) (#pp) 2 * (% Emply ) (0.0082) significant (all are for 2002) Economic variables that are significant (all are for 2002) (with p-values)

48 48 What was happening in the U.S. economy in 2001-2003? Economic recovery – beginning to emerge from a recession.

49 49

50 50 What was happening in the U.S. economy in 2001-2003? Economic recovery – beginning to emerge from a recession. – GDP increasing sharply – Employment declining, bottoming out Unstable economy –Sharp opposite movements in GDP and Employment

51 51 What is the nature of the interaction between economic conditions and #pages? GDP and Employment effects tend to be offsetting. – GDP moderates the page effect on response rates. – Emply exacerbates the page effect on response rates. Magnitude of Employment effect is larger.

52 52 What is the nature of the interaction between economic conditions and #pages? continued Magnitude of Employment effect is larger. –The more negative the change in employment, the more negative is the effect of #pages on response rates. –The curve representing the relationship between #pages and response rates would be flatter.

53 53 What is the nature of the interaction between economic conditions and #pages? continued Employment effect Number of pages Response Rates

54 54 Economic Theory of Lagged Expectations Economic Effects Postscript: Economic Theory of Lagged Expectations In-depth examination of effects of quarterly employment change on response rates –Statistically significant impact of changes 4 quarters and 3 quarters prior to beginning of economic census data collection.

55 55 Interpreting the Economic Effect Unstable economy –Employment decline and GDP growth Uncertainty –Riskier to divert resources to non- productive activities Cost-benefit –Survey response results in a cost with no associated productive benefit.

56 56 Interpreting the Economic Effect continued The bottom line –In a risky economic environment… –Employees are too valuable to devote to filling out forms. –Forms that appear longer (#pages) are presumed to take longer and be more costly to complete… –Response rates are in jeopardy!

57 57 Summary: Number of Pages Negative effect –#pages response rates –Cubic functional relationship Negative effect of #pages on economic census response appears to have diminished over time

58 58 Summary: Economic Conditions Evidence that economic conditions interact with form length to affect response rates –Mainly through changes in employment Survey response = non-productive cost –Corroborates qualitative research findings


Download ppt "1. 2 The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates Diane K. Willimack U.S. Census Bureau Economic Census."

Similar presentations


Ads by Google