1. Aggregate and Systemic Components of Risk in Total Survey Error Models John L. Eltinge U.S. Bureau of Labor Statistics International Total Survey Error Workshop June 20, 2011

2. Acknowledgements and Disclaimer The author thanks Ted Chang, Moon Jung Cho, Mike Davern, John Dixon, Malay Ghosh, Jeffrey Gonzalez, Rachel Harter, Bill Iwig, Mike Horrigan, Brandon Kopp, Phil Kott, Bill Mockovak, Danny Pfeffermann, Jay Ryan, George Stamas and Michael Sverchkov for many helpful discussions of the topics considered in this paper. The views expressed here are those of the author and do not necessarily reflect the policies of the U.S. Bureau of Labor Statistics. 2

3. Overview I. Survey Risk, Data Quality and Total Survey Error Models II. Aggregate and Systemic Components of Risk III. Impact of, and Recovery from, Systemic Errors IV. Adjusting Design Features to Account for Systemic Error Components 3

4. I. Survey Risk, Data Quality and Total Survey Error Models A. Survey Risk 1. Degradation of one or more dimensions of survey quality 2. Six dimensions from Brackstone (1999, Survey Methodology) Accuracy, Timeliness, Relevance, Interpretability, Accessibility, Coherence 4

5. I. Survey Risk, Data Quality and Total Survey Error Models (Continued) B. Total Survey Error Model: Detailed decomposition of the “accuracy” component of quality (Estimator) – (True value) = (frame error) + (sampling error) + (incomplete data effects) + (measurement error) + (processing effects) 5

6. II. Aggregate and Systemic Components of Survey Risk A. Types of Survey Risk: For degradation in any component of data quality (and especially for TSE components), distinguish between aggregate risk and systemic risk 1. Aggregate risk: Combined effects of a large number of (infinitesimal) random events a. Superpopulation effects b. Local imperfections in frame, listing operation c. Selection mechanism d. Incomplete data e. Measurement error 6

7. II. Aggregate and Systemic Components of Survey Risk (Continued) 2. Systemic risk: Large shocks with widespread impact on our data collection and estimation work Ex: Change in administrative record system used to produce frame, auxiliary data Ex: Change in field management; training problems Ex: Errors in programming collection instrument 7

8. II. Aggregate and Systemic Components of Survey Risk (Continued) Additional Examples: Ex: Lack of fit in implicit or explicit models used for imputation, measurement error adjustments Ex: Failure to identify important sources of nonsampling error in preliminary lab studies, pilot tests, field operations Ex: Problems in implementation of rules for edit, imputation, allocation 8

9. II. Aggregate and Systemic Components of Survey Risk (Continued) B. Historical Pattern: 1. Research community has focused on characterization, modeling and management of aggregate risk 2. Survey managers often worry as much, or more, about systemic risk 9 A. Origins of Systemic Errors: Mixed linear or hierarchical Bayes models Note especially that many systemic components riskill affect many (or all) data collectedA. Origins of Systemic Errors: Mixed linear or hierarchical Bayes models Note especially that many systemic components riskill affect many (or all) data collected

10. II. Aggregate and Systemic Components of Survey Risk (Continued) C. Potential models for systemic risk: 1. Generalized mixed linear models, e.g., Y_ij = X_ij A + Z_ij B + u_i + e_ij X: Design variables Z: Observed paradata A, B: Coefficient vectors u_i: Regional or other coarse random effects e_ij: Deviation specific to unit j in region I Reflect systemic effects in coefficients A, B or coarse random effect u_i 10

11. II. Aggregate and Systemic Components of Survey Risk (Continued) 2. Hierarchical Bayes models 3. Life-testing models for identification of software problems 4. For all potential models, issues with a. Model identification information b. Limited effective sample sizes 11

12. III. Impact of, and Recovery from, Systemic Errors A. Four Possible Outcomes from Systemic Error 1. Perfect resilience 2. Substantial degradation in quality with slow recovery 3. Moderate degradation, again with slow recovery 4. Substantial degradation with rapid recovery (cf. literature on “recoverable computing”) 12

14. III. Impact of, and Recovery from, Systemic Errors (Continued) B. Features of outcomes in the four graphs arising from: 1. Population characteristics (e.g., decay rate of systemic shocks) 2. Survey design (design features to buffer systemic shocks) 3. Ad hoc direct intervention 14

15. IV. Adjusting Design Features to Account for Systemic Error Components A. Potential Goals: 1. Prevent systemic errors from arising 2. Limit the effects of systemic errors in current production sample (cf. discussion of weighting methods that are robust against model misspecification) 3. Timely identification of, and direct adjustment for, systemic errors (cf. responsive or multi-phase designs) 15

16. IV. Adjusting Design Features to Account for Systemic Error Components (Continued) B. Tools to Prevent, or Reduce the Impact of, Systemic Errors (cf. “The Three Es” of Risk Management) 1. Enforcement - Formal management directives - Incentive structures in organization 16

17. IV. Adjusting Design Features to Account for Systemic Error Components (Continued) 2. Education: Training of methodologists, managers, field personnel 3. Engineering: Modification of Methodological and Technological Features of Survey Processes 17

18. V. Closing Remarks A. Survey Risk: Likelihood and impact of failure in one or more dimensions of survey quality, including TSE components B. Important for methodologists to study both aggregate and systemic components of risk C. Consider design work to identify and ameliorate systemic components 18

19. V. Closing Remarks (Continued) D. Discussion Questions 1. Have you (or your survey organization) encountered forms of systemic errors? 2. If so: a. What are their dominant features? b. What models have you used to describe the errors, and their effect on data quality? c. What changes in design or estimation methods have you used to ameliorate their effects? 19

20. Contact Information John L. Eltinge Associate Commissioner Office of Survey Methods Research www.bls.gov 202-691-7404 Eltinge.John@bls.gov