1. IAOS 2014 Conference – Meeting the Demands of a Changing World Da Nang, Vietnam, 8-10 October 2014 Diagnosing the Imputation of Missing Values in Official Economic Statistics via Multiple Imputation: Unveiling the Invisible Missing Values National Statistics Center (Japan) Masayoshi Takahashi Notes: The views and opinions expressed in this presentation are the authors’ own, not necessarily those of the institution.

2. Outline 1. Problems of Missing Values and Imputation 2. Theory of MI and the EMB Algorithm 3. Mechanism Behind the Diagnostic Algorithm 4. Data and Missing Mechanism 5. Assessment of the Diagnostic Algorithm 6. Conclusions and Future Work 1

3. Problems of Missing Values  Prevalence of missing values  Effects of missing values Reduction in efficiency Introduction of bias  Assumptions and solution Missing At Random (MAR) Imputation 2 1. Problems of Missing Values and Imputation

4. Problematic Nature of Single Imputation (SI) 3 1. Problems of Missing Values and Imputation Deterministic SI Stochastic SI There is only one set of regression coefficients. Random noise ^ = OLS estimate

5. Multiple Imputation (MI) Comes for Rescue 4 2. Theory of Multiple Imputation and the EMB Algorithm ~ = random sampling from a posterior distribution Multiple sets of regression coefficients Need multiple values of

6. Likelihood of Observed Data 5 2. Theory of Multiple Imputation and the EMB Algorithm Random sampling from observed likelihood  Various computation algorithms  Not easy!! Solution

7. Computational Algorithms  EMB algorithm Expectation-Maximization Bootstrapping Most computationally efficient  Other MI algorithms MCMC FCS 6 2. Theory of Multiple Imputation and the EMB Algorithm

8. Graphical Presentation of the EMB Algorithm 7 2. Theory of Multiple Imputation and the EMB Algorithm

9. Paradox in Imputation  Imputed values Estimates, not true values Diagnosis  True values Always missing Cannot compare the imputed values with the truth  How do we go about imputation diagnostics? 8 3. Mechanism Behind the Diagnostic Algorithm

10. Solution to the Paradox  Indirect diagnostics of imputation Abayomi, Gelman, and Levy (2008) Honaker and King (2010)  MI Within-imputation variance Between-imputation variance 9 3. Mechanism Behind the Diagnostic Algorithm

11. Disadvantage of multiple imputation  Dozens of imputed datasets  Computational burden  Multiple values for one cell  Unrealistic to directly use in official statistics 10 3. Mechanism Behind the Diagnostic Algorithm

12. Proposal in this Research  Two-step procedure Imputation step: Stochastic SI Diagnostic step: MI  Advantage Can have only one imputed value  Advantage of SI Can know the confidence about each imputed value  Advantage of MI 11 3. Mechanism Behind the Diagnostic Algorithm  New!!

13. Multiple Imputation as a Diagnostic Tool  Variation among M imputed datasets Estimation uncertainty in imputation  Our diagnostic algorithm Utilizes this variability Can examine the stability & confidence of imputation models  What does this mean? See the next slide for illustration 12 3. Mechanism Behind the Diagnostic Algorithm

14. Illustration: Two Cases of Variation in Imputations 13 3. Mechanism Behind the Diagnostic Algorithm

15. Mathematical Representation 14 3. Mechanism Behind the Diagnostic Algorithm Imputation Step: Stochastic SI Diagnostic Step: MI If, then no uncertainties What we actually check is whether

16. Data  Multivariate log-normal distribution  Mean vector & variance-covariance matrix Simulated dataset Manufacturing Sector 2012 Japanese Economic Census  Number of observations 1,000  Variables turnover, capital, worker 15 4. Data and Missing Mechanism

17. Missing Mechanism  Target variable turnover  Missing rate 20%  Missing mechanism MAR A logistic regression to estimate the probability of missingness according to the values of explanatory variables (capital and worker) 16 4. Data and Missing Mechanism

18. R-Function diagimpute  New function developed in R  Graphical detection of problematic imputations as outliers  Graphical presentation of the stability of imputation via control chart  Not yet publicly available A work in progress Once finalized, planning to make it publicly available 17 5. Assessment of the Diagnostic Algorithm

19. Preliminary Result 1 18 5. Assessment of the Diagnostic Algorithm

20. Preliminary Result 2 19 5. Assessment of the Diagnostic Algorithm

21. Conclusions  MI as a diagnostic tool A novel way  Diagnostic algorithm Still a work in progress A preliminary assessment given Useful to detect problematic imputations  Help us strengthen the validness of official economic statistics. 20 6. Conclusions and Future Work

22. Future Work  Intend to further refine the algorithm  Test it against a variety of real datasets  Use several imputation models 21 6. Conclusions and Future Work

23. References 1 1. Abayomi, Kobi, Andrew Gelman, and Marc Levy. (2008). “Diagnostics for Multivariate Imputations,” Applied Statistics vol.57, no.3, pp.273-291. 2. Allison, Paul D. (2002). Missing Data. CA: Sage Publications. 3. Congdon, Peter. (2006). Bayesian Statistical Modelling, Second Edition. West Sussex: John Wiley & Sons Ltd. 4. de Waal, Ton, Jeroen Pannekoek, and Sander Scholtus. (2011). Handbook of Statistical Data Editing and Imputation. Hoboken, NJ: John Wiley & Sons. 5. Honaker, James and Gary King. (2010). “What to do About Missing Values in Time Series Cross-Section Data,” American Journal of Political Science vol.54, no.2, pp.561–581. 6. Honaker, James, Gary King, and Matthew Blackwell. (2011). “Amelia II: A Program for Missing Data,” Journal of Statistical Software vol.45, no.7. 7. King, Gary, James Honaker, Anne Joseph, and Kenneth Scheve. (2001). “Analyzing Incomplete Political Science Data: An Alternative Algorithm for Multiple Imputation,” American Political Science Review vol.95, no.1, pp.49-69. 8. Little, Roderick J. A. and Donald B. Rubin. (2002). Statistical Analysis with Missing Data, Second Edition. New Jersey: John Wiley & Sons. 22

24. References 2 9. Oakland, John S. and Roy F. Followell. (1990). Statistical Process Control: A Practical Guide. Oxford: Heinemann Newnes. 10. Rubin, Donald B. (1978). “Multiple Imputations in Sample Surveys — A Phenomenological Bayesian Approach to Nonresponse,” Proceedings of the Survey Research Methods Section, American Statistical Association, pp.20-34. 11. Rubin, Donald B. (1987). Multiple Imputation for Nonresponse in Surveys. New York: John Wiley & Sons. 12. Schafer, Joseph L. (1997). Analysis of Incomplete Multivariate Data. London: Chapman & Hall/CRC. 13. Scrucca, Luca. (2014). “Package qcc: Quality Control Charts,” http://cran.r- project.org/web/packages/qcc/qcc.pdf.http://cran.r- project.org/web/packages/qcc/qcc.pdf 14. Statistics Bureau of Japan. (2012). “Economic Census for Business Activity,” http://www.stat.go.jp/english/data/e-census/2012/index.htm. http://www.stat.go.jp/english/data/e-census/2012/index.htm 15. Takahashi, Masayoshi and Takayuki Ito. (2012). “Multiple Imputation of Turnover in EDINET Data: Toward the Improvement of Imputation for the Economic Census,” Work Session on Statistical Data Editing, UNECE, Oslo, Norway, September 24-26, 2012. 23

25. References 3 16. Takahashi, Masayoshi and Takayuki Ito. (2013). “Multiple Imputation of Missing Values in Economic Surveys: Comparison of Competing Algorithms,” Proceedings of the 59 th World Statistics Congress of the International Statistical Institute, Hong Kong, China, August 25-30, 2013, pp.3240-3245. 17. Takahashi, Masayoshi. (2014a). “An Assessment of Automatic Editing via the Contamination Model and Multiple Imputation,” Work Session on Statistical Data Editing, United Nations Economic Commission for Europe, Paris, France, April 28-30, 2014. 18. Takahashi, Masayoshi. (2014b). “Keiryouchi Data no Kanrizu (Control Chart for Continuous Data),” Excel de Hajimeru Keizai Toukei Data no Bunseki (Statistical Data Analysis for Economists Using Excel), 3 rd edition. Tokyo: Zaidan Houjin Nihon Toukei Kyoukai.. 19. van Buuren, Stef. (2012). Flexible Imputation of Missing Data. London: Chapman & Hall/CRC. 24

26. Thank you 25