1. Output Quality of Multisource Statistics
Ton de Waal
15 March, 2017

2. Overview Komuso: ESSnet on quality of multisource statistics
Basic data configurations (BDCs) and some work done
For all BDCs literature reviews have been carried out
For most BDCs suitability tests have been done
Conclusions

3. Komuso: ESSnet on quality of multisource statistics
Komuso is part of ESS.VIP Admin Project
During first Specific Grant Agreement (January 2016 until April 2017) four Work Packages (WPs) have been defined:
WP 1: Evaluating the quality of input data
WP 2: Methodology for the assessment of the quality of frames for social statistics
WP 3: Framework for the quality evaluation of statistical output based on multiple sources
WP 4: Communication with respect to the ESSnet

4. Komuso: ESSnet on quality of multisource statistics
Komuso is part of ESS.VIP Admin Project
During first Specific Grant Agreement (January 2016 until April 2017) four Work Packages (WPs) have been defined:
WP 1: Evaluating the quality of input data
WP 2: Methodology for the assessment of the quality of frames for social statistics
WP 3: Framework for the quality evaluation of statistical output based on multiple sources
WP 4: Communication with respect to the ESSnet

5. BDC 1: The baseline
Several data sources with non-overlapping units that together cover complete population
Estimates from data sources can simply be added
Even in this “simple” case important problems occur, such as
Progressiveness of data
Unit problems, e.g. classification into domains

6. Some results for BDC 1
Statistics Netherlands has examined the effect of errors in the NACE code classification on growth rates of enterprise statistics broken down by NACE code

7. BDC 2: Partly overlapping units/variables
Part of variables and units in data sources overlap
Observed value in one data source may differ from observed value in other data source
Options:
Micro-integration
Latent class models
Structural equation models

8. Some results for BDC 2
ISTAT has examined multiple administrative and survey sources that provide the value of the same variable of interest
A Latent Class model can be used to estimate the true values
Estimates of the probabilities Pr⁡( 𝑌 𝑔 =𝑖|𝑋=𝑖), where 𝑌 𝑔 is the observed value in data source 𝑔 and 𝑋 is the true (latent) value, can be used to evaluate the accuracy of data source 𝑔

9. Some results for BDC 2
Statistics Austria has analysed a quality framework that can be used when several data sources with possibly conflicting values for common variables are available.
The quality framework models errors in variables in these data sources as well as systematically uses expert knowledge.

10. BDC 3: Partly overlapping units/variables with under-coverage
Part of variables and units in data sources may overlap
Under-coverage occurs
Options:
Capture-recapture methods

11. BDC 4: Microdata and aggregated data
Microdata are combined with aggregated data
Inconsistencies between microdata and aggregated data should be avoided
Especially complicated if aggregated data are estimates
Example: Dutch virtual Population Census
Options:
Repeated weighting
Calibrated imputation
Macro-integration

12. Some results for BDC 4
Statistics Netherlands and Statistics Norway have been working on quality measures that can be applied to BDC 4
Many macro-economic figures are connected by constraints (“accounting equations”)
Input estimates usually do not automatically satisfy accounting equations due to measurement and sampling errors
Estimation involves a reconciliation step by which the input estimates are modified
An accounting equation is considered as a single entity and scalar quality measures have been defined
These measures capture the adjustment effect as well as the relative contribution of the various input estimates to the final estimated account

13. BDC 5: Only aggregated data
Sometimes only aggregated data are combined with each other
Example: National Accounts
Options:
Macro-integration

14. Some results for BDC 5
Same method that has been developed by Statistics Netherlands and Statistics Norway for BDC 4 can be applied to BDC 5

15. BDC 6: Longitudinal data
Combining longitudinal data with different frequencies
Example: combining turnover data from monthly survey with (more accurate) quarterly data from Tax Office
Problem: calibrate monthly data on quarterly data while preserve month-to-month growth
Option:
Benchmarking techniques

16. Conclusions Much work has been done More work is needed
simplifying some of the quality measures, methods to compute them, and the use of these measures/methods in practice
extending the range of situations in which the quality measures and methods to compute them can be applied
examining quality measures relating to “coherence” in more detail