1. Better data quality through global data and metadata sharing
Agne Bikauskaite and Håkan Linden
Eurostat
European Conference on Quality in Official Statistics (Q2014)
Vienna, 3-5 June 2014

2. Outline Context A data sharing model The necessary preconditions
Implementing Eurostat's data sharing strategy
Conclusions and outlook

3. Context General objectives Reduce reporting burden on NSIs
More efficient use of resources in International Organisation (IO)
Ensure high quality and consistency of data of official statistics
Improve global data exchange and dissemination

4. European statistics: From national to Eurostat
A data sharing model
Eurostat
Data Validation
EU Member state
EU Member state
EU Member state
EU Member state
European statistics: From national to Eurostat

5. A data sharing model U S E R
EU countries
OECD countries (non-EU countries only)
Other countries (non-OECD countries only)
Eurostat - ECB
OECD
IMF, UN, WB, ILO, BIS, other IOs U S E R
Eurostat as international hub for European statistics

6. The necessary pre-conditions
Internationally agreed technical and statistical standards
Internationally agreed data structures
Maintenance agreements
Internationally agreed data validation
Streamlined data exchange processes

7. Statistical Data and Metadata Exchange (SDMX)
It consists of technical and statistical standards, guidelines, an IT service infrastructure and IT tools.
SDMX provides
technical/statistical standards
new exchange modes (hubs)
clear rules and responsibilities
SDMX
ISO IS 17369

8. SDMX describes the data and metadata exchange
Provision Agreement
Organisation scheme
SDMX Registry
maintainer
Concept Schemes
Code lists
DSDs
Concepts 8

9. Describing the data exchange
Who?
When?
Who?
How?
Where?
What?
What? 9 9 2

10. Content-Oriented guidelines
Cross-domain concepts and code lists
Statistical subject-matter domains
Metadata common vocabulary
Recommendations to harmonise implementations
Organisation 1
Organisation 2
Organisation 3
interoperability 10

11. Implementing Eurostat's data sharing strategy
Standardisation of structural metadata
Code lists describe dimensions in data tables, giving a meaning to the data.
Code lists are based on:
official statistical classifications such as NACE, NUTS, ISCO, etc.
The ESS and SDMX Content Oriented Guidelines
domain specific codifications
A standard code list is a code list already harmonised
Standard code lists should be used all along the statistical business process: data design, collection, aggregation, dissemination, exchange, archiving.

12. Implementing Eurostat's data sharing strategy Recommendations for the SCL creation
RECOMMENDED RULES
ESS
SDMX
COMMENTS
Input: Official information ⱴ
Coding
A-Z _
A-Z _
In SDMX “–“ (dash) is not allowed (to avoid confusion with operator "minus")
Codes starting with letter
With some exceptions
Meaningful coding
Less homogeneity in coding in SDMX (due to involvement of several different partners)
Aggregates are possible
To be used all along the statistical business process
May be referenced by several statistical concepts
Based on clear guidelines
Maintenance agency
ESS: Eurostat Unit B5
SDMX: Statistical Working Group (SWG)
Versioning system
In future registries
Generic concept
In SDMX is special CL for generic codes
In ESS generic codes are implemented in each SCL when it is needed

13. Implementing Eurostat's data sharing strategy SDMX standards into ESS structural metadata
In purpose to improve quality of the data comparability and clarity is needed:
To use identical SCLs in the ESS and in the SDMX
To transpose the SDMX guidelines into the ESS code lists
To adapt the ESS standard codes into the SDMX DSDs

14. Implementing Eurostat's data sharing strategy Overview of the ESS SCLs
504 ESS CLs
194 ESS SCLs released in Ramon
12 fully SDMX compliant
110 SDMX compliant (except Generic codes)

15. Implementing Eurostat's data sharing strategy Standardisation of Reference Metadata
ESMS
Euro SDMX Metadata Structure
ESQRS
ESS Standard for Quality Reports Structure
EPMS
Eurostat Process Metadata Structure

16. Implementing Eurostat's Reference metadata sharing strategy
WASTE (end of life vehicles, packaging, electronic waste)
WINE
FARM STRUCTURE
MIP STATISTICS
HICP/ Compliance monitoring
EHIS (Education, health and social protection)
R&D (CIS 2012)
Annual crops
PRAG
ESAW
AES (Education, Science and Culture)
LCI (Labour Cost Index)
INFOSOC (Information Society)
BUSINESS REGISTER
HICP
LFS-Q, LFS-A
EU-SILC
FATS
STS (Short Term Statistics)
WASTE
AEI (Pesticides)
EDUCAT
JVC (Job Vacancy Stats)
PRODCOM
EXTERNAL TRADE (3rd countries)
COSAEA
URBANREG
R&D
TOURISM
PERMANENT CROPS
CENSUS
HOUSING PRICES HPS
Over 30 Eurostat domains are in various phases of ESS Reference metadata standardisation.
This concerns about 35% of all eligible Eurostat processes.

17. Implementing Eurostat's data sharing strategy The Eurostat established methodology17

18. Implementing Eurostat's data sharing strategy in ESS

19. Implementing Eurostat's data sharing strategy Development of the technical infrastructure
Key components:
SDMX Registries
The Euro-SDMX Registry
The Global SDMX Registry
SDMX Reference Infrastructure (SDMX-RI)

20. Implementing Eurostat's data sharing strategy What is the EuroSDMX Registry(SER)?
Eurostat's implementation of the SDMX Registry specifications as published by the SDMX initiative sdmx.org.
Based on SDMX 2.1 (as published on April 2011) Also capable of importing and exporting SDMX 2.0 artefacts.
Allows browsing, searching, editing and subscribing to artefacts.
Advanced access control mechanism for distributed maintenance of artefacts controlling also their visibility.

21. Access to the content of the Registry advanced search
Home page
Access to the content of the Registry advanced search
Access to the content of the Registry text search
Access to the content of the Registry by type
Most recent items

22. Conclusions
International data co-operation improves the production of accurate, comparable and coherent statistics;
SDMX promotes an incremental movement toward the data and metadata sharing model;
The increasing use of SDMX based statistical standards improves the quality of the underlying statistical processes;
The SDMX technical standards pave the ways for simplified exchange and dissemination processes helping to improve also timeliness and accessibility;
Statistical integration needs to go hand-in-hand with technical integration and standardisation.

23. Outlook
Much more global data and metadata sharing in the years to come;
Common data validation and processing procedures are required (from structural validation to content information validation);
Better metadata driven statistics production systems: the use of standards throughout the processes in combination with common metadata registries ;
Better harmonised international reference metadata frameworks and templates;
Broadening the scope of SDMX (versioning of codes, disabling of dimensions, other formats like CSV, flat files etc.);
Interoperability between information models (GSIM, SDMX, DDI etc.).