1. ESTP course on Statistical Metadata – Introductory course –
Statistical Data and Metadata eXchange
ESTP course on Statistical Metadata – Introductory course –
The ESS Metadata Standards
Statistics Netherlands The Hague, February 2013

2. Overview 0. Introduction to the different types of metadata
1. The ESS Standards for Reference Metadata
1.1. The Euro SDMX Metadata Structure (ESMS)
1.2. The Quality Reporting within Eurostat and the ESS (ESQRS)
1.3. The relation between ESMS and ESQRS
2. The ESS Standards for Structural Metadata
2.1. Harmonisation within Eurostat and the ESS
2.2. Harmonisation within SDMX
3. Conclusion

3. 0. Introduction to the different types of metadata
Reference metadata
Acts only as descriptors of the data, they don’t help to actually identify the data;
They can be of different kinds:
conceptual metadata;
methodological metadata;
quality metadata.
Reference metadata can be exchanged independently from the data they are related to, but are however often linked to them.

4. 0. Introduction to the different types of metadata
Structural metadata
Act as identifiers and descriptors of the data, such as:
dimensions of statistical cubes;
Variables;
titles of tables.
Structural metadata must always be associated with the data to allow their identification, retrieval and browsing.

5. 1. The ESS Standards for Reference Metadata 1. 1
1. The ESS Standards for Reference Metadata 1.1. The Euro SDMX Metadata Structure (ESMS)
Is the standard format for reference metadata in the ESS;
Replaces the former SDDS format since December 2008
All former SDDS files disseminated on Eurostat’s website have been converted into ESMS format.
Is the format to be used for the reporting of national reference metadata files to Eurostat (Commission Recommendation 2009/498/EC of June 2009);
Covers 21 concepts selected from the 66 SDMX cross-domain concepts (also the main quality related concepts);
Is fully SDMX compliant.

6. 1. The ESS Standards for Reference Metadata 1. 1
1. The ESS Standards for Reference Metadata 1.1. The Euro SDMX Metadata Structure (ESMS)

7. 1. The ESS Standards for Reference Metadata 1. 1
1. The ESS Standards for Reference Metadata 1.1. The Euro SDMX Metadata Structure (ESMS)

8. 1. The ESS Standards for Reference Metadata 1. 2
1. The ESS Standards for Reference Metadata 1.2. The Quality reporting within Eurostat and the ESS
Within the European Statistical System (ESS) reporting on statistical data quality exists in many statistical domains…

9. 1. The ESS Standards for Reference Metadata 1. 2
1. The ESS Standards for Reference Metadata 1.2. The Quality reporting within Eurostat and the ESS
… BUT
Quality reports do not exist for all statistical processes within the ESS;
No homogeneity between the different report structures used for data quality reporting;
Not all the quality related information is made publicly available;
No common and standard IT infrastructure is used within the ESS.
The new Eurostat vision: “Improving the production method of EU statistics” requires an improvement action.

10. 1. The ESS Standards for Reference Metadata 1. 2
1. The ESS Standards for Reference Metadata 1.2. The Quality reporting within Eurostat and the ESS
Progress made since 2008
01/2009
Release of the new version of the ESS quality reporting documents:
ESS Standard for Quality Reports (ESQR);
ESS Handbook for Quality Reports (EHQR).
Detailed requirements following the European Statistics Code of Practice;
ESS Quality and Performance Indicators (QPI’s) defined.
03/09
EP/Council Regulation 223/2009
Article 12 defining the quality criteria to be reported.

11. 1. The ESS Standards for Reference Metadata 1. 2
1. The ESS Standards for Reference Metadata 1.2. The Quality reporting within Eurostat and the ESS
The ESQRS (ESS Standard for Quality Reports Structure)
Based on the ESQR, a new report structure - the ESQRS - was created for harmonising the reporting on statistical data quality within the ESS;
The ESQRS is using the main statistical data quality criteria as listed in EP/Council Regulation 223/2009 and as being part of the ESMS and details them further:
Relevance;
Accuracy;
Timeliness and Punctuality;
Accessibility and Clarity;
Comparability;
Coherence.
A subset of the Quality Performance Indicators (QPI’s) is also covered in the new ESQRS.

12. 1. The ESS Standards for Reference Metadata 1. 2
1. The ESS Standards for Reference Metadata 1.2. The Quality reporting within Eurostat and the ESS

13. 1. The ESS Standards for Reference Metadata 1. 3
1. The ESS Standards for Reference Metadata 1.3. Relation between ESMS and ESQRS
ESMS is more oriented to the USERS of statistics
To understand the statistical data released;
There is no need for too detailed information on data quality.
ESQRS is more oriented to the PRODUCERS of statistics
To monitor the quality of the statistics produced in detail;
Concentrating on the main quality concepts (being also part of the ESS Statistics Regulation No 223/2009);
Goes further than the ESMS in terms of quality criteria.
However, there is information on quality criteria which is common to both ESMS and ESQRS.
The metadata produced in the ESMS and ESQRS need to be kept consistent.

14. 1. The ESS Standards for Reference Metadata
Main conclusions
The ESMS is the European standard for the production of reference metadata files within the ESS; it is fully SDMX compliant;
Standards for quality metadata (ESQRS) have been developed and are now also successively used for standard quality reporting;
Some common concepts between ESMS and ESQRS but the target population is not the same:
ESMS for end-users of statistics;
ESQRS for producers of statistics.
An ESMS and ESQRS in parallel is probably not needed in each statistical domain;
Reference metadata need to be reported once and can be reused for different purposes, if needed.

15. 2. The ESS Standards for Structural Metadata
Why do we need code lists?
Collect, disseminate, exchange and organise statistical information;
Aggregate and disaggregate datasets in a meaningful way for complex analysis;
Present statistical information in a standard way;
Support policy and decision making;
Build formulas for calculation and validation;
Standard code lists should be used all along the statistical business process, (i.e. during the phases of data design, data collection, data aggregation, data dissemination and data archiving)

16. 2. The ESS Standards for Structural Metadata
Why is the harmonisation necessary?
To improve and further integrate the statistical business processes within the ESS;
To save costs and increase productivity; however harmonisation means always change from non-standard to standard;
To be able to implement SDMX in statistical domains (use of common code list in Data Structure Definitions).

17. 2. The ESS Standards for Structural Metadata
Two processes for harmonising structural metadata
Harmonisation within the ESS
Eurostat produces and releases harmonised structural metadata (Standard Code Lists – SCLs) covering all the statistical domains;
The main code lists are then proposed for inclusion into the SDMX standards.
Harmonisation within SDMX
The SDMX Content-oriented Guidelines (version 2009) also deals with the harmonisation of structural metadata. Annex 2 recommends some cross-domain code lists to institutions using SDMX.
The two processes run in parallel with a “mutual” impact.
The results of both processes need to be fully consistent.

18. 2. The ESS Standards for Structural Metadata 2. 1
2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat
The harmonisation of code lists (and their maintenance) is made in one Unit in Eurostat (B5) in agreement with all statistical production Units;
The production of standard code lists will also lead to an overall reduction of the code lists in use in the Eurostat Reference database (down from 700 lists at the beginning of the process);
These standard code lists will be gradually included into the domain specific SDMX Data Structure Definitions and they will then also impact NSIs for the data transmission.

19. 2. The ESS Standards for Structural Metadata 2. 1
2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat
Impact and advantages of working with SCLs
Achieve a better quality of the business processes and data with less mapping, less conversions, better comparability across statistical domains;
Each code list, code and code description is clearly defined;
Reduce the workload of the domain managers because they can use pre-defined standards and do not need to create own lists;
Taking into consideration all the needs of statistical domains when creating SCLs;
Necessary pre-condition for the ESS business process harmonisation.

20. 2. The ESS Standards for Structural Metadata 2. 1
2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat
Basic principles used for the standardisation
Each Code List entry must be associated to one single concept;
Cover the requirements of the statistical domains thanks to the close cooperation with Production Units;
Standard Code Lists are defined on the basis of official classifications or widely used other standards;
Additional codes e. g. for aggregates are included whenever needed by statistical domains.

21. 2. The ESS Standards for Structural Metadata 2. 1
2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat
The processes for code list standardisation
Action 1 – Analysis
Analyse code lists content, identify statistical concept, the structure of multidimensional tables and others sources;
Action 2 – Harmonisation
Coherence with other statistical domains;
Discussion with Domain Managers e.g. on the coverage of each code;
Agreement on meaningful short codes and labels.
Action 3 – Finalisation
Management and implementation in the Eurostat Reference database;
Upload on the Eurostat webpage for public dissemination;
Implementation in SDMX data structure definitions (DSDs);
Action 4 – Maintenance
Regular reviewing and possible extension of Standard Code lists.

22. 2. The ESS Standards for Structural Metadata 2. 1
2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat
Basic rules to be used for harmonising code lists
Codes should be meaningful and short (preferably <12 digits). Characters to be used:
Alphanumerical : A, B, C, D, …, Z; 1, 2, 3, 4, …, 0;
Dash “-” (which means “until”);
Underscore ‘_’ (which means “plus”/”and” or simply separation):
e.g. B-E36_F (NACE Rev. 2)
Y_LT1 (AGE).
Avoid the use of ‘leading zeroes’ (codes like ‘000', '01', '001' which may cause problems in Excel);
Positions must respect an appropriate order (especially if code list is hierarchical).

23. 2. The ESS Standards for Structural Metadata 2. 1
2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat
Generic codes remain unchanged in SCL
TOTAL Total
TOT_.... Subtotals
OTH Others
…_OTH Other …..
NAP Not applicable
NRP No response
NSP Not specified
UNK Unknown

24. 2. The ESS Standards for Structural Metadata 2. 1
2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat
Example of a harmonised code list (NACE Rev. 1.1)

25. 2. The ESS Standards for Structural Metadata 2. 1
2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat
Example of the geographical code list (GEO)
It contains codes with labels in 3 language versions;
It covers:
aggregates: %;
countries: %;
regions: %;
organisations: 3 %.
It is used for different synonym code lists like
PARTNER Geopolitical entity (partner);
 C_BIRTH Country/region of birth;
C_LOAD Country/region of loading/embarking;
 C_TRANS Country of transit;
 C_WORK Country/region of work.

26. 2. The ESS Standards for Structural Metadata 2. 1
2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat
Harmonisation of 26 code lists for the 2011 Census
AGE Age
AMENITY Amenities
AREA Area
AREA_OCC Floor area per occupant
BUILDING Type of building
C_BIRTH Country/region of birth
C_WORK Country/region of work
CITIZEN Citizenship
GEO Geopolitical entity (declaring)
HHCOMP Composition of households
HHSTATUS Individuals by household status
HOUSING Housing
ISCED97 International Standard Classification of Education
(1997) …

27. 2. The ESS Standards for Structural Metadata 2. 1
2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat
Metadata – Standard Code Lists stored in RAMON
Different formats available: HTML, CSV, XML

28. 2. The ESS Standards for Structural Metadata 2. 2
2. The ESS Standards for Structural Metadata 2.2. The harmonisation at SDMX level
Responsible: The SDMX Statistical Working Group (SWG)
Composed of representatives from several SDMX organisations and NSIs;
Revision of the existing SDMX cross-domain code lists (Annex 2 of the COG);
Approval of further standard code lists proposed by the organisations;
Advice/recommendation on the use of code lists for global DSDs shared by several institutions.

29. 2. The ESS Standards for Structural Metadata 2. 2
2. The ESS Standards for Structural Metadata 2.2. The harmonisation at SDMX level
Annex 2 of the SDMX Content-oriented Guidelines recommends the following 9 harmonized code lists to institutions applying SDMX

30. 2. The ESS Standards for Structural Metadata 2. 2
2. The ESS Standards for Structural Metadata 2.2. The harmonisation at SDMX level
The next steps at Eurostat-ESS level
Further harmonisation of the code lists, and implementation in the ESTAT database for dissemination;
Further use of SCLs in DSDs when implementing SDMX;
SCLs in preparation, priority to statistical domains where SDMX is being implemented:
National Accounts;
Balance of Payments;
Research&Development;
etc…
Further promote newly harmonised cross-domain code lists to the SDMX SWG for approval in the COG.

31. 2. The ESS Standards for Structural Metadata 2. 2
2. The ESS Standards for Structural Metadata 2.2. The harmonisation at SDMX level
The next steps at SDMX level
The main tasks concern:
Further work within the SDMX SWG;
Further adoption of additional cross-domain code lists (extension of the COG);
Ensure the use of standard code lists in Global DSDs shared by several organisations.

32. 3. Conclusions
Harmonisation of Metadata is crucial to facilitate the exchange of Data and Metadata between International organisations and National Statistical Institutes within the ESS and beyond;
The standards for Reference Metadata and Quality Reporting are defined (ESMS, ESQRS…) and more and more implemented within the ESS;
The harmonisation of Structural Metadata (standard code lists) is progressing and needs further work and implementation at ESS and SDMX world-wide levels;
National Statistical Institutes will be highly involved in the use of all these Standards.

33. Questions