ESTP course on Statistical Metadata – Introductory course – Statistical Data and Metadata eXchange ESTP course on Statistical Metadata – Introductory course – The ESS Metadata Standards Statistics Netherlands The Hague, 18-19 February 2013

Overview 0. Introduction to the different types of metadata 1. The ESS Standards for Reference Metadata 1.1. The Euro SDMX Metadata Structure (ESMS) 1.2. The Quality Reporting within Eurostat and the ESS (ESQRS) 1.3. The relation between ESMS and ESQRS 2. The ESS Standards for Structural Metadata 2.1. Harmonisation within Eurostat and the ESS 2.2. Harmonisation within SDMX 3. Conclusion

0. Introduction to the different types of metadata Reference metadata Acts only as descriptors of the data, they don’t help to actually identify the data; They can be of different kinds: conceptual metadata; methodological metadata; quality metadata. Reference metadata can be exchanged independently from the data they are related to, but are however often linked to them.

0. Introduction to the different types of metadata Structural metadata Act as identifiers and descriptors of the data, such as: dimensions of statistical cubes; Variables; titles of tables. Structural metadata must always be associated with the data to allow their identification, retrieval and browsing.

1. The ESS Standards for Reference Metadata 1. 1 1. The ESS Standards for Reference Metadata 1.1. The Euro SDMX Metadata Structure (ESMS) Is the standard format for reference metadata in the ESS; Replaces the former SDDS format since December 2008 All former SDDS files disseminated on Eurostat’s website have been converted into ESMS format. Is the format to be used for the reporting of national reference metadata files to Eurostat (Commission Recommendation 2009/498/EC of June 2009); Covers 21 concepts selected from the 66 SDMX cross-domain concepts (also the main quality related concepts); Is fully SDMX compliant.

1. The ESS Standards for Reference Metadata 1. 1 1. The ESS Standards for Reference Metadata 1.1. The Euro SDMX Metadata Structure (ESMS)

1. The ESS Standards for Reference Metadata 1. 1 1. The ESS Standards for Reference Metadata 1.1. The Euro SDMX Metadata Structure (ESMS)

1. The ESS Standards for Reference Metadata 1. 2 1. The ESS Standards for Reference Metadata 1.2. The Quality reporting within Eurostat and the ESS Within the European Statistical System (ESS) reporting on statistical data quality exists in many statistical domains…

1. The ESS Standards for Reference Metadata 1. 2 1. The ESS Standards for Reference Metadata 1.2. The Quality reporting within Eurostat and the ESS … BUT Quality reports do not exist for all statistical processes within the ESS; No homogeneity between the different report structures used for data quality reporting; Not all the quality related information is made publicly available; No common and standard IT infrastructure is used within the ESS. The new Eurostat vision: “Improving the production method of EU statistics” requires an improvement action.

1. The ESS Standards for Reference Metadata 1. 2 1. The ESS Standards for Reference Metadata 1.2. The Quality reporting within Eurostat and the ESS Progress made since 2008 01/2009 Release of the new version of the ESS quality reporting documents: ESS Standard for Quality Reports (ESQR); ESS Handbook for Quality Reports (EHQR). Detailed requirements following the European Statistics Code of Practice; ESS Quality and Performance Indicators (QPI’s) defined. 03/09 EP/Council Regulation 223/2009 Article 12 defining the quality criteria to be reported.

1. The ESS Standards for Reference Metadata 1. 2 1. The ESS Standards for Reference Metadata 1.2. The Quality reporting within Eurostat and the ESS The ESQRS (ESS Standard for Quality Reports Structure) Based on the ESQR, a new report structure - the ESQRS - was created for harmonising the reporting on statistical data quality within the ESS; The ESQRS is using the main statistical data quality criteria as listed in EP/Council Regulation 223/2009 and as being part of the ESMS and details them further: Relevance; Accuracy; Timeliness and Punctuality; Accessibility and Clarity; Comparability; Coherence. A subset of the Quality Performance Indicators (QPI’s) is also covered in the new ESQRS.

1. The ESS Standards for Reference Metadata 1. 2 1. The ESS Standards for Reference Metadata 1.2. The Quality reporting within Eurostat and the ESS

1. The ESS Standards for Reference Metadata 1. 3 1. The ESS Standards for Reference Metadata 1.3. Relation between ESMS and ESQRS ESMS is more oriented to the USERS of statistics To understand the statistical data released; There is no need for too detailed information on data quality. ESQRS is more oriented to the PRODUCERS of statistics To monitor the quality of the statistics produced in detail; Concentrating on the main quality concepts (being also part of the ESS Statistics Regulation No 223/2009); Goes further than the ESMS in terms of quality criteria. However, there is information on quality criteria which is common to both ESMS and ESQRS. The metadata produced in the ESMS and ESQRS need to be kept consistent.

1. The ESS Standards for Reference Metadata Main conclusions The ESMS is the European standard for the production of reference metadata files within the ESS; it is fully SDMX compliant; Standards for quality metadata (ESQRS) have been developed and are now also successively used for standard quality reporting; Some common concepts between ESMS and ESQRS but the target population is not the same: ESMS for end-users of statistics; ESQRS for producers of statistics. An ESMS and ESQRS in parallel is probably not needed in each statistical domain; Reference metadata need to be reported once and can be reused for different purposes, if needed.

2. The ESS Standards for Structural Metadata Why do we need code lists? Collect, disseminate, exchange and organise statistical information; Aggregate and disaggregate datasets in a meaningful way for complex analysis; Present statistical information in a standard way; Support policy and decision making; Build formulas for calculation and validation; Standard code lists should be used all along the statistical business process, (i.e. during the phases of data design, data collection, data aggregation, data dissemination and data archiving)

2. The ESS Standards for Structural Metadata Why is the harmonisation necessary? To improve and further integrate the statistical business processes within the ESS; To save costs and increase productivity; however harmonisation means always change from non-standard to standard; To be able to implement SDMX in statistical domains (use of common code list in Data Structure Definitions).

2. The ESS Standards for Structural Metadata Two processes for harmonising structural metadata Harmonisation within the ESS Eurostat produces and releases harmonised structural metadata (Standard Code Lists – SCLs) covering all the statistical domains; The main code lists are then proposed for inclusion into the SDMX standards. Harmonisation within SDMX The SDMX Content-oriented Guidelines (version 2009) also deals with the harmonisation of structural metadata. Annex 2 recommends some cross-domain code lists to institutions using SDMX. The two processes run in parallel with a “mutual” impact. The results of both processes need to be fully consistent.

2. The ESS Standards for Structural Metadata 2. 1 2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat The harmonisation of code lists (and their maintenance) is made in one Unit in Eurostat (B5) in agreement with all statistical production Units; The production of standard code lists will also lead to an overall reduction of the code lists in use in the Eurostat Reference database (down from 700 lists at the beginning of the process); These standard code lists will be gradually included into the domain specific SDMX Data Structure Definitions and they will then also impact NSIs for the data transmission.

2. The ESS Standards for Structural Metadata 2. 1 2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat Impact and advantages of working with SCLs Achieve a better quality of the business processes and data with less mapping, less conversions, better comparability across statistical domains; Each code list, code and code description is clearly defined; Reduce the workload of the domain managers because they can use pre-defined standards and do not need to create own lists; Taking into consideration all the needs of statistical domains when creating SCLs; Necessary pre-condition for the ESS business process harmonisation.

2. The ESS Standards for Structural Metadata 2. 1 2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat Basic principles used for the standardisation Each Code List entry must be associated to one single concept; Cover the requirements of the statistical domains thanks to the close cooperation with Production Units; Standard Code Lists are defined on the basis of official classifications or widely used other standards; Additional codes e. g. for aggregates are included whenever needed by statistical domains.

2. The ESS Standards for Structural Metadata 2. 1 2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat The processes for code list standardisation Action 1 – Analysis Analyse code lists content, identify statistical concept, the structure of multidimensional tables and others sources; Action 2 – Harmonisation Coherence with other statistical domains; Discussion with Domain Managers e.g. on the coverage of each code; Agreement on meaningful short codes and labels. Action 3 – Finalisation Management and implementation in the Eurostat Reference database; Upload on the Eurostat webpage for public dissemination; Implementation in SDMX data structure definitions (DSDs); Action 4 – Maintenance Regular reviewing and possible extension of Standard Code lists.

2. The ESS Standards for Structural Metadata 2. 1 2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat Basic rules to be used for harmonising code lists Codes should be meaningful and short (preferably <12 digits). Characters to be used: Alphanumerical : A, B, C, D, …, Z; 1, 2, 3, 4, …, 0; Dash “-” (which means “until”); Underscore ‘_’ (which means “plus”/”and” or simply separation): e.g. B-E36_F (NACE Rev. 2) Y_LT1 (AGE). Avoid the use of ‘leading zeroes’ (codes like ‘000', '01', '001' which may cause problems in Excel); Positions must respect an appropriate order (especially if code list is hierarchical).

2. The ESS Standards for Structural Metadata 2. 1 2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat Generic codes remain unchanged in SCL TOTAL Total TOT_.... Subtotals OTH Others …_OTH Other ….. NAP Not applicable NRP No response NSP Not specified UNK Unknown

2. The ESS Standards for Structural Metadata 2. 1 2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat Example of a harmonised code list (NACE Rev. 1.1)

2. The ESS Standards for Structural Metadata 2. 1 2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat Example of the geographical code list (GEO) It contains 4 200 codes with labels in 3 language versions; It covers: aggregates: 4 %; countries: 6 %; regions: 86 %; organisations: 3 %. It is used for different synonym code lists like PARTNER Geopolitical entity (partner);  C_BIRTH Country/region of birth; C_LOAD Country/region of loading/embarking;  C_TRANS Country of transit;  C_WORK Country/region of work.

2. The ESS Standards for Structural Metadata 2. 1 2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat Harmonisation of 26 code lists for the 2011 Census AGE Age AMENITY Amenities AREA Area AREA_OCC Floor area per occupant BUILDING Type of building C_BIRTH Country/region of birth C_WORK Country/region of work CITIZEN Citizenship GEO Geopolitical entity (declaring) HHCOMP Composition of households HHSTATUS Individuals by household status HOUSING Housing ISCED97 International Standard Classification of Education (1997) …

2. The ESS Standards for Structural Metadata 2. 1 2. The ESS Standards for Structural Metadata 2.1. The harmonisation within Eurostat Metadata – Standard Code Lists stored in RAMON Different formats available: HTML, CSV, XML

2. The ESS Standards for Structural Metadata 2. 2 2. The ESS Standards for Structural Metadata 2.2. The harmonisation at SDMX level Responsible: The SDMX Statistical Working Group (SWG) Composed of representatives from several SDMX organisations and NSIs; Revision of the existing SDMX cross-domain code lists (Annex 2 of the COG); Approval of further standard code lists proposed by the organisations; Advice/recommendation on the use of code lists for global DSDs shared by several institutions.

2. The ESS Standards for Structural Metadata 2. 2 2. The ESS Standards for Structural Metadata 2.2. The harmonisation at SDMX level Annex 2 of the SDMX Content-oriented Guidelines recommends the following 9 harmonized code lists to institutions applying SDMX

2. The ESS Standards for Structural Metadata 2. 2 2. The ESS Standards for Structural Metadata 2.2. The harmonisation at SDMX level The next steps at Eurostat-ESS level Further harmonisation of the code lists, and implementation in the ESTAT database for dissemination; Further use of SCLs in DSDs when implementing SDMX; SCLs in preparation, priority to statistical domains where SDMX is being implemented: National Accounts; Balance of Payments; Research&Development; etc… Further promote newly harmonised cross-domain code lists to the SDMX SWG for approval in the COG.

2. The ESS Standards for Structural Metadata 2. 2 2. The ESS Standards for Structural Metadata 2.2. The harmonisation at SDMX level The next steps at SDMX level The main tasks concern: Further work within the SDMX SWG; Further adoption of additional cross-domain code lists (extension of the COG); Ensure the use of standard code lists in Global DSDs shared by several organisations.

3. Conclusions Harmonisation of Metadata is crucial to facilitate the exchange of Data and Metadata between International organisations and National Statistical Institutes within the ESS and beyond; The standards for Reference Metadata and Quality Reporting are defined (ESMS, ESQRS…) and more and more implemented within the ESS; The harmonisation of Structural Metadata (standard code lists) is progressing and needs further work and implementation at ESS and SDMX world-wide levels; National Statistical Institutes will be highly involved in the use of all these Standards.

Questions