7. Introduction to the main SDMX objects for metadata exchange Fernando H MORENTE ORIA Eurostat, Unit B.5 Data and metadata services; standards European Statistical Training Programme (ESTP) SDMX Basics course, 5-7 March 2019, Luxembourg

What’s metadata? Information that is given to describe or help you use other information (Cambridge dictionary)

Types of metadata Structural metadata Acting as identifiers and descriptors of the data, such as: dimensions of statistical cubes titles of tables nomenclatures (code lists) Always associated with the data enabling its identification, retrieval and browsing

Types of metadata Reference metadata Acting only as descriptors of the data, they don’t help to actually identify the data They can be of different kinds: conceptual metadata methodological metadata quality metadata (process and output) Can be exchanged independently from the data they are related to, but are however often linked to them The part concerning Metadata assisting interpretation of statistical information is strongly content-oriented. The requirements for metadata depend very much on the subject area and target user groups. A driving force for the minimum metadata set formulated in this group is that as much information as necessary should be made available to ensure a correct interpretation and to avoid misuse. As a result, the list of recommended metadata in this case is quite long covering both the metadata required for the correct interpretation of statistics and the metadata for better assessing the quality and comparability of statistics.

Common artefacts in the implementation of SDMX for data and reference metadata Concept Scheme Code list Dataflow DSD Metadata Metadaflow MSD

Main SDMX metadata objects (similarities) Concept scheme Concept Id, Name (compulsory) and Description (optional)*. Code list Concept Id, Name (compulsory) and Description (optional). Metadataflow - Concept Id, Name (compulsory) and Description (optional). The metadataflow will be linked to a MSD rather than a DSD. * Eurostat uses a SDMX standard reporting structure (SIMS) which is used across domains

Main SDMX metadata objects (similarities) MSD Concepts same role: metadata Attribute The MSD can target any object that can be identified such as a code or concept DSD Concepts different roles: Dimensions, Measures or Attributes

MSD Metadata Target: One or more target objects (Dimension, partial/full key, Dataset, etc) identifying where the metadata information will be available. Report Structure: Comprises a set of metadata attributes. Each metadata attribute identifies a concept that is reported Report Structure ID Concept ref Text format Data_Desc String STAT_UNIT

Eurostat SDMX implementation solution

Why reference metadata? Metadata exists in different formats and comprises different pieces of information ESSMH The ESS’s solution for SDMX based reference metadata Single Integrated Metadata Structure Eurostat’s reporting structure for reference metadata

Eurostat’s reporting structures: SIMS or its subset ESMS structure

Identify/Define Code Lists Like in the case of SDMX data implementation, code lists may be required. Each code list will comprise an unique: ID, Agency ID, Version and Name. Ask the question Reveals first set of minor bullets on mouse click Then reveals the second major bullet

Concept Scheme Domain specific concepts which are not present on the SIMS structure will be included on a different Concept Scheme.

MSD (TOUR) Report Structure The MSD will include those concepts from SIMS or its subset structure plus domain specific concepts included in the Concept Scheme and Code List and Format

MSD (TOUR) Target Objects Report structure is linked to : Dataflow (dataset) Data provider (EU/MSs) Time period (reference year)

Example for European and national reference metadata

ESS Metadata Handler

What is the ESS Metadata Handler? Is a web based application for national and European reference metadata Allows production, exchange and dissemination of metadata across the ESS Implements the ESS metadata standards (SIMS or ESMS) Been in production since 31 January 2014

The ESS Metadata Handler The business process Common user Interface Output produced for the Eurostat Web Other output for Eurostat or external users ESS-MH IT application ESS – Metadata Handler Euro SDMX Registry Input from national metadata Metadata from the Eurostat Domain manager Eurostat as main administrator

National Statistical Institute EUROSTAT EDAMIS National Statistical Institute EUROSTAT ESS Metadata Handler (ESS MH) ESS MH Database Eurostat Website PRODUCTION TREATMENT & ANALYSIS DISSEMINATION National Metadata File

Result National and European reference metadata files that include under a standard structure, sources and summary information regarding data quality and the production process in general

Summary Main SDMX objects for metadata exchange: Concept Scheme, Code lists, Metadataflow and MSD MSD: Concepts same role (metadata attribute) and can target any object that can be identified such as a code or concept Eurosta’s SDMX reference metadata implementation offers SIMS or ESMS reporting structures Eurostat uses ESSMH for metadata compilation, exchange and dissemination

Questions?