Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, March 2016

Eurostat The SDMX Components 2 IT Infrastructure for exchange and sharing Content Oriented Guidelines SDMX Information Model

Eurostat Who? What? When? Who? How? Describing the data exchange What's next?

Eurostat EUROSTAT SDMX TOOLS FOR DESIGNERS 4

Eurostat Eurostat SDMX tools for designers 5 Tools offeredUse Data Structure WizardUsed to create, edit and test SDMX objects Maintenance of SDMX registry SDMX RegistryA central repository for storing and sharing SDMX objects

Eurostat Eurostat SDMX tools for designers 6 Data Structure Wizard SDMX Registry Create/Edit Store/Edit Web service

Work roles and process SDMX Registry Create DSD Publish DSD Format / structure SDMX messages Consult and get DSD Send SDMX messages COLLECTING ORGANISATION Consult and get DSD Interpret SDMX messages 7

Data Structure Wizard Creation / edition of DSDs Validation of DSD Local repository Artefacts are stored in a local repository Remote repository Can be used for retrieving existing artefacts 8

DSW: Management of SDMX artefacts 9

DSW: Artefact example: the codelists 10

Eurostat SDMX Registry 11 Metadata registry which implements the SDMX registry specificationsSDMX Tasked with providing structure, organisation, maintenance and query interfaces for the SDMX components required to support the data sharing vision Provides a web-based user interface as well as web services for interacting with the SDMX structural metadata objects in use within the collecting organisation and with statistical partners.

Eurostat SDMX Registry 12 Only concerned with providing information need to access the data and reference metadata sets. An application which wants a particular data or metadata set would then query the registry for the structure, and then go and retrieve the data or metadata set directly from the provider's web server. Example of existing SDMX registries: Euro SDMX Registry (2.1) SDMX Global Registry (2.1)

Eurostat New Euro SDMX Registry (2.1)  Eurostat's implementation of the SDMX Registry  Based on SDMX 2.1 (as published on April 2011) Also capable of importing and exporting SDMX 2.0 artefacts.  Allows browsing, searching, editing and subscribing to artefacts.  Advanced access control mechanism of artefacts controlling their visibility (Public/Private).

Eurostat New Euro SDMX Registry (2.1) Home page Most recent items Access to the content of the Registry by type Access to the content of the Registry text search

Eurostat Registry Content Number of objects of type selected Filtering list New Euro SDMX Registry (2.1)

SDMX Global Registry(SGR) Launched in 2011 by the SDMX Sponsoring institutions Originally to support the world-wide technical implementation of Balance of Payments and National Account implementations To make metadata material (DSDs, concept schemes, MSDs, code lists, etc.) publicly and centrally available used and maintained by national and international organisations.

Who should use the SDMX Global Registry (SGR)? Any national or international agency that wants to use SDMX standards for data management or for exchanging data with other organisations using SDMX standard. Any end-user of statistical data can access the data structures posted on the SGR.

How to use the SDMX Global Registry (SGR)? All information published in the SGR is publicly available and freely viewable and downloadable. No need to log in to view or to download information - SDMX Global Registry Content.SDMX Global Registry Content Users can view the full content of the Global Registry

Exploring by agency Available SDMX objects Click on a type Available SDMX objects Click on a type Enter a text to be found in the Registry Click on Export to choose a SDMX object to download Access to the web service

Codelist Content 2. Select an object

Dimensions, measure and attributes Corresponding list of codes 1. Choose a DSD in the list

Example of result for "NA-MAIN" 5 versions found Example of result for "NA-MAIN" 5 versions found

Web Service Query Select the SDMX version, the Object type and the identifiers of the SDMX object (ID, agency, version) Select the SDMX version, the Object type and the identifiers of the SDMX object (ID, agency, version) View the xml file in the browser or Download the file Select which referenced objects (e.g. Code lists) you want to include The URL generated is the SDMX query to be sent to the web service to get the xml file corresponding to the selected object

View the xml file in your browser (DSD NA_MAIN)

Eurostat SDMX TOOLS FOR DATA PROVIDERS 25

Eurostat SDMX Tools for Data Providers 26 Tools offeredUseAction Web Forms (EU)Excel-like templates Transmission of low volumes of data No costs for EU organisations Manual work for senders (type, copy/paste) SDMX Converter (sender’s PC) Converts data files between SDMX formats and other file formats Installation on sender’s PC Manual work for senders (convert) SDMX Converter (batch mode) Installation on server SDMX Reference Infrastructure A set of tools that allows to connect your IT systems to the SDMX world Installation on server Mapping of database to DSDs Own SDMX Implementation Local development in organisations

Eurostat Data provider view: One goal – different possibilities Webforms EDAMIS Single Entry Point SDMX-ML file Database export SDMX ConverterSDMX-ML file Database SDMX Reference Infrastructure SDMX-ML file SDMX Web Service Same DSD  Same Format Excel sheets SDMX ConverterSDMX-ML file

Eurostat Data provider view: One goal – different IT Architectures Webforms Web-based Push mode Database export SDMX Converter Local conversion Push mode Database SDMX Reference Infrastructure Push or Pull mode Excel sheets SDMX Converter Local conversion Push mode

Eurostat SDMX Converter File based conversion Open source and platform independent (Java) Different ways of using it: Graphical user interface Batch file (server or client side) Web Service interface Reusing source code in your own Java application Formats: SDMX-ML 2.0 & 2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, predefined Excel templates

Eurostat GUI (Converter 5.2.1) 2.a Select the DSD in the local drive 2.b Identify a DSD to download from the SDMX Registry (configuration required) 2.c Identify a dataflow linked to the DSD to download from the SDMX Registry (configuration required) CSV parameters XML parameters for SDMX output formats 1. Selection of the input/output files and their format 6. SDMX (output) validation 2. If the local DSD includes multiple versions, we can specify the one derired 2. If the local DSD includes multiple versions, we can specify the one desired 3. SDMX header (.prop file) Only for flat and excel files 4. Mapping and Transcoding 5. CSV quotation 3. Excel parameter file

Eurostat 31

Eurostat 32

Eurostat Exercice: conversions 33 From CSV to SDMX Compact From XLS to SDMX Compact From SDMX Compact to CSV

Eurostat What is SDMX-RI 34 Framework for SDMX data provision Set of pick-and-choose reusable building blocks allowing a statistical office to disseminate SDMX data to the external world Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse Uses SDMX standards including the one for Web Services

Eurostat Why SDMX-RI 35 Developed to simplify the exchange of data Provides standard software and components, allowing individual statistical organizations to interact and exchange their data using the same software and methodology Modular approach, users can o use part or the entire infrastructure, o extend it by adding new modules or modify it in any other way to suit their own purposes Developed in both Java and.NET

Eurostat SDMX Query (XML File) SDMX Reference Infrastructure Census-HUB architecture – Eurostat to NSI SDMX DataMessage (XML File) Eurostat Census HubNational Statistics Institute SDMX -RI Web service

Eurostat Supported DBs 37 Oracle SQL Server MySQL Pc Axis ODBC Supported DBs Oracle SQL Server MySQL PC Axis ODBC

Eurostat SDMX-RI components 38 Mapping Assistant  Graphical tool to create the mapping between the DSD and the dissemination database. Test Client  Used to test your dataflow locally NSI Web Service  Allows you to share your dataflow NSI Client  Web interface to interact with the Web Service

Eurostat SDMX-RI process 39 Dissemin ation DB DSD Mapping Store Mapping Store Test Client (Desktop Software) Test Client (Desktop Software) Web Service Data Consumer (external) NSI Client Local environment Is my dataflow working? Is my WS working?

Eurostat Dissemination Database 40 The SDMX-RI is configured to directly map the dissemination Database An SDMX output is generated by the SDMX-RI using the PULL mode (the Receiver Organisation directly requests data and receives an SDMX file with no intervention of the Sender Organisation)

Eurostat Dissemination Database 41

Eurostat Mapping Assistant 42 Facilitates the mapping between the structural metadata provided by an SDMX-ML Data Structure Definition (DSD) and those that reside in a database of a dissemination environment Maintains a Mapping Store for keeping the mappings between the SDMX and the local data storage scheme In the SDMX Reference Infrastructure, provides mapping information to the Data Retriever

Eurostat 43

Eurostat 44

Eurostat

Transcoding 46

Eurostat Test Client 47 Test mappings of data flows configured through the Mapping Assistant Test the NSI Web Service is correctly set up Export data sets to file in SDMX format

Eurostat

Web Client 49 Web Interface to directly request data. Graphical view of the results Generates the output in different formats

Eurostat NSI Web Client : Welcome screen 50

Eurostat

Support for data providers SDMX Reference Infrastructure Installed in ~32 countries for Census Hub Can be extended to other domains 1st contact point - Local Transmission Coordinators In National Statistical Institutes Experience with EDAMIS and SDMX 2nd contact point - Eurostat support teams

Eurostat Tools Relationships – Example 53 DSW SDMX Converter SDMX Converter SDMX Registry SDMX-RI Mapping Assistant DSD National DB CSV File SDMX-ML Dataset Mapping Produce use Store Produce use DSD

Eurostat SDMX Tools – Summary Eurostat tools  Eurostat SDMX Info Space SDMX Converter SDMX Reference Infrastructure & Mapping Assistant SDMX Registry & Data Structure Wizard Other tools available in the community Fusion Weaver (validation, transformation, creation) Fusion Transformer (command line conversion) Fusion Registry ECB Checker/Loader (conversion and validation) FAO OpenSDMX project

Eurostat Thank you for your attention! Questions