SDMX Reference Infrastructure Introduction Jean-Francois LEBLANC Christian SEBASTIAN 11-13 May 2016 Eurostat, Unit B3 – IT solutions for statistical production

Table of Contents Objectives What is SDMX-RI Why SDMX-RI Prerequisites SDMX-RI components SDMX-RI process SDMX-RI implementations

1 Objectives Understand what is the SDMX-RI and its architecture Main functionalities Why to choose SDMX-RI Understand the tools under SDMX-RI and how they work Offer a clear overview of the workflow

2 What is SDMX-RI Framework for SDMX data provision (dissemination and reporting) Set of pick-and-choose reusable building blocks, allowing the data provider to expose SDMX data to the external world Designed to translate internal data storages into SDMX compliant data files using mappings Uses SDMX standards specifications (including the one for Web Services)

3 Why SDMX-RI Developed to simplify the exchange of data Provides standard software and components, allowing individual statistical organizations to interact and exchange their data using the same software and methodology Modular and extendable approach (modify, use part or the entire infrastructure)to suit different purposes WS is developed in both Java and .NET SDMX RI Development Roadmap 2012, unit B-3

4. Prerequisites A web application server Supported DBs Oracle SQL Server MySQL PC Axis ODBC A web application server .net Framework 4.0 (or higher) / a java environment A database server (Oracle, Sql server, MySql, Pc Axis, ODBC, Maria DB) Windows environment (the WS is available in Java but the tools are .net based) For details, consult the installation manual of the tool SDMX RI Development Roadmap 2012, unit B-3

5. SDMX-RI components Mapping Assistant Test Client NSI Web Service Graphical tool to create the mapping between the DSD and the disseminetion database. Test Client Used to test your dataflow localy NSI Web Service Allow you to share your dataflow NSI Client Web interface to interact with the Web Service

Data Consumer (external) 6. SDMX-RI process Data Consumer (external) Web Service NSI Client Dissemination DB Mapping Store Mapping DSD Is my dataflow working? Is my WS working? Local environment Test Client (Desktop Software)

7.1 Ultimate solution key features Common API All modules use the SDMX Common API The main blocks are the Structure and the Data Retriever Interchangeable implementation Fosters component reusability Support for SDMX 2.1 New messages (data representation, queries) Web Service extended to support SDMX 2.1 standardized SOAP and RESTful APIs Web Service endpoints were added above the Controller New endpoints co-exist with SDMX 2.0 endpoint Will support SDMX 2.1 error handling

SDMX API Implementation 7.2 Ultimate solution getGenericData getGerneric getCompactData getStructureSpecific getCrossSectionalData queryStructure Web Service Provider Web Service SR API Structure Retriever DR API Data Retriever (streaming) (3)Data Retriever (streaming) Components with dashed line are API’s, i.e. Application Programming Interfaces. Components with normal lines are solid implementations. Web Service Provider This component is responsible for exposing the data using a Web Service interface that provides SDMX-ML messages. It will be capable of exposing three different interfaces, i.e. SDMX 2.0 SOAP, SDMX 2.1 SOAP & REST. (1) Structure Retriever This component is responsible for serving the “queryStructure” calls, i.e. to handle SDMX Structure related queries. Currently, SRI responds to queries for DSDs, Codelists, Concept Schemes, Category Schemes and Dataflows. In the context of the “Ultimate” solution, its API is packaged separately. (3) Data Retriever This component is responsible for querying the dissemination database, getting the respective recordset and populating the sdmx data model with the data retrieved, which is then returned. (6) Common SDMX API A set of interfaces for handling data and metadata based on the SDMX information model. It provides methods for reading and writing from/to SDMX-ML messages. Eurostat’s plans are to use this in several SDMX SW components/tools, e.g. SDMX Converter, Euro SDMX Registry, DSW. (7) SDMX API Implementation This component is an implementation of the interfaces specified in (6) Common SDMX API; combined with the latter, it replaces the component (5) SDMX Model/IO. In the “Ultimate” solution it is based on a Metadata Technology implementation. Mapping Store This component (database) is responsible for keeping the mappings between the SDMX structural metadata and the native format (a file or a DB schema). The mappings are created and edited off-line by the Mapping Assistant. In other words, the Mapping Store is responsible for creating the mappings between an SDMX Data Structure Definition (DSD) and a DB schema (dissemination database) or a set of dissemination data files (PC-Axis files). It maps the DB schema from the database to the SDMX DSD. Dissemination database This is the final storage data warehouse maintained by the Data Provider. It stores data that can be published to potential Data Consumers. PC-Axis files This is the PC-Axis dissemination environment file format (aka px-files). A custom driver has been implemented for loading the data from px-files into a temporary in-memory database so as to be queried by the Data Retriever. Common SDMX API Mapping Store PC-Axis <implements> SDMX API Implementation Dissemination DB

7.3 WS extension to SDMX 2.1 interfaces SOAP Request 2.0 SOAP Request 2.1 REST Request 2.1 NSI_Service_2.0 NSI_Service_2.1 NsiRestService Web Service Provider Controller Web Service Provider This module is responsible for exposing the data using a Web Service interface that provides SDMX-ML messages. It offers 3 Web Service interfaces: SOAP SDMX v2.0, SOAP v2.1, REST v2.1 NSI_Service_2.0 It is a module of the Web Service Provider component. It implements the Web Service SOAP interface according to the SDMX v2.0 Web Service guidelines. It is responsible for serving such requests, that are passed to the Controller. NSI_Service_2.1 It is a module of the Web Service Provider component. It implements the Web Service SOAP interface according to the SDMX v2.1 Web Service guidelines (SDMX v2.1 provides a standardised WSDL). It is responsible for serving such requests, that are passed to the Controller. NsiRestService It is a module of the Web Service Provider component. It implements the Web Service Restful API according to the SDMX v2.1 Web Service guidelines. It is responsible for serving such requests, that are passed to the Controller. Controller It is a module of the Web Service Provider component that has all the logic of the Web Service provider. It coordinates the calls to the rest of the modules (SR, DR, common SDMX API reader/writers) in order to carry out the request so as its result is streamed back to the interface that was called i.e. v2.0, v2.1 and Rest services. Data Retriever This module is responsible for querying the dissemination database and getting the respective recordset, which is then streamed to the caller. The DR is provided with the query to process and a Streaming Writer that depends on the type of the message (i.e. Generic, Compact, XS). Common SDMX API This is an API that provides interfaces of objects for storing data and metadata based on the SDMX information model. Also, it provides interfaces of methods for reading and writing from/to SDMX-ML messages. It is the common SDMX API that is intended to be used in a inter-organisation scope in order to foster reusability of components. The SRI Web Service uses from this API the reading of SDMX-ML Data Query, reading writing of the SDMX-ML RegistryInterface for structure query request/responses and finally the streaming writers of SDMX-ML Datasets. SDMX API implementation This is the implementation of the Common API used in the context of the SRI. The modules are dependent on the API that provides the interfaces - however they should use an implementation of the API that provides the actual functionality. The MT implementation for Java will be used in the SRI. An implementation for the .NET is pending yet. Structure Retriever Common SDMX API Data Retriever (streaming) SDMX 2.0 Implementation SDMX API Implementation SDMX 2.1 Implementation SDMX RI Development Roadmap 2012, unit B-3

7.4 Ultimate solution - Impacts Organizations with Mapping Store in production, will have to Install new Mapping Assistant Upgrade Mapping Store automatically within MA Organizations that have a Web Service installation in place, will have to Install the new Web Service package Existing clients of Web Service will not be affected The SDMX 2.0 SOAP interface will remain SDMX RI Development Roadmap 2012, unit B-3

7.5 Ultimate solution - Impacts Organisations using the SRI components APIs Migration will be required Migration guidelines will be provided Organisations that have already done modifications to the source code Will have to make the changes again using the “Ultimate” solution SDMX RI Development Roadmap 2012, unit B-3

SDMX Reference Infrastructure SDMX RI Development Roadmap 2012, unit B-3