SDMX IT Tools SDMX Reference Infrastructure Jean-Francois LEBLANC Christian SEBASTIAN November 2015 Eurostat Unit B3 – IT and standards for data and metadata exchange

Table of Contents Objectives What is SDMX-RI Why SDMX-RI Supported DBs Data process SDMX-RI components SDMX-RI process SDMX-RI implementations

1 Objectives Present all the components of the SDMX-RI Show their capabilities SDMX-RI advantages Exhibit the relationship between its components Offer a clear working workflow

2 What is SDMX-RI Framework for SDMX data provision Set of pick-and-choose reusable building blocks allowing a statistical office to expose SDMX data to the external world Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse Uses SDMX standards including the one for Web Services

3 Why SDMX-RI Developed to simplify the exchange of data Provides standard software and components, allowing individual statistical organizations to interact and exchange their data using the same software and methodology Modular approach, use part or the entire infrastructure, extend it by adding new modules or modify it in any other way to suit their own purposes Developed in both Java and .NET

4. Supported DBs Oracle SQL Server MySQL Pc Axis ODBC Supported DBs

5. Data process

6. SDMX-RI components Mapping Assistant Test Client NSI Web Service Graphical tool to create the mapping between the DSD and the disseminetion database. Test Client Used to test your dataflow localy NSI Web Service Allow you to share your dataflow NSI Client Web interface to interact with the Web Service

6.1 SDMX-RI components example Firewall Firewall Firewall       Workstation App Server Mapping Assistant NSI-WS External Client Test Client NSI-Client DB Server(s) MSDB Store In this example we can see firewalls but note that the security configuration is customized by every organization and it is completely independent of SDMX. DDB Internal Network Secure DMZ DMZ Internet SDMX RI Development Roadmap 2012, unit B-3

Data Consumer (external) 7 SDMX-RI process Data Consumer (external) Web Service NSI Client Dissemination DB Mapping Store Mapping DSD Is my dataflow working? Is my WS working? Local environment Test Client (Desktop Software)

7.1 SDMX-RI overview SDMX RI Development Roadmap 2012, unit B-3

8 SDMX-RI implementations SDMX-RI First implementation Overview, shortcomings SDMX-RI Intermediate solution Overview, rationale, changes, benefits, impact to users SDMX-RI “Ultimate” solution SRI Initial Solution SRI Intermediate Solution SRI Ultimate Solution SDMX RI Development Roadmap 2012, unit B-3

8.1 Ultimate solution key features Common API All modules use the SDMX Common API Interchangeable implementation Fosters component reusability Support for SDMX 2.1 New messages (data representation, queries) Web Service extended to support SDMX 2.1 standardized SOAP and RESTful APIs Web Service endpoints were added above the Controller New endpoints co-exist with SDMX 2.0 endpoint Will support SDMX 2.1 error handling

SDMX API Implementation 8.2 Ultimate solution getGenericData getCompactData getCrossSectionalData queryStructure Web Service Provider Web Service SR API Structure Retriever DR API Data Retriever (streaming) (3)Data Retriever (streaming) Components with dashed line are API’s, i.e. Application Programming Interfaces. Components with normal lines are solid implementations. Web Service Provider This component is responsible for exposing the data using a Web Service interface that provides SDMX-ML messages. It will be capable of exposing three different interfaces, i.e. SDMX 2.0 SOAP, SDMX 2.1 SOAP & REST. (1) Structure Retriever This component is responsible for serving the “queryStructure” calls, i.e. to handle SDMX Structure related queries. Currently, SRI responds to queries for DSDs, Codelists, Concept Schemes, Category Schemes and Dataflows. In the context of the “Ultimate” solution, its API is packaged separately. (3) Data Retriever This component is responsible for querying the dissemination database, getting the respective recordset and populating the sdmx data model with the data retrieved, which is then returned. (6) Common SDMX API A set of interfaces for handling data and metadata based on the SDMX information model. It provides methods for reading and writing from/to SDMX-ML messages. Eurostat’s plans are to use this in several SDMX SW components/tools, e.g. SDMX Converter, Euro SDMX Registry, DSW. (7) SDMX API Implementation This component is an implementation of the interfaces specified in (6) Common SDMX API; combined with the latter, it replaces the component (5) SDMX Model/IO. In the “Ultimate” solution it is based on a Metadata Technology implementation. Mapping Store This component (database) is responsible for keeping the mappings between the SDMX structural metadata and the native format (a file or a DB schema). The mappings are created and edited off-line by the Mapping Assistant. In other words, the Mapping Store is responsible for creating the mappings between an SDMX Data Structure Definition (DSD) and a DB schema (dissemination database) or a set of dissemination data files (PC-Axis files). It maps the DB schema from the database to the SDMX DSD. Dissemination database This is the final storage data warehouse maintained by the Data Provider. It stores data that can be published to potential Data Consumers. PC-Axis files This is the PC-Axis dissemination environment file format (aka px-files). A custom driver has been implemented for loading the data from px-files into a temporary in-memory database so as to be queried by the Data Retriever. Common SDMX API Mapping Store PC-Axis <implements> SDMX API Implementation Dissemination DB

8.3 WS extension to SDMX 2.1 interfaces SOAP Request 2.0 SOAP Request 2.1 REST Request 2.1 NSI_Service_2.0 NSI_Service_2.1 NsiRestService Web Service Provider Controller Web Service Provider This module is responsible for exposing the data using a Web Service interface that provides SDMX-ML messages. It offers 3 Web Service interfaces: SOAP SDMX v2.0, SOAP v2.1, REST v2.1 NSI_Service_2.0 It is a module of the Web Service Provider component. It implements the Web Service SOAP interface according to the SDMX v2.0 Web Service guidelines. It is responsible for serving such requests, that are passed to the Controller. NSI_Service_2.1 It is a module of the Web Service Provider component. It implements the Web Service SOAP interface according to the SDMX v2.1 Web Service guidelines (SDMX v2.1 provides a standardised WSDL). It is responsible for serving such requests, that are passed to the Controller. NsiRestService It is a module of the Web Service Provider component. It implements the Web Service Restful API according to the SDMX v2.1 Web Service guidelines. It is responsible for serving such requests, that are passed to the Controller. Controller It is a module of the Web Service Provider component that has all the logic of the Web Service provider. It coordinates the calls to the rest of the modules (SR, DR, common SDMX API reader/writers) in order to carry out the request so as its result is streamed back to the interface that was called i.e. v2.0, v2.1 and Rest services. Data Retriever This module is responsible for querying the dissemination database and getting the respective recordset, which is then streamed to the caller. The DR is provided with the query to process and a Streaming Writer that depends on the type of the message (i.e. Generic, Compact, XS). Common SDMX API This is an API that provides interfaces of objects for storing data and metadata based on the SDMX information model. Also, it provides interfaces of methods for reading and writing from/to SDMX-ML messages. It is the common SDMX API that is intended to be used in a inter-organisation scope in order to foster reusability of components. The SRI Web Service uses from this API the reading of SDMX-ML Data Query, reading writing of the SDMX-ML RegistryInterface for structure query request/responses and finally the streaming writers of SDMX-ML Datasets. SDMX API implementation This is the implementation of the Common API used in the context of the SRI. The modules are dependent on the API that provides the interfaces - however they should use an implementation of the API that provides the actual functionality. The MT implementation for Java will be used in the SRI. An implementation for the .NET is pending yet. Structure Retriever Common SDMX API Data Retriever (streaming) SDMX 2.0 Implementation SDMX API Implementation SDMX 2.1 Implementation SDMX RI Development Roadmap 2012, unit B-3

8.4 Ultimate solution - Impacts Organizations with Mapping Store in production, will have to Install new Mapping Assistant Upgrade Mapping Store automatically within MA Organizations that have a Web Service installation in place, will have to Install the new Web Service package Existing clients of Web Service will not be affected The SDMX 2.0 SOAP interface will remain SDMX RI Development Roadmap 2012, unit B-3

8.5 Ultimate solution - Impacts Organisations using the SRI components APIs Migration will be required Migration guidelines will be provided Organisations that have already done modifications to the source code Will have to make the changes again using the “Ultimate” solution SDMX RI Development Roadmap 2012, unit B-3

SDMX Reference Infrastructure SDMX RI Development Roadmap 2012, unit B-3