1. SDMX IT Tools SDMX Reference Infrastructure
Jean-Francois LEBLANC
Christian SEBASTIAN
November 2015
Eurostat Unit B3 – IT and standards for data and metadata exchange

2. Table of Contents Objectives What is SDMX-RI Why SDMX-RI Supported DBs
Data process
SDMX-RI components
SDMX-RI process
SDMX-RI implementations

3. 1 Objectives Present all the components of the SDMX-RI
Show their capabilities
SDMX-RI advantages
Exhibit the relationship between its components
Offer a clear working workflow

4. 2 What is SDMX-RI Framework for SDMX data provision
Set of pick-and-choose reusable building blocks allowing a statistical office to expose SDMX data to the external world
Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse
Uses SDMX standards including the one for Web Services

5. 3 Why SDMX-RI Developed to simplify the exchange of data
Provides standard software and components, allowing individual statistical organizations to interact and exchange their data using the same software and methodology
Modular approach, use part or the entire infrastructure, extend it by adding new modules or modify it in any other way to suit their own purposes
Developed in both Java and .NET

6. 4. Supported DBs Oracle SQL Server MySQL Pc Axis ODBC Supported DBs

7. 5. Data process

8. 6. SDMX-RI components Mapping Assistant Test Client NSI Web Service
Graphical tool to create the mapping between the DSD and the disseminetion database.
Test Client
Used to test your dataflow localy
NSI Web Service
Allow you to share your dataflow
NSI Client
Web interface to interact with the Web Service

9. 6.1 SDMX-RI components example
Firewall
Firewall
Firewall
Workstation
App Server
Mapping
Assistant
NSI-WS
External Client
Test Client
NSI-Client
DB Server(s)
MSDB Store
In this example we can see firewalls but note that the security configuration is customized by every organization and it is completely independent of SDMX.
DDB
Internal Network
Secure DMZ
DMZ
Internet
SDMX RI Development Roadmap 2012, unit B-3

10. Data Consumer (external)
7 SDMX-RI process
Data Consumer (external)
Web Service
NSI Client
Dissemination DB
Mapping
Store
Mapping
DSD
Is my dataflow working?
Is my WS working?
Local
environment
Test Client
(Desktop Software)

11. 7.1 SDMX-RI overview
SDMX RI Development Roadmap 2012, unit B-3

12. 8 SDMX-RI implementations
SDMX-RI First implementation
Overview, shortcomings
SDMX-RI Intermediate solution
Overview, rationale, changes, benefits, impact to users
SDMX-RI “Ultimate” solution
SRI Initial Solution
SRI Intermediate Solution
SRI Ultimate Solution
SDMX RI Development Roadmap 2012, unit B-3

13. 8.1 Ultimate solution key features
Common API
All modules use the SDMX Common API
Interchangeable implementation
Fosters component reusability
Support for SDMX 2.1
New messages (data representation, queries)
Web Service extended to support SDMX 2.1 standardized SOAP and RESTful APIs
Web Service endpoints were added above the Controller
New endpoints co-exist with SDMX 2.0 endpoint
Will support SDMX 2.1 error handling

14. SDMX API Implementation
8.2 Ultimate solution
getGenericData
getCompactData
getCrossSectionalData
queryStructure
Web Service Provider
Web Service
SR API
Structure Retriever
DR API
Data Retriever
(streaming)
(3)Data Retriever
(streaming)
Components with dashed line are API’s, i.e. Application Programming Interfaces.
Components with normal lines are solid implementations.
Web Service Provider
This component is responsible for exposing the data using a Web Service interface that provides SDMX-ML messages. It will be capable of exposing three different interfaces, i.e. SDMX 2.0 SOAP, SDMX 2.1 SOAP & REST.
(1) Structure Retriever
This component is responsible for serving the “queryStructure” calls, i.e. to handle SDMX Structure related queries. Currently, SRI responds to queries for DSDs, Codelists, Concept Schemes, Category Schemes and Dataflows.
In the context of the “Ultimate” solution, its API is packaged separately.
(3) Data Retriever
This component is responsible for querying the dissemination database, getting the respective recordset and populating the sdmx data model with the data retrieved, which is then returned.
(6) Common SDMX API
A set of interfaces for handling data and metadata based on the SDMX information model.
It provides methods for reading and writing from/to SDMX-ML messages.
Eurostat’s plans are to use this in several SDMX SW components/tools, e.g. SDMX Converter, Euro SDMX Registry, DSW.
(7) SDMX API Implementation
This component is an implementation of the interfaces specified in (6) Common SDMX API; combined with the latter, it replaces the component (5) SDMX Model/IO. In the “Ultimate” solution it is based on a Metadata Technology implementation.
Mapping Store
This component (database) is responsible for keeping the mappings between the SDMX structural metadata and the native format (a file or a DB schema). The mappings are created and edited off-line by the Mapping Assistant.
In other words, the Mapping Store is responsible for creating the mappings between an SDMX Data Structure Definition (DSD) and a DB schema (dissemination database) or a set of dissemination data files (PC-Axis files). It maps the DB schema from the database to the SDMX DSD.
Dissemination database
This is the final storage data warehouse maintained by the Data Provider. It stores data that can be published to potential Data Consumers.
PC-Axis files
This is the PC-Axis dissemination environment file format (aka px-files).
A custom driver has been implemented for loading the data from px-files into a temporary in-memory database so as to be queried by the Data Retriever.
Common SDMX API
Mapping Store
PC-Axis
<implements>
SDMX API Implementation
Dissemination DB

15. 8.3 WS extension to SDMX 2.1 interfaces
SOAP Request 2.0
SOAP Request 2.1
REST Request 2.1
NSI_Service_2.0
NSI_Service_2.1
NsiRestService
Web Service Provider
Controller
Web Service Provider
This module is responsible for exposing the data using a Web Service interface that provides SDMX-ML messages. It offers 3 Web Service interfaces: SOAP SDMX v2.0, SOAP v2.1, REST v2.1
NSI_Service_2.0
It is a module of the Web Service Provider component. It implements the Web Service SOAP interface according to the SDMX v2.0 Web Service guidelines. It is responsible for serving such requests, that are passed to the Controller.
NSI_Service_2.1
It is a module of the Web Service Provider component. It implements the Web Service SOAP interface according to the SDMX v2.1 Web Service guidelines (SDMX v2.1 provides a standardised WSDL). It is responsible for serving such requests, that are passed to the Controller.
NsiRestService
It is a module of the Web Service Provider component. It implements the Web Service Restful API according to the SDMX v2.1 Web Service guidelines. It is responsible for serving such requests, that are passed to the Controller.
Controller
It is a module of the Web Service Provider component that has all the logic of the Web Service provider. It coordinates the calls to the rest of the modules (SR, DR, common SDMX API reader/writers) in order to carry out the request so as its result is streamed back to the interface that was called i.e. v2.0, v2.1 and Rest services.
Data Retriever
This module is responsible for querying the dissemination database and getting the respective recordset, which is then streamed to the caller. The DR is provided with the query to process and a Streaming Writer that depends on the type of the message (i.e. Generic, Compact, XS).
Common SDMX API
This is an API that provides interfaces of objects for storing data and metadata based on the SDMX information model. Also, it provides interfaces of methods for reading and writing from/to SDMX-ML messages. It is the common SDMX API that is intended to be used in a inter-organisation scope in order to foster reusability of components. The SRI Web Service uses from this API the reading of SDMX-ML Data Query, reading writing of the SDMX-ML RegistryInterface for structure query request/responses and finally the streaming writers of SDMX-ML Datasets.
SDMX API implementation
This is the implementation of the Common API used in the context of the SRI. The modules are dependent on the API that provides the interfaces - however they should use an implementation of the API that provides the actual functionality. The MT implementation for Java will be used in the SRI. An implementation for the .NET is pending yet.
Structure Retriever
Common SDMX API
Data Retriever (streaming)
SDMX 2.0 Implementation
SDMX API Implementation
SDMX 2.1 Implementation
SDMX RI Development Roadmap 2012, unit B-3

16. 8.4 Ultimate solution - Impacts
Organizations with Mapping Store in production, will have to
Install new Mapping Assistant
Upgrade Mapping Store automatically within MA
Organizations that have a Web Service installation in place, will have to
Install the new Web Service package
Existing clients of Web Service will not be affected
The SDMX 2.0 SOAP interface will remain
SDMX RI Development Roadmap 2012, unit B-3

17. 8.5 Ultimate solution - Impacts
Organisations using the SRI components APIs
Migration will be required
Migration guidelines will be provided
Organisations that have already done modifications to the source code
Will have to make the changes again using the “Ultimate” solution
SDMX RI Development Roadmap 2012, unit B-3

18. SDMX Reference Infrastructure
SDMX RI Development Roadmap 2012, unit B-3