1. Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015

2. Eurostat The SDMX Components 2 IT Infrastructure for exchange and sharing Content Oriented Guidelines SDMX Information Model

3. Eurostat Who? What? When? Who? How? Describing the data exchange What's next?

4. Eurostat EUROSTAT SDMX TOOLS FOR DESIGNERS 4

5. Eurostat Eurostat SDMX tools for designers 5 Tools offeredUse Data Structure WizardUsed to create, edit and test SDMX objects Maintenance of SDMX registry SDMX RegistryA central repository for storing and sharing SDMX objects

6. Eurostat Eurostat SDMX tools for designers 6 Data Structure Wizard SDMX Registry Create/Edit Store/Edit Web service

7. Work roles and process SDMX Registry Create DSD Publish DSD Format / structure SDMX messages Consult and get DSD Send SDMX messages COLLECTING ORGANISATION Consult and get DSD Interpret SDMX messages 7

8. Data Structure Wizard Creation / edition of DSDs Validation of DSD Local repository Artefacts are stored in a local repository Remote repository Can be used for retrieving existing artefacts 8

9. DSW: Management of SDMX artefacts 9

10. DSW: Artefact example: the codelists 10

11. Eurostat SDMX Registry 11 Metadata registry which implements the SDMX registry specificationsSDMX Tasked with providing structure, organisation, maintenance and query interfaces for the SDMX components required to support the data sharing vision Provides a web-based user interface as well as web services for interacting with the SDMX structural metadata objects in use within the collecting organisation and with statistical partners.

12. Eurostat SDMX Registry 12 Only concerned with providing information needed to access the data and reference metadata sets. An application which wants a particular data or metadata set would then query the registry for the URL, and then go and retrieve the data or metadata set directly from the provider's web server. Example of existing SDMX registries: Euro SDMX Registry (2.1) SDMX Global Registry (2.1)

13. Eurostat New Euro SDMX Registry (2.1) Eurostat's implementation of the SDMX Registry specifications as published by the SDMX initiative sdmx.org. sdmx.org Based on SDMX 2.1 (as published on April 2011) Also capable of importing and exporting SDMX 2.0 artefacts. Allows browsing, searching, editing and subscribing to artefacts. Advanced access control mechanism for distributed maintenance of artefacts controlling also their visibility.

14. Eurostat New Euro SDMX Registry (2.1) Home page Most recent items Access to the content of the Registry by type Access to the content of the Registry text search

15. Eurostat Registry Content Number of objects of type selected Filtering list New Euro SDMX Registry (2.1)

16. SDMX Global Registry(SGR) Launched in 2011 by the SDMX Sponsoring institutions Originally to support the world-wide technical implementation of Balance of Payments and National Account implementations  all international implementations To make metadata material, DSDs and related artefacts (concept schemes, MSDs, code lists, etc.), publicly and centrally available

17. SDMX Global Registry(SGR) Contains other SDMX DSD and artefacts that are used and maintained by national and international organisations. Designed in such a way that it can be developed into a global portal for SDMX structural data and metadata, providing not only information about how statistical information is structured, but also where the related data can be accessed.

18. Who should use the SDMX Global Registry (SGR)? Any national or international agency that wants to use SDMX standards for data management or for exchanging data with other organisations using SDMX standard. Any end-user of statistical data can access the data structures posted on the SGR.

19. How to use the SDMX Global Registry (SGR)? All information published in the SGR is publicly available and freely viewable and downloadable. No need to log in to view or to download information - SDMX Global Registry Content.SDMX Global Registry Content Users can view the full content of the Global Registry, see how the various objects are interlinked and use the SDMX Global Registry Technical viewer.SDMX Global Registry Technical viewer Help is available in the Global Registry WikiGlobal Registry Wiki

20. Registry Content Navigate the content of the registry Search for an item that contains the text entered Technical view of SDMX objects

21. Registry Content Exploring by agency Click on the agency to see the items maintained by it

22. Registry Content Available SDMX objects Click on a type Available SDMX objects Click on a type

23. Registry Content 2. Select a version 1. Choose a DSD in the list 3. Click on View

24. Registry Content Dimensions, measure and attributes Clicking on a concept on the left frame, displays the corresponding list of codes (if exists) Download the DSD (XML file) Download the DSD (XML file)

25. Registry Content Enter a text to be found in the Registry content

26. Registry Content Example of result for "NA-MAIN" Two versions found Example of result for "NA-MAIN" Two versions found

27. Registry Content Click to access to the technical view

28. Technical view Click on the icon to download a SDMX object

29. Technical view Select one or more SDMX objects Click to download the xml file (unique file containing all the selected objects)

30. Technical view Access to the technical view of the Registry content

31. Technical view Select an agency Select an object type Select an object version View of the content (here the DSD elements)

32. Technical view Access to the web service

33. Technical view Select the SDMX version, the Object type and the identifiers of the SDMX object (ID, agency, version) Select the SDMX version, the Object type and the identifiers of the SDMX object (ID, agency, version) Click to view the xml file in your browser Select which referenced objects (e.g. Code lists) you want to include Same selection as above, the URL generated is the SDMX query to be sent to the web service to get the xml file corresponding to the selected object

34. View the xml file in your browser (DSD NA_MAIN)

35. Eurostat SDMX TOOLS FOR DATA PROVIDERS 35

36. Eurostat SDMX Tools for Data Providers 36 Tools offeredUseAction Web Forms (EU)Excel-like templates Transmission of low volumes of data No costs for EU organisations Manual work for senders (type, copy/paste) SDMX Converter (sender’s PC) Converts data files between SDMX formats and other file formats Installation on sender’s PC Manual work for senders (convert) SDMX Converter (batch mode) Installation on server SDMX Reference Infrastructure A set of tools that allows to connect your IT systems to the SDMX world Installation on server Mapping of database to DSDs Own SDMX Implementation Local development in organisations

37. Eurostat Data provider view: One goal – different possibilities Webforms EDAMIS Single Entry Point SDMX-ML file Database export SDMX ConverterSDMX-ML file Database SDMX Reference Infrastructure SDMX-ML file SDMX Web Service Same DSD  Same Format Excel sheets SDMX ConverterSDMX-ML file

38. Eurostat Data provider view: One goal – different IT Architectures Webforms Web-based Push mode Database export SDMX Converter Local conversion Push mode Database SDMX Reference Infrastructure Pull mode Excel sheets SDMX Converter Local conversion Push mode

39. Eurostat SDMX Converter https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Converter https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Converter File based conversion Open source and platform independent (Java) Different ways of using it: Graphical user interface Batch file (server or client side) Web Service interface Reusing source code in your own Java application Formats: SDMX-ML 2.0 & 2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, predefined Excel templates

40. Eurostat Select the Input fileSelect the output file Select the input and output formats Select the DSD on the local drive Identify a DSD to download from the SDMX Registry Identify a dataflow linked to the DSD to download from the SDMX Registry Select / manage headers for CSV input formats Select mapping / transoding tables CSV parameters GESMES representation for GESMES output formats Load / save the current settings XML parameters for SDMX output formats

41. Eurostat Exercice: conversions 41 From CSV to SDMX Compact From XLS to SDMX Compact From SDMX Compact to CSV

42. Eurostat SDMX Reference Infrastructure SDMX-RI 42 Generalized service infrastructure that can be re-used partially or completely by any organisation interested in starting SDMX projects for data exchange.SDMX Set of pick-and-choose building blocks allowing a statistical office to expose data to the external world based on access rights Developed in both Java and.NET with well defined API

43. Eurostat SDMX Reference Infrastructure SDMX-RI 43 Provides data and structural metadata based on mappings to an organizations data warehouse Conform to SDMX Web Service guidelines Open Source package under the EUPL licence Support Census Hub and similar Eurostat projects

44. Eurostat SDMX Query (XML File) SDMX Reference Infrastructure Census-HUB architecture – Eurostat to NSI SDMX DataMessage (XML File) Eurostat Census HubNational Statistics Institute SDMX -RI Web service

45. Eurostat Data ProviderData Collector Non-SDMX local database SDMX data set SDMX Registry DSD SDMX-RI Mapping Assistant Web Svc Test Client SDMX Reference Infrastructure https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastr ucture_SDMX-RI https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastr ucture_SDMX-RI Web Client

46. Eurostat Mapping Assistant 46 Facilitates the mapping between the structural metadata provided by an SDMX-ML Data Structure Definition (DSD) and those that reside in a database of a dissemination environment. Maintains a Mapping Store for keeping the mappings between the SDMX and the local data storage scheme In the SDMX Reference Infrastructure, provides mapping information to the Data Retriever

48. Eurostat

50. Test Client 50 Test mappings of data flows configured through the Mapping Assistant Test the NSI Web Service is correctly set up Export data sets to file in SDMX format

51. Eurostat

52. Web Client 52 Result of a SDMX query

53. Eurostat

54. Support for data providers SDMX Reference Infrastructure Installed in ~32 countries for Census Hub Can be extended to other domains 1st contact point - Local Transmission Coordinators In National Statistical Institutes Experience with EDAMIS and SDMX 2nd contact point - Eurostat support teams

55. Eurostat Tools Relationships – Example 55 DSW SDMX Converter SDMX Converter SDMX Registry SDMX-RI Mapping Assistant DSD National DB CSV File SDMX-ML Dataset Mapping Produce use Store Produce use DSD

56. Eurostat SDMX Tools – Summary http://www.sdmxtools.org http://www.sdmxtools.org Eurostat tools  Eurostat SDMX Info Space SDMX Converter SDMX Reference Infrastructure & Mapping Assistant SDMX Registry & Data Structure Wizard Other tools available in the community Fusion Weaver (validation, transformation, creation) Fusion Transformer (command line conversion) Fusion Registry ECB Checker/Loader (conversion and validation) FAO OpenSDMX project

57. Eurostat Questions & Answers HELP – We do not have a database! No worries. Excel templates are a possible use Conversion is supported through the SDMX converter. However: use the opportunity to start!

58. Eurostat Questions & Answers I want to have a database. What should I do? Simple DB creation possible from Excel sheets Sharing experience from other domains in your NSI (e.g. Census hub)

59. Eurostat Questions & Answers Which versions / formats should we use? SDMX-EDI (Gesmes/TS) supported, deprecated SDMX-ML 2.0 (compact) most used SDMX-ML 2.1 (structure specific)

60. Eurostat Questions & Answers Where are the DSDs managed? DSDs will be managed in the SDMX registry

61. Eurostat Questions & Answers Other questions?