SDMX IT Tools SDMX use in practice in NA National Account use case Jean-Francois LEBLANC Christian SEBASTIAN November 2015 Eurostat Unit B3 – IT and standards for data and metadata exchange

Table of contents Objectives National Accounts (NA) NA business needs NA Domains NA Technical Integration NA Data Validation Process NA Project Cycle

Table of contents SDMX Tools in practice SDMX Tools for Data Providers Data Structure Wizard SDMX Converter SDMX Reference Infrastructure Tools relationship SDMX Tools – Summary SDMX Tools for Data Providers Possibilities for Data Providers IT Architectures for Data Providers How to use the tools Conclusion

1 Objectives To reflect how National Accounts work To present the SMDX IT tools To expose how to work with the data and SDMX in real situations To give a base understanding of the tools and their relationship that will be of help for further explanations

2 National Accounts (NA) BOP National Accounts Main Aggregates Sector Accounts Supply & Use Input / Output GFS FDI Excel questionnaires GESMES files Excel templates will be supported for a 2 years period. From then on, the NSI have to create their info in SDMX format because there won’t be any more support for excel templates. The Converter could be a solution but not relying in new templates whatsoever.

2 Why SDMX for NA? Within Eurostat Within the ESS Internationally

2.1 NA business needs Main trigger: new methodology SNA 93  SNA 2008 ESA 95  ESA 2010 Business process reengineering Corporate production systems Integration of production chains Harmonised coding Within Eurostat Within the ESS Internationally

2.1 NA business needs Cross-domain integration Balance of Payments Foreign Direct Investment ESS Business process reengineering Standardisation Improved validation Reuse of tools Within Eurostat Within the ESS Internationally

2.1 NA business needs Data quality Lowering reporting burden 1 country  1 year  1 GDP Shared validation rules Across all organisations Lowering reporting burden 1 data flow from national to international International cooperation Inter-Agency Group on Economic and Financial Statistics = SDMX Sponsors Example: PGI Website http://www.principalglobalindicators.org Within Eurostat Within the ESS Internationally

2.2 NA Domains Shared Concepts and Code Lists National Accounts SNA 2008 ESA 210 Other related domains FDI statistics Other domains Balance of Payment Statistics Shared Concepts and Code Lists Cross Domain Concepts and Code Lists National accounts have 5 domains with one DSD each (MAIN aggregates, SECtor accounts and government finance, Supply/use and input/output statistics, REGional statistics, and PENSion schemes. FDI and BoP (Balance of Payment) are not using the NA DSDs, these domains have their own DSD. They only share some codelists with NA DSDs.

2.3 NA Technical Integration National data sources Source National statistical organizations National DBs Mapping International organizations SDMX DSDs & Data Flows Metadata driven process based on SDMX standards and tools Full automation of the data exchange possible

2.4 NA Data Validation Process Before/during transmission “First Level” Currently covered by SDMX Format Check (SDMX-ML) Code Check (SDMX DSD) Code used correctly (SDMX Dataflow & Constraint) After Transmission “Second Level” Not covered by SDMX Detailed value check Validation expressions …

2.5 NA Project Cycle All projects go through the same process which is shown in this cycle

3. SDMX Tools in practice Find DSDs Edit/Create DSDs Registry DSW Converter SDMX-RI Find DSDs Edit/Create DSDs Data files -> SDMX files Database -> SDMX WS/files The DSDs, where can they find them? In the registry! How can they edit/create the DSD? The DSW is the solution. The first conversions can be realised with the converter. It is an intermediate solution for the dissemination. The converter also implements limited validation functionalities. Remember that the converter is limited when it comes to the dissemination phase. How can the user export the data directly from the database to the client? The RI is the solution. The conclusion is that Eurostat tools cover each need and it is illustrated by the NA example.

Any oranization can create their EURO SDMX registry The Data Structure Wizard (DSW) application provides an improved solution for the management of Data Structure Definitions (DSDs) within the SDMX-ML v2.0 and 2.1 standards. DSW covers and completes the following functionalities for handling structural metadata artefacts from the KeyFamilyDB Access tool, SDMX Registry, Eurostat GESMES Structural Metadatabase and ECB Structural Metadatabase (the functionality of the two GESMES tools is under development at the time being). Specifically, DSW supports: Maintenance of the following SDMX v2.0 and v2.1 artefacts: Code Lists, Concept Schemes, Data Structure Definitions, Data Flows, Category Schemes, Hierarchical Code lists and Organization Schemes Dynamic creation of a template for a specified DSD in SDMX v2.0 and v2.1 format Import and export of artefacts from/to SDMX-ML v2.0 and v2.1 format Import and export of DSDs from/to GESMES/TS structure files Interaction with any standard-compliant SDMX Registry (using SDMX RegistryInterface messages) Connection to SDMX Registry with "query" privilege and update of the local repository of artefacts Update of the SDMX Registry, using the "maintain" privilege, with an artefact that was edited or created locally Print of summary reports for Codelists, Concepts from ConceptSchemes, DSDs, Dataflows Print reports for the DSDs components and for the Codes from Codelists Advanced search (search Dataflow for a specified DSD, search DSD for a specified Dataflow, search all DSDs using a code list and search all the DSDs using a given concept) SDMX-ML Data Authoring feature (creating sample SDMX-ML datasets from DSDs) 3.1 Data Structure Wizard Java tool (open source and platform independent) Supports Maintenance of SDMX v2.0 and v2.1 structures Import and export of artefacts from/to SDMX-ML v2.0, v2.1 and GESMES format Interaction with Euro SDMX Registry (retrieval, modification and search of artefacts) Sample dataset creation

3.2 SDMX Converter Java tool (open source and platform independent) File based conversion Different ways of using it: Graphical user interface Command Line interface Web Service interface Common API interface Formats*: SDMX-ML 2.0, SDMX-ML 2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, predefined Excel templates *Exceptions may apply

3.3 SDMX Reference Infrastructure Data Provider Data Collector SDMX Registry Mapping Assistant DSD Web Svc Web Client Test Client Non-SDMX local database SDMX data set SDMX-RI

3.4 Tools relationship produces stores uses DSD DSW SDMX Registry ESS-MH MSD MSD produces DSD SDMX MetaDataset uses produces uses uses SDMX Converter SDMX-RI Mapping Assistant SDMX Dataset produces exposes produces Mapping Census Hub CSV or other datafile National DB

3.4.1 Tools relationship – Example Eurostat produces NA DSDs Eurostat stores NA DSDs DSW SDMX Registry DSD DSD NSI uses NA DSDs SDMX-RI NSI maps NA DSDs National DB Mapping Assistant Mapping The XSD Generator is responsible for producing XML Schema Definitions (XSD) based on a DSD received. The present version of the tool produces XSDs for Compact, Cross-Sectional or Utility formats of SDMX standard v2.0 and can be used • As a re-usable building block, through its API. • Through a standalone Graphic User Interface (GUI) • Through Command Line Interface (CLI) • Through a web GUI The tool is licenced under the terms of the European Union Public Licence V.1.1 A complete Euro SDMX Registry package is stored on CIRCA repository and can be downloaded here: XSD Generator_app_v1.0.0_2011.06.10 (ZIP) The package contains an application, source code and also documentation such as a User Manual, Installation Guide, Analysis and Design as well as Testing Documentation. All versions of the tool can be downloaded from CIRCA repository. SDMX-ML Dataset NSI produces NSI uses NA DSDs SDMX Converter NSI exports NA data (CSV, XLS)

3.5 SDMX Tools – Summary Eurostat tools  Eurostat SDMX Info Space SDMX Converter SDMX Reference Infrastructure & Mapping Assistant SDMX Registry & Data Structure Wizard Eurostat SDMX Info Space is a wiki with Information about the tools Trainings Videos References to CIRCABC Eurostat SDMX Info Space is a wiki used mainly by ESTAT B3 and B5.

4. SDMX Tools for Data Providers Tools offered Use Action Web Forms (EU) Excel-like templates Transmission of low volumes of data No costs for EU organisations Manual work for senders (type, copy/paste) SDMX Converter Converts data files from different file formats to SDMX format Local installation on sender’s PC Installation on server SDMX Reference Infrastructure A set of tools that allows to transform directly your data from a database to SDMX Mapping of database to DSDs Common API Supports the development of local SDMX systems Local development in organisations

4.1 Possibilities for Data Providers Webforms EDAMIS Single Entry Point SDMX-ML file Database export SDMX Converter Database SDMX Reference Infrastructure SDMX-ML file SDMX Web Service Same DSD  Same Format Excel sheets SDMX Converter SDMX-ML file

4.1.1 Possibilities for Data Providers. NA. Not applicable in National Accounts Webforms EDAMIS Single Entry Point SDMX-ML file Database export SDMX Converter Database SDMX Reference Infrastructure SDMX-ML file SDMX Web Service Will be phased out in National Accounts Same DSD  Same Format Excel sheets SDMX Converter SDMX-ML file

4.2 IT Architectures for Data Providers Webforms Web-based Push mode (EDAMIS) Excel sheets SDMX Converter Local conversion Push mode (EDAMIS) Database export All domains in Eurostat use push mode except Census HUB SDMX Converter Local conversion Push mode (EDAMIS) Database SDMX Reference Infrastructure Pull mode

4.3 How to use the tools If you are a Data Collector Design your DSD using the DSW Make the DSD available in the SDMX Registry Inform your data providers for the available DSD

4.3 How to use the tools Download/pull the DSD from the SDMX Registry If you are a Data Provider Download/pull the DSD from the SDMX Registry If you have a DDB Data stores as files DDB Install and use the SDMX-RI Install and use the Converter Expose the data to be pulled (WS, HUB) Push via EDAMIS

4.4 Conclusion SDMX statistical and technical standards & SDMX IT tools and IT infrastructure provide the "toolkit" for the harmonisation and integration of data structures the rationalisation of national/international data flows finally improving the data quality of official statistics.

SDMX in practice in NA