SDMX Converter Raynald PALMIERI June 2015 IPA Workshop on ITS Statistics

Table of Contents Objectives What is the Converter Interfaces Steps of a conversion Minimum requirements Supported formats Interfaces GUI CLI

Table of Contents Where to find the SDMX Converter Hands-on exercise Installation Simple conversions From GESMES to SDMX Compact From CSV to SDMX Compact From SDMX Compact to CSV

1. Objectives Illustrate how to use the Converter List all the supported formats Explain its interfaces

2. What is the Converter The SDMX converter is an application that converts files between different formats File based conversion (No DB) Open source and platform independent (Java) Different ways of using it: Graphical user interface Batch file … Formats: SDMX-ML 2.0, SDMX-ML 2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, Excel The SDMX Converter has been developed in the context of the SODI project. It is a Java tool that allows converting from/to various dataset types based on SDMX-ML DSDs. The allowed types between which the Converter is capable of converting are: SDMX-ML formats (in order to go from/to Cross-Sectional type the DSD should contain Cross-Sectional information and TimeDimension) GESMES/TS (aka SDMX-EDI) (not capable of converting from/to SDMX-ML Cross-Sectional format) GESMES/2.1 and GESMES/DSIS CSV, FLR, EXCEL, DSPL (supports mapping mechanism and parametric delimiter for CSV. Converting to CSV/FLR from other formats may result in loss of attributes attached at a higher level than observation.)

2.1 Steps of a conversion Steps to proceed a conversion Reading the input message Reading the DSD Writing the converted message <pagebreak> The conversion process comprises two main activities; reading an input data message and writing out the converted data message. The are specific modules that read and write datasets i.e SDMX-ML (Compact, Generic, Utility, Cross-Sectional) Gesmes (TS, 2.1, DSIS) Flat files (CSV FLR). The information of a dataset to converter is stored in classes that are based on the SDMX Information Model v2.0. These classes play the role of an intermediate format between readers and writers. The Data Structure Definition related to the converted datasets is needed for performing a conversion. SDMX Converter, if it’s is not provided manually, can retrieved the DSD from the Registry.

2.2 Minimum requirements Input file Output file (complete path) Format for input and output files Specify DSD DSD file Reference to a DSD file in the Registry

2.3 Supported formats SDMX 2.0 GENERIC SDMX COMPACT SDMX UTILITY SDMX CROSS-SECTIONAL SDMX SDMX 2.1 GENERIC DATA 2.1 GENERIC TS DATA 2.1 STRUCTURE SPECIFIC DATA 2.1 STRUCTURE SPECIFIC TS DATA 2.1 GESMES GESMES TS GESMES 2.1 GESMES DSIS OTHERS CSV FLR DSPL EXCEL <pagebreak> The conversion process comprises two main activities; reading an input data message and writing out the converted data message. The are specific modules that read and write datasets i.e SDMX-ML (Compact, Generic, Utility, Cross-Sectional) Gesmes (TS, 2.1, DSIS) Flat files (CSV FLR). The information of a dataset to converter is stored in classes that are based on the SDMX Information Model v2.0. These classes play the role of an intermediate format between readers and writers. The Data Structure Definition related to the converted datasets is needed for performing a conversion. SDMX Converter, if it’s is not provided manually, can retrieved the DSD from the Registry. Limitations may apply. Please, check the User Manual.

3. Interfaces User interface Command line

3.1 GUI 1. Selection of the input/output files and their format 2. Select the DSD in the local drive 2. Identify a DSD to download from the SDMX Registry (configuration required) 2. If the local DSD includes multiple versions, we can specify the one derired 2. If the local DSD includes multiple versions, we can specify the one desired 2. Identify a dataflow linked to the DSD to download from the SDMX Registry (configuration required) There are some mandatory fields, e.g. the DSD is mandatory, therefore, either loading from a file or from the Registry, it must be present. A message will appear if there are mandatory fields missing. 3. SDMX header (.prop file) Only for flat and excel files CSV parameters 4. Mapping 5. SDMX (output) validation XML parameters for SDMX output formats

3.1.1 Example 1. Input and output files 2. Format 3. DSD file or reference

3.1.2 Header

3.1.3 Change mapping

3.1.4 Transcoding

3.2 CLI Converter Options Windows OS: converter.bat Converter [Options] InputFile OutputFile InputFormat OutputFormat Converter Windows OS: converter.bat Unix OS: converter.sh Options -reg, -dsd_file, -dsd_id, -dsd_agency, -dsd_version, -df, -df_id, -df_version, -df_agency, -header_file, -date_format, -level, -mapping_file, -ordered_input, -trans_file, -delimiter, header_row For further information check the User Manual page 84. converter.bat -dsd_file "C:\ProjectsSharp\myOutputs\First_NA_MAIN_DSD.xml" -header_file "C:\ProjectsSharp\myOutputs\Input_DatasHeader.prop" -header_row DISREGARD_COLUMN_HEADER -delimiter ; "C:\ProjectsSharp\myOutputs\Input_Datas.csv" "C:\ProjectsSharp\myOutputs\CLI_Compact.xml" CSV COMPACT_SDMX

4. Where to find the SDMX Converter You can download the latest version of the SDMX converter on CIRCABC (Current 4.5.0) https://circabc.europa.eu/w/browse/76a63fc2-3d22-42cc-85f2-4f6f30b4d8eb Available packages SDMX Converter Platform Independent SDMX Converter Web Service SDMX Converter installer for Windows 32-bit SDMX Converter Documentation

5. Hands-on exercise Installation Conversion using the Graphical Interface From GESMES to SDMX Compact From CSV to SDMX Compact From SDMX Compact to CSV

SDMX Converter