1. SDMX Converter Raynald PALMIERI June 2015
IPA Workshop on ITS Statistics

2. Table of Contents Objectives What is the Converter Interfaces
Steps of a conversion
Minimum requirements
Supported formats
Interfaces
GUI
CLI

3. Table of Contents Where to find the SDMX Converter Hands-on exercise
Installation
Simple conversions
From GESMES to SDMX Compact
From CSV to SDMX Compact
From SDMX Compact to CSV

4. 1. Objectives Illustrate how to use the Converter
List all the supported formats
Explain its interfaces

5. 2. What is the Converter
The SDMX converter is an application that converts files between different formats
File based conversion (No DB)
Open source and platform independent (Java)
Different ways of using it:
Graphical user interface
Batch file …
Formats: SDMX-ML 2.0, SDMX-ML 2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, Excel
The SDMX Converter has been developed in the context of the SODI project.
It is a Java tool that allows converting from/to various dataset types based on SDMX-ML DSDs.
The allowed types between which the Converter is capable of converting are:
SDMX-ML formats (in order to go from/to Cross-Sectional type the DSD should contain Cross-Sectional information and TimeDimension)
GESMES/TS (aka SDMX-EDI) (not capable of converting from/to SDMX-ML Cross-Sectional format)
GESMES/2.1 and GESMES/DSIS
CSV, FLR, EXCEL, DSPL (supports mapping mechanism and parametric delimiter for CSV. Converting to CSV/FLR from other formats may result in loss of attributes attached at a higher level than observation.)

6. 2.1 Steps of a conversion Steps to proceed a conversion
Reading the input message
Reading the DSD
Writing the converted message
<pagebreak>
The conversion process comprises two main activities; reading an input data message and writing out the converted data message.
The are specific modules that read and write datasets i.e SDMX-ML (Compact, Generic, Utility, Cross-Sectional) Gesmes (TS, 2.1, DSIS) Flat files (CSV FLR). The information of a dataset to converter is stored in classes that are based on the SDMX Information Model v2.0. These classes play the role of an intermediate format between readers and writers.
The Data Structure Definition related to the converted datasets is needed for performing a conversion. SDMX Converter, if it’s is not provided manually, can retrieved the DSD from the Registry.

7. 2.2 Minimum requirements Input file Output file (complete path)
Format for input and output files
Specify DSD
DSD file
Reference to a DSD file in the Registry

8. 2.3 Supported formats
SDMX 2.0
GENERIC SDMX
COMPACT SDMX
UTILITY SDMX
CROSS-SECTIONAL SDMX
SDMX 2.1
GENERIC DATA 2.1
GENERIC TS DATA 2.1
STRUCTURE SPECIFIC DATA 2.1
STRUCTURE SPECIFIC TS DATA 2.1
GESMES
GESMES TS
GESMES 2.1
GESMES DSIS
OTHERS
CSV
FLR
DSPL
EXCEL
<pagebreak>
The conversion process comprises two main activities; reading an input data message and writing out the converted data message.
The are specific modules that read and write datasets i.e SDMX-ML (Compact, Generic, Utility, Cross-Sectional) Gesmes (TS, 2.1, DSIS) Flat files (CSV FLR). The information of a dataset to converter is stored in classes that are based on the SDMX Information Model v2.0. These classes play the role of an intermediate format between readers and writers.
The Data Structure Definition related to the converted datasets is needed for performing a conversion. SDMX Converter, if it’s is not provided manually, can retrieved the DSD from the Registry.
Limitations may apply. Please, check the User Manual.

9. 3. Interfaces
User interface
Command line

10. 3.1 GUI 1. Selection of the input/output files and their format
2. Select the DSD in the local drive
2. Identify a DSD to download from the SDMX Registry (configuration required)
2. If the local DSD includes multiple versions, we can specify the one derired
2. If the local DSD includes multiple versions, we can specify the one desired
2. Identify a dataflow linked to the DSD to download from the SDMX Registry (configuration required)
There are some mandatory fields, e.g. the DSD is mandatory, therefore, either loading from a file or from the Registry, it must be present. A message will appear if there are mandatory fields missing.
3. SDMX header (.prop file)
Only for flat and excel files
CSV parameters
4. Mapping
5. SDMX (output) validation
XML parameters for SDMX output formats

11. 3.1.1 Example 1. Input and output files 2. Format
3. DSD file or reference

12. 3.1.2 Header

13. 3.1.3 Change mapping

14. 3.1.4 Transcoding

15. 3.2 CLI Converter Options Windows OS: converter.bat
Converter [Options] InputFile OutputFile InputFormat OutputFormat
Converter
Windows OS: converter.bat
Unix OS: converter.sh
Options
-reg, -dsd_file, -dsd_id, -dsd_agency, -dsd_version,
-df, -df_id, -df_version, -df_agency, -header_file,
-date_format, -level, -mapping_file, -ordered_input,
-trans_file, -delimiter, header_row
For further information check the User Manual page 84.
converter.bat -dsd_file "C:\ProjectsSharp\myOutputs\First_NA_MAIN_DSD.xml" -header_file "C:\ProjectsSharp\myOutputs\Input_DatasHeader.prop" -header_row DISREGARD_COLUMN_HEADER -delimiter ; "C:\ProjectsSharp\myOutputs\Input_Datas.csv" "C:\ProjectsSharp\myOutputs\CLI_Compact.xml" CSV COMPACT_SDMX

16. 4. Where to find the SDMX Converter
You can download the latest version of the SDMX converter on CIRCABC (Current 4.5.0)
Available packages
SDMX Converter Platform Independent
SDMX Converter Web Service
SDMX Converter installer for Windows 32-bit
SDMX Converter Documentation

20. 5. Hands-on exercise Installation
Conversion using the Graphical Interface
From GESMES to SDMX Compact
From CSV to SDMX Compact
From SDMX Compact to CSV

21. SDMX Converter