1. 1 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX Basics Core Elements Information Model Data Structure Definition (DSD) SDMX-ML Messages Major changes in SDMX v 2.1

2. 2 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 THE SDMX COMPONENTS Technical Specifications The SDMX Information Model Technical Specifications The SDMX Information Model Guidelines to Hamonise Content The Content Oriented Guidelines (COG) Guidelines to Hamonise Content The Content Oriented Guidelines (COG) Tools IT Architectures for data exchange SDMX compliant tools Tools IT Architectures for data exchange SDMX compliant tools

3. 3 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 The SDMX Information Model is a meta-model describing the objects involved in: The collection The dissemination The publication of aggregated statistics and related metadata The abstract model is like a structured set of containers Everything in SDMX is model-driven: All messages and interfaces are implementations of the information model THE SDMX INFORMATION MODEL

4. 4 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX INFORMATION MODEL – SCOPE DATA & METADATA FLOWS Structure Definition Category Scheme Category Constraint Provision Agreement Data Provider Data & Metadata set

5. 5 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX Information Model

6. 6 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 STATISTICAL DATA & METADATA Time series data representation Cross-sectional data representation Statistical Data (Figures) Statistical Metadata (Identifiers, Descriptors) Structural metadata Reference metadata Statistical Metadata (Methodology, Quality)

7. 7 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Statistical data - Cube Time 2005 2006 Country FRITESAT Tourism activity A100 B010 B020 2007 Time series Cross-section for 2006 8174 8138 8052 542 1216 8138 2510 STATISTICAL DATA & METADATA Two different ways to represent data

8. 8 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 STATISTICAL DATA - TIME SERIES REPRESENTATION

9. 9 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 STATISTICAL DATA - CROSS-SECTIONAL REPRESENTATION

10. 10 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 From a number to statistical data 11353511 STRUCTURAL METADATA Introduction

11. 11 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 CONCEPTS STRUCTURAL METADATA Identify and describe data Dimension, Attribute or Measure in a DSD to define a Data set’s structure Attributes in a MSD to define the structure of a Metadata set

12. 12 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 STRUCTURAL METADATA From a statistical table to its descriptor concepts

13. 13 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 STRUCTURAL METADATA – CONCEPTS AND ROLES

14. 14 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 DSD STRUCTURAL METADATA: DATA STRUCTURE DEFINTION To easily exchange and process data, we first define a standard container based on the structure of the real statistical table: The Data Structure Definition (DSD) Code lists Dimensions Attributes Measures Concepts UNIT TIME_PERIOD COUNTRY OBSERVATIONS The DSD can be seen as a "logical container" for a specific set of data that we want to exchange. It includes the concepts that represent the data, gives them roles (Dimension, Measure, Attributes) and links them to code lists.

15. 15 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 ELEMENTS OF A DATA STRUCTURE DEFINITION

16. 16 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – 10-11 and 14-15 March 2011 Dataset DSD SDMX does not introduce any new concept for statisticians. It just provides a framework for what statisticians already know. Code lists Observations Table structure The SMDX dataset is a standard container in which statistical data are represented together with the structural metadata, according to the DSD. SDMX INFORMATION MODEL - DATA SET Now you have an easy way to exchange and process data and metadata automatically.

17. 17 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 DATA SET KEY GROUP KEY KEY VALUES TIME PERIOD OBSERVATION VALUE OBSERVATION VALUE ATTRIBUTE VALUE ATTRIBUTE VALUE Attribute attachment Cross-section Time series SDMX INFORMATION MODEL - DATA SET

18. 18 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX INFORMATION MODEL - DATA SET

19. 19 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX INFORMATION MODEL - DATA SET

20. 20 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 REFERENCE METADATA

21. 21 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Reference Metadata Set SDMX INFORMATION MODEL - METADATA SET Concepts

22. 22 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX INFORMATION MODEL – DATA & METADATA FLOW DATA & METADATA FLOWS Structure Definition Category Scheme Category Constraint Provision Agreement Data Provider Data & Metadata set

23. 23 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX INFORMATION MODEL – CATEGORIES

24. 24 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX IM – DATA PROVIDERS & PROVISION AGREEMENT Production and dissemination of Statistical data Production and dissemination of Reference Metadata

25. 25 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 DATA & METADATA FLOWS Constraint Provision Agreement SDMX IM - CONSTRAINTS

26. 26 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX IM - CONSTRAINTS Example: A data provider can restrict his reporting of monthly data to only some months. Example: A data provider can restrict his reporting of data to subsets of statistical cubes.

27. 27 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX IM - SUMMARY

28. 28 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 THE SDMX COMPONENTS Technical Specifications The SDMX Information Model Technical Specifications The SDMX Information Model Guidelines to Hamonise Content The Content Oriented Guidelines (COG) Guidelines to Hamonise Content The Content Oriented Guidelines (COG) Tools IT Architectures for data exchange SDMX compliant tools Tools IT Architectures for data exchange SDMX compliant tools

29. 29 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 IT ARCHITECTURES FOR DATA EXCHANGE

30. 30 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX REGISTRY REGISTRY

31. 31 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX REGISTRY DEMONSTRATION

32. 32 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX Data Structure Definition (DSD)

33. 33 COMPLIANCE & IMPLEMENTATION Generally the following four steps need to be done: 1.Preparation: The statisticians from the organisations involved in the data exchange describe the data and the different dataflows, dataset and provision agreements. 2.Compliance: you create all the necessary objects according to the SDMX Technical Specifications. 3.Implementation: Now we put into practice. Standard software is installed and configured to use the DSDs. The exchange process is set up and tested. 4.Production: use the objects in the production process. SDMX implementation is achieved when the data and metadata exchanges within the domain are carried out according to SDMX- compliant specifications.

34. 34 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Define the DSD –List of concepts (Concept scheme) –Roles of concepts (Dimension, Attribute, Measure) –Code lists Provide the related Dataflows (e.g. STSRTD_TURN_M, DEMOGRAPHY_RQ) CREATE ALL THE NECESSARY OBJECTS

35. 35 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 THE STEPS TO BUILD A DATA STRUCTURE DEFINITION Identification of the descriptor concepts for the data Choose the type of data representation (Time Series and Cross-sectional ) Identification of the descriptor concepts for the data Choose the type of data representation (Time Series and Cross-sectional ) Choice of Cross Domain code lists or definition of specific code lists for coded concepts Definition of the text format for non coded concepts Definition of the concept role (Dimension, Attribute or Measure) Define Dimensions for Time Series and Cross-sectional data representation Define Attributes with the attachment levels Time Series and Cross-sectional data representation Define Time Series primary measure and/or Cross- sectional measures with their measure concepts Create the defined artefacts in a SDMX Data Structure Definition tool (e.g. DSW) 1 2 3 4 5

36. 36 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 1- IDENTIFICATION OF THE DESCRIPTOR CONCEPTS

37. 37 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 2 – DEFINE THE CODE LISTS

38. 38 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Cross-sectional slice Time series slice Statistical data - Cube Country ESITFRAT Tourism activity A100 B010 B020 Time 2005 2006 2007 Time series Cross-section for 2006 1250 1216 1220 542 1216 8138 2510 3- CHOOSE THE TYPE OF DATA REPRESENTATION TIME SERIES (TS) / CROSS-SECTIONAL (CS)

39. 39 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 DATA REPRESENTATION – TIME SERIES

40. 40 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 DATA REPRESENTATION – CROSS-SECTIONAL

41. 41 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 4- DEFINE ROLES OF CONCEPTS AND LIST OF CONCEPTS

42. 42 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 5 – DEFINE GROUPS AND ATTRIBUTE ATTACHEMENTS

43. 43 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Eurostat Unit B5 – Statistical Information Technologies SDMX Training for Statisticians – March 2010 6 – DEFINE THE VIEW OF THE DATA STRUCTURE

44. 44 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 EXAMPLE: STS SAMPLE DATASET Dimensions Attributes Primary Measure Dimensions

45. 45 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 EXAMPLE: STS SAMPLE DATASET STS_INDICATOR TITLE STS_ACTIVITY REFERENCE_AREA FREQSTS_ BASE_YEAR ADJT

46. 46 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 OBS_STATUSOBS_VALUE REFERENCE_PERIOD OBS_CONF STS_INSTITUTION EXAMPLE: STS SAMPLE DATASET

47. 47 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 M;GR;N;TOTV;NS5201;1;2000;200201;88.8;A;F M;GR;N;TOTV;NS5201;1;2000;200202;84.7;A;F M;GR;N;TOTV;NS5201;1;2000;200203;88.8;A;F M;GR;N;TOTV;NS5201;1;2000;200204;93.0;A;F M;GR;N;TOTV;NS5201;1;2000;200205;60.8;A;F M;GR;N;TOTV;NS5201;1;2000;200206;78.2;A;F M;GR;N;TOTV;NS5201;1;2000;200207;89.9;A;F AttributesPrimary Measure Dimensions M;GR;N;TOTV;NS5201;1;2000;200201;88.8;A,F Reference PeriodGroup EXAMPLE: STS SAMPLE DATASET IDENTIYING CONCEPTS AND GROUPING SERIES IN CSV FILES

48. 48 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 DSD OF DATAFLOW STSRTD_IND_M ConceptConcept ID frequency FREQ reference area REF_AREA adjustment ADJUSTMENT type of index STS_INDICATOR activity STS_ACTIVITY type of institution STS_INSTITUTION base year STS_BASE_YEAR reference period TIME_PERIOD turnover idex OBS_VALUE status OBS_STATUS confidentiality OBS_CONF time duration set TIME_FORMAT Title TITLE decimals DECIMALS Example of value Remark MMonthly GRGreece NNo TOVV Turnover deflated (volume of sales) NS5201Retail trade 1 1=NSI or 2=National Bbank 2000 200201CCYYMM 108.6observation Aactual data FFree of publication P1MISO8601 1One Code List CL_FREQ CL_AREA_EE CL_ADJUSTMENT CL_STS_INDICATOR CL_STS_ACTIVITY CL_STS_INSTITUTION CL_STS_BASE_YEAR CL_OBS_STATUS CL_OBS_CONF CL_TIME_FORMAT CL_DECIMALS Dimensions Measure Attributes Attachment level Obs Series Group List of variablesValuesCodesRolesFootnotes

49. 49 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 STRUCTURE OF THE DATASET FOR TIME SERIES Group of series Series M;GR;N;TOTV;NS5201;1;2000;200201;88.8;A;F M;GR;N;TOTV;NS5201;1;2000;200202;84.7;A;F M;GR;N;TOTV;NS5201;1;2000;200203;88.8;A;F M;GR;N;TOTV;NS5201;1;2000;200204;93.0;A;F REF_AREA="GR" ADJUSTMENT="N" STS_INDICATOR="TOTV" STS_ACTIVITY="NS5201" STS_INSTITUTION="1" STS_BASE_YEAR="2000" DECIMAL="1" TITLE="Retail trade" Attributes and attachment level: group M;GR;N;TOTV;N15220;1;2000;200201;60.8;A;F M;GR;N;TOTV;N15220;1;2000;200202;78.2;A;F M;GR;N;TOTV;N15220;1;2000;200203;89.9;A;F Group of series REF_AREA="GR" ADJUSTMENT="N" STS_INDICATOR="TOTV" STS_ACTIVITY="N15220" STS_INSTITUTION="1" STS_BASE_YEAR="2000" DECIMAL="1" TITLE="Retail sale of food" Attributes can be attached to groups Series

50. 50 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Definition of Series 1 M;GR;N;TOTV;NS0006;1;2000;200201;88.8;A;F M;GR;N;TOTV;NS0006;1;2000;200202;84.7;A;F M;GR;N;TOTV;NS0006;1;2000;200203;88.8;A;F FREQ="M" REF_AREA="GR" ADJUSTMENT="N" STS_INDICATOR="TOTV" STS_ACTIVITY="NS0006" STS_INSTITUTION="1" STS_BASE_YEAR="2000" TIME_FORMAT="P1M" Attributes and attachment level: series M;GR;N;TOTV;N14500;1;2000;200201;60.8;A;F M;GR;N;TOTV;NS0006;1;2000;200202;78.2;A;F M;GR;N;TOTV;NS0006;1;2000;200203;89.9;A;F Definition of Series 2 FREQ="M" REF_AREA="GR" ADJUSTMENT="N" STS_INDICATOR="TOTV" STS_ACTIVITY="N14500" STS_INSTITUTION="1" STS_BASE_YEAR="2000" TIME_FORMAT="P1M" Attributes can be attached to series Series 1 Series 2 STRUCTURE OF THE DATASET FOR TIME SERIES

51. 51 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Definition of Series 1 FREQ="M" REF_AREA="GR" ADJUSTMENT="N" STS_INDICATOR="TOTV" STS_ACTIVITY="NS0006" STS_INSTITUTION="1" STS_BASE_YEAR="2000" TIME_FORMAT="P1M" Attributes and attachment level: series Attributes can be attached to observations Definition of Observation 1 TIME_PERIOD="200201" OBS_VALUE="88.8" OBS_STATUS="A" OBS_CONF="F" Definition of Observation 2 TIME_PERIOD="200202" OBS_VALUE="84.7" OBS_STATUS="A" OBS_CONF="F" Definition of Observation 2 TIME_PERIOD="200203" OBS_VALUE="88.8" OBS_STATUS="A" OBS_CONF="F" M;GR;N;TOTV;NS0006;1;2000;200201;88.8;A;F M;GR;N;TOTV;NS0006;1;2000;200202;84.7;A;F M;GR;N;TOTV;NS0006;1;2000;200203;88.8;A;F Observation 1 Observation 2 Observation 3 CSV STRUCTURE OF THE DATASET FOR TIME SERIES

52. 52 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 EXAMPLE 2: DEMOGRAPHY SAMPLE DATASET Measures Attributes DimensionsDimensions

53. 53 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 TITLE TIME_PERIODTIME_PERIOD TAB_NUM REV_NUMOBS_STATUS FREQFREQ COUNTRYCOUNTRY Dimensions attached to the dataset level Dimensions attached to the group level EXAMPLE 2: DEMOGRAPHY SAMPLE DATASET

54. 54 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 OBS-VALUE DEMODEMO SEXSEX UNIT MALE Dimensions attached to the observation level Measure Dimension FEMALE TOTAL EXAMPLE 2: DEMOGRAPHY SAMPLE DATASET

55. 55 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 DSD FOR DATAFLOW: DEMOGRAPHY_RQ

56. 56 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Dataset Attributes and attachment level Attribute attached to group COUNTRY="FI" Group REF_PERIOD="2005" FREQ="A" TIME_FORMAT="P1Y" Section DECI="0" UNIT="PERS" UNIT_MULT="0" Dimension attached to dataset Attributes attached to sections Dimension attached to group Observation FEMALE OBS_VALUE="35" DEMO="ADJT" OBS_STATUS="P" Cross–sectional measure Dimensions attached to observation Attribute attached to observation MALE OBS_VALUE="29400" DEMO="LBIRTHST" OBS_STATUS="P" TOTAL OBS_VALUE="8986" DEMO="NETMT" OBS_STATUS="P" Observation STRUCTURE OF THE DATASET FOR CROSS SECTIONAL

57. 57 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Organisation Schemes Organisation Schemes DSDs Concept Schemes Category Schemes DataFlows Code lists CREATION OF THE DSD THE SDMX OBJECTS RELATED TO THE DATA STRUCTURE

58. 58 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 DSW – “standalone” desktop application (replaced KeyFamily AccessDB tool) Offline version of Eurostat’s SDMX Registry Maintenance of SDMX v2.0 data and meta data structures (create, modify, delete, query) Import/Export SDMX-ML structures (validate structure messages) Import/Export GESMES/TS structure files Reporting of structures Advanced search features Export metadata for use with the GENEDI tool Data Authoring (building SDMX-ML sample datasets) Interaction with any SDMX v2.0 compliant Registry Query SDMX v2.0 Registry Submit data structures to SDMX v2.0 Registry Interaction with any SDMX v2.0 compliant Registry Query SDMX v2.0 Registry Submit data structures to SDMX v2.0 Registry SDMX Registry Import/Export SDMX-ML messages CREATION OF THE DSD: DATA STRUCTURE WIZARD

59. 59 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Example - DSD import / creation using the DSW

60. 60 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 LIFE DEMONSTRATION - DSD IMPORT / CREATION USING THE DATA STRUCTURE WIZARD

61. 61 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 DATA STRUCTURE DEFINITION IDFISH_CATCH_A NameCatches for all fishing areas Version1.0 AgencyIDESTAT Valid From Valid To EXERCISE: CREATION OF THE DSD: FISH_CATCH_A

62. 62 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 DIMENSIONS Position in Key CONCEPTREPRESENTATION Dimension Type IDName CONCEPT SCHEMECODELIST TEXT FORMAT IDVERAGENCYIDVER AGENC Y 1FREQFrequencyCS_FISHERIES1.0ESTATCL_FREQ1.1ESTATFrequency 2REPORTING_AREA Country ISO3 codes (extended) CS_FISHERIES1.0ESTAT CL_REPO RTING_AR EA 1.0ESTAT 3 PRODUCTION_AR EA Production Area (from major area to sub-unit) CS_FISHSTAT1.0FAO CL_PROD UCTION_A REA 1.0FAO 4SPECIES ASFIS Species Alpha 3 Code CS_FISHSTAT1.0FAO CL_SPECI ES 1.0FAO TIMETIME_PERIODReference yearCS_FISHERIES1.0ESTAT EXERCISE: CREATION OF THE DSD: FISH_CATCH_A

63. 63 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 MEASURES TYPE CONCEPTREPRESENTATION MEASUR E DIMENSI ON CODE IDName CONCEPT SCHEMECODELIST TEXT FORMAT IDVERAGENCYIDVERAGENCY PrimaryOBS_VALUEValue of the measure CS_FISH ERIES 1.0ESTATN/A ATTRIBUTES ATTACHMENT LEVEL CONCEPTREPRESENTATION ATTRIBUTE TYPE ASSIGNMENT STATUS IDName CONCEPT SCHEMECODELIST TEXT FORMAT IDVERAGENCYIDVERAGENCY ObservationUNITunitCS_FISHERIES1.0ESTATCL_UNIT1.1ESTATC EXERCISE: CREATION OF THE DSD: FISH_CATCH_A

64. 64 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX Converter Data Structure Wizard SDMX Technical Standard v2.0 (http://www.sdmx.org/index.php?page_id=16) Help-desk: SDMX-support@sogeti.lu USEFUL LINKS

65. 65 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX-ML Messages

66. 66 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Based on a common Information Model –SDMX-EDI (GESMES/TS) EDIFACT syntax Time-series oriented – One format for Data Sets –SDMX-ML XML syntax Four different formats for Data Sets Easier validation (XML based) SYNTAXES FOR SDMX MESSAGES

67. 67 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX DATA COMMON HEADERS

68. 68 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Equivalent representations for reporting Datasets SDMX DATA MESSAGES Version 2.0Version 2.1 4 data messages, each with a distinct format. GenericData CrossSectional Data Compact Data UtilityData Therefore, there are now 4 data messages which are based on two general formats: GenericData GenericTimeSeriesData StructureSpecificData StructureSpecificTimeSeriesData Phased out

69. 69 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 EXAMPLE OF GENERIC SDMX-ML MESSAGE

70. 70 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 EXAMPLE OF COMPACT SDMX-ML MESSAGE

71. 71 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 EXAMPLE OF CROSS-SECTIONAL SDMX-ML MESSAGE

72. 72 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Equivalent formats Generic SDMX-ML Cross-sectional SDMX-ML Compact SDMX-ML Can be expanded to other formats (e.g. CSV, GESMES) Based on the same IM Exceptions: If a Cross-Sectional DSD does NOT contain a time dimension Exceptions: If a Cross-Sectional DSD does NOT contain a time dimension CONVERSIONS SDMX V2.0

73. 73 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Read the input message Parsing Populate the data model of the tool (based on the SDMX v2.0 information model) Write the converted message Uses the data model to write the output message in the required target format. Information retrieved from the Registry Data flow ID is used to retrieve the data flow definition from the Registry. The DSD ID, version and agencyID are retrieved from the data flow definition and are used to acquire the DSD SDMX CONVERTER

74. 74 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Possible conversions CSV Compact SDMX-ML Generic SDMX-ML Utility SDMX-ML Cross-sectional SDMX-ML * SDMX-EDI (GESMES/TS) CSV Compact SDMX-ML Generic SDMX-ML Utility SDMX-ML Cross-sectional SDMX-ML * SDMX-EDI (GESMES/TS) CSV Compact SDMX-ML Generic SDMX-ML Utility SDMX-ML Cross-sectional SDMX-ML SDMX-EDI (GESMES/TS) CSV Compact SDMX-ML Generic SDMX-ML Utility SDMX-ML Cross-sectional SDMX-ML SDMX-EDI (GESMES/TS) Main use: Conversion CSV  Compact SDMX-ML SDMX CONVERTER MAIN FUNCTIONALITY

75. SDMX training session on basic principles, Major Changes in version 2.1 Fabien JACQUET SDMX Basics MMMM 2011 Select the Input fileSelect the output file Select the input and output formats Select the DSD on the local drive Identify a DSD to download from the SDMX Registry Identify a dataflow linked to the DSD to download from the SDMX Registry Select / manage headers for CSV input formats Select mapping / transoding tables CSV parameters GESMES representation for GESMES output formats Load / save the current settings XML parameters for SDMX output formats

76. 76 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Conversion Example

77. 77 Major changes in SDMX v 2.1

78. 78 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Overview of the changes Structural Metadata –Data Structure Definition (DSD) –Metadata Structure Definition (MSD) –Constraint –Code List –Organisation Scheme –Categorising Structures –Process –Provision Agreement –Transformations and Expressions Data Set –Message Changes –Structured Data Mechanism Revised Metadata Set –Message Changes –Alignment of Formats –Structured Metadata Mechanism Revised

79. 79 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Data structure Definition (DSD) Support for non-time-series data structures Measure Dimension DSD Code lists Dimensions And Measure dimension Dimensions And Measure dimension Attributes Measures Concepts DSD Version 2.0Version 2.1 Measure Dimension Dimensions Attributes Primary Measure Concepts Concept Scheme Code lists

80. 80 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Maintainable artefact Constraint Version 2.0 Version 2.1 Dataflow Provision agreement Constraint Registry Constraint DataflowCode list Provision agreement DSD

81. 81 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Code List Common Code list Common Code list Constraint 1 Partial DSD Constraint 2 Version 2.1

82. 82 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Categorising Structures Version 2.0 Version 2.1 Category Scheme Data/Metadata flow Reference Categorisation Data/MetadataflowCode list Category Reference Provision agreement DSD Category Only Maintainable artefact

83. 83 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 Version 2.0Version 2.1 Message Changes Data Set 4 data messages, each with a distinct format. GenericData CrossSectionalData CompactData UtilityData Therefore, there are now 4 data messages which are based on two general formats: GenericData o GenericTimeSeriesData StructureSpecificData o StructureSpecificTimeSeriesData Phased out