Eurostat SDMX and Global Standardisation Marco Pellegrino Eurostat, Statistical Office of the European Union Bangkok, September 2015SDMX Global Conference

Eurostat Evolution of SDMX Standards integration - Examples Opportunities and challenges - All good standards change 2 Outline

Eurostat 3 A model to describe statistical data and metadata A standard for automated communication from machine to machine A technology supporting standardised IT tools A common language for statistics Statisticians agree to use a common description for data and metadata The data exchange process is then driven by this common description Data descriptions are made available for everybody who wants to understand and reuse the data SDMX provides

Eurostat The same information is needed for exchange between different steps in a statistical production process. The use of SDMX throughout the process, in combination with a metadata registry (central storage of definitions, classifications, etc.) makes it more efficient and coherent to implement changes, e.g. in definitions Metadata-driven systems Broadening the scope of SDMX 4

Eurostat 5  Standard metadata layer for the description and use of data and metadata throughout the process Broadening the scope of SDMX

Eurostat GSBPM and SDMX: towards a more complete picture 6

Eurostat SDMX and standards integration SDMX promotes an incremental movement towards a data and metadata sharing model with the production of comparable and accurate statistics. The increasing use of SDMX: a) improves the quality of the statistical process b) enables simplified exchange and dissemination processes, improving timeliness and accessibility Statistical integration goes hand-in-hand with technical integration and standardisation. 7

Eurostat Building bridges 8 …not walls

Eurostat 9 Building bridges

Eurostat SDMX and Linked Open Data Based on RDF - Resource Description Framework - a family of specifications published by W3C allowing for machine-actionable, semantically rich linking of things found on the Web. Main RDF vocabulary for statistical data: → Data Cube Vocabulary Simplified version of the SDMX model covering data structures 10 Building bridges

SDMX Data Structure Definition RDF Data Cube Vocabulary SDMX Data Set structured by dimensionality

SDMX and RDF: Scenario Triple Store (DataCube) Statistical Dissemination System RDF Service SPARQL SDMX-ML File SDMX-ML File to RDF Transformer Either Or Using SDMX Component Architecture Data Cube Writer

Eurostat Data validation  “Technical” - Covered by SDMX today - Format Check (SDMX-ML) - Codes exist (SDMX DSD) - Codes used correctly (Dataflow & Constraint)  “Statistical Domain” - Not yet covered by SDMX (VTL) - Value check - Time series - Revisions - Validation expressions Building bridges

Eurostat 14 Standard language for defining validation and transformation rules Validation (now) Transformation (partially now, to be enriched at a later stage) Main goals Define and preserve validation and transformation rules Exchange and share rules Apply rules in industrialized processes Apply to several standards (e.g. SDMX, DDI, GSIM) thanks to a generic information model VTL: Validation and Transformation Language

Eurostat 15 SDMX and DDI  DDI Lifecycle can provide a very detailed set of metadata, covering: Surveys and processing of microdata Structure of data files, including hierarchical files and complex relationships Archiving of data files and their metadata Tabulation and processing of data into tables Link between microdata variables and resulting aggregates SDMX can provide: Metadata describing the structure of dimensional data Stand-alone metadata sets (“reference metadata”) Formats for dimensional data A model of data reporting and dissemination Standard registry interfaces, providing a catalogue of resources Guidelines for deploying standard web services A way of describing statistical processes Building bridges

Eurostat SDMX and DDI: similarities and differences Both standards use a similar model for identifiable, versionable and maintainable artefacts Both standards use “schemes”, as packages for lists of items, and XML “schemas” Both standards are designed to support reuse DDI has much more detailed metadata at the level of the study domain, and provides more complete descriptions of the processing of data SDMX provides more architectural components to support registration, reporting/collecting and exchange, and has a solid information model 16

17

Other relevant standards Geospatial standards DDI SDMX GSIM Conceptual model Implementation standards 18

Eurostat Opportunities and challenges SDMX is interacting well with other standards (GSIM, DDI, RDF Linked Open Data, JSON) and this “complementarity” opens us new perspectives for the innovation of statistical processes. Common data validation and processing procedures are required (from structural validation to content). Better metadata-driven statistical production systems, with the use of standards throughout the processes in combination with a metadata registry. Better maintenance and developments of SDMX (e.g. support to use cases, new functions, more formats, etc.) using the wealth of its Information Model. 19

Eurostat All good standards change 20 September 2004April 2011November 2005 Version 2.0 SDMX-EDI SDMX-ML SDMX Registry Version 2.0 SDMX-EDI SDMX-ML SDMX Registry Version 1.0 GESMES/TS Version 1.0 GESMES/TS Too much change may discourage adoption But… not giving users the functionalities they want would also discourage adoption

Eurostat Thanks for your attention! 21 SDMX and Global Standardisation « If you are not sure where you are going you will finish someplace else »