SDMX Basics course, March 2016 Eurostat SDMX Basics course, March Introducing the Roadmap Marco Pellegrino Eurostat Unit B5: “Data and Metadata Services and Standards” SDMX Basics course, March 2016

Eurostat SDMX Basics course, March 2016 «If you are not sure where you are going, you will finish someplace else» The SDMX Roadmap 2020 The Action List, TWG/SWG work programmes Eurostat Homework Building bridges – the future

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 SDMX Roadmap 2020 Endorsed by the sponsors (SDMX level) - It follows the Action plan High-level global strategy Strategic context for future work - Vision and strategic goals, priorities Key priorities for priority areas 3

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 SDMX 2020 Main challenges for the years to come: Strengthening implementation Facilitating data consumption Supporting statistical process innovation Enhancing communication 4

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March

Eurostat SDMX Basics course, March

Eurostat SDMX Basics course, March

Eurostat SDMX Basics course, March

Eurostat SDMX Basics course, March 2016 Actions List ( ) 9 “ A vision is void if it is not followed by a strategy and concrete actions towards implementation” ESS Vision 2020

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 Technical/Statistical Working Group Workplan

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 Eurostat SDMX roadmap More global data and metadata sharing (standard concepts/codes across domains and across organisations) Lower the “entry threshold” More sophisticated SDMX IT infrastructure Broadening SDMX use cases (e.g. SDMX-based validation) Synergies with other standards (e.g. DDI, Linked Open Data) Eurostat/ESS SDMX implementation strategy 11

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 Initial focus on data collection from countries Shared DSDs with ESS countries for regular data exchange SDMX-compliant reference metadata (ESMS, ESQRS,…) DSDs for global use with international org. (ESA, BoP) SDMX dissemination web services Eurostat: from start-up phase… 12

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March Exposed using SDMX-RI Web dissemination Census Hub Census Hub: opened in December countries connected (EU & EFTA) 30 using the SDMX-RI SPECIFY NEEDS DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE SDMX-RI infrastructure enabling buy-in and use of web services for exposing data Data Loaded in DDB

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 Input Through- put Output Process 1 Process 2 14 Standard for describing data and metadata throughout the exchange, production and dissemination process Broadening the use of SDMX SPECIFY NEEDS DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March  Standard metadata layer for the description and use of data and metadata throughout the process Broadening the scope of SDMX

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 Building bridges 16 …not walls

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 SDMX and Linked Open Data Based on RDF - Resource Description Framework - a family of specifications published by W3C allowing for machine-actionable, semantically rich linking of things found on the Web. Main RDF vocabulary for statistical data: → Data Cube Vocabulary Simplified version of the SDMX model covering data structures 17 Building bridges

SDMX Basics course, March 2016 SDMX Data Structure Definition RDF Data Cube Vocabulary SDMX Data Set structured by dimensionality

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 Data validation  “Technical” - Covered by SDMX today - Format Check (SDMX-ML) - Codes exist (SDMX DSD) - Codes used correctly (Dataflow & Constraint)  “Statistical Domain” - Not yet covered by SDMX (VTL) - Value check - Time series - Revisions - Validation expressions Building bridges

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March SDMX and DDI  DDI Lifecycle can provide a very detailed set of metadata, covering: Surveys and processing of microdata Structure of data files, including hierarchical files and complex relationships Archiving of data files and their metadata Tabulation and processing of data into tables Link between microdata variables and resulting aggregates SDMX can provide: Metadata describing the structure of dimensional data Stand-alone metadata sets (“reference metadata”) Formats for dimensional data A model of data reporting and dissemination Standard registry interfaces, providing a catalogue of resources Guidelines for deploying standard web services A way of describing statistical processes Building bridges

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 SDMX and DDI: similarities and differences Both standards use a similar model for identifiable, versionable and maintainable artefacts Both standards use “schemes”, as packages for lists of items, and XML “schemas” Both standards are designed to support reuse DDI has much more detailed metadata at the level of the study domain, and provides more complete descriptions of the processing of data SDMX provides more architectural components to support registration, reporting/collecting and exchange, and has a solid information model 21

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March

SDMX Basics course, March 2016 Other relevant standards Geospatial standards DDI SDMX GSIM Conceptual model Implementation standards 23

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 Opportunities and challenges SDMX is interacting well with other standards (GSIM, DDI, RDF Linked Open Data, JSON) and this “complementarity” opens us new perspectives for the innovation of statistical processes. Common data validation and processing procedures are required (from structural validation to content). Better metadata-driven statistical production systems, with the use of standards throughout the processes in combination with a metadata registry. Better maintenance and developments of SDMX (e.g. support to use cases, new functions, more formats, etc.) using the wealth of its Information Model. 24

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March A model to describe statistical data and metadata A standard for automated communication from machine to machine A technology supporting standardised IT tools A common language for statistics Statisticians agree to use a common description for data and metadata The data exchange process is then driven by this common description Data descriptions are made available for everybody who wants to understand and reuse the data SDMX provides

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 All good standards change 26 September 2004April 2011November 2005 Version 2.0 SDMX-EDI SDMX-ML SDMX Registry Version 2.0 SDMX-EDI SDMX-ML SDMX Registry Version 1.0 GESMES/TS Version 1.0 GESMES/TS Too much change may discourage adoption But… not giving users the functionalities they want would also discourage adoption ?

SDMX Basics course, March 2016 Eurostat SDMX Basics course, March 2016 Thanks for your attention! “Would you tell me, please, which way I ought to go from here?” “That depends a good deal on where you want to get to” said the Cat. “I don’t much care where” said Alice. “Then it doesn’t matter which way you go” said the Cat. “So long as I get SOMEWHERE” Alice added as an explanation. “Oh, you’re sure to do that” said the Cat, “if you only walk long enough”.