Eurostat achievements and challenges Emanuele Baldacci, Director European Commission - Eurostat Director Methodology; Corporate statistical and IT services 29 September 2015SDMX Global Conference

Eurostat Start-up phase: implementation Maturity phase: consolidation Moving to the future: modernization 2 Outline

Eurostat 3 Why SDMX? To improve the exchange of statistical data and metadata, which is complex, resource intensive and expensive To provide a standardisation layer replacing or reconciling specific approaches and solutions To take advantage of opportunities related to new technologies for machine to machine exchange In the ESS: 2007: Endorsement at ESS Committee 2008: UNSC recognition : SDMX inserted in legal acts for data-metadata exchange

Eurostat 4 SPECIFY NEEDS DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE Initial focus on data collection from countries This helped the integration of statistical processes in the European Statistical System Phase 1

Eurostat 5 SPECIFY NEEDS DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE Eurostat: Data collection by file format (2014) SDMX-ML + EDI = 47.5% exchanged files. SDMX-EDI declining, XML increasing CSV/Excel (mappable to SDMX)= 33.5% Examples of lowering the burden to data providers, with webforms and CSV/SDMX Some issues: Business buy-in slow, as considerable initial investment needed Real advantages not clear enough for the business Heavy project management needed

Eurostat 6 SPECIFY NEEDS DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE A Hundred DSDs shared with ESS countries DSDs used for regular data exchange within the European Statistical System DSDs for global use with international org. (national accounts, BoP,…)

Eurostat Reference Metadata: full compliance with SDMX ESMS Euro SDMX Metadata Structure ESQRS ESS Standard for Quality Reports Structure SIMS Single Integrated Metadata Structure 7 SDMX concepts and codes for creating Metadata Structure Definitions ESS Metadata Standards covering all Eurostat domains on the web SPECIFY NEEDS DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE

Eurostat 8 Exposed using SDMX-RI Web dissemination Census Hub Census Hub opening: 8 December countries connected (EU & EFTA) 30 using the SDMX-RI Until now > data requests SPECIFY NEEDS DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE SDMX-RI infrastructure enabling buy-in and use of web services for exposing data Data Loaded in DDB

Eurostat SDMX Dissemination Web Services 9 Directly from our dissemination chain: complete list of available datasets complete structure definition of a given dataset download of a subset or a full dataset, customizing requests for da ta SPECIFY NEEDS DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE

Eurostat Input Through- put Output Process 1 Process 2 10 Standard for describing data and metadata throughout the exchange, production and dissemination process Metadata as the decisive abstraction layer to be carried through Broadening the use of SDMX Phase 2 SPECIFY NEEDS DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE

Eurostat Eurostat SDMX roadmap Lower the “entry threshold” More global data sharing (standard concepts/codes across domains and across organisations) Metadata sharing for global use More sophisticated SDMX IT infrastructure Broadening SDMX use cases (e.g. SDMX-based validation) Achieve synergies with other standards (e.g. DDI, Linked Open data) 11

Eurostat SDMX challenges Main challenges for the years to come: Strengthening implementation Facilitating data consumption Supporting statistical process innovation (collection, analysis, dissemination) Broadening SDMX scope (e.g. content validation) Enhancing communication and training 12

Eurostat “Innovation is about making mistakes and learning from them” 13