1 Meeting on the Management of Statistical Information Systems (MSIS 2010) SDMX architecture for data sharing and interoperability Francesco Rizzo, ISTAT, Italy Adam Wronski, Eurostat Daejeon, Republic of Korea, April 2010

2 SDMX service infrastructure: rational and keywords The idea is based on the analysis of the current level of SDMX implementations within Member States and their needs expressed during several TF, WG and other meetings on the possibility to re-use software already developed with the main aims of reducing costs and increasing productivity

3 SDMX can help by …. improving quality and efficiencies in the exchange and dissemination of data and metadata; reducing national reporting burden; reducing costs through the re-use of the software; facilitating and standardizing the use of new technologies as XML and Web services.

4 Data Repository (Warehousing) Architecture NSI Eurostat Pull Requestor eDAMIS Data reception SDMX Registry Loading preparation Verification / Conversion o SDMX Received data in SDMX-ML Loader register Data Warehouse Database query Dissemination XSL for SDMX-ML PULLPULL PUSHPUSH

5 Data Hub Architecture Dissemination XSL for SDMX-ML Data Hub Query SDMX messages cache Web Service Web Service Web Service GUI Data Providing Organizations Data collector Organization

6 Eurostat strategies to foster SDMX implementations within NSIs A project aiming to design an SDMX service infrastructure for NSIs and developing related building blocks; Finance and support, through SDMX ESSnet, a group of Member States that have joined their resources in order to develop SDMX re-usable software. Capacity building actions

7 SDMX NSI reference service infrastructure: main objectives support Census Hub and other Eurostat project facilitate SDMX implementation within NSIs with a particular attention to PC-AXIS community stimulate a “SDMX community of developer”

8 SDMX NSI reference service infrastructure: deliverables SDMX NSI reference service architecture document  It represents the syntheses of several experiences worldwide and may be considered not as a strict specification but rather a guide or “best practice” document;  It provides a description/specification of a generalized infrastructure that could be re-used partially or entirely by NSIs interested in SDMX projects; A set of software building blocks; Mapping Assistant tool; Capacity building actions (Training and Technical Workshops).

9 SDMX NSI reference service infrastructure Hub Pull Requestor Data Collector SDMX NSI Infrastructure NSI Dissemination environment Mapping Store Mapping Assistant NSI Web Service Web Service Provider Data Retriever SDMX Data Generator SDMX Query Parser DDB Pull DSDs

10 The mapping process Data within Data Providers’ dissemination databases are described using “local” structural metadata (concepts, code lists, formats); “local” structural metadata and SDMX structural metadata must be mapped:  concepts mapping  codes mapping

11 Concept and code mapping: an example LOCAL CONCEPTSLOCAL CODESSDMX CODESSDMX CONCEPTS Freq12MFREQ ITREF_AREA TypemWADJUSTMENT Domain Set Category e ip 63 PRODSTS_INDICATOR AtecoDA DB N100DA N100DB STS_ACTIVITY 1STS_INSTITUTION Umpe2000STS_BASE_YEAR PURE_NUMBUMIS 0UMIS_MULT Year, Month2005, TIME_PERIOD P1MTIME_FORMAT ValueOBS_VALUE AOBS_STATUS

12 The Mapping Assistant tool: workflow Map local Database schema (Dataset) Map local Concepts To DSD (Mapping Set) Map local Codes To DSD (Transcoding)

13 The Mapping Assistant tool: Dataset

14 The Mapping Assistant tool: Mapping Set

15 The Mapping Assistant tool: Transcoding

16 Software maintenance and governance The software is offered as open source package under the EUPL licence Up to now Eurostat is managing both the evolutive and adaptive maintenance. How the versioning will be handled when NSIs will re-use and improve the software? Is the model used by many “open source” community applicable to the “statistical community”? know-how could come from NSIs participating in two ESSnet projects