1. Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 IT architectures for data exchange SDMX-RI and the Hub approach Nadezhda Vlahova Marco Pellegrino

2. 2 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 Data Repository (Warehousing) Architecture NSI Eurostat Pull Requestor eDAMIS Data Input SDMX Registry Intermediate storage Verification / Conversion To SDMX Received data in SDMX-ML Loader register Warehouse storage Eurobase query Dissemination XSL for SDMX-ML PULLPULL PUSHPUSH

3. 3 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012

4. 4 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 The European Census Hub: key issues  Dissemination of the data from the 2011 population and housing censuses in the European Union  Data that are methodologically comparable and structured according to “hypercubes” agreed with Member States (Census Regulation)  Providing users with an easy access to detailed census data (advanced functionalities)  Management of massive amounts of data produced and controlled by Member States  High accessibility to data and metadata  Harmonised concepts and definitions  Maximum flexibility to cross-tabulate data from different sources  Easy to use 4

5. 5 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 EU Census: Implementing measures Regulation (EC) 763/2008 on population and housing censuses authorises the European Commission to adopt implementing measures on: –technical specifications of the topics and their breakdown (Regulation (EC) 1201/2009) –programme of the statistical data and metadata to be transmitted to Eurostat (Regulation (EU) 519/2010) –quality reporting and technical format of data transmission (Regulation (EU) 1151/2010)

6. 6 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 Article 6 of Regulation (EU) 1151/2010 –Member States shall transmit the required data conforming to the data structure definitions and related technical specifications provided by the Commission (Eurostat) –The technical format to be used for the transmission of data and metadata for the reference year 2011 shall be the Statistical Data and Metadata eXchange (SDMX) format –Member States shall store until 1 January 2025 the required data and metadata for any later transmission requested by the Commission (Eurostat) Technical format for data transmission

7. 7 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 Based on Census Regulation - Data Hub is: System of DSDs built on SDMX 2.0 (standard concepts and codes) and in use for 31 countries of the European Statistical System data dissemination portal based on SDMX data model –communicating with data providers via SDMX Web Service no data processing (no editing or aggregation) additional reusable modules for –LAU / NUTS management –Tool for handling SDMX structural metadata innovative user interface allowing user to extract data by starting from the statistical concept Tests on-going: started with dummy data and sample hypercube, now to continue with real hypercubes MSs status: 22 up and running

8. 8 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 Population topics Sex SEX AgeAGE Legal marital statusLMS Country/place of birth POB Country of citizenship COC Place of usual residence - one year prior to the censusROY (Size of the) LocalityLOC Household statusHST Type of private householdTPH Size of private householdSPH Family statusFST Type of family nucleusTFN Size of family nucleusSFN Topics required for all geographical levels down to Local Administrative Units (LAU = municipalities) Housing topics Occupancy status of conventional dwellingsOCS Number of occupantsNOC Useful floor space and/or Number of roomsUFS/NOR Density standardDFS/DRM Dwellings by type of buildingTOB Dwellings by period of constructionPOC Type of living quartersTLQ Which data for which geographical area?

9. 9 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 Topics required for aggregated geographical levels NUTS 2, NUTS 1 and nation Population topics Year of arrival in the country YAE / YAT Educational attainmentEDU Location of place of workLPW Current activity status CAS OccupationOCC Industry IND Status in employmentSIE Tenure status of households TSH Housing topics Housing arrangements HAR Type of ownership (of dwellings) OWS Water supply system WSS Toilet facilities TOI Bathing facilities BAT Type of heating TOH Which data for which geographical area?

10. 10 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 Example: DSD for Table 6 (Marital Status) IDCONCEPTCODELIST TIMETime period or rangeCL_TIME GEOGeographical areaCL_GEO SEXSexCL_SEX FSTFamily statusCL_FST LMSLegal marital statusCL_LMS CASCurrent activity statusCL_CAS POBCountry/place of birthCL_POB COCCountry of citizenshipCL_COC AGEAgeCL_AGE FREQFrequencyCL_FREQ IDATTACHMENT LEVEL CODELIST OBS_STATUSObservationCL_OBS_STATUS OBS_LEVELObservationCL_OBS_LEVEL OBS_NOTEObservation HC_NOTESeries IDNAME OBS_VALUEObservation value Dimensions Measures Attributes

11. 11 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 The components

12. 12 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 Through the SDMX Hub, a data user can… Browse the Hub to define a dataset of interest, navigating via structural metadata: - Select hypercube or search by topic (filters) - Select data (level of detail, breakdowns) - Select layout (axes) View a table Save a query Export a file (CSV, Excel, SDMX-ML)

13. 13 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 How the Hub works Eurostat Census Hub National Statistical Institute

14. 14 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012

15. 15 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 Lesson learnt and benefits in participating Statistical needs first. Then, technological aspects. Capacity-building is a must –Participating organisations are gaining a good in-house experience in SDMX and its implementation A system of distributed databases is harmonised through the use of SDMX standards and content guidelines SDMX-RI can be reused for sharing data in other domains –Limited cost for installations, development costs can be reduced –Step forward towards generic solutions for statistical domains

16. 16 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 For more information Nadezhda.Vlahova@ec.europa.eu CIRCA: http://circa.europa.eu/Public/irc/dsis/x-dis-xensus-hub/library http://circa.europa.eu/Public/irc/dsis/x-dis-xensus-hub/library