Census Hub in practice Working Group "European Statistical Data Support" Luxembourg, 29 April 2015

Contents The Census Hub: motivation and opportunities Broad overview of the census data collection The Graphic User Interface of the Census Hub Feedback received from users

The EU programme for the 2011 census Massive effort of data harmonization: first time that population and housing censuses are regulated at this level of detail High level of geographical data detail Possibility to cross-tabulate several social variables  focus on specific subgroups of the resident population Huge volume of data involved  decentralized database

The Census Hub The Census Hub: an innovative dissemination tool A distributed data warehouse based on agreed data structures Data are physically resident in Member States and retrieved dynamically upon user request

The Census Hub Central and national nodes can automatically exchange data among each other Solution that can be reused (either for next census or for other data collections) First full implementation of SDMX in the ESS

The Census Hub System designed to treat multiple data accesses National Statistical Institute SDMX-RI National Statistical Institute Eurostat Census Hub System designed to treat multiple data accesses Peak of 1,000 users/hour Very good performance level with fast response time A data hub is a pull mode based architecture for common data sharing. Data is not previously collected and stored in a central repository but it is [A>] directly accessed from the data providers’ databases through a central hub upon [B> ]request of a data collector:[/A][/B] [C>] The data collector browses the Hub to define the dataset of interest via its structural metadata. The Hub converts the user’s request into an SDMX Query message and [D>] sends it to an National Statistical Institute’s Web Service. [/C] [/D] [E>}The NSI Web Service converts the SDMX Query in a set of SQL queries, fetches the data from the NSI database, dynamically constructs the SDMX-ML file and [F>] sends it back to the hub. If the request concerns data of several NSIs, the steps are executed simultaneously for each of them. [/E][/F] [G>] The Hub assembles all the SDMX-ML files received from the NSIs and presents the result to the user in a readable format.[/G] The workflow is: Step 1: a “data user” browses the Hub to define a dataset of interest via structural metadata. He browses the dimensions and selects a dataset. Then he chooses the organization of the output layout specifying which dimension wll match X-axis and Y-axis and which dimension will vary item after item to generate new tables Step 2: The Hub converts the user request into an SDMX Query and sends the SDMX Query to an interested NSI Web Service Step 4: The NSI Web Service converts the SDMX Query in a set of SQL queries and sends them to the NSI data warehouse Step 5: The NSI data warehouse sends the result to the NSI web service Step 6: The NSI Web Service converts the result in a SDMX-ML Data message and sends it to the Hub Step 7: The same steps are repeated if the user has requested data from different MSs Step 8: the Hub puts together all the SDMX-ML data messages proceeding from the interested NSIs and presents the result to the “data user” in the web browser in readable format. SDMX-RI 6 6 6

Topics covered by the census Location Usual residence Place of work Demographic characteristics Sex Age Data on persons Family & household characteristics (Legal) marital status Family status Household status Employment and education characteristics Current activity status Industry Occupation Status in employment Educational attainment Migration and mobility Place of birth Country of citizenship Year of arrival Residence one year before List of topics collected on persons Housing arrangements

Census datasets (hypercubes) Distribution of census hypercubes by type of statistical unit Distribution by type of statistical unit

Census datasets (hypercubes) Distribution of census hypercubes by level of geographical detail Distribution by level of geographical detail

Census datasets (hypercubes) Distribution of census hypercubes by number of dimensions Distribution by number of dimensions

Data selection in Eurostat database Overview of the data selection process in Eurobase

Difficulty to implement "traditional" data selection with census hypercubes Very high level of dimensionality: impossibility to find meaningful names to datasets Two dataset might have 6 dimensions in common and differ only for the remaining 2 The same dataset can cover several domains (e.g. economic status, migration background, etc.)  difficulty to build a meaningful "data tree"

Data selection in the Census Hub The data selection process in the Census Hub GUI

Data selection in the Census Hub The data selection process in the Census Hub GUI

Data selection in the Census Hub The data selection process in the Census Hub GUI

Data selection in the Census Hub The data selection process in the Census Hub GUI

Data selection in the Census Hub The data selection process in the Census Hub GUI

Data selection in the Census Hub 18

Feedback from users The data selection mechanism implemented in the Census Hub is very user-friendly Data selection is completely user-oriented and not producer-oriented Codelists and available breakdowns are immediately visible to users The system is well performing, despite the high volume of data involved  response time similar to Eurostat database

Thanks for your attention