Vision Infrastructure Project (VIP) Enhanced Dissemination Chain

Vision Infrastructure Project (VIP) Enhanced Dissemination Chain
Point 2.3 of the agenda Vision Infrastructure Project (VIP) Enhanced Dissemination Chain Results of Action 4 of the VIP-EDC 2nd SISAI meeting, June 2012 Eurostat Unit B6 SISAI Meeting June 2012

VIP “Enhanced Dissemination Chain”
Actions Deliverables Development towards a Eurostat / ESS data warehouse 1 Strategy (to be worked out with NSIs) Simplification and integration of the Eurostat Dissemination Chain More functionality for producing and disseminating Eurostat policy indicators 2 3 COLOR CODING ORANGE = NSI’s involved / impacted / can use deliverables BLUE = EUROSTAT action or deliverable We work in the context of a project with: - 4 tasks, with specific deliverables. - 2 of the tasks are in collaboration with NSIs SISAI Meeting June 2012

VIP “Enhanced Dissemination Chain”
Actions Deliverables 1 Development towards a Eurostat / ESS data warehouse Strategy (to be worked out with NSIs) 2 Simplification and integration of the Eurostat Dissemination Chain 3 More functionality for producing and disseminating Eurostat policy indicators In the context of development and use of common ESS tools and services for dissemination, the 4th task is the one to look at, and this the focus of today’s presentation 2 targets : Embeddable visualisation tool = “widgets” Usage of web services for the purpose of Eurostat data dissemination More integration of Eurostat data and metadata in NSI dissemination 4 Embeddable visualisation tools Dissemination web services SISAI Meeting June 2012

Development & use of common ESS tools and services for dissemination
Embeddable visualisation tools Widgets are embeddable visualisation tools Small web applications To be embedded in any website Showing a relatively small amount of Eurostat data In a simple and understandable way Easy to configure Which data Display Widgets definition “To be embedded in any website” = in the beginning, only NSI and Commission services, later this could be a much broader audience, and this could also take the form of an iOS app e.g. “Showing a relatively small amount of Eurostat data” = this is about presenting key facts Attention point « Easy to configure » Which data – the set of data which will be supported for display in a widget is going to be based on a short list, cfr later slide SISAI Meeting June 2012

Progress Initial analysis of needs & features “Paper for discussion” Working widgets prototype Short Demo Summarize our main areas of focus during the past months The « Paper for discussion » was an input for the Task Force meeting, and will be updated in the coming weeks to include all the valuable inputs received Show a short demo of the working prototype Attention points on the prototype are This is not the final look and feel, this is just a standard look & feel applicable in the context of Eurostat’s portal Main characteristics of widgets are : SISAI Meeting June 2012

Progress Initial analysis of needs & features “Paper for discussion” Working widgets prototype Initial architecture design Project planning “Initial architecture design” = here with have a synergy with the 2nd topic, i.e. web services, as the data will be consumed through this type of technology “Project planning” was reviewed in the task force SISAI Meeting June 2012

Task Force meeting 29 & 30 March Agreed on Over 2 half days we reviewed together the topics related to widgets an web services Thank you to all involved NSIs who provided interesting insights and participated proactively to the discussions. SISAI Meeting June 2012

Task Force meeting 29 & 30 March Agreed on 2 top priority widgets Presentation of main results from the TF meeting SISAI Meeting June 2012

Task Force meeting 29 & 30 March Agreed on 2 top priority widgets Iterative work approach, delivering as quickly as possible tangible results and proceeding with review cycles Presentation of main results Delivering as quickly as possible tangible results and proceeding with review cycles -> this applies currently only to the 6 NSIs involved in the Task Force -> information about availability of other deliverables will be provided at the next DWG meeting SISAI Meeting June 2012

Work approach, the waterfall way We are not going to apply a classic waterfall approach where NSI would have to validate late inputs – this has a too negative impact on the schedule SISAI Meeting June 2012

Work approach, the iterative way Results Results NSI review NSI review We will work with short iterations and limited, yest specific, review cycles SISAI Meeting June 2012

Task Force meeting 29 & 30 March Agreed on 2 top priority widgets Iterative approach, delivering as quickly as possible tangible results and proceeding with review cycles Reviewed Dashboards examples – FR Widgets – CH & IT Needs & features, taking on board additional requirements, e.g. Behaviour for offline consultation, Presentation & user interaction, Accessibility, Linking to data etc. We had during the TF maating lots of interactions and discussions, and major other outputs are …. SISAI Meeting June 2012

Theme code Name of the table Economy/ National Accounts tec00114 GDP per capita in PPS tsdec100 Real GDP per capita, growth rate and totals tec00001 Gross domestic product at market prices Economy/Government statistics tsdde410 General government gross debt teina200 General government deficit (-) and surplus (+) tec00023 Total general government expenditure Economy/ Prices teicp000 HICP-all items Population/ Labour force teilm020 Harmonised unemployment rate by gender t2020_10 Employment rate by gender, age group 20-64 Population/ Demography tps00010 People by age classes tps00025 Life expectancy at birth, by gender tps00112 Crude birth rate Industry teiis080 Industrial production - total industry (excluding construction) teiis600 Industrial new orders Construction teiis540 Building permits Energy tsier040 Electricity prices by type of user tsdcc330 Electricity generated from renewable sources Environment tsdcc210 Greenhouse gas emissions by sector (source: EEA) THERE ARE NOW 18 TABLES INSTEAD OF 20 BECAUSE 2 are not retained as Total number / absolute figures are not suitable for widgets Population by citizenship-foreigners tps00157 Removed from the list of tables for widgets: data are in total number (absolute figures) and therefore not suitable for presentation in a widget. The other dataset contained too much missing data (food consumption) Consumption of certain foodstuffs per inhabitant tsdpc330 Removed from the list of tables for widgets: data are incomplete and there are obvious quality issues in the data; therefore not suitable for widget presentation. Few code changes Needed from the DWG - agreement on the indicators to keep for development and delivery We provide here a shortlist of potential indicators that could be supported in the Eurostat widgets. At this stage this is a basic proposal and we actively invite Task Force Members to come up with request about Eurostat data they would like to see in widgets. For quality control reasons we will however have to agree on the exact list of supported indicators, in order to keep testing activities during the project under control. SISAI Meeting June 2012

… next steps … Process all the valuable inputs from the March Task Force meeting Updated “Points for discussion” document Specifications (use cases per widget, user interface, etc.) Architecture & in particular how data will be provided to the widgets (web services) Defining the testing approach (Test Management Plan) Eurostat will proceed with all activities relevant in this phase, and in particular those listed here. Also, as we will work interatively, we will work on the test management plan, clarifying how NSIs can functionally validate deliverables The document « Points for discussion” will be renamed … SISAI Meeting June 2012

Dissemination web services An additional Eurostat dissemination product Allowing to perform machine-to-machine data download In a format which is already available through the user interface of the Bulk Download For whole datasets, or subsets of datasets Free of charge Open to all users, no authentication Dynamically generating Data Structure Definition (DSD) & pulling data, at request Compact SDMX What is the dissemination web service … today? Notice that in the title web services is in plural form, a quick word about that in a minute SISAI Meeting June 2012

Progress since last DWG meeting Load testing Web service is now publicly available Update of system documentation, including a user’s guide With SOAP 1.1 With SOAP 1.2 Summarize our main areas of focus during the past months SISAI Meeting June 2012

Bottom up approach, based on an existing technical implementation BulkDnld Full dataset The dissemination web service comes on top of an existing infrastructures and set of functionalities available on the Eurostat portal NavTree Full dataset Dissem. Reference SISAI Meeting June 2012 17 17 17

Bottom up approach, based on an existing technical implementation DSWS Full & partial dataset runCustomQuery BulkDnld Full dataset NavTree Full dataset Dissem. Reference SISAI Meeting June 2012 18 18 18

Task Force meeting 29 & 30 March IT showcase about usage of SDMX-RI for dissemination purposes Exchange of views on a number of topics … Over 2 half days we reviewed together the topics related to widgets an web services Thank you to all involved NSIs who provided interesting insights and participated proactively to the discussions. SISAI Meeting June 2012

Current dissemination web service is a b release Identified improvements for SDMX data delivery Increase compliance with guidelines, i.e. use SDMX-ML messages and exchange patterns as specified by the standard Need for a resulting SDMX output to be compliant with existing SDMXConverter tool(s) Discussed @ The current version of the dissemination web service is to be viewed as a beta release and should at this stage not be used for real IT production Several needed improvements are already identified We had a very interesting presentation form ISTAT about the usage of SDMX for dissemination purposes, which also highlighted the challenges of this approach. SISAI Meeting June 2012

Dissemination web service V2 Supporting SDMX guidelines for the use web services, for SDMX version 2.0 and SDMX version 2.1 REST & SOAP Online HTML documentation Investigate / prepare for the support of formats other than SDMX The current version of the dissemination web service is to be viewed as a beta release and should at this stage not be used for real IT production Several needed improvements are already identified We had a very interesting presentation form ISTAT about the usage of SDMX for dissemination purposes, which also highlighted the challenges of this approach. SISAI Meeting June 2012

Widgets & web services … how will this work ? We have 1. Users, working with different types of devices 2. A NSI 3. A widget provided by Eurostat, which is embedded into a NSI’s (home)page 4. Eurostat, which has a database 5. Containing the 18 datasets retained for the widgets How will all this work together ? NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012

Widgets WS These are our major building blocks ON NSI side, there is the web server supporting the NSI portal On Eurostat side there is a Widgets dedicated WS NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012

Widgets WS www The users issue requests to view the NSI’s homepage NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012

Widgets WS www The NSI’s server get the request, when the Homepage is called the widget in turn triggers a call. The widget in this case is just a minimal piece of code NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012

Widgets WS www The widget request is sent to a dedicated Eurostat widgets service NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012

Widgets WS Let’s step back a minute and consider what do we have on Eurostat side NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012

Widgets WS Data prepared for widgets At Eurostat, we will have internal logic to extract when appropriate (EMBARGO) the 18 datasets from the database where they are kept, and save them already in a format that is appropriate for the widgets JSON = Javascript Object Notation NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012

Widgets WS www So, remember, we just received at Eurostat a call from a widget on a NSI Portal, and this is processed at Eurostat by the Widgets WS NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012

Widgets WS Data prepared for widgets Data is pulled from the JSON repository NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012

Widgets WS Data Data is pulled from the JSON repository, for the sake of clarity I show how the data travels back NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012

Widgets WS www & Data Now notice that what we send back is the JSON data, as well as rendering information NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012 13

Widgets WS www & Data the JSON data, as well as rendering information, travel back to the initial requestor, in this case the NSI NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012 13

Widgets WS & Data the JSON data, as well as rendering information, are now built into the NSI page NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012 13

Widgets WS www & Data And the whole information is returned to the user who requested the NSI page NSI teina200 tsdec100 tec00114 tsdde410 tec00001 SISAI Meeting June 2012

Widgets WS & Data www And the whole information is returned to the user who requested the NSI page NSI teina200 tsdec100 tec00114 tsdde410 & Data tec00001 SISAI Meeting June 2012 13

& Data Widgets WS What is interesting here, is that the user now has in his or her browser a cache with the widget information If the user navigates to another page of the NSI portal and then gets back to the page with the widget, that is not going to be requested again The user will just see what is cached locally NSI teina200 tsdec100 tec00114 tsdde410 tec00001 & Data SISAI Meeting June 2012

& Data Widgets WS After some time interval, a request may be issued to refresh the data, e.g. when hitting embargo time areas NSI teina200 tsdec100 tec00114 tsdde410 tec00001 & Data SISAI Meeting June 2012

& Data Widgets WS www www After some time interval, a request may be issued to refresh the data, e.g. when hitting embargo time areas NSI teina200 tsdec100 tec00114 tsdde410 tec00001 & Data SISAI Meeting June 2012

& Data Widgets WS www www Data prepared for widgets After some time interval, a request may be issued to refresh the data, e.g. when hitting embargo time areas NSI teina200 tsdec100 tec00114 tsdde410 tec00001 & Data SISAI Meeting June 2012

& Data Widgets WS www www Now, let’s imagine the Eurostat site goes down NSI teina200 tsdec100 tec00114 tsdde410 tec00001 & Data SISAI Meeting June 2012

& Data Because the user still has his / her local cache the issue stays unnoticed This approach would also allow to work offline It is only in the case at the very first request things go wrong that the widget could not show data, in this case an information message would be displayed instead of the widget and the time would take care of making a new call after some time to try to get information NSI & Data SISAI Meeting June 2012

… next steps … Dissemination WS & SDMX file format Analysis of required evolutions, then implementation In combination with Widgets: data consumption Architecture validation Data consumption prototype for early validation Back to WS only topic, the next steps are … SISAI Meeting June 2012

… next Task Force meeting in September … For information, the next Task Force meeting is in september SISAI Meeting June 2012

Point 2.3 of the agenda Marc VANDERPERREN Project Manager - IT
Unit B6 - Dissemination Tel: SISAI Meeting June 2012

Vision Infrastructure Project (VIP) Enhanced Dissemination Chain

Similar presentations

Presentation on theme: "Vision Infrastructure Project (VIP) Enhanced Dissemination Chain"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Vision Infrastructure Project (VIP) Enhanced Dissemination Chain

Similar presentations

Presentation on theme: "Vision Infrastructure Project (VIP) Enhanced Dissemination Chain"— Presentation transcript:

Similar presentations

About project

Feedback