1. IS4STAT Vittorio Viaggi Eurostat
Unit B3: “IT for statistical production”
SDMX Basics course, October 2015

2. Typical Estat production process
Structural Validation
Format Conversion
Data Load
Data processing and calculation
Export for dissemination and sharing
October 2015

3. IS4STAT approach Production system October 2015 Structural Validation
Format Conversion
Data
Load
Data processing and calculation
Export for dissemination and sharing
Production
system
October 2015

4. IS4STAT approach Production system IS4STAT Workflow manager Today
Structural Validation
Format Conversion
Data
Import
Data processing and calculation
Export for dissemination and sharing
Production
system
Today
Format Conversion
OLAP Data
Loading
IS4STAT
Workflow
manager
October 2015

5. IS4STAT approach Production system IS4STAT Workflow manager 2016 Q1
Format Conversion
Data
Import
Data processing and calculation
Export for dissemination and sharing
Production
system
2016 Q1
Today
Structural
Validation
Format Conversion
OLAP Data
Loading
IS4STAT
Workflow
manager
October 2015

6. IS4STAT approach Production system IS4STAT Workflow manager 2016 Q1
Format Conversion
Data
Import
Data processing and calculation
Production
system
2016 Q1
Today
Structural
Validation
Format Conversion
OLAP Data
Loading
Export for dissemination and sharing
IS4STAT
Workflow
manager
October 2015

7. Wrapping existing tools
Structural Validation
Format Conversion
Data
Load
Data processing and calculation
Export for dissemination and sharing
October 2015

8. Wrapping existing tools
Data
Load
Data processing and calculation
Export for dissemination and sharing
Structural Validation
Format Conversion
SDMX Converter
October 2015

9. Wrapping existing tools
Data
Load
Data processing and calculation
Export for dissemination
Structural Validation
Format Conversion
Export for sharing
SDMX Converter
SDMX Converter
October 2015

10. STRUctural VALidation (NAPS-S project)
Call STRUVAL Service
Valid SDMX-ML file
Coded according to the DSD
Mandatory fields present
Correct data types
Dataflow definition
DSD
Polling EDAMIS for new data-flows
SDMX-ML
Validation report
Notify sender NSI through EDAMIS feedback channel
Deliver data-flow to production system
Notify:
IS4STAT support, Domain manager, Corresponsible(s)
Validation Not OK
Validation OK
Validation impossible
October 2015

11. IS4STAT suite
SDMX Tools Task Force
October 2015

12. IS4STAT suite
It's a generic dashboard for the pre-processing and processing of incoming data. Single operations are synchronized and orchestrated by a process manager.
It connects to EDAMIS and retrieves new incoming data flows.
Data flows are then processed according to the statistical domain operational workflow.
It uses SDMX Converter features (Web service) to provide:
Structural correctness validation of SDMX-ML against Data Structure Definition.
Conversion to a production-friendly format.
Feedback notifications to sender NSIs in case of validation issues.
Direct data loading to production system (currently OLAP, MDT by end 2016).
Unified logging of all operations, direct download of all intermediate versions of a data flow.
SDMX Tools Task Force
October 2015

13. IS4STAT suite
It is a generic dashboard to manage the export process for: sharing data with other international institutions and for dissemination (Eurobase).
It uses SDMX converter features (Web service) in order to convert from production format to SDMX-ML.
Repetitive tasks are automated by the use of exporting templates.
All operations are synchronized and orchestrated by the process manager.
SDMX Tools Task Force
October 2015

14. Thank you!
October 2015 14