2. eDAMIS Validation Possibilities
Validation at the Single Entry Point; based on SDMX
No installation or configuration in Member States
eDAMIS Web Forms: Real-Time Validation (in Production for some years)
eDAMIS Web Portal: New Validation Engine (available since eDAMIS 3.0, July 2010)
eDAMIS Web Application
Server side validation for all eWA versions
Local validation in eWA 3.1 (using rules from the server)
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

3. eDAMIS Validation Engine (eVE)
Batch Validation for eWP and eWA Transmissions
New version available in eDAMIS 3.0
For SDMX-ML formatted files
XML format validation
Code validation using SDMX Data Structure Definitions
Data Validation
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

4. eVE – Data Validation Features
Same Validation Rule Syntax as Web Forms
Within one file and reference period
Different rule sets per reference period possible
Country specific rules
Mandatory values, Range checks
Basic expressions, comparison (+ - * / < > =)
Mathematical expressions (SUM, AVG, MIN, MAX, …)
Conditional checks (IF…THEN…ELSE)
Logical expressions (AND, OR, NOT)
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

5. eDAMIS Validation Engine for …
Domain Managers
Based on SDMX DSDs
Links to SDMX Registry
Same syntax as eWF for Data Validation
Less iterations for transmissions lowers workload
Data Senders
All transmission channels
Support confidential datasets
Validation transparent to data senders
Full automatic transmission and level 1 validation workflow
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010 5

6. Workflow eDAMIS Validation Engine
SDMX Registry
DSDs
Web Service
Browser
SDMX Converter
CSV
Settings
MS Database
Eurostat Production Unit
eDAMIS Server
Validation
SDMX
eWP
eWA
Report
Report
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

7. Successful Pilot Projects in Member States
Fisheries Pilot (May 2010)
Eurostat Unit E2 (Matthew Elliott)
Workshops in Sweden, Latvia, UK, Romania
Remote Testing in CBS, Netherlands
Aviation Pilot (September 2010)
Eurostat Unit E6 (Hubertus Cloodt)
Workshop in Statistik Austria
Good Feedback from both Pilot Projects
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

8. Situation in Fishery Statistics
Submission of many different formats to Eurostat:
Time consuming to process;
Difficult to validate.
Eurostat responsibility for sharing data with international organisations (FAO and regional fisheries organisations)
Conflict with other formats adopted by the EC for fisheries data (DG MARE and JRC);
Challenge - simplify collection and share data (reducing MS burdens, also being looked at by the Standing Committee for Agricultural Statistics);
Opportunity – changes to legislation and move to new production database (MDT/Oracle).
Information required is set out in technical annexes to various legislation but it allows MSs some flexibility in the format they send data in. About half send data in flat file format. The remainder use an Excel format provided by FAO and in some cases of their own design.
Validation of data is particularly difficult for fisheries data. There are around 10,000 valid species codes. Many species are specific to certain geographical regions and plausibility checking of the species/area combinations can be laborious. The greater the automation of validation and particularly pre-validation, the more time saved in processing and contacts with MSs.
MSs send data to ESTAT for stats purposes, JRC for scientific assessments and MARE for control. Data collection mechanisms are different and at different levels of sophistication. DG MARE under the umbrella of CFP reform are looking to update their data collection to make it more efficient and facilitate improved quality. MSs are also required to share information (sales and catch) between themselves and a common format will help this (and possibly allow development of common IT solutions).
Having a common format gives greater scope for comparing what is sent to different institutions. Minimising duplication saves their time in preparing reports and ours allowing more time on data quality and making the best use of the data.
SDMX as a solution feeds in to work to improve the Eurostat fisheries data collections generally and in particular as part of the revamp of the entire production process from collection to dissemination. A central part of this is the development of a new MDT production databse.
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

9. Overview SDMX – for Fisheries
From here
Various file formats
Various code lists
Between
Member States
European Institutions
Other organisations
Going there
Single file format SDMX
Shared Data Structures
Harmonized Code Lists
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

10. Overview – Transition to SDMX in Fisheries
SDMX Workshop in March:
Benefit of SDMX for senders?
SDMX too complicated?
Links with Coordinating Working party for Fisheries Statistics established: development of code lists.
Pilot Project with some Member States launched
Cooperation with DG MARE
Workshop Road Trip to SE, LV, UK in May, RO in September.
Remote Testing for NL
Fisheries Statistics Working Group - June
This provides an overview of what we have achieved from formally announcing the SDMX initiative for the March Workshop (though we had give advance warning through the SCAS)
The Workshop on 5 March was very detailed and technical.
It was well received but it was very difficult to arrange this for the right audience – attendance was diverse – managers, IT specialists, fisheries specialists, NSIs and some fulfilling several roles.
We got the impression that many of you thought that it looked overly complicated – “you have taken something simple”
Workshop launched the Pilot for testing of generation and processing of SDMX in MSs. Was achieved in spite of the Icelandic Volcano (whose name I will not try to pronounce)
This was also used to examine the possibilities for wider SDMX use – by Commission Services and within MSs. Useful meeting in London with the UK MMO.
Visits to SE, LV and UK
Testing packs also sent to NL and RO. Need to discuss results bilaterally and also possibility of additional visits (RO)
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

11. Pilot - Technical Scope
Catch for Major Fishing Area 27 (NE Atlantic) used for the pilot;
eDAMIS will be used as transmission system;
eDAMIS Validation performed for:
Format validation, Code list validation ( DSD)
Value validation:
Detect some species that should not be in the area
Check for mandatory fields and duplicates if possible
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

12. Technical Workflow – Tested during Pilot
SDMX Converter
CSV to SDMX using the DSD
eDAMIS
Upload SDMX to web portal
eDAMIS Validation
Get validation report
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

13. Feedback from Member States
SDMX is no “Rocket Science”
Number 1 Issue is to harmonize Code Lists
The SDMX Registry is seen as a useful tool to manage data structures and code lists centrally
Deadlines and Reports should be harmonized between organizations
A Single Entry Point for data for the whole Commission would make life easier for data senders
Tools and Guides were easy to use
The information on the Validation reports was considered easy to interpret and use for data correction
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

14. eDAMIS Validation Engine – Live Demo
Based on pilot project dataset
View of a Data Sender
Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010