1. SISAI STATISTICAL INFORMATION SYSTEMS ARCHITECTURE AND INTEGRATION
WORKING GROUP 3rdMEETING MAY 2013 ITEM Evaluation of software from ESSnets

2. Items 2.2.1 - 2.2.3 Evaluation of software from ESSnets
SISAI Meeting, May Håkan Linden, Eurostat (B5) Bengt-Åke Lindblad, Eurostat (B3) Denis Grofils, Eurostat (ESS.VIP TF)
Items Evaluation of software from ESSnets

3. Outline
Context: The ESSnets on SDMX; and the conclusions of ITDG November 2012
Overview of SDMX Data Express and the Eurostat evaluation
Overview of eCollect-X and the Eurostat evaluation
Expression of interest for further evaluation of the software for possible use in the ESS

4. Conclusions from ITDG 27-28 Nov 2012 (item 6.1)
The SDMX IT application for micro-data handling needs to be forwarded to SISAI for checking and further testing; then a decision needs to be taken with regard to its further use.
The maintenance of the SDMX Data Explorer needs to be addressed to the SISAI (in a broader context).
Some Member States also expressed interest in testing/ using these ESSnets deliverables.

5. The ESSnets on SDMX
The ESSnets on SDMX aimed to improve the SDMX standards and guidelines; to create or improve statistical IT applications used for SDMX based data/metadata exchange; and to contribute to SDMX capacity building.
The ESSnets on SDMX were organized in 2 phases: phase I and phase II.
The deliverables should support business process integration in the ESS, in line with the ESS vision and strategy. The ESSnets on SDMX were part of a broader ESSnet program launched since 2009.

6. ESSnet on SDMX - phase I
Finished

7. Results of the ESSnet on SDMX – phase I (co-ordinator: Portugal)
Leading country
Topic
Results
Portugal
MCV Ontology
Ontology used as input to GSIM, SDMX COG etc.
SDMX Data Express
Web application released for the visualization of statistical data using SDMX data files.
Italy
Generalised SDMX Framework
Some modules of the framework used for the SDMX-RI
Sweden
SDMX integration into PC- Axis
Intermediate stage on integration reached
Slovak Republic
SDMX in survey design and data collection
First draft IT application for micro-data collection designed

8. Overview of SDMX Data Express
Main objective WP3:
To provide a way to convert among different SDMX data messages formats (generic, cross sectional, compact) and provide an interface to help users to build a SDMX query (based on a given DSD file).
Main output WP3:
An application that allows users to visualize SDMX data files and the related DSD under different perspectives and in an user-friendly way.

9. Overview of SDMX Data Express

10. ESSnet on SDMX – phase II
Finished

11. Results of the ESSnet on SDMX – phase II (co-ordinator Italy)
Leading country
Topic
Results
Italy
Harmonise the existing SDMX tools and software
Further proposals for improving the SDMX-RI elaborated
SDMX capacity building
Training actions carried out, documentation for statisticians produced
Sweden
SDMX integration into PC- Axis
Final integration of SDMX into PC-Axis reached
Slovak Republic
SDMX in survey design and data collection
IT application for micro-data piloted and delivered

12. Overview of eCollect-X
Main objective WP3:
To implement SDMX standard at the NSI level in the phase of statistical micro-data processing (survey preparation, data collection, data processing, and eventual dissemination);
Main output WP3:
A pilot application to verify the use of SDMX in micro-data description, its further processing and eventual dissemination.

13. Overview of eCollect-X

14. Functionalities of SDMX Data Express
View & convert user-loaded SDMX data with corresponding data structure definition (DSD)
Validate input SDMX
View tabular data as HTML with user defined table layout
Export as CSV
Chart visualization
Data conversion
Query builder
 More info on

15. Evaluation of SDMX Data Express
Evaluation of software of interest to the ESS – SDMX Data Express
Quality criteria
Rating*
Comment
Functionality
Are the required functions available in the software?
   - Suitability
Can software perform the tasks required? 5
No missing functionalities compared to terms of reference. Direct SDMX data visualizers are not so common.
   - Accuracy
Is the result as expected?
Results are accurate.
   - Interoperability
Can the system interact with another system?
SDMX open format is available as input and output via the GUI and via web service.
   - Security
Does the software prevent unauthorised access? 2
Under the responsibility of end-user
Support
Can users rely on supporting ?
   - Documentation
Documentation available, clear and accurate? 4
Detailed documentation is available, installation guide needs to be looked into to ensure interoperability with newer versions of the operating systems (Windows XP only now).
   - Self-support
Is self-support of the user community organized?
No user community.
   - Formal support
Is there a body formally in charge of support?
No formal support.
Reliability
How reliable is the software? 3
Crashes sometimes when bad operations are done (following the user guide).
Usability
Is the software easy to use?
The software is very intuitive and easy to use
Efficiency
How efficient is the software? ?
Not tests performed with large datasets
Maintainability
How easy is to modify the software?
Application is modular, developer documentation is available
Portability
How easy is to transfer the software to another environment?
.NET app, best fit on Microsoft environments. Installation guide only for windows XP.
* Based on section Quality assessment of ESS software sharing policy and manual proposed to SISAI
under agenda point 2.3 ESS software sharing. Scores are from 0 to 5.

16. Functionalities of eCollect-X
eCollect-X is an application designed for micro-data collection with integration into SDMX technology.
User management
Create questionnaire
Web app, WYSIWYG editor
Create modules, questions, modalities
Create and associate validation rules
Fill-in questionnaire
Check/Validate filled-in questionnaires
SDMX Export
More info on

17. Evaluation of eCollect-X
Evaluation of software of interest to the ESS - eCollect-X version 4.0
Quality criteria
Rating*
Comment
Functionality
Are the required functions available in the software?
   - Suitability
Can software perform the tasks required? 4
No missing functionalities compared to terms of reference. Many other survey data collection tools exist.
   - Accuracy
Is the result as expected?
Results are accurate (for the pilots)
   - Interoperability
Can the system interact with another system? 3
SDMX open format is available as output for questionnaires and data but only via the GUI so it is not possible to submit or extract data from another IT system without human intervention; eCollect-X interacts with Mapping Assistant software and also manually through export of XML files containing codelists with Data Structure Wizard.
   - Security
Does the software prevent unauthorised access? 2
User management is provided but login and data transmissions are not secured.
Support
Can users rely on supporting ?
   - Documentation
Documentation available, clear and accurate?
Detailed documentation is available but room for further improvement.
   - Self-support
Is self-support of the user community organized?
N/A
It's a prototype there is no user community…
   - Formal support
Is there a body formally in charge of support?
It's a prototype there is no formal support…
Reliability
How reliable is the software?
Sometimes a bit unstable…
Usability
Is the software easy to use?
Basic functions of creating questionnaire, modules questions and modalities is relatively straightforward but the workflow correspond to the data structure what can be counter-intuitive. Still some bugs…
Efficiency
How efficient is the software?
Knowing it is a pilot application no stress test done. Some screens will not scale for a complete survey sample (load sample data, view responses, …)
Maintainability
How easy is to modify the software?
Application is modular, developer documentation is available
Portability
How easy is to transfer the software to another environment?
.NET app, best fit on Microsoft environments…
* Based on section Quality assessment of ESS software sharing policy and manual proposed to SISAI
under agenda point 2.3 ESS software sharing. Scores are from 0 to 5.

18. Member States are invited to…
Report/ provide feed-back on any tests done in their NSI's of these two applications;
Express their interest in further evaluation of these two SDMX compliant IT applications;
Express users' needs for these two applications (in the view of their further use in the ESS).