1. ESS technical standards and tools for quality reporting
Giorgia Simeoni
Istat,
ESTP Training Course
“Advanced course on quality reporting”
Luxembourg, November 2016

2. Contents Overview on SDMX for reference metadata
Introduction to ESS Metadata Handler (ESS-MH)
Interoperability between national metadata systems and ESS-MH
ESS-MH Demonstration

3. Reference metadata in SDMX
SDMX: the technical standard for Statistical Data and Metadata eXchange
Based on the exchange of xml messages (SDMX-ml), and including an Information Model.
The implementation of the SDMX standard is supported by a series of SDMX tools.
Introduction of SDMX for the transmission of reference metadata and:
supported quality reporting harmonisation
influenced the structure for ESS quality reports

4. SDMX Information Model*
* SDMX self-learning package No. 2 Student book

5. Metadata Structure Definition - MSD
The Metadata Structure Definition is the SDMX artifact that defines the structure of the quality report.
It corresponds to the template that the quality report should follow.
It is made up of 2 main parts:
Identifiers: describe the object which describes reference metadata.
Report structure: a set of report attributes. The report attributes correspond to the statistical concepts required from the template.
ESMS and ESQRS are implemented through MSDs.

6. Metadata Structure Definition
ESMS
ESMS_REPORT_FULL
FULL_ESMS
DATAFLOW > Domain-subdomain
DATA_PROVIDER > NSI
TIME_PERIO > reference period
ESTAT_DATAFLOWS_SCHEME: List of Domains
ESTAT_ORGANISATIONS: List of NSIs and other organisations

7. Classification system
Metadata Structure Definition
SDMX cross domain concepts
ESMS …
Data description
Classification system
Coverage - sector ….
FULL_ESMS
ESMS_REPORT_FULL

8. Metadataflow
The Metadataflow is the conjuction between the template (MSD) and the statistical domain (reported as a CATEGORY of a CATEGORY SCHEME in SDMX standard).
It provides also the information on the reporting frequency (annual, quarterly…)
e.g. the metadataflow AEI_ESQRS_A means that for the AEI domain (Agri-environmental statistics) the ESQRS template should be used with annual (A) reporting frequency

9. Metadataset
The Metadataset is the SDMX artifact that contains the filled in quality report
It refers to a metadataflow
It reports the values of the identifiers of the MSD
It reports the values of the metadata attributes of the MSD

10. What is ESS Metadata Handler (ESS-MH)?
The ESS-MH is an IT application provided by Eurostat allowing the compilation, management and transmission of reference metadata files and quality reports following ESS standards
SDMX compliant: it produces metadatasets in SDMX-ml format for the statistical domains included in a category scheme for which one ore more metadataflow(s) has been opened referring to specific MSDs

11. Reference metadata transmission
Reference metadata in the ESS are mainly exchanged in SDMX format via the ESS-MH. Thus, what is implemented in the ESS-MH defines the rules for the exchange.
E.g.:
At the moment, the version of SDMX used for reference metadata in ESS-MH is 2.0
For IT experts: there are 2 types of SDMX messages for metadatasets
the generic metadata set
the metadata structure specific metadata set
The generic metadata set is required in the ESS-MH
CLOODT Hubertus (ESTAT) 16/09/2016
OK to let it in, but if interfaces that consume reference metadata using the SDMX standard are well defined they should be agnostic for what concerns the SDMX message type used. The type of message should then not be a show stopper for further processing

12. Access to ESS-MH
At NSI level there are 2 different user profiles that access to ESS-MH:
The National Administrator:
manage the metadata files from all the statistical domains and from all the national Institutions
manage national user profiles
The National User:
manage the metadata files for specific statistical domain(s)
To access ESS-MH an ECAS account is needed

13. Interoperability between national metadata system and ESS-MH*
Three possible scenarios
Only ESS-MH is used to produce national metadata files
* Nielsen et al.(2015) ESS reference metadata: implementation of standards and interoperability of national and European metadata systems Workshop of the Modernisation Committee on Standards: International Collaboration for Standards-Based Modernisation Meeting, 5 – 7 May 2015

14. Interoperability between national metadata system and ESS-MH*
National metadata system is used to produce the metadata file and then it is manually imported in ESS-MH
* Nielsen et al.(2015) ESS reference metadata: implementation of standards and interoperability of national and European metadata systems Workshop of the Modernisation Committee on Standards: International Collaboration for Standards-Based Modernisation Meeting, 5 – 7 May 2015

15. Interoperability between national metadata system and ESS-MH
Metadata file is produced using the National metadata system and automatically imported in ESS-MH via EDAMIS
* Nielsen et al.(2015) ESS reference metadata: implementation of standards and interoperability of national and European metadata systems Workshop of the Modernisation Committee on Standards: International Collaboration for Standards-Based Modernisation Meeting, 5 – 7 May 2015

16. Interoperability between national metadata system and ESS-MH
Technical specification for file to be produced by the national metadata systems and imported in the ESS-MH has been produced by Eurostat
E.g.:
- only the SDMX-ML file with the metadata set or a zip file including the metadataset, a directory with the annexes, a directory with the images, and the related MSD (optional) can be imported
- the name of the metadataset as well as the tag on the header should follow specific rules
- the «annotations» tag can be used to transmit information on annexes and on the restriction for publication
CLOODT Hubertus (ESTAT) 16/09/2016
will you explain these rules?

17. Interoperability between national metadata system and ESS-MH
In order to accomodate domain specific reporting needs, html specific templates have been introduced in ESS-MH within the SDMX metadata attribute values

18. Interoperability between national metadata system and ESS-MH
In practice, there are domain specific quality reporting elements included below SIMS level
The interoperability between national metadata system and ESS-MH is more difficult
CLOODT Hubertus (ESTAT) 16/09/2016
suggestion: may be explain this with a small example to make it concrete for a participant?

19. Interoperability between national metadata system and ESS-MH
One of the objectives of this course is to promote cross-domain harmonisation on quality reports, reducing domain specific accomodation as far as possible

20. DEMO of ESS-MH Production environment
Demo on the «acceptance environment»
Webinar on ESS-MH are periodically organised by Eurostat

21. References
Eurostat / ESS SDMX info space:
SDMX website:
SDMX content oriented guidelines:
SDMX information model detailed information:
SDMX checklist for sdmx design projects:
Nielsen et al.(2015) ESS reference metadata: implementation of standards and interoperability of national and European metadata systems Workshop of the Modernisation Committee on Standards: International Collaboration for Standards-Based Modernisation Meeting, 5 – 7 May 2015
Eurostat. Metadata Extra SDMX-ML annotations for ESS-MH