ESS conceptual standards for quality reporting Giorgia Simeoni Istat, simeoni@istat.it ESTP Training Course “Advanced course on quality reporting” Luxembourg, 12-13 June 2018

Outline ESS quality reporting: commitment The process of harmonisation of ESS quality reporting ESS standards for metadata and quality reporting

ESS Quality reporting: commitment Starting point:“Quality” section on EU Regulations Domain specific requirements Mandatory, but only for some domains Comparability of Quality Reports among Countries and not among domains Producer-oriented reports (internal, monitoring purposes)

ESS Quality reporting: commitment European statistics Code of Practice Indicator 4.3: Output quality is regularly monitored, assessed with regard to possible trade-offs, and reported according to the quality criteria for European Statistics Harmonisation of contents Application to all European Statistics

ESS Quality reporting: commitment Quality Assurance Framework of the European Statistical System Indicator 4.3 - Methods at product/survey level User oriented quality reports. User oriented quality reports are made available to the public. Producer oriented quality reports. Producer oriented quality reports are published regularly […]. Product quality monitoring. Users and producers quality reporting are used for regular quality monitoring over time. Promotion of dissemination to improve transparency Distinction user/producer oriented Useful for quality monitoring purposes

ESS Quality reporting: commitment European Statistical Law Regulation (EU) 2015/759 of 29 April 2015 amending Regulation (EC) No 223/2009 on European statistics. Art. 12 (3): “Member States shall provide the Commission (Eurostat) with reports on the quality of data transmitted…”

The process of harmonisation of ESS quality reporting National implementation of SQR starts (ESS-MH) How to make a quality report Standard quality reports 2003 Standard quality indicators 2005 ESMS ESS Handbook for quality reports 2009 ESS Quality and Performance Indicators ESQRS 2010 WG “Assessment of Quality in Statistics” Task force on quality indicators Eurostat starts implementing ESMS Expert Group on Quality Indicators

Grants to support national implementation The process of harmonisation of ESS quality reporting SIMS ESS Handbook for quality reports 2014 2014 SIMS, ESMS, ESQRS v.2.0 Implementation plan (2018) Interoperability No html in ESMS 2015 ESTP Advanced course on quality reporting ESS Metadata and Quality Reporting Handbook Alternative solution for domain specific needs 2016-18 Task force on quality reporting Grants to support national implementation Task force on metadata implementation

The Single Integrated Metadata Structure (SIMS) SIMS is a dynamic and unique inventory of ESS standard reference metadata statistical concepts with definitions and reporting guidelines.

SIMS 2.0

ESS Quality reporting: commitment European Statistical Law Regulation (EU) 2015/759 of 29 April 2015 amending Regulation (EC) No 223/2009 on European statistics. Art. 12 (3): “Member States shall provide the Commission (Eurostat) with reports on the quality of data transmitted…” The Single Integrated Metadata Structure (SIMS) version 2.0 and the ESS Handbook for Quality report 2014 are approved as ESS standard by the ESSC SIMS can be considered the standard for quality reporting according to Article 12 of Regulation 223/2009 on European statistics

What are ESMS and ESQRS? ESMS and ESQRS are 2 structures (templates) that can be derived from SIMS Euro SDMX Metadata Structure (ESMS) recommended for user-oriented quality reporting ESS Standard for Quality Reports Structure (ESQRS) recommended for producer-oriented quality reporting

ESMS 2.0

ESQRS 2.0

SIMS 2.0 ESQRS 2.0 ESMS 2.0

ESS quality reporting: tools and standards

Methodological manuals Further methodological indications are needed to fill-in quality reports… ESS Handbook for Quality Reports

ESS Handbook for quality reports The new version of the Handbook will be probably available in mid 2019. Eurostat will provide more information on this topic tomorrow

ESS handbook for quality reports 2014 Aim: to provide guidelines for the preparation of comprehensive quality reports for a full range of statistical processes and their outputs Types of statistical processes: Sample survey Census Statistical process using administrative source(s) Statistical process involving multiple data sources Price or other economic index process Statistical compilation

The structure of the new handbook will be aligned with SIMS 2.0 ESS handbook for quality reports 2014 The chapters follow the (old) ESQRS structure because it is the standard for producer-oriented quality reports Main sections: Introduction Relevance Accuracy and Reliability Timeliness and Punctuality Coherence and Comparability Accessibility and Clarity Cost and burden Confidentiality Statistical processing The structure of the new handbook will be aligned with SIMS 2.0

Annex I: ESS Quality and Performance Indicators Set of standard quality indicators developed by an expert group Indicators for each ESS quality dimension, if possible “product” oriented

ESS Quality and Performance Indicators Timeliness and punctuality Time lag - first results Time lag - final results Punctuality - delivery and publication Comparability and coherence Asymmetry for mirror flows statistics - coefficient Length of comparable time series Accessibility and clarity Data tables - consultations Metadata - consultations Metadata completeness - rate Relevance Data completeness - rate Accuracy and reliability Sampling error - indicators Over-coverage - rate Common units - proportion Unit non-response - rate Item non-response - rate Imputation - rate Data revision - average size

Annex II: Technical manual of the SIMS 1. Introduction 2. Use and general characteristics of the SIMS 3. Distinction between the short user- (U) and the detailed producer- (P) oriented quality reports 4. Integration of the Quality and Performance Indicators in the SIMS 5. Implementation of the "once for all purposes" quality reporting strategy Annex 1 Creation of the Single Integrated Metadata Structure from the ESMS and ESQRS Annex 2 ESS guidelines for SIMS Annex 3 New ESMS guidelines as derived from SIMS Annex 4 New ESQRS guidelines as derived from SIMS Annex 5 How to create and submit a report in NRME SIMS v 2.0 – ESMS 2.0 and ESQRS 2.0 – 2015 ESS Metadata Handler

References Eurostat (2015) Single Integrated Metadata Structure v 2.0 – ESMS 2.0 and ESQRS 2.0 – 2015 http://ec.europa.eu/eurostat/documents/64157/4373903/SIMS-2-0-Revised-standards-November-2015-ESSC-final.pdf/47c0b80d-0e19-4777-8f9e-28f89f82ce18 Eurostat (2015) Single Integrated Metadata Structure and its Technical Manual – 2014 http://ec.europa.eu/eurostat/documents/64157/4373903/03-Single-Integrated-Metadata-Structure-and-its-Technical-Manual.pdf/6013a162-e8e2-4a8a-8219-83e3318cbb39 Eurostat (2014) ESS Handbook for Quality reports (2014 edition) http://ec.europa.eu/eurostat/documents/64157/4373903/01-ESS-Handbook-for-Quality-Reports-2014.pdf/d6152567-a007-4949-a169-251e0ac7c655 Eurostat (2014) ESS Guidelines for the implementation of the ESS quality and performance indicators (QPI) http://ec.europa.eu/eurostat/documents/64157/4373903/02-ESS-Quality-and-performance-Indicators-2014.pdf/5c996003-b770-4a7c-9c2f-bf733e6b1f31 ESS Standard for Quality Reports Structure (release 1, October 2010) Euro SDMX Metadata Structure (release 3, March 2009) Eurostat (2009). Eurostat Methodologies and Working papers “ESS Standards for Quality Reports”. http://epp.eurostat.ec.europa.eu/portal/page/portal/ver-1/quality/documents/ESQR_FINAL.pdf