Overview of the ESS quality framework and context Remi Prual Statistics Estonia, Remi.Prual@stat.ee ESTP Training Course “Advanced course on quality reporting” Luxembourg, 24-25 November 2016

Contents Treaty of the Functioning of the European Union Regulation 223 European Statistics Code of Practice Quality Assurance Framework ESS Vision 2020

Treaty of the Functioning of the European Union Article 1: 1. This Treaty organises the functioning of the Union and determines the areas of, delimitation of, and arrangements for exercising its competences. Article 338: 1. … (EU) shall adopt measures for the production of statistics where necessary for the performance of the activities of the Union. 2. The production of Union statistics shall conform to impartiality, reliability, objectivity, scientific independence, cost-effectiveness and statistical confidentiality; it shall not entail excessive burdens on economic operators.

223/2009 Regulation on European Statistics Accepted by the Council and Parliament in March and came into force 1 April 2009 (updated in 2015). Article 1: This Regulation establishes a legal framework for the development, production and dissemination of European statistics. 223/2009 contains specific section on quality starting from the Code of Practice and containing the ESS quality dimensions. In practice it broadens the quality reporting scope to all statistics which so far did not have specific requirements, and unify the contents.

223/2009 Regulation (amended by 759/2015) sets out: Statistical principles (art. 2): professional independence, impartiality, objectivity, reliability, statistical confidentiality and cost effectiveness. Quality criteria (art.12 (1)): relevance, accuracy, timeliness, punctuality, accessibility and clarity, comparability and coherence. Specific quality requirements (art.12 (2)), such as target values and minimum standards for the production of statistics, may also be laid down in sectoral legislation / Commission implementing acts in specific statistical domains, including the modalities, structure and periodicity of quality reports.

223/2009 Regulation (amended by 759/2015) sets out: Quality reporting (Art.12 (3) and (4)): Member States have to provide Eurostat with reports on the quality of the data transmitted; Eurostat has to assess the quality of the data and has to prepare and publish reports on the quality of European statistics Link to the Code of Practice principles (art.11)

European Statistics Code of Practice According to art. 11 of Regulation 223/2009, ES CoP aims at ensuring public trust in European statistics. It establishes how European statistics should be developed, produced and disseminated in conformity with the statistical principles of professional independence, impartiality, objectivity, reliability, statistical confidentiality and cost effectiveness. It promotes the application of best international statistical practices, principles and methods.

European Statistics Code of Practice Quality is defined by the ES CoP in terms of the Institutional environment (6 principles) Statistical processes (4 principles) Statistical outputs (5 principles)

ES CoP: Institutional environment Institutional environment is the whole context in which the statistical authority operates and within which a programme of statistical processes is conducted. Some of the quality criteria of the institutional environment also concern the statistical processes – these criteria have a dual applicability. 1. Professional independence: professional independence of statistical authorities from other policy, regulatory or administrative departments and bodies, as well as from private sector operators, ensures the credibility of European Statistics. 2. Mandate for data collection: statistical authorities have a clear legal mandate to collect information for European statistical purposes. Administrations, enterprises and households, and the public at large may be compelled by law to allow access to or deliver data for European statistical purposes at the request of statistical authorities.

ES CoP: Institutional environment Institutional environment is the whole context in which the statistical authority operates and within which a programme of statistical processes is conducted. Some of the quality criteria of the institutional environment also concern the statistical processes – these criteria have a dual applicability. 3. Adequacy of resources: the resources available to statistical authorities are sufficient to meet European Statistics requirements. 4. Commitment to quality: statistical authorities are committed to quality. They systematically and regularly identify strengths and weaknesses to continuously improve process and product quality. 5. Statistical confidentiality: the privacy of data providers (households, enterprises, administrations and other respondents), the confidentiality of the information they provide and its use only for statistical purposes are absolutely guaranteed.

ES CoP: Statistical processes In the context of the ESS and in line with the principles of the CoP, the quality criteria of the statistical processes are as follows. Some of the quality criteria of the statistical processes also concern the institutional environment – these criteria have a dual applicability. 7. Sound methodology: sound methodology, including adequate tools, procedures and expertise, underpins quality statistics. 8. Appropriate statistical procedures: appropriate statistical procedures, implemented from data collection to data validation, underpin quality statistics. 9. Non-excessive burden on respondents: the reporting burden is proportionate to the needs of the users and is not excessive for respondents. The statistical authorities monitor the response burden and sets targets for its reduction over time. 10. Cost effectiveness: resources are used effectively.

ES CoP: Statistical outputs In line with the last five ES Code of Practice Principles, output quality in the ESS is assessed in terms of the following quality criteria: 11. Relevance: outputs, i.e. European Statistics meet the needs of users. 12. Accuracy and Reliability: outputs accurately and reliably portray reality. 13. Timeliness and Punctuality: outputs are released in a timely and punctual manner. 14. Coherence and Comparability: outputs are consistent internally,over time and comparable between regions and countries; it is possible to combine and make joint use of related data from different sources. 15. Accessibility and Clarity: outputs are presented in a clear and understandable form, released in a suitable and convenient manner, available and accessible on an impartial basis with supporting metadata and guidance.

Quality assurance Compliance with the ES Code of Practice is regularly monitored through the ESS-wide exercise of peer reviews which start with a national self-assessment questionnaire – improvement actions identified in the peer review exercise are then monitored and reported upon on an annual basis. As a 3rd level of quality assurance, the ESS Quality Assurance Framework (QAF) has been developed in 2011-2012. ESS QAF provides methods and tools for implementation at institutional and process level for the indicators of the ES Code of Practice as well as links to relevant reference documentation. Therefore, it provides clear guidance to compliance assessors.

Quality Assurance Framework The 3rd level of quality assurance Not part of the CoP, a complementary / underlying document Like the ES Code of Practice, it is applicable across all statistical domains It was endorsed by the ESSC in May 2015

Quality Assurance Framework of the ESS - Quality Reporting (I) Principle 4: Commitment to Quality Statistical authorities are committed to quality. They systematically and regularly identify strengths and weaknesses to continuously improve process and product quality. Indicator 4.3 Product quality is regularly monitored, assessed with regard to possible trade-offs, and reported according to the quality criteria for European Statistics Methods of implementation – institutional level Procedures to monitor product quality: procedures based on quality reporting are in place to internally monitor product quality. Results are analysed regularly and senior management is informed in order to decide improving actions User satisfaction surveys: user satisfaction surveys or other indirect methods are implemented on a regular basis and their results are made public and incorporated where useful in Quality Reports, since they monitor “Relevance”, amongst other dimensions

Quality Assurance Framework of the ESS - Quality Reporting (II) Principle 4: Commitment to Quality Statistical authorities are committed to quality. They systematically and regularly identify strengths and weaknesses to continuously improve process and product quality. Indicator 4.3 Product quality is regularly monitored, assessed with regard to possible trade-offs, and reported according to the quality criteria for European Statistics Methods of implementation – product/process level User oriented quality reports: User oriented quality reports are made public, bearing in mind the standards for reference metadata and quality indicators, in particular the Single Integrated Metadata Structure (SIMS). Producer oriented quality reports: producer oriented quality reports are published regularly (periodicity to be determined: e.g. by the specific Regulation and the survey life cycle), bearing in mind the standards for reference metadata and quality indicators, in particular the Single Integrated Metadata Structure (SIMS). Product quality monitoring: users and producers quality reporting is used for regular quality monitoring over time.

Indicator 4.3: Product quality is regularly monitored Links between quality reporting, quality assessment and the overall framework Indicator 4.3: Product quality is regularly monitored

Quality reporting in Code of Practice Principle 1 - Professional independence   Principle 2 - Mandate for Data Collection Principle 3 - Adequacy of Resources Principle 4 - Commitment to Quality Principle 5 - Statistical Confidentiality Principle 6 - Impartiality and Objectivity Principle 7 - Sound methodology Principle 8 - Appropriate statistical procedures Principle 9 - Non-excessive burden on respondents Principle 10 - Cost Effectiveness Principle 11 - Relevance Principle 12 - Accuracy and Reliability Principle 13 - Timeliness and Punctuality Principle 14 - Coherence and Comparability Principle 15 - Accessibility and Clarity

Any questions so far?

ESS Vision 2020 Common strategic response of the ESS to the challenges that official statistics is facing. Adopted by the ESSC in May 2014. Five key areas: Focus on users + Strive for quality + Harness new data sources + Promote efficiency in production processes + Improve dissemination and communication Complemented by supporting frameworks: Quality + Cooperation Models + ESS Enterprise Architecture + Information Models and Standards

Q in the ESS Vision 2020 Action line 1. COORDINATE QUALITY IN THE ESS VISION 2020 PORTFOLIO AND PRESERVE TRUST IN EUROPEAN STATISTICS by continuously PROVIDING HIGH-QUALITY PRODUCTS AND SERVICES – Coordinate quality in the ESS Vision 2020 Portfolio, ensure consistency with the ESS quality framework and enhance the ESS quality framework where appropriate. Action line 2. ENHANCE QUALITY MANAGEMENT IN THE ESS, DEVELOP THE CONCEPTUAL ELEMENTS OF A COMMON QUALITY FRAMEWORK FOR THE ESS – develop further the quality approach which is applicable to the statistical institutions as a whole, inspired by a holistic approach to quality and embed risk management in quality management.

Q in the ESS Vision 2020

Q in the ESS Vision 2020 The deliverables of AL2 (like e.g. the knowledge base with experiences, lessons learned and good practices) will provide the necessary elements, core values, principles and models to manage the quality in the ESS in an integrated manner. These elements are of crucial importance to the implementation of the Quality key area of the ESS Vision 2020, as emphasised in the February 2015 ESSC meeting.

Q in the ESS Vision 2020

Any questions?

References Treaty of the Functioning of the European Union 223/2009 Regulation on European Statistics Eurostat quality framework European Statistics Code of Practice Quality Assurance Framework of the ESS (v.1.2) ESS Vision 2020