1. Quality assessment
ESTP Training Course “Quality Management and survey Quality Measurement” Rome, 24 – 27 September 2013
Giorgia Simeoni
Researcher
Unit “Auditing, Quality and Harmonisation“
Istat
Marina Signore
Director of Research
Chief, Division "Metadata, Quality and R&D Projects“
Istat

2. Content
Data quality assessment map
Quality reporting
Self-assessment, auditing, peer review
Labelling and certification

3. Elements of a quality management system*
Quality assessment
Assessment
Institution
Statistical products and processes
Elements of a quality management system*
User needs
Institutional
environment
Production processes
Statistical products
Management systems & leadership
Support processes
*Bergdahl et al. (2007) Handbook on DatQAM

4. Data quality assessment map*
External Environment
Users Requirements
Standards
Assessment
methods and tools
Labelling
Certification (ISO 20252)
Self-Assessment
Audits
Process Variables
Quality indicators
Quality Reports
User Surveys
Preconditions
Measurements of processes and products
*Bergdahl et al. (2007) Handbook on DatQAM

5. Main features of a quality report:
Quality reporting
A quality report provides information on the main quality characteristics of a product so that the user should be able to assess product quality. In the optimal case quality reports are based on quality indicators (Bergdahl et al., 2007)
Main features of a quality report:
Object: a statistical product or statistical process
Target recipients: users or producers of official statistics
Purposes: to accompany official statistics dissemination, to support quality assessment
Contents: quality dimensions, quality indicators
Further characteristics: standard structure, different levels of detail tailored on recipient and purpose

6. ESS standard structures for metadata and quality reporting
Euro SDMX Metadata Structure (ESMS)
Development of the technical standard «Statistical Data and Metadata eXchange (SDMX)»
Commission Recommendation of 23 June 2009 on reference metadata for the European Statistical System: ESMS is to be used by NSIs when compiling reference metadata in the different statistical areas and exchanging them within the European Statistical System or beyond.
Definition of reference metadata: Metadata describing the contents and the quality of the statistical data. Preferably, reference metadata should include all of the following: a) "conceptual" metadata, describing the concepts used and their practical implementation, allowing users to understand what the statistics are measuring and, thus, their fitness for use; b) "methodological" metadata, describing methods used for the generation of the data (e.g. sampling, collection methods, editing processes); c) "quality" metadata, describing the different quality dimensions of the resulting statistics (e.g. timeliness, accuracy).

7. Euro SDMX Metadata Structure (ESMS)

8. Type of statistical processes: Main section headings:
ESS standard structures for metadata and quality reporting ESS Handbook and Standard for Quality Reports
Guidelines for preparation of comprehensive producer oriented quality reports for different types of statistical process.
Type of statistical processes:
Main section headings:
Introduction
Relevance
Accuracy
Timeliness and Punctuality
Accessibility and Clarity
Coherence and Comparability
Sample survey
Census
Statistical process using administrative source(s)
Statistical process involving multiple data sources
Price or other economic index process
Statistical compilation
Trade-offs
Assessment of user needs and perceptions
Performance, cost and respondent burden
Confidentiality, trasparency and security
Conclusions

9. ESS Standard for Quality Reports Structure (ESQRS)

10. Self-assessment
Comprehensive, systematic and regular review of an organisation’s activities and results against a reference model/framework
«Do it yourself» evaluation, subjective
More general, not based on quantitative measures
It does not require extensive production of ad hoc documentation
Based on the compilation of check-list or questionnaires
It stimulates the quality awareness and continuous improvement
Self-assessment can be applied also at program/process level

11. Self Assessment Checklist for Survey Managers (DESAP)
generic checklist for a systematic quality assessment of surveys in the European Statistical System (ESS)
tool to support survey managers in assessing the quality of their statistics and in identifying improvement measures
fully compliant with the ESS quality criteria
comprises the main aspects relevant to the quality of statistical data
two versions: long and condensed

12. Statistical auditing
Systematic, independent and documented process for obtaining verifiable evidence and evaluating it objectively to determine the extent to which the audit criteria (policies, procedures or requirements) are fulfilled
External or internal auditors (external to the audited process)
It may require the compilation of reports, computation of quality indicators, preparation of ad hoc documentation
It may use supporting questionnaires
It results in the identification of strength and weaknesses of the process, and definition of improvement actions with a time scheduling for their implementation

13. Main tools for data quality assessment
Statistical auditing experiences
In October 2006, Istat interviewed some European Statistical Institutes and some overseas Statistical Offices on Statistical and IT Auditing activities.
At that time almost more than 1/3 of the institutions had started statistical auditing activities and the same amount of institutions were planning to adopt it.
Auditing is generally performed with reference to:
Standards and Guidelines for Statistical Surveys
Standard Quality Reports
ISO norms and Statistical production procedures manuals
Tools
Auditing questionnaire, Quality reports, …
Assessment report
Improvement actions report
Monitoring systems are built

14. Main tools for quality assessment
Peer reviews
Systematic examination and assessment of the performance of a statistical agency by other statistical agencies, with the objective of helping the reviewed institution to improve its quality policy, adopt best practices and comply with given standards and principles
More informal and less structured respect to auditing
May require to have extensive documentation at hand
Generally oriented to assess at high level of the institution
Generally they are not aimed at controlling conformity towards detailed checklist
Usually they produce recommendations and an implementation programme

15. Eurostat peer reviews
In Eurostat and all National Statistical Institutes were subject to a peer review to evaluate the compliance with the areas related to the institutional environment and dissemination of statistics (principles 1-6 and 15 of the Code of Practice
The exercise was complementary to a prior self-assessment evaluation
A general report and country-specific reports with the evidence of the results of the review process and a Summary of the good practices are publicly available on Eurostat website
Country specific reports included:
- Good practices to be highlighted
- Recommendations of the peer review team
- List of improvement actions by principle of the Code of Practice
Implementation of the action was monitored by means of a questionnaire in the next years
Eurostat is planning a new round of reviews (methodology not yet publicly available)

16. Labelling
It consists on the assignment of an attribute (short message) conferring a quality connotation to the product/service, based on a procedure to evaluate compliance to ex-ante (before statistics are produced) or ex-post (after statistics are produced) standards or requirements.
In official statistics field, the message is usually referred to the product or to the producer
Need for careful selection of the message
Need for existence of a procedure to guarantee the message is appropriate
It provides guarantees on the product/service
It increases the producer’s authoritativeness
Example of labels
“Official statistics”; “European statistics”

17. Certification
Method aimed at certifying the compliance to a international standard combining the external audit approach with the labelling one
ISO 20252:2006 Market, opinion and social research is an international standard on quality of data in the sector of the marketing, opinion and social research studies
It requires the adoption of a quality management system
It is a very transparent system
It may increase the Institution credibility
Usually, it requires the production of extensive documentation
It increases the awareness on process quality and it stimulates the quality improvement

18. References
Brancato G. (2007). Review on IT and Statistical Auditing Procedures, Istat, Italy
Eurostat ( ). Code of Practice Peer review reports by country
Handbook on Data Quality Assessment Methods and Tools (2007),
ESS Standard for Quality Reports Structure (release 1, October 2010)
Euro SDMX Metadata Structure (release3, March 2009)
Eurostat (2010), ESS guidelines for the implementation of the ESS quality and performance indicators
Eurostat (2009). Eurostat Methodologies and Working papers “ESS Standards for Quality Reports”.
Eurostat (2003a), Definition of Quality in Statistics. Eurostat Working Group on Assessment of Quality in Statistics, Luxembourg, October 2-3.
Eurostat (2003b), Glossary. Eurostat Working Group on Assessment of Quality in Statistics, Luxembourg, October 2-3.
Eurostat (2003). Checklist for Survey Managers (DESAP)
D’Orazio M, Carbini R, Signore M. (2012) Quality assessment in ISTAT: The combined use of standard quality indicator analysis and audit procedures. European conference on Quality in Official Statistics (Q2012) Athens, 29 May – 1 June
Götzfried A., Linden H., and Clement E. (2011) Standards and processes for integrating metadata in the European Statistical System. UNECE, Workshop on Statistical Metadata (METIS), Geneva, Switzerland 5-7 October 2011
International Organisation for Standardisation (ISO) (2006). Market, opinion and social research – Vocabulary and service requirements (ISO 20256:2005)