1. Introduction to Quality Concepts
Measurement of the quality of statistics
3-5 October 2012
Marina Signore
Istat
Division "Metadata, Quality and R&D Projects", Chief

2. Topics
Definition of quality
Quality management

3. The definition of quality
The concept of quality
multi-faceted concept
likely to change over time
The definition of quality
ESS adopted a common definition
Quality is defined by Eurostat with reference to six criteria

4. Definition of quality for the ESS (ISO 8402, 1986)
The totality of features and characteristics of a product or service that bear on its ability to satisfy a given need

5. Eurostat Quality dimensions
Relevance
Accuracy
Timeliness and Punctuality
Accessibility and Clarity
Comparability
Coherence
Eurostat (2003), Definition of Quality in Statistics. Eurostat Working Group on Assessment of Quality in Statistics, Luxembourg, October 2-3.

6. Quality dimensions Relevance overall statistical activity
Relevance is the degree to which statistics meet current and potential users’ needs. It refers to whether all statistics that are needed are produced and the extent to which concepts used (definitions, classifications etc.) reflects user needs
overall statistical activity
users involvement and feedback from planning to dissemination

7. Quality dimensions Accuracy old meaning of quality
Accuracy in the general statistical sense denotes the closeness of computations or estimates to the exact or true values
old meaning of quality
effects of both sampling and non-sampling errors
several sub-components

8. Quality dimensions Timeliness and Punctuality increasing importance
Timeliness of information reflects the length of time between its availability and the event or phenomenon it describes
Punctuality refers to the time lag between the release date of data and the target date when it should have been delivered, for instance, with reference to dates announced in some official release calendar, laid down by Regulations or previously agreed among partners
increasing importance
trade-off with accuracy
release calendar

9. Quality dimensions Accessibility and Clarity users needs
Accessibility refers to the physical conditions in which users can obtain data: where to go, how to order, delivery time, clear pricing policy, convenient marketing conditions (copyright, etc.), availability of micro or macro data, various formats (paper, files, CD-ROM, Internet…), etc.
Clarity refers to the data’s information environment whether data are accompanied with appropriate metadata, illustrations such as graphs and maps, whether information on their quality also available (including limitation in use…) and the extent to which additional assistance is provided by the NSI.
users needs
metadata and quality

10. Quality dimensions Comparability
Comparability aims at measuring the impact of differences in applied statistical concepts and measurement tools/procedures when statistics are compared between geographical areas, nongeographical domains, or over time. We can say it is the extent to which differences between statistics are attributed to differences between the true values of the statistical characteristic
Comparability in the National statistical system
Comparability in the ESS

11. Quality dimensions Coherence Comparison between different sources
Coherence of statistics is their adequacy to be reliably combined in different ways and for various uses. It is, however, generally easier to show cases of incoherence than to prove coherence
Comparison between different sources

12. Eurostat Quality definition
Disadvantages
The dimensions are in conflict with each other
Not an easy task to assess each dimension
No successful attempt to derive an overall index
Advantages
Harmonised framework at European level for quality measuring and reporting
Easier for the users to judge and compare the quality of statistics
Can be used as a framework for the assessment of user perception of a statistical product

13. Quality management systems
The commitment for quality is increasing in most NSIs
The quality definition sets the requirements that statistical outputs have to meet in order to satisfy users needs
Need to have in place a set of coordinated and integrated tools and procedures for assessing and reporting quality but also to improve it
Quality management systems
To deliver “quality” products and services to users
To assess and systematically improve the quality of products, processes and services to users

14. Institutional environment
Elements of a quality management system
User needs
Management systems & leadership
Statistical products
Support processes
Statistical processes
Institutional environment
Handbook on Data Quality Assessment Methods and Tools (2007)

15. Product Quality
Refers to the quality of statistical output (or products)
It is usually expressed as a set of components which have some attributes or requirements that are to be met (e.g. Eurostat quality vector with 6 dimensions)
Essential for any assessment of data quality
Reporting to users can be organised following quality dimensions

16. Process Quality
Refers to the quality of the operations and instruments used in the data production process
The way by which statistical information is produced affects the output quality
To monitor and improve the statistical production process is a way to improve data quality
Focus on the survey operations and process quality indicators

17. Process Quality requirements
No standard criteria have been defined at ESS level
Efficiency - produces the desired outcomes cost efficiently
Effectiveness - successful in delivering the desired outcomes
Robustness - delivers results against challenging demands
Flexibility -readily adaptable to changing needs and demands
Transparency - open, visible and easily understood
Integration - complementary and consistent, both with other processes, and with meeting business needs
Haworth M., Bergdall M., Booleman M., Jones T., Madaleno M. (2001), “LEG chapter: Quality Framework for the European Statistical System”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, May 2001, CD-ROM

18. How to improve process quality
Standardisation of survey operations and tools
Use of best practices
To decrease the unnecessary variation
To adopt practices that have proved to be successful in improving quality or reducing costs
Process monitoring
- identification, measurement and analysis of key process variables
- tools from statistical quality control

19. Process monitoring Key process variables Examples
Those factors that can vary with each repetition of the process and have the largest effect on critical product characteristics, i.e. those characteristics that best indicate the quality of the products*
Examples
non-response of different types
interviewer performance
costs and use of time for different processes
*Jones and Lewis (2003), “Handbook on improving quality by analysis of process variables”. Eurostat

20. Continuous Quality improvement
the PDCA cycle introduced by Deming
plan
act do
check

21. Some remarks Quality management is a long term-commitment
Support from top management is crucial
Design an overall strategy for quality
Implementation can be done step by step
High initial costs and benefits are not immediate therefore quality work should be seen as an investment
May obstacles to face, culture changes might be necessary therefore training staff is important

22. References
Bergdahl M., Japec L., Jones T., Signore M. (2001), " LEG chapter: Tools for Standardising and Improving the Quality of Statistics Production Processes and Other Operations”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, May 2001, CD-ROM
Biemer P.P. and Lyberg L. (2003) “Introduction to Survey Quality”, Wiley
Blanc M., Lundholm G. and Signore M. (2001), “LEG chapter: Documentation”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, May 2001, CD-ROM
Blanc M., Radermacher W. and Koerner T. (2001), “LEG chapter: Quality and Users”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, May 2001, CD-ROM
Booleman M., van Brakel R., Jones T. and Tzougas I. (2001), “LEG chapter: Assessment tools”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, May 2001, CD-ROM
Eurostat (2003), Definition of Quality in Statistics. Eurostat Working Group on Assessment of Quality in Statistics, Luxembourg, October 2-3.
Haworth M., Bergdall M., Booleman M., Jones T., Madaleno M. (2001), “LEG chapter: Quality Framework for the European Statistical System”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, May 2001, CD-ROM
Jones and Lewis (2003), “Handbook on improving quality by analysis of process variables”. Eurostat 22 22

23. References
Eurostat (2003), Definition of Quality in Statistics. Eurostat Working Group on Assessment of Quality in Statistics, Luxembourg, October 2-3.
Handbook on Data Quality Assessment Methods and Tools (2007),
Haworth M., Bergdall M., Booleman M., Jones T., Madaleno M. (2001), “LEG chapter: Quality Framework for the European Statistical System”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, May 2001, CD-ROM
Lyberg L. et alt. (2001), “Summary Report from the Leadership Group (LEG) on Quality”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, May 2001, CD-ROM
Madaleno M., Koerner T., Radermarcher W., Booleman M., van Brakel R. (2001), “LEG chapter: Implemetation of Quality Management in National Statistical Institutes, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, May 2001, CD-ROM
M.J. Zilhão, M. Madaleno, (2001), "Quality Management in Statistics and Performance Indicators“, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, May 2001, CD-ROM
WESTAT, (1992) Statistical thinking. Rockville, Maryland 23 23