4.1. Data Quality 1

Introduction The objective of quality measurement of international merchandise trade statistics is twofold: to provide producers with the appropriate information to monitor and further enhance data quality and to provide users with sufficient information to judge whether the data are adequate for their intended use. The development of a Quality framework is needed to fulfill properly users needs. It can be considered as a third pillar, in addition to the delivery of data and metadata. Various Quality frameworks have been developed at national and international level (IMF, EUROSTAT, OECD, ECB,…) Quality of statistics is a multidimensional concept and is not so easy to measure. There is no unique structure between the various frameworks but many common or related dimensions. The ISO 9000 standards gives a frame for quality management but need to be adapted to the specific domain of statistics.

Definition and preliminary remarks Quality is generally defined as “the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs” A quality assessment programme should cover: the quality of outputs (measured on the basis of a “quality model” composed of various dimensions/components) the analysis of procedures included in the collection/production process in order to ensure the quality of outputs. 3

IMTS 2010 recommendations on quality Outline Quality models IMTS 2010 recommendations on quality MEDSTAT III Draft Quality questionnaire 4

Quality models 1.1 European Statistical System (ESS) Standard for Quality Reports Provides recommendations for preparation of comprehensive Quality reports Recommendations are linked to the European Statistics Code of Practice Recommended standard structure: Introduction - brief history of the statistical process - overview of all outputs - reference to documentation, especially on methodology Relevance Degree to which statistical outputs meet current and potential user needs. It depends on whether all the statistics that are needed are produced and the extent to which concepts used (definitions, classifications,…) reflect user needs Accuracy Accuracy of outputs is the degree of closeness of estimates to the true value Should be covered: (i) coverage errors, (ii) measurement errors, (iii) non response errors, (iv) processing errors

Quality models 1.1 European Statistical System (ESS) Standard for Quality Reports Accuracy coverage errors: are due to divergences between the target population and the frame population Example: free zones excluded from the statistical territory, trade below threshold… What should be included on Coverage Errors: An assessment, preferably quantitative, on the extent of undercoverage and the bias risks associated with it. Actions taken for reduction of undercoverage and associated bias risks, measurement errors: are errors that occur during data collection and cause recorded values of variables to be different from the true ones Example: Wrong typing by declarants, misclassification of goods,… What should be included on Measurement Errors: Identification and general assessment of the main risks in terms of measurement error. Information on failure rates during data editing. Efforts made in the collection process on error reduction. Forms used should be annexed (if very long by hyperlink)

Quality models 1.1 European Statistical System (ESS) Standard for Quality Reports Accuracy non response errors: Non response is the failure of a sample survey or a census to collect data for all data items in the survey questionnaire from all the population units designated for data collection. Example: missing declarations, missing data (supplementary units,…) What should be included on Non response errors: Non response rates according to the most relevant variables A qualitative statement on the bias risks associated with non response Measures to reduce non response Technical treatment of non response at the estimation stage processing errors: errors generated during data entry, data editing (checks and corrections), coding or imputation. Example : errors introduced in the manual coding of declarations What should be included on Processing Errors: Identification of the main issues regarding processing errors for the statistical process and its outputs. Where relevant and available, an analysis of processing errors affecting individual observations should be presented, else a qualitative assessment should be included.

Quality models 1.1 European Statistical System (ESS) Standard for Quality Reports Timeliness and Punctuality Timeliness of statistical outputs is the length of time between the event or phenomenon statistical process they describe and their availability Punctuality is the time lag between the release date of data and the target date on which they were scheduled for release as announced in an official release calendar Accessibility and Clarity Accessibility and clarity refer to the simplicity and ease with which users can access statistics with the appropriate supporting information and assistance Coherence and Comparability The coherence of two or more statistical outputs refer to the degree to which the statistical processes by which they are generated used the same concepts and harmonised methods. Comparability is a special case of coherence, and refers to the aim of combining outputs to make comparisons over time, or across regions, or across domains.

Quality models 1.1 European Statistical System (ESS) Standard for Quality Reports Trade-offs between output quality Components The report should deal with the trade-offs that have to be made assuming that the budget is limited and that factors contributing to improvements with respect to one component lead to deterioration with respect to another Examples: trade-off relevance – timeliness; accuracy – timeliness,… Assessment of users needs and perceptions The most effective method of evaluating user perceptions is a full scale user satisfaction survey Performance, Cost and Respondent Burden Resources must be effectively used. Respondent burden should be proportional to the needs of users and not excessive for respondents. Respondent burden should be measured and targets set for its reduction over time. Confidentiality, Transparency and Security The privacy of data providers, the confidentiality of the information and its use only for statistical purposes must be absolutely guaranteed. Statistical authorities must produce and disseminate statistics respecting scientific independence ad in an objective, professional and transparent manner in which all users are treated equitably.

Quality models 1.2 Other Quality Models: IMF DQAF IMF Data Quality Assessment Framework First developed in 2001 by the IMF Statistics Department, its aim is to complement the quality dimension of the IMF Special Data Dissemination Standard and General Data Dissemination System and to support assessment of the quality of data provided by countries as background for IMF Reports on the Observance of Standards and Codes A generic DQAF has been designed in 2003. It is based on five dimensions (assurances of integrity, methodological soundness, accuracy and reliability, serviceability, and accessibility) + a set of prerequisites Several specific frameworks have been defined, including Balance of Payments statistics

Quality models 1.2 IMF DQAF : generic model

Quality models 1.2 IMF DQAF

Quality models 1.2 IMF DQAF

Quality models 1.2 IMF DQAF 14

Quality models 1.2 IMF DQAF

Quality models 1.2 IMF DQAF

IMTS Recommendations on data quality (Chapter 9) It is recommended that countries develop quality standards and related good practices covering the institutional arrangements, the statistical processes and outputs. It is recommended that countries develop a standard for regular quality reports which cover the full range of statistical processes and their outputs. Such reports can be either producer-oriented with the aim to identify strengths and weaknesses of the statistical process and lead to or contain the definition of quality improvement actions or user-oriented with the aim to keep users informed on the methodology of the statistical process and the quality of the statistical output. It is recommended that the quality reports of international merchandise trade statistics should be completed or updated at least every five years or more frequently if significant methodological changes or changes in the data sources occur. It is recommended that countries base their quality reports on a set of quantitative and qualitative indicators for international merchandise trade statistics and on a checklist covering data collection, processing and dissemination to allow for an assessment of strengths and weaknesses in the statistical process and to identify possible quality improvement actions.

Quality report on trade statistics IMTS 2010 Suggested indicators

Quality report on trade statistics IMTS 2010 Suggested indicators

MEDSTAT III Quality questionnaire Proposed draft revised questionnaire … New questions are highlighted in red.