Survey phases, survey errors and quality control system Measurement of the quality of statistics 3-5 October 2012   Marina Signore Istat Division "Metadata, Quality and R&D Projects", Chief

Topics Survey phases and survey errors Quality control system

Survey phases Set of homogenous operations from an organisational and chronological point of view Each phase is a potential source of errors Error sources and methods to prevent or reduce errors impact for each phase of the statistical production process

GSBPM: Generic Statistical Business Process Model

GSBPM main features Developed within the Joint UNECE/Eurostat/OECD Work Session on Statistical Metadata (METIS) Born as a documentation model, with the aim of standardise terminology Organised in four levels with increasing level of detail Over-arching processes Flexibility Final version April 2009

Over-arching processes The Generic Statistical Business Process Model Level 0 Over-arching processes Level 1 Level 2 5.2. Classify and code - This sub-process classifies and codes the input data. For example automatic (or clerical) coding routines may assign numeric codes to text responses according to a pre-determined classification scheme. Level 3

Questionnaire design and testing planning Sample selection Data processing Data collection Documentation Quality control system

Quality control system Survey phases Planning Questionnaire design and testing Sample selection Data collection Data processing Documentation Quality control system

Planning Determine needs for information Design outputs Design variables descriptions Design data collection methodology Design frame and sample methodology Design production system & workflow Design the quality control system Take into account constraints in terms of available resources, costs, timeliness, national or international regulations.

Planning Determine needs for information Design outputs Impact on relevance, comparability and coherence Planning Determine needs for information Design outputs Design variables descriptions Design data collection methodology Design frame and sample methodology Design production system & workflow Impact on accuracy but also on timeliness

Survey phases Questionnaire design and testing Sample selection Data collection Data processing Impact on accuracy but also on timeliness

Questionnaire design and testing The questionnaire is a communication tool highly influential for data quality question wording response categories and their order the way and context in which questions are presented Source of measurement errors and item non response In some cases also of unit non response

Sample selection The selection from a frame of the population units given the sample design and size the population to be investigated should have been specified and the frame from which the units are sampled and contacted should have been chosen Source of coverage or frame errors

Data collection data collection is any process whose purpose is to acquire or assist in the acquisition of data* the impact of data collection is both direct and critical as these data are the primary input. The quality of the operation has a very high impact on the quality of the final product, in particular on accuracy the mode of administration of the questionnaire (e.g. mail, telephone, in person) has an influence on data quality. Respondents may answer questions differently in the presence of an interviewer, over the phone, or by themselves Source of unit non response and measurement errors (respondent and interviewer errors) But also data collection mode effect *Statistics Canada (2003)

Data processing - data entry - editing and imputation - coding Coding: operation of converting an actual response for a survey question into a category (open-ended responses e.g. “other specify” or “specify occupation” Data entry: process of transferring collected data to an electronic medium Editing and Imputation: data editing is the application of checks to detect missing, invalid or inconsistent entries or to point to data records that are potentially in error and the imputation is the process used to determine and assign replacement values for missing, invalid, inconsistent data that have failed edits

Data processing Source of processing errors the errors range from simple recording errors to mispecification of the editing and imputation model. different impact for different data collection mode. e.g. different coders are likely to interpret and code the same response differently. Even the same coder may change over time (get experienced or bored). They affect the accuracy of final estimates Source of processing errors

Survey phases and survey errors Questionnaire design Measurement errors Item non response Sample selection Coverage or frame errors Data collection Unit non response Data processing Processing errors

Quality Control System Set of tools and methods to improve data quality Set of quality control actions for improving the quality of each survey phase and operations Quality control activity should be part of the survey process itself

Quality Control Actions Preventive actions: actions put in place in order to avoid potential errors usually before the operation is carried out Monitoring actions: actions put in place in order to reduce errors during the execution of an operation Ex-post actions: actions put in place in order to estimate the impact on the final data of non sampling errors

Some examples Quality control system Training of interviewers Advance letter to respondents Interviewers response rate Questionnaire testing Follow-up on non respondent units Edit and imputation procedures Sampling variance estimation

Quality Control System Planning quality Prevent Monitor Evaluate Run the checks Process Indicators Direct Measures Analyse results

Quality Measures Direct measures Quality indicators + estimate of error impact - very expensive - no overall estimate Quality indicators - indirect estimate of error impact + by-product of the production process + alarm bell

Documentation It is important to document the survey operations and procedures, the quality controls done and their results It is a demanding and time-consuming task and usually has not a high priority for survey managers

Documentation It should be considered as an investment: Save time in future survey occasions Easier to train new staff Share good practices and know-how Transparency

References Biemer P. and Lyberg L. (2003), Introduction to Survey Quality, J. Wiley, N.Y. Groves R.M. (1989), Survey Errors and Survey Costs, J. Wiley, N.Y. Lyberg L. and Kasprzyk D. (1991), “Data Collection Methods and Measurement Errors: An Overview” in Measurement errors in surveys, Biemer et alt. (eds.), J. Wiley, N.Y. Lyberg L. and Kasprzyk D. (1997), “Some Aspects of Post-Survey Processing” in Survey Measurement and Process Quality, Lyberg et alt. (eds.), J. Wiley, N.Y. ONS (2004), Guidelines for Measuring Statistical Quality, version 1.0, http://www.statistics.gov.uk/methods_quality/publications.asp Statistics Canada (2003), Quality Guidelines, fourth edition 25 http://www.statistics.gov.uk/methods_quality/publications.asp 25