Documentation and survey quality

Introduction

Quality

A definition Quality can be defined simply as "fitness of use". In the context of a survey this translate to a requirement for survey data to be as accurate as necessary to achieve their intended purpose, be available at the time it is needed and be accessible to those for whom the survey was conducted. Accuracy, timeliness and accessibility then, are three dimensions of survey quality. (Biemer & Lyberg, 2003, p. 13)

accuracy Minimizing survey error? Or minimizing potential bad use of survey, that means documenting survey error

availability A typical task for the archives

timeliness Also a subject of preoccupation for the archives

Survey errors

Types of survey errors Coverage and Non-response error Measurement process

Coverage and Non-response error Design –Sampling Stratification Clustering –Coverage Issue –Weighting As a correction for the design As a correction of other bias Non-response –Unit-non-response The sources of non response How to report –Item non-response Estimation of missing values

Measurement process Question Design –Scales –Randomized experiences –Reference to past experiences Translation Interviewer effect Coding effect

Does it matters? One example: the debate between "reluctant" and "hard to reach"

Documentation

Biemer & Lyberg Frame and frame development –Target population and frame population –Description of the development of the frame –Information available on the frame –Process for constructing the frame –Coverage rates Sampling –Stratification and its purpose –Sampling design used –Measures of size used for multistage sampling –Sampling sizes by stratum and for each sampling size –Auxiliary information used –Anticipated precision of the estimates –Problem in sampling and any deviations from the sampling plan during implementation Data collection –Description of modes used –Procedures to contact sampled units including non-response follow-up –The interview or data collection process –Interviewer hiring, training, supervision, monitoring, and observation –Special approaches to reduce non-response, including incentives –Non-response rates by key respondent groups, including refusal, non-contact, and other non-response rates –Questionnaire and interview/data collection instructions

From the AAPOR guidelines Have specific goals for the survey Consider alternatives to using a survey to collect information Select samples that well represent the population to be studied Use design that balance costs with errors Take great care in matching question wording to the concepts being measured and the population studied Pretest questionnaire and procedures to identify problem prior the survey Train interviews carefully on interviewing techniques and the subject matter of the survey Construct quality checks for each stage of the survey Maximise cooperation or response rate within the limits of ethical treatment of human subjects Use statistical analytic and reporting techniques appropriate to the data collected Carefully develop and fulfill pledges of confidentiality given to respondents Disclose all methods of the survey that permit evaluation and replication

Construct quality checks for each stage of the survey

Any consequences for the archives?

Inside: how to take it in the data process Use of standard form –AAPOR form for response rate –Elaboration of forms for Interviewer training Pre-test Incentive Other form of promotion of the study: advance letter... Documentation of experiences? Other? Use of a question databank –Documenting history of the question –Documenting translations –Other?

Outside: how to give attention to this? Example: Ess and weighting