3 STEPS IN DESIGNING AND CONDUCTING HEALTH SURVEYS
4 TYPES OF SURVEY ERRORSSystematic Error (bias): Difference (positive or negative) between survey estimate and actual population valueVariable (random) Error: Range or variation (variance) in values of an estimate across observations (or cases)
6 OVERVIEW OF SURVEY ERRORS SURVEY PROCEDURESYSTEMATIC ERRORVARIABLE ERRORStudy DesignPoor internal validityPoor external validityDesign specification ambiguitySample DesignNoncoverage biasWeighting errorsStandard errorsDesign effectsData CollectionUnit nonresponse biasItem nonresponse biasInterviewer variabilityMode effectsQuestionnaire DesignUnder/over-reportingYea-sayingOrder & context effectsMeasurementLow or poor validityLow or poor reliabilityData PreparationImputation/Estimation errorsData coding, editing, or data entry errorsAnalysis PlanPoor statistical conclusion validityLow statistical precision or power
7 SURVEY ERRORS: Study Design SystematicPoor internal validity. The study design does not adequately and accurately address the study’s hypotheses, particularly with respect to demonstrating a causal relationship between the independent (predictor) and dependent (outcome) variables.Poor external validity. Findings based on the study design cannot be widely or universally applied to related populations or subgroups.VariableDesign specification ambiguity. The statement of the study objectives and related concepts to be measured in the survey are not clearly and unambiguously stated, particularly in relationship to the underlying study design and data analysis plan for the study.
8 SURVEY ERRORS: Sample Design SystematicNoncoverage bias. All units of the target population (e.g., households, individuals) are not included in the sampling frame (frame bias) or the respondent is not selected randomly (respondent selection bias).Weighting errors. Respondents are disproportionately represented in the survey sample by failing to weight each of the cases by the disproportionate probability of their falling into the sample (sampling fraction).VariableStandard errors. The standard error measures random sampling variation in an estimate (e.g., mean or proportion) across all possible random samples of a certain size that could theoretically be drawn from the target population.Design effects. The design effect, computed as the ratio of the variance of a complex sample to that of a simple random sample, measures the increase in random sampling variation in an estimate due to the complex nature of a sample design.
9 SURVEY ERRORS: Data Collection SystematicUnit nonresponse bias. Selected units of the study sample (e.g., households, individuals) are not included in the final study due to respondent refusals or unavailability during the data collection process.Item nonresponse bias. Selected questions on the survey questionnaire are not answered due to respondent refusals or interviewer or respondent errors or omissions during the data collection process.VariableInterviewer variability. Survey interviewers or data collectors vary in how they ask or record answers to the survey questions.Mode effects. The responses to comparable questions by respondents vary across different data collection methods (e.g., personal interview, telephone interview, mail self-administered questionnaire, web survey, etc.). [Note: If these effects differ in a particular direction across mode, they become systematic errors.]
10 SURVEY ERRORS: Questionnaire Design SystematicUnder/over-reporting. An estimate (e.g., mean or proportion) across samples differs in a particular (negative or negative) direction from the underlying actual (or true) population value for the estimate, i.e., is lower (underreporting) or higher (overreporting).Yea-saying. Respondents tend to agree rather than disagree with statements as a whole (acquiescent response set) or with what are perceived to be socially desirable responses (social desirability bias).VariableOrder & context effects. Answers to selected survey questions vary depending on whether they are asked before or after other questions and/or appear at the beginning or the end of the survey questionnaire.
11 SURVEY ERRORS: Measurement SystematicLow or poor validity. Systematic departures exist in answers to the content of a survey question from the meaning of the concept itself (content validity), a criterion for what constitutes an accurate answer based on another data source (criterion validity), and/or hypothesized relationships of the concept being measured with other measures or concepts (construct validity).VariableLow or poor reliability. Random variation exists in answers to a survey question due to when it is asked (test-retest reliability), who asked it (inter-rater reliability), and/or that it is simply one of a number of questions that could have been asked to obtain the information (internal consistency reliability).
12 SURVEY ERRORS: Data Preparation SystematicImputation/Estimation errors. Procedures for assigning values to survey questions for which answers are not available (missing values due to item nonresponse) either from data internal or external to the survey (imputation or estimation, respectively) introduce systematic errors (biases) in estimating or examining relationships between variables.VariableData coding, editing, or data entry errors. Data coding, editing, or data entry personnel or procedures introduce random errors in producing data files based on the survey questionnaires.
13 SURVEY ERRORS: Analysis Plan SystematicPoor statistical conclusion validity. The accuracy of statistical conclusions is compromised due to the application of statistical procedures that do not meet underlying assumptions related to the study design and objectives, level of measurement of study variables, and/or the underlying population distribution.VariableLow statistical precision or power. There are insufficient cases in the study sample to estimate population parameters with a reasonable level of precision or to have enough statistical power to detect statistically (and substantively) significant relationships between variables if they do exist.