1. Total Survey Error Design, Implementation, and Evaluation
Paul P. Biemer
RTI International and
University of North Carolina at Chapel Hill

2. Modern View of Survey Design
Surveys should be designed to maximize total survey quality within timeliness and budget constraints.
But how when…
survey budgets are severely constrained,
data must be produced and disseminated in a timely fashion,
public interest in participating in surveys has been declining world-wide for years, and
even when participation is obtained, responses may not be accurate.
This is the challenge for survey research in the 21st century

3. Outline What is total survey quality?
How does it differ from total survey error?
How can surveys be designed to maximize total survey quality?
What is the total survey error paradigm and what does it say about the design, implementation, and evaluation of survey?

4. User and Producer Have Very Different Perspectives on Survey Quality
Producers place high priority on
Accuracy – total survey error is minimized
Credibility – credible methodologies; trustworthy data
Users place higher priority on
Timeliness – data deliveries adhere to schedules
Relevance – data satisfy user needs
Accessibility – access to data is user friendly
Interpretability – documentation is clear; meta-data are well-managed

5. Users Also Demand…
Comparability – valid demographic, spatial and temporal comparisons
Coherence – estimates from different sources can be reliably combined
Completeness – data are rich enough to satisfy the analysis objectives without undue burden on respondents

6. TSQ Optimally Balances Producer and User Requirements
Timeliness
Accessibility
Comparability
Coherence
Completeness
Relevance
Accuracy
Credibility
Interpretability
Producer

7. The Total Survey Quality Paradigm
Identifies measurable and achievable objectives for each user-defined dimension of quality
Determines costs and resources required to achieve these objectives
Maximizes survey accuracy within remaining budget

8. Accuracy is maximized by minimizing total survey error
Sampling Error
Sampling scheme
Sample size
Estimator choice
Nonsampling Error
Specification
Nonresponse
Frame
Measurement
Data processing
Systematic
Variable
Bias
Variance
Mean Squared Error (MSE)
MSE = Bias2 + Variance

9. Optimal Design for Total Survey Quality
Minimize total survey error
Accuracy
Subject to
Sampling error
budget
time
Timeliness
Specification error
Accessibility
Nonresponse error
Interpretability
Frame error
constraints
Measurement error
Comparability
Data processing error
Credibility

10. Designing Surveys to Minimize Total Survey Error
Objective – minimum mean squared error (MSE) subject to cost and timeliness constraints
Major bias contributors
concept misspecification
frame noncoverage
nonresponse
measurement bias
editing errors
Major variance contributors
sampling error
measurement unreliability
interviewer error

11. Key Design Principles
Design robustness – accuracy does not change appreciably as the survey design features change; i.e. optimum is “flat” over a range of alternate designs
Effect generalizability – design features found to be optimal for one survey are often generalizable to other similar surveys
optimum accuracy

12. Key Design Principles
Design robustness – accuracy does not change appreciably as the survey design features change; i.e. optimum is “flat” over a range of alternate designs
Effect generalizability – design features found to be optimal for one survey are often generalizable to other similar surveys
loss in accuracy

13. Implications for Design
Compile information on TSE (e.g., quality profiles)
Identify major contributors to TSE
Allocate resources to control these errors
Use results from the literature and other similar surveys to guide the design
Develop an effective process for modifying the design during implementation to achieve optimality
Embed experiments and conduct studies to obtain data on TSE for future surveys

14. Design Implementation Strategies
The initial survey design must modified or adapted during implementation to control costs and maximize quality.
Four strategies for reducing costs and errors in real-time:
Continuous quality improvement
Responsive design
Six Sigma
Adaptive total design and implementation
Initial quality
Final quality

15. Continuous Quality Improvement (CQI)
Prepare a workflow diagram of the process and identify key process variables.
Identify characteristics of the process that are critical to quality (CTQ).
Develop real-time, reliable metrics for the cost and quality of each CTQ.
Continuously monitor costs and quality metrics during the process.
Intervene as necessary to ensure that quality and costs are within acceptable limits.

16. Responsive Design Strategy
Developed for face to face data collection (Groves & Heeringa, 2006)
Similar to CQI but includes three phases:
Experimental phase – tests major design options
For e.g., split sample designs to test incentive levels
Main data collection phase – implements design selected in first phase
Continues until “phase capacity” is reached
NRFU phase – special methods implemented to reduce nonresponse bias and control data collection costs
NR double sampling, higher incentives, more intensive followup
Phase capacity – point at which efforts to reduce NR bias under current protocol are no longer cost effective
Innovative uses of paradata for CQI

17. Six Sigma Developed by Motorola in the 1980’s
Definition (from Pande, et al, 2000, p. xi)
Extends ideas of Total Quality Management (TQM) and continuous quality improvement (CQI)
Has mostly been applied in business and manufacturing.
“A comprehensive and flexible system for achieving, sustaining and maximizing business success,…uniquely driven by a close understanding of customer needs, disciplined use of facts, data, and statistical analysis, and diligent attention to managing, improving, and reinventing business processes.” – Pande, et al (2000, p. xi)

18. Strengths of Six Sigma
Provides a systematic, highly effective approach for quality improvement (DMAIC).
Focuses on attributes of a process that are most important to the client.
Emphasizes decision making based on data analysis.
Strives for verifiable and sustainable improvements for both costs and quality.
Contains a rich set of techniques and tools for monitoring, controlling, and improving a process.

19. Weaknesses of Six Sigma
Can be expensive to implement.
Achieving 3.4 defects per million opportunities is an impossible goal for many survey processes.
Often requires data that do not exist and cannot be obtained affordably.
Terminology and some techniques are too business and manufacturing oriented. This obscures its applicability to survey work.
Uses a lot of jargon.

20. Six Sigma’s DMAIC Strategy
Define the problem.
Measure key aspects of the process and collect relevant data.
Analyze the data to determine root causes of the problem.
Improve the process based upon results from the data analysis.
Control the process by continuously monitoring metrics from the process.

21. Typical Survey Design and Implementation Process
Develop a survey design with design options A, B, etc
Monitor critical-to-quality design attributes (CTQs) 1
Post-survey processing, adjustment, and file preparation
Modify design to maximize accuracy while meeting cost and timeliness objectives
Pretest design and options
Pre-release quality evaluations
Budget or schedule exhausted?
Data release
Select and Implement best design option no
STOP
yes 1

22. Six Sigma Focuses Primarily on these Activities
Develop a survey design with design options A, B, etc
Monitor critical-to-quality design attributes (CTQs) 1
Post-survey processing, adjustment, and file preparation
Modify design to maximize accuracy while meeting cost and timeliness objectives
Pretest design and options
Pre-release quality evaluations
Budget or schedule exhausted?
Data release
Select and Implement best design option no
STOP
yes 1

23. Adaptive Total Design and Implementation
An approach for continuously monitoring survey processes to control errors, improve quality, and reduce costs.
Adaptive in that it combines the real-time error control features of CQI, responsive design, and Six Sigma strategies.
Total in that it simultaneously monitors multiple sources; for e.g.,
Sampling frame and sampling
Response quality
Nonresponse bias reduction
Field production
Costs and timeliness

24. Six Sigma Tools and Concepts
Workflow diagram
Common vs. special cause variation
Process control chart
Dashboard
Fishbone diagram
Pareto chart
Many others are available (see Breyfogle, 2003))

25. Workflow Diagram for Sampling and Initial Interview Attempt1 1 1 2 3
Compute domain sample size
Compute current eligibility rate
Compute required sample per PSU 2 3 4 3
Select sample lines to send to field
Assign case priorities
Transmit to FS’s in field 4 5 4 5 6
Assign cases to FIs
Conduct travel efficiency sessions
Optimize work sequence order 7
Contact?
Interview?
FI places initial contact attempt
Yes
Yes 2 8 9
Complete ROC log 1
Set appointment
Critical to quality key:
Achieve target sample sizes for each domain
Distribute sample to PSUs to minimize design effects
FI workloads must be adequate
Ensure high response propensity for high priority cases
Minimize FI travel costs through work sequence optimization
Ensure that high priority cases are worked fully
Ensure good cooperation at first contact
Record of calls (ROC) is completed accurately
Schedule an firm appointment after each contact

26. CTQs and Metrics for Frame Construction and Sampling
Maximize frame coverage
Maximize within unit coverage
Detect/control duplications and
ineligibles
Effectively post-stratified sample
for bias reduction
Use auxiliary data and efficient
estimators to minimize sampling
error
Minimize design effects for key
analytic domains
Achieve target sample allocations
for key domains
Optimally allocate sample to strata
and sampling stages
Process Metrics
Ineligibility rates by domain, PSU,
and overall
Achieved # interviews by domain
# HU's identified by q.c.
Projected vs. actual coverage
Design effects by domain
Screener mean propensity
Interview mean propensity
# active cases

27. CTQs and Metrics for Observation Quality
Detect/control post-survey
measurement errors
Identify/repair problematic survey
questions
Detect/control response errors
Minimize interviewer biases and
variances
Process Metrics
CARI results by interviewer and
overall
Interviewer exception report
Missing data item frequency by
interviewer
Replicate measurement analysis
summary
Interview length by interviewer
CARI refusal rate by FI, by
phase

28. CTQs and Metrics for Nonresponse Followup
Maximize response rates
Minimize nonresponse bias
Effectively adjust for unit
nonresponse
Effectively impute missing data for
key items
Process Metrics
Overall Phase 3 sampling rate by
PSU and overall
Response rate for high priority
cases
Hours per converted
nonrespondent (refusal vs. other)
Projected WRR by PSU and overall
(actual vs. expected)
Projected design effects by domain
Budgeted vs actual hours charged
for Phase II

29. CTQs and Metrics for Costs, Production, and Timeliness
Maximizing interviewing efficiency
Maximize effectiveness of refusal
conversions attempts
Complete call histories accurately
and completely
Minimize hours per completed
screener
Minimizes hours per completed
interview
Maintain planned costs per quarter
Maintain planned schedule for
sample completion per quarter
Process Metrics
Cost per interview
Dollars spent vs. dollars budgeted
by interviewer
Dollars spent vs. value of work
conducted by interviewer
Cost breakdown (by phase and
overall)
Number of cases interviewed
(actual vs. budgeted)
Calls per hour (actual vs. expected)
Refusal conversion rates by
interviewer
Hours charged (actual vs.
expected)
Level of effort per case by
interviewer and overall
Hours per completed screener
Hours per completed interview

30. Special vs. Common Cause Variation
Special causes – assignable to events and circumstances that are extraordinary, rare and unexpected
e.g., frame was not sorted prior to sampling
Addressed by actions specific to the cause leaving the design of the process essentially unchanged
Common causes – naturally occurring random disturbances that are inherent in any process and cannot be avoided.
e.g., normal fluctuations of response across regions and months
Actions designed to address a common cause is neither required nor advisable; this lead to process “tampering”

31. Chart of Screening Response Rates by County

32. Chart of Screening Response Rates by County
Problem counties?

33. Chart of Screening Response Rates by County

34. Process Control Chart with More Extreme Values

35. Process Control Chart with More Extreme Values
Special cause

36. Interviewer Efficiency - Contacting and Locating
Example of a Dashboard
Interviewer Efficiency - Contacting and Locating

37. Dashboard Showing Weighted Response Rates, Interview Costs, Interviewer Exceptions and Production

38. Other Useful Tools Cause and effect (fishbone) diagrams
Helps to identify all possible root causes of a problem
An important component of the measure stage of DMAIC.
Economy
Supervision
Availability of higher paying jobs
Reward system
Poor supervision
Lack of benefits
Interviewer turnover
Misinformation from other FIs
Lack of steady work
Family situation
Low pay
Conflict with supervisor
Inadequate training
Job difficulty
Job characteristics
Personal reasons
Unrealistic employee expectations

39. Other Useful Tools (cont’d)
Pareto chart
Useful for identify the “vital few” sources of process deficiencies

40. Total Survey Error Evaluation
Addresses several dimensions of total survey quality.
Essential for optimizing resource allocations to reduce the errors.
In experimentation, needed to compare the quality of alternative methods.
Provides valuable information on data quality for gauging uncertainty in estimates, interpreting the analysis results, and building confidence and credibility in the data.

41. Primary Methods
Nonresponse bias studies (required by OMB for some surveys)
Evaluates differences between respondents and nonrespondents for key survey items
Frame data or prior waves provides data on nonrespondents
Model-based approaches for nonignorable nonresponse bias
Measurement bias studies
Record check studies
Reconciled reinterviews
Internal and external consistency checks
Test-retest reinterview approaches
Embedded repeated measures analysis (e.g. structural equation modeling, latent class analysis)

42. Primary Methods (continued)
Other methods
Frame undercoverage evaluations
Editing error (pre- and post-editing comparisons)
Cognitive methods for detecting comprehension errors, recall problems, data sensitivity, etc.
Subject matter expert reviews of concepts vs. question meaning
Process data summaries
Response rate analysis
Data entry error rates
Edit failure rates
Missing data rates
Post-survey adjustment factors

43. Major Take-Home Points
Survey quality is multi-dimensional including both data user and producer dimensions.
Accuracy is maximized subject to cost and timeliness constraints
Survey design optimization begins with the initial design and extends throughout implementation and post-survey processing.
ATDI combines CQI, responsive design and Six Sigma strategies to provide a comprehensive approach for real-time reduction of total survey error and costs.
Survey evaluation is an essential component of the total survey error framework.

44. It is not enough to do your best; you must know what to do, and then do your best.
– W. Edwards Deming