The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010

Early thinkers Hansen and colleagues, U.S. Bureau of the Census Deming, U.S. Bureau of the Census and consultant Kish, University of Michigan Dalenius, Statistics Sweden and Stockholm University

What were they thinking about? Nonsampling errors Balancing errors and costs Design criteria The limitations of sampling theory Standards Similarities between survey implementation and the assembly line

4 Deming (1944) On Errors in Surveys American Sociological Review! First listing of sources of problems, beyond sampling, facing surveys The 13 factors

Demings 13 factors The 13 factors that affect the usefulness of a survey -To point out the need for directing effort toward all of them in the planning process with a view to usefulness and funds available -To point out the futility of concentrating on only one or two of them -To point out the need for theories of bias and variability that correlate accumulated experience

6

Their difficult position They had to promote Neymans theory But his theory basically assumes very small nonsampling errors They were in a first-things-first situation They promoted vigorous controls hopefully leading to small biases They discussed what a Bayesian approach might offer

Lines of thought I There is as yet no universally accepted survey design formula that provides a solution to the design problem (Dalenius 1967) Thats why textbooks devote little space to design Important to control specific error sources

Lines of thought II The U.S. Bureau of the Census is a statistical factory. The main product is statistical tables (Deming and Geoffrey 1941) Concentration on QC of error sources, evaluation, and survey models Disentangling the design process

Lines of thought III Hansen-Hurwitz-Pritzker 1967 Take all error sources into account Minimize all biases and select a minimum-variance scheme so that Var becomes an approximation of (a decent) MSE The zero defects movement that later became Six Sigma Dalenius 1969 Total survey design

The design process Criterion of effectiveness: Minimum MSE per unit of cost while meeting other requirements such as timeliness of results (not just minimum variance) Good survey design calls for reasonably effective control of the accuracy through appropriate specifications for survey procedures and adequate control of the operations, i.e. proper design of the total system

Mean squared error (MSE) MSE=Var+B 2 +(Relevance error) 2 +Interaction MSE Z (y)=E(y-Y) 2 +(Y-X) 2 +(X-Z) 2 +2(Y-X)(X-Z) Z is the ideal goal, X is the defined goal, and y is the actual result Hansen-Hurwitz-Pritzker call them requirements (Z), specifications (X) and operations (y)

Design issues X-Z is crucial in the design situation Do we want an approximate solution to the right problem or an exact solution to the wrong problem?

The design approach (Dalenius and Hansen et al) Specify the ideal goal Z Analyze the survey situation (financial, methodological and information resources) Construct a small number of alternative designs Evaluate the alternatives by reference to associated MSE and costs

The design approach (contd) Make a decision Use one of the alternatives Use a modification of one of them Do not conduct the survey Develop the administrative design Feasibility The signal system A self-contained design document (tree) Plan B

What does this tell us? All error sources should be taken into account There is very little process talk such as the need for CQI However, the common situation was: no process view, no controls Concern about costs and effectiveness of all these controls The user is a somewhat distant player

The user The user was hiding under terms such as subject matter problem, study purpose or the four key functions of a statistical system (reporting, analytic, consulting, research) Tukey 1949 But there were federal statistics users conferences in the U.S. from Dalenius provides more than 200 references on users in a 1967 ISI paper

Who identifies the requirements? Usually seen as one fictive person An official An administrator A statistician acting as a subject-matter specialist Requirements define the population, types of measurement, time dimensions and statistics needed

The designers role vis-à-vis the requirements To critique the suggested requirements To suggest QC procedures, construct dummy tables to check the decision- making and perform sensitivity analysis To act as the devils advocate and discuss specific result interpretations with the user

Kishs contributions The neo-Bayesian view Appreciates the literature by Schlaifer, Ericson, Edwards, Lindman and Savage on Bayesian methods in survey sampling and psychometrics For instance, judgment estimates of measurement biases may be combined with sampling variances to construct more realistic estimates of the total survey error

More from Kish Experiments and sample surveys might not be sufficient. Other investigations collecting data with considerable care and control but without randomization and probability sampling might be necessary.

Kishs view on design Multipurpose is great from an economical point of view. If one principal statistic can be identified that alone can decide the design If a small number of principal statistics can be identified a reasonable design compromise is possible If statistics are too disparate a joint design might not be possible

Kish on economic design Requires joint consideration of sampling and nonsampling errors Sometimes demands prior or pilot studies of sufficient size Requires information about unit variance Emphasizes a small total error Appreciates the fact that a reduction of one source might increase total error

Examples of decisions Frame needs updating? Reference period? Acceptable respondent rules? Number of callbacks? Allocation of callbacks? How much and what kind of editing? Mix of modes?

Kish summed up Get a good balance between different error sources We need to know how error structures behave under different design alternatives Relevant information should be recorded during implementation (paradata) Many practical constraints The multipurpose nature calls for a compromise

Hansen, Dalenius and colleagues on standards General standards Measurable survey plan, self-contained plan, replications should generate similar results, cost- efficient, sufficiently simple plan Standards for error control Relevance control, control of accuracy (should be dominated by variance terms) Minimum performance standards Check that standards yield the results expected

Hansen, Dalenius and colleagues summed up One should be guided by common sense, experience and theory Design and execution is a management and systems analysis problem A survey is an economic production process Survey goals must be identified Standards must be dynamic End the practice that sampling error is viewed as the total error They predicted the CASM movement

More from Hansen, Dalenius and colleagues The examination of design alternatives is costly and time-consuming There is a risk of overcontrol and inadequate control. Consequences of large errors must guide any relaxation but they dont talk about CQI One might have to compromise relevance to get controllable measurements or abstain from the survey Keep bias near zero and allow variance at expected levels

What happened? Still no design formula General design principles exist for some areas Still a concentration on some error sources more than others CASM happened We got standards The TSE paradigm accepted but has some promotional problems Many of the early thoughts were just that, very little practice, but still useful