1. QMSS, Lugano, 13-8-2004 Lynn Control of Sampling Error Peter Lynn Institute for Social and Economic Research, University of Essex, UK

2. QMSS, Lugano, 13-8-2004 Lynn Presentation Structure Objectives of Survey Design Survey Error Framework Coverage Error: Control Sampling Error: Control Design Effects: a key tool Examples from European Social Survey

3. QMSS, Lugano, 13-8-2004 Lynn Survey Design: Objectives Appropriate accuracy (cf. budget): For what estimates? What is appropriate? Estimates: Typically, descriptives for sub- domains and total, and comparisons between sub- domains (inc. models) Appropriate: high enough, cost effective

4. QMSS, Lugano, 13-8-2004 Lynn Survey Errors ProcessError source Population ↓Coverage Sampling Frame ↓Sampling Sample ↓Non-response Responding sample ↓Measurement Data

5. QMSS, Lugano, 13-8-2004 Lynn Coverage Error Ideal aim: Complete coverage Implies zero coverage error Practical aim: Very high coverage Similar coverage for each sub- domain Hopefully similar (and small) coverage bias

6. QMSS, Lugano, 13-8-2004 Lynn Example: ESS Target population in each nation (domain): all persons 15 years or older resident in private households within the borders of the nation, regardless of nationality, citizenship, language or legal status. Under-coverage in practice: If language is barrier to interviewing; If some addresses/households excluded (e.g. If electoral registers used as frame of addresses); If illegal residents excluded (e.g. If population register used as frame).

7. QMSS, Lugano, 13-8-2004 Lynn Sampling Error Aim: Maximum precision of estimates of between-domain differences Implication: same precision for each domain (for a given estimate)

8. QMSS, Lugano, 13-8-2004 Lynn Sampling Error Affected by: Sample size (n) Population variance (S 2 ) Sample clustering Sample stratification Variable sampling fractions For SRS: Var(y)=(S 2 /n)(1-(n/N)) Effect of other 3 design features can be summarised by design effect

9. QMSS, Lugano, 13-8-2004 Lynn Design Effect: A useful tool where ;

10. QMSS, Lugano, 13-8-2004 Lynn Example: Use of Design Effects on ESS and where and So, challenge was to predict and for each nation

11. QMSS, Lugano, 13-8-2004 Lynn Sample Design Process ESS sampling panel set up Each nation allocated to a panel member for bilateral liaison Panel met 3 times and communicated regularly, to ensure consistency of approach Each design had to be approved by whole panel

12. QMSS, Lugano, 13-8-2004 Lynn Predicting Unclustered designs (5) trivial: For other designs, necessary to decide upon n and number of clusters and to predict eligibility rate and response rate Under-estimated if RR under-estimated. E.g. Greece Over-estimated if RR over-estimated. E.g. Italy, Spain, Czech Rep

13. QMSS, Lugano, 13-8-2004 Lynn Predicting Many countries assumed default value of 0.02 A few countries assumed values between 0.03 and 0.05, either based on estimates from earlier surveys or because clustering units were particularly small Post-fieldwork estimates showed large range across variables

14. QMSS, Lugano, 13-8-2004 Lynn Predicting Planned variation in sampling fractions over strata (only NR is uncertain); Additionally, 3 forms of uncertainty encountered: Re. Distribution of # persons aged 15+ per household Re. Relationship between a proxy size measure used at PSU level and actual size measure of relevance Re. Relationship between a proxy size measure at hhd/address level and actual number of persons aged 15+

15. QMSS, Lugano, 13-8-2004 Lynn Predictions of Zero variation (SE, DK, FI, HU, SI, BE) Known SFs only (DE, NO, PL) From recent surveys (CH, NL, UK) PSU size measures & SFs (IL) Hhold size dist. (AT, CZ, ES, GR) Hhold size measure & SF (LU)

16. QMSS, Lugano, 13-8-2004 Lynn Overall Deff Predictions Good for many countries Some under-estimates compensated by larger m (GR, SI) Some modest under-estimates, e.g. CZ AT Two severe under-estimates: IL NO Poor RR prediction also affected m, e.g. IT, CZ

17. QMSS, Lugano, 13-8-2004 Lynn Quality Improvement Round 1 procedures improved quality in several countries and cross-nationally Round 1 estimates will influence round 2 predictions (etc.) Guidelines to be amended in light of round 1 experience (e.g. m, dual designs, default roh)

18. QMSS, Lugano, 13-8-2004 Lynn Final Comments None of these ideas are unique to cross- national surveys: apply to any comparative survey (= all surveys) We are forced explicitly to consider domain precision aims when fieldwork is organised separately for each domain But we should always do this: sample design should be appropriate for analysis aims

19. QMSS, Lugano, 13-8-2004 Lynn Control of Sampling Error Peter Lynn Institute for Social and Economic Research, University of Essex, UK