How representative are the samples? Sabine Häder and Peter Lynn European Social Survey (ESS) – Launch Conference Brussels, 25/26 November 2003

Häder/Lynn Representativeness of Samples 2 Sampling: Important part of a survey! Sampling: Important part of a survey! Precondition for comparability of countries Precondition for comparability of countries What do you need for achieving high-quality comparable samples? What do you need for achieving high-quality comparable samples? In an ideal world you have In an ideal world you have complete frames, from which you can select randomly (with known probabilities) many individuals who all like to respond.

Häder/Lynn Representativeness of Samples Sounds easy Sounds easy..... But: Could be found only in a minority of But: Could be found only in a minority of ESS countries ESS countries In reality: In reality: Considerable variation between countries Considerable variation between countries in constraints (e.g. availability of sampling in constraints (e.g. availability of sampling frames) and survey practice frames) and survey practice

Häder/Lynn Representativeness of Samples 4 Task: Minimising the impact of this variation Requirements for sample designs: The same population (residents 15 years and older) Same precision of results (Effective sample size of 1.500) Highest achievable response rates Recording sample design characteristics (e.g. inclusion probabilities)

Häder/Lynn Representativeness of Samples 5 Organisation of the work package Organisation of the work package Expert Panel Sabine Häder (Centre for Survey Research and Methodology, Germany) (Centre for Survey Research and Methodology, Germany) Siegfried Gabler (Centre for Survey Research and Methodology, Germany) (Centre for Survey Research and Methodology, Germany) Seppo Laaksonen (Statistics Finland) (Statistics Finland) Peter Lynn (University of Essex, U.K.) (University of Essex, U.K.)

Häder/Lynn Representativeness of Samples 6 Guidance: Guidance: Identifying suitable sampling frames (e.g. Austria) Developing new sample designs (e.g.Greece) Maximising response rates (e.g. Switzerland) Implementing concept of design effects (e.g. France) Finally: ‚Signing off‘ the sample designs Result: Best probability samples in all countries —› As close to the ideal world as possible —› As close to the ideal world as possible

Häder/Lynn Representativeness of Samples 7 Variation in procedures to achieve equivalence of outcomes: 2 examples Advances in survey design and practice: 2 examples Use of design weights in data analysis Some lessons for the future

Häder/Lynn Representativeness of Samples 8 Variation in procedures to achieve equivalence of outcomes: Variation in procedures to achieve equivalence of outcomes: Example 1: Age range 15+ (no upper cut-off) Example 1: Age range 15+ (no upper cut-off) Address-based samples (e.g.UK, Greece) Population register samples (e.g. Finland, Denmark) Electoral register sample (Italy)

Häder/Lynn Representativeness of Samples 9 Variation in procedures to achieve equivalence of outcomes: Variation in procedures to achieve equivalence of outcomes: Example 2: Effective sample size 1,500+ Example 2: Effective sample size 1,500+ Equal probability unclustered samples (e.g. Finland, Denmark) Variable probabilities due to address-based sampling (e.g. UK, Greece, Portugal) Variable probabilities due to local aims (e.g. Luxembourg) Clustered samples (most, but to different extents)

Häder/Lynn Representativeness of Samples 10 Example 2: Effective sample size, continued Example 2: Effective sample size, continued Design effects ClustAddOtherTot Equiv. precision Slovenia ,100 Netherlands ,785 Germany ,280 Ireland ,400 Israel ,385 Sweden ,500

Häder/Lynn Representativeness of Samples 11 Advances in survey design and practice: Advances in survey design and practice: Example 1: National probability sample of households in Greece Example 1: National probability sample of households in Greece Probability samples previously used only by NSI No list of persons or addresses available to other organisations Quota sampling common Exclusion of rural areas and most islands common

Häder/Lynn Representativeness of Samples 12 Advances in survey design and practice: Advances in survey design and practice: Example 1: Greece (continued) Example 1: Greece (continued) Area-based probability sampling approach Census EAs as PSUs Field listing of addresses In-office selection of addresses from the list Carried out successfully - and good response rate!

Häder/Lynn Representativeness of Samples 13 Advances in survey design and practice: Advances in survey design and practice: Example 2: Inclusion of year-olds in electoral register based sample in Italy Example 2: Inclusion of year-olds in electoral register based sample in Italy Equal probability sampleof electors (18+) Interviewer lists all residents 15+ (r) at address of selected elector and randomly selects one Interviewer also asks number of electors at address (e) Selection probability is Ke/r: all persons 15+ have known selection probability

Häder/Lynn Representativeness of Samples 14 Design weights Design weights In most countries, selection probabilities were unequal We ensured that the probabilities are known We have converted these into a „design weight“ that should be used in all analyses e.g. address-based samples (Greece, UK, Portugal...): probability is inversely proportional to number of persons in household

Häder/Lynn Representativeness of Samples 15 Design weights: Example Design weights: ExampleCZESGRFISE V1, unw V1, wtd V2, unw V2, wtd V1 = Single-person household (%) V2 = 3+ hrs/week watching TV (%)

Häder/Lynn Representativeness of Samples 16 Lessons for the future Lessons for the future Central co-ordination of sample design and close liaison with national teams necessary Necessary not only to „agree“ the design, but also to monitor and discuss implementation Central team must be seen as „helping,“ not „controlling“ Specification of effective sample size was successful Methodological research into sample design