Quality strategies in cross- national surveys The case of the European Social Survey Ineke Stoop

European Conference on Quality in Official Statistics 8-11 July 2008 Quality and comparability Harmonisation strategies and cross-national surveys Optimal comparability European Social Survey Nonresponse

European Conference on Quality in Official Statistics 8-11 July 2008 Harmonisation strategies (Körner and Meyer, 2005) Harmonised concept Harmonised concept Harmonised concept Measurement procedure Measurement procedure Measurement procedure National survey/ Specific concept National survey/ Specific concept National survey/ Specific concept Input harmonisation Ex-ante output harmonisation Ex-post output harmonisation

European Conference on Quality in Official Statistics 8-11 July 2008 Type of harmonisation Ex post output harmonisation Use existing sources Ex ante output harmonisation Set up new survey (or develop new instrument) Output harmonisation Best national quality Or, national survey tradition But, what about optimal comparability Input harmonisation Design new survey Optimal comparability Or, are identical methods and instruments really equivalent in different countries? But, what about optimal quality

European Conference on Quality in Official Statistics 8-11 July 2008 Strategies in comparative survey Quality Optimal national approach Comparability? Poor performance accepted Some restrictions and standards Consistency Identical approach Lowest standards Possible? Pull down level Highest standards Possible? Pull up level High standards and optimal national approach

European Conference on Quality in Official Statistics 8-11 July 2008 Mixed approach Focus on those key aspects that may hamper comparability Data collection mode Sampling approach Response rates Question formulation

European Conference on Quality in Official Statistics 8-11 July 2008 European Social Survey Attitudes, values and beliefs Bi-annual: 2002, 2004, 2006, 2008 Fixed core, rotating modules Effective size 1500 or more Central co-ordination National implementation 30 National Coordinators responsible for fieldwork Face-to-face

European Conference on Quality in Official Statistics 8-11 July 2008 Aims European Social Survey To monitor and interpret public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions Provide data on beliefs, attitudes and values for scientific and policy making purposes Measure attitude change in a changing Europe To advance and consolidate improved methods of cross- national survey measurement in Europe and beyond Collect data according to highest standards Generate methodological research Develop and disseminate new best practices Develop and improve social survey research infrastructure in Europe

European Conference on Quality in Official Statistics 8-11 July 2008 EVERYTHING DOCUMENTED EVERYTHING AVAILABLE

European Conference on Quality in Official Statistics 8-11 July 2008 Nonresponse: consistency Strict probability sampling No substitution Fieldwork monitoring Guidelines response enhancement Incentives Brochure Guidelines interviewer training Target noncontact rate: 3% Minimum number of calls Timing of calls (evening, weekend) Target response rate: 70% Refusal conversion suggested Contact form and call records Standard response rate calculation

European Conference on Quality in Official Statistics 8-11 July 2008 Nonresponse: divergence Population Residential population Minority languages Sampling frame Addresses, households Individuals At home pattern Female employment When does the evening start? Fieldwork organisation Field director’s meeting Interviewers Experience F2F Random sampling Training and briefing Remuneration Response, hours, staff Mode of contacting Use of advance letters Incentives Contact forms

European Conference on Quality in Official Statistics 8-11 July 2008 Response rates ESS (deviation 70%)

European Conference on Quality in Official Statistics 8-11 July 2008 Why do response rates differ? Survey climate Used to surveys? Popularity topic Europe? Country size, urbanicity and contactability Survey modes Predominantly telephone? Fieldwork efforts

European Conference on Quality in Official Statistics 8-11 July 2008 UP TILL NOW NO UNIVOCAL PREDICTOR OF RESPONSE RATES ACROSS COUNTRIES

European Conference on Quality in Official Statistics 8-11 July 2008 % noncontact and refusal (R1, R2, R3) % noncontact % refusal

European Conference on Quality in Official Statistics 8-11 July 2008 Age, gender, education (Vehovar & Zupanic) Age: underrepresented 35+ (Austria) 55+ (Belgium and Luxemburg) (Spain, Ireland, Netherlands, UK) Education: underrepresented Middle (Austria, France, Iceland, Luxembourg, the Netherlands, Norway, Sweden and the United Kingdom) Less (Switzerland, Czech Republic, Germany, Estonia, Hungary, Norway, Slovenia, Slovakia and Ukraine)

European Conference on Quality in Official Statistics 8-11 July 2008 Results multivariate analysis: reluctance (Beullens, Billiet and Loosveldt) Switzerland Members of larger families Germany Female, aged, city dweller Internet, unemployment history Less political participation Estonia Female, not in village Employed Slovakia Older, average education, church life Never job, less safe, comfortable family income Netherlands Female, average education, more TV, internet Paid job, healthy Immigrants threat, trust political institutions Civil obedience, political participation Socially isolated, dissatisfied

European Conference on Quality in Official Statistics 8-11 July 2008 Nonresponse bias in cross-national studies How to measure bias? Different type of auxiliary information available Sample frame/register information? Reluctant respondents? Neighbourhood information? Different size of bias? Different type of bias? Different bias over time? ESS-experiments in several countries Mixed results

European Conference on Quality in Official Statistics 8-11 July 2008 Is output harmonisation better? Maybe, but in that case do you know about Response rates (standard calculation) Intensity and effect of field efforts Nonresponse composition Underrepresented groups Nonresponse bias Black box: Differences between countries due to methodology or substance?

European Conference on Quality in Official Statistics 8-11 July 2008 Are comparative surveys possible? If not, national surveys are not possible either Subgroups Socio-demographic Survey interest Topic relevance Regions National languages Interviewers There is so much we don’t know, and now we know so much more

European Conference on Quality in Official Statistics 8-11 July 2008 Thank you for your attention