1. Representativity Indicators for Survey Quality Programme: Cooperation Theme: Socio-economic sciences and Humanities Activity: Socio-economic and scientific indicators Area: Provision for underlying official statistics Costs:€ 827,000 Length:28 months (from 1-3-2008 to 30-06-2010)

2. Is the response rate a good quality indicator?

3. Is more response always better?  The accuracy of survey estimates is determined by the precision (variance) and the bias of estimators.  A higher response rate is only better if the bias is smaller. This is not always the case! Integrated Survey on Household Living Conditions 1998 The composition of the sample deteriorated in month 2. ResponseResponse after 1 month Response after 2 months Complete sample % social allowance10.5 %10.4 %12.1 % % non-natives12.9 %12.5 %15.0 %

4. Is more response always better? Other examples  The composition of the response deteriorates after a call- back survey among non-respondents.  The composition of the response may deteriorate if incentives are used to increase response.

5. There is a need for better quality indicators  Such quality indicators should reflect how well the composition of the survey response reflects the population (or complete sample).  The RISQ project intends to develop and test such indicators.  These indicators will be called R-indicators (short for: Representativity Indicators).  Such indicators should be comparable over space (countries, regions) and time.  They should be useful in the data collection phase and the analysis phase of the survey.  They should also be useful for processing register data.  They should not depend on the survey variables.

6. An example of an R-indicator  The bias (due to non-response) of the response mean is equal to  Y is the survey variable and ρ is the response probability.  Cor(Y, ρ) is the correlation between Y and ρ.  S(Y) and S(ρ) are the standard deviations of Y and ρ.  The bias vanishes if all response probabilities are equal. Then

7. An example of an R-indicator (continued)  Definition of an indicator:  M(ρ) = 1: All response probabilities are equal. The response is representative.  M(ρ) = 0: Maximum possible deviation from representativity.  Computation of M(ρ): Required: auxiliary variables. Fit logit (or other) model for response probabilities. Estimate response probabilities.

8. Research issues  Development of R-indicators based on variance of response probabilities.  Development of R-indicators based on g-weights.  Analysis of dependency on sample size.  Analysis of dependency on auxiliary variables.  Estimation of response probabilities if only population distribution of auxiliary variables is available.  Development of partial R-indicators to identify groups at risk.  Use of paradata (fieldwork data) in response probability models.

9. An example of an R-indicator (continued)  Example: Dutch Labour Force Survey.  Sample of non-respondents re-approached (call-back) with complete questionnaire.  Sample of non-respondents re-approached with small questionnaire (basic question approach).  The composition of the response improves more after the call-back approach. ResponseRateM(ρ) LFS62 %0.79 LFS + Basic question approach76 %0.77 LFS + Call-back approach77 %0.85

10. The RISQ Project Objectives of the project:  Develop R-indicators based on variance of response probabilities and g-weights.  Determine statistical properties of indicators.  Develop tools to compute indicators.  Explore use for monitoring data collection.  Explore use for controlling data collection.  Explore use in correction (selection of weighting variables).  Explore use in analysis (space and time).  Test on real data sets (surveys and registers).  Social and economic surveys.

11. The RISQ Project Project partners:  Statistics Netherlands (Netherlands, co-ordinator)  University of Southampton (UK)  Statistics Norway (Norway)  University of Leuven (Belgium)  Statistical Office of the Republic of Slovenia (Slovenia) Website  www.r-indicator.eu