The European Statistical Training Programme (ESTP)

Chapter 2: The nonresponse problem Handbook: chapter 1 What is nonresponse? Causes of nonresponse Response rates Some international trends

Nonresponse Nonresponse Elements that are selected in the sample, and that are also eligible for the survey, do not provide the required information. Two types of nonresponse: Unit nonresponse: a selected element does not provide any information at all. Item nonresponse: a selected element does answer some questions, but not all of them. Why is nonresponse a problem? Smaller sample size. Nonresponse bias due to selective nonresponse.

Selective nonresponse Example: Dutch Housing Demand Survey 1981 Initial sample size: 82,849 persons. Response: 71.2%. Follow-up survey among nonrespondents: Percentage of movers in response: 29.7%. Percentage of movers in nonresponse: 12.8%. Do you intend to move within 2 years Response Nonresponse Yes 17,515 3,056 No 41,457 20,821 Total 58,972 23,877

Selective nonresponse in Dutch Surveys Victimization Survey People who are afraid to be home at night are less inclined to participate in the survey. Mobility Survey The more mobile people were harder to contact, and thus under-represented in the survey, Labour Force Survey Refusal rates are higher among the unemployed. Election Survey Participation rates are higher among voters. Voters are over-represented.

Selective nonresponse Simulation Estimation of the mean income in Samplonia. Simulate 1,000 samples of size 50. Case 1: No nonresponse. Case 2: Nonresponse increases with income. Conclusion: estimates are systematically too low in case of nonresponse. Confidence intervals are wrong. No nonresponse Selective nonresponse

Causes of nonresponse (CAPI) Contact? No Eligible? Yes Nonresponse Non-contact Over-coverage No Eligible? Over-coverage Yes No Participates? Nonresponse Refusal Yes No Able? Nonresponse Not-able Yes Response

Causes of nonresponse Non-contact Nobody home, phone not answered. Make more contact attempts (up to 6). Refusal Can be permanent or temporary. Role of interviewers important. Use of incentives? Not-able Physical or mental problems. Language problems. Unprocessed Interviewers are unable to handle their workload.

Response rates Definition Lynn et al. (2002) Proportion of eligible elements in the sample for which a questionnaire has been completed: Notation nE = Number of eligible elements in the sample nR = Number of eligible respondents nE = nR + nNC + nRF + nNA Initial sample size n = nE + nOC. So nE = n – nOC. Problem: over-coverage unknown for non-contacts.

Response rates Estimating the response rate Assumption 1: all non-contacts are eligible: Assumption 2: the proportion of eligibles among non-contacts is the same as the proportion of eligible among contacts:

Response rates - Example Fieldwork results for the Dutch Integrated Survey on Household Living Conditions in 1998: If all non-contacts are eligible, the response rate equals If the proportion of eligibles among contacts and non-contacts is the same, the response rate is Outcome Frequency Over-coverage 129 Response 24,008 Non-contact 2,093 Refusal 8,918 Not-able 1,151 Other nonresponse 3,132 Total 39,431

Response rates Response percentages in some Dutch surveys

Response rates Some other response rate problems Self-administered surveys (no interviewers): It is unclear who is completing the questionnaire. Is the person eligible? The cause of nonresponse is unclear. Unequal probability sampling: How to compute the response rate if the sample is selected with unequal probabilities? Unweighted response rate as a quality indicator of the performance of interviewers. Weighted response rate as a quality indicator of estimates. Household survey: How to define response/nonresponse of a household?

International trends De Heer (1999) compared Labor Force Surveys for official statistical offices Large difference in response rates and trends Rates vary from around 60 (NL) to around 90 (USA) percent. UK, USA stable pattern. Finland, Sweden downward trend. Belgium seems to be recovering. The Netherlands world champion nonresponse.

International trends Also a large differences in the composition of nonresponse in non-contacts and refusals UK: non-contact decreasing whereas in most countries it is increasing. Refusals mostly differ in rate, not so much in trend.

International trends Possible explanations towards differences: Difference in general design   Belgium NL UK Finland Sweden USA Data collection mode face-to-face CAPI CATI Observational unit household person Substitution yes no Call scheduling Refusal conversion Mandatory no*

International trends Stoop (2005) analyses data from the first round of the European Social Survey, in 2002/2003. In the European Social Survey: One sampling design Response target: 70%; non-contact target: 3% Central fieldwork specifications The results are very different: The response rate varies from 33% (Switzerland) to 80% (Greece) The non-contact rates varies from 2% (Poland) to 15% (France) The refusal rate varies from 14% (Hungary) to 55% (Switzerland) The Netherlands is doing well, with a response rate of 68%, non-contact of 3% and refusal rate of 24%.

International trends Results of the first round of the ESS

International trends The second round from the ESS in 2004/2005 showed and improvement of low response countries with respect to the first round (Stoop and Billiet, 2007): Response rate in the European Social Survey in the first two rounds

International trends Response rate and refusal/ (refusal+noncontact) in the European Social Survey Round 2 Relationship between the response rate and the composition of the nonresponse. Refusal is more important than non-contact in most countries No relationship between response rate and response composition