1. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Fieldwork effort, response rate and the distribution of survey outcomes Joel Williams, TNS BMRB Patrick Sturgis & Ian Brunton-Smith, University of Southampton ESRC Research Methods Festival, July 8 th 2014

2. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 The value of probability samples 2 Bedrock of social research Uncertainty can be quantified via confidence intervals Confidence intervals assume all sampled cases have same response propensity If this assumption does not hold, a high response rate will limit NR bias

3. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Maximising the response rate 3 Need to tackle multiple reasons for not responding Make repeated visits/calls on different days/times of day ‘Reminders’ for self-completion surveys Respondent engagement strategies Incentives ‘Late’ respondents can be costly to recruit so they need to add value…

4. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 The increasing difficulty of fieldwork (BCS 2011) 4 (% of sampled households)

5. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 A formula for non response bias 5 R xp S x S p p NR bias (x) =

6. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 A general metric for the risk of non-response bias 6 R xp S x S p p CV(p) =

7. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Estimating response propensity variation (S p ) 7 Generally estimated using logistic regression models May estimate using calibration weights if sample frame is uninformative Low variation may be real or due to poor model CV(p*): partial estimate of p Single variable versions: e.g. age group

8. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 BCS: Response rate progression (age groups) 8 Response rate (%) BCS CV(p*) =.53

9. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 BCS: Response rate progression (age groups) 9 Response rate (%) BCS CV(p*) =.13

10. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 BCS v Taking Part (age groups) 10 Response rate (%) BCS CV(p*) =.13 TP CV(p*) =.11

11. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Uses simple paradata (‘visits’ per case) to assess impact of effort on a continuum from ‘barely any’ to ‘a great deal’ Assesses how impact varies by topic and question/response structure [early stages] Our analysis 11 Four major FTF surveys Representative set of variables from each (total = 393) 1.2.3.4.

12. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Survey details 12 British Crime Survey Taking PartBritish Election Study Skills for Life PopulationE+W 16+E 16+GB 18+E 16-65 TimingIssued in 2011 Interviewed in 2011 Spring 20102010-11 Sample size46,78510,9941,8117,230 Response rate76%59%54%~57% Incentives?Stamps (U) Stamps (U) +£5 (C) £5-10 (C)£10 (C)

13. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Model structure (depvar: x v –x) 13 Question x S.E 1 visit Up to 2 visits Up to 3 visits Up to 5 visits Response category x=1 S.E 1 visit Up to 2 visits Up to 3 visits Up to 5 visits Response category x=2

14. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Early findings 14 Coding of question and response structure complete Some evidence that questions about beliefs are most sensitive to fieldwork effort …particularly if a multi-point scale is used Topic level coefficients constrained by limited set of topics in 4 surveys Impact of fieldwork effort (# of visits) not particularly substantial even with empty model…

15. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Estimates of bias after 1 visit to each address (n=393) 15

16. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Estimates of bias after 1 visit to each address (n=393) 16 Mean ‘bias’ = 1.41% w/controls = 1.14%

17. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Estimates of bias after 2 visits to each address (n=393) 17

18. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Estimates of bias after 2 visits to each address (n=393) 18 Mean ‘bias’ = 0.97% w/controls = 0.70%

19. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Estimates of bias after 3 visits to each address (n=393) 19

20. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Estimates of bias after 3 visits to each address (n=393) 20 Mean ‘bias’ = 0.71% w/controls = 0.44%

21. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Estimates of bias after 5 visits to each address (n=393) 21

22. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Estimates of bias after 5 visits to each address (n=393) 22 Mean ‘bias’ = 0.39% w/controls = 0.12%

23. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Points for discussion I 23 Efforts to maximise response are costly Generally lead only to marginal changes in survey estimates Modelling method can help quantify by question/topic typologies Important to remember that bias may be present even if maximised response Potential for larger bias scores for sub- groups “ “

24. 3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Points for discussion II 24 Is this evidence in favour of using cheaper data collection modes with lower response rates? Not straightforward to generalise these findings to other modes But possible that non-response bias is not as great as feared Is the non-statistical value of a high response rate enough to warrant the expenditure? “ “