1. Obtaining information on non-responders: a development of the basic question approach for surveys of individuals
Patten Smith (Ipsos MORI)
Richard Harry (Sport Wales)
Colin Gardiner (Ipsos MORI)
John D’Souza (Ipsos MORI)

2. Overview of basic question approach Sport Wales Active Adults Survey
Outline
Overview of basic question approach
Sport Wales Active Adults Survey
Adaptation to Active Adults
Results and indicative estimates of non-repose bias
Conclusions

3. Basic Question Approach
Basic question: ask small number of key survey questions from survey non-respondents (eg see Bethlehem, Cobben and Schouten, 2011; Lynn, 2003)
Do this for two reasons:
Examine whether non-respondents differ from respondents; if so, provides evidence of non-response bias
Improved non-response weights
Approach likely to be especially valuable when risk that survey topic will affect propensity to respond
With standard basic question approach, two constraints:
Basic Questions (BQs) restricted to household level characteristics (BQs often asked before individual selection in household)
BQs restricted to survey questionnaire items – cannot deal with key survey variables that are constructed from multiple questions
Remainder of presentation: application of BQ approach to a survey of sports participation

4. Active Adults Survey
Household survey measuring levels of sports participation, club membership and volunteering plus lifestyle, health and other sport related behaviours
Adults (aged 15 years and above) living in Wales
Sponsored by Sport Wales – organisation responsible for increasing participation and improving performance in sport in Wales
Random probability sample of addresses stratified by Local Authority and key demographics
Two-stage address sample design:
Sample frame: Postcode Address File (PAF)
PSUs formed from Census Output Areas
Stratified (disproportionately) by Local Authority (LA) and by the percentage of managers / professionals among household reference persons and by population density (proportionately within LA)
Single adult randomly selected at each address through Kish grid

5. Active Adults Survey
Fieldwork between January 2012 and January 2013, spread over one year to enable full coverage of seasonal activities
Questionnaire administered using Computer Assisted Personal Interviewing (CAPI) and a total of 13,145 interviews achieved
Of 27,704 sampled addresses, nine per cent classified as ineligible as they did not contain an occupied private household
25,108 addresses classified as in-scope, giving a response rate of 52%
Weights:
Inverse probability (design) weights
Modelled non-response weights (based on geodemographic variables)
Calibration to population distributions for age, gender and LA (including interlocking) and to ensure even quarterly distributions

6. Active Adults Survey response table

7. Adapting the basic question approach to the Active Adults Survey
Previously mentioned restrictions cause problems for Active Adults Survey:
Key survey variables relate to individual characteristics and not household ones
Single most important sports participation variable constructed from series of questions
Revised basic question approach
Ask new household-level question that is not in questionnaire:
question must correlate highly with key survey questions
must be asked of respondents as well as non-respondents (as new question not in main questionnaire)
Experimental work reported here
Focuses on four key survey variables (next slide); BQs need to correlate with these
Two basic questions attempted with first person contacted at all contacted households:
BQ1: In the last two weeks have you or any other adult in your household taken part in sports or other physical exercise as a member of a sports club or leisure or fitness centre?
BQ2: Apart from this, in the last two weeks, have you or any other adult in your household done any exercise mainly for the purpose of getting or keeping fit?

8. Key survey variables

9. Response rates for basic questions
*Contact with household but not with selected adult

10. Using BQs to estimate non-response bias
Non-zero values of non-response bias will only be observed if main survey respondents and non-respondents differ in their answers to basic questions
Answers to basic questions allow us to estimate likely levels of non-response bias if:
answers to basic questions correlate with answers to key survey questions
non-respondents answering basic questions are representative of those not answering them;
Today, reporting our preliminary analyses in which we:
examine these conditions and
use a weighting approach to estimate non-response bias

11. Do main survey respondents and non-respondents differ in answers to basic questions?

12. Do answers to basic questions correlate with those to key survey variables?

13. Do answers to basic questions correlate with those to key survey questions?

14. Do basic question respondents represent main survey non-respondents?
Compared main survey respondents who did and did not answer basic questions on selected geodemographic variables
Significant differences, but magnitude relatively small – not enough substantially to undermine our analyses
Logistic regression of BQ response on geodemographic variables gives poor fit: Nagelkirke pseudo R2 = .018

15. Do basic question respondents represent main survey non-respondents?

16. Estimating non-response bias
General approach taken:
Use basic questions to construct weights
Compare key survey estimates:
With design weights only
With design weights plus basic question weights
With design weights and standard NR+calibration weights
With design weights and standard NR+calibration weights supplemented with basic question weights
Problem: no BQ data for 14% of main survey respondents
Dealt with this by imputing BQ answers from main questionnaire variables (Chained Equations method)

17. Estimating non-response bias
BQ weights: four cells formed by cross classifying the two BQs:

18. Estimating non-response bias
Two basic question weights:
BQ weight A (conservative assumptions): Weight respondents to total number of cases answering BQs (after imputing values for respondents’ missing BQ data)
BQ weight B (heroic assumptions): assume main survey non-respondents who answer BQs are representative of all main survey non-respondents
Bias estimates calculated as difference between BQ weighted estimates and design weighted estimates
Findings:
Bias estimate greatest for variable most similar to BQs – individual sport participation
Bias estimates greater with heroic weight
Evidence that non-response bias is substantial 0.7% to 2.6% with conservative assumptions; 1.5% to 5.7% with heroic assumptions

19. Comparison of weighting schemes

20. Does calibration weighting deal with the bias?
Calculated three calibration weights:
Standard calibration weight: design weights + modelled NR weights + calibration to age, gender and LA totals
Design weights (+ modelled NR weights) + BQ A (conservative) + calibration weight
Design weights (+ modelled NR weights) + BQ B (heroic) + calibration weight
Bias estimates calculated as difference between BQ + calib. weighted estimates and standard calib. weighted estimates
Findings:
Similar pattern to before, but magnitude of estimated bias reduced
Again bias estimates greater for heroic weight
Bias remains substantial: 0.5% to 1.9% with conservative assumptions; 1.2% to 4.2% with heroic assumptions
Calibration weighting reduces bias but leaves a lot uncorrected

21. Does calibration weighting deal with the bias?

22. Conclusions Conclusions:
Basic question method has potential for being very useful method for estimating likely non-response bias for key survey variables
At best, capable of delivering relatively direct non-response bias estimates
Cheap and simple to implement
Easily adapted to cases where key variables relate to individuals and not households and where key variables are constructed from multiple questions
Caveats
Teething problems likely while it is treated as “experimental” and interviewers are not completely used to it – eg missing cases for main survey respondents. Overcome these with suitable training
Effectiveness of method depends on:
Quality of answers to basic questions; need to collect further evidence on validity and reliability of BQ respondents’ answers
Representativeness of BQ sample – maximise BQ RR; also wise to run checks using geodemographic and interviewers observation data