Obtaining information on non-responders: a development of the basic question approach for surveys of individuals Patten Smith (Ipsos MORI) Richard Harry (Sport Wales) Colin Gardiner (Ipsos MORI) John D’Souza (Ipsos MORI)

Overview of basic question approach Sport Wales Active Adults Survey Outline Overview of basic question approach Sport Wales Active Adults Survey Adaptation to Active Adults Results and indicative estimates of non-repose bias Conclusions

Basic Question Approach Basic question: ask small number of key survey questions from survey non-respondents (eg see Bethlehem, Cobben and Schouten, 2011; Lynn, 2003) Do this for two reasons: Examine whether non-respondents differ from respondents; if so, provides evidence of non-response bias Improved non-response weights Approach likely to be especially valuable when risk that survey topic will affect propensity to respond With standard basic question approach, two constraints: Basic Questions (BQs) restricted to household level characteristics (BQs often asked before individual selection in household) BQs restricted to survey questionnaire items – cannot deal with key survey variables that are constructed from multiple questions Remainder of presentation: application of BQ approach to a survey of sports participation

Active Adults Survey Household survey measuring levels of sports participation, club membership and volunteering plus lifestyle, health and other sport related behaviours Adults (aged 15 years and above) living in Wales Sponsored by Sport Wales – organisation responsible for increasing participation and improving performance in sport in Wales Random probability sample of addresses stratified by Local Authority and key demographics Two-stage address sample design: Sample frame: Postcode Address File (PAF) PSUs formed from Census Output Areas Stratified (disproportionately) by Local Authority (LA) and by the percentage of managers / professionals among household reference persons and by population density (proportionately within LA) Single adult randomly selected at each address through Kish grid

Active Adults Survey Fieldwork between January 2012 and January 2013, spread over one year to enable full coverage of seasonal activities Questionnaire administered using Computer Assisted Personal Interviewing (CAPI) and a total of 13,145 interviews achieved Of 27,704 sampled addresses, nine per cent classified as ineligible as they did not contain an occupied private household 25,108 addresses classified as in-scope, giving a response rate of 52% Weights: Inverse probability (design) weights Modelled non-response weights (based on geodemographic variables) Calibration to population distributions for age, gender and LA (including interlocking) and to ensure even quarterly distributions

Active Adults Survey response table

Adapting the basic question approach to the Active Adults Survey Previously mentioned restrictions cause problems for Active Adults Survey: Key survey variables relate to individual characteristics and not household ones Single most important sports participation variable constructed from series of questions Revised basic question approach Ask new household-level question that is not in questionnaire: question must correlate highly with key survey questions must be asked of respondents as well as non-respondents (as new question not in main questionnaire) Experimental work reported here Focuses on four key survey variables (next slide); BQs need to correlate with these Two basic questions attempted with first person contacted at all contacted households: BQ1: In the last two weeks have you or any other adult in your household taken part in sports or other physical exercise as a member of a sports club or leisure or fitness centre? BQ2: Apart from this, in the last two weeks, have you or any other adult in your household done any exercise mainly for the purpose of getting or keeping fit?

Key survey variables

Response rates for basic questions *Contact with household but not with selected adult

Using BQs to estimate non-response bias Non-zero values of non-response bias will only be observed if main survey respondents and non-respondents differ in their answers to basic questions Answers to basic questions allow us to estimate likely levels of non-response bias if: answers to basic questions correlate with answers to key survey questions non-respondents answering basic questions are representative of those not answering them; Today, reporting our preliminary analyses in which we: examine these conditions and use a weighting approach to estimate non-response bias

Do main survey respondents and non-respondents differ in answers to basic questions?

Do answers to basic questions correlate with those to key survey variables?

Do answers to basic questions correlate with those to key survey questions?

Do basic question respondents represent main survey non-respondents? Compared main survey respondents who did and did not answer basic questions on selected geodemographic variables Significant differences, but magnitude relatively small – not enough substantially to undermine our analyses Logistic regression of BQ response on geodemographic variables gives poor fit: Nagelkirke pseudo R2 = .018

Do basic question respondents represent main survey non-respondents?

Estimating non-response bias General approach taken: Use basic questions to construct weights Compare key survey estimates: With design weights only With design weights plus basic question weights With design weights and standard NR+calibration weights With design weights and standard NR+calibration weights supplemented with basic question weights Problem: no BQ data for 14% of main survey respondents Dealt with this by imputing BQ answers from main questionnaire variables (Chained Equations method)

Estimating non-response bias BQ weights: four cells formed by cross classifying the two BQs:

Estimating non-response bias Two basic question weights: BQ weight A (conservative assumptions): Weight respondents to total number of cases answering BQs (after imputing values for respondents’ missing BQ data) BQ weight B (heroic assumptions): assume main survey non-respondents who answer BQs are representative of all main survey non-respondents Bias estimates calculated as difference between BQ weighted estimates and design weighted estimates Findings: Bias estimate greatest for variable most similar to BQs – individual sport participation Bias estimates greater with heroic weight Evidence that non-response bias is substantial 0.7% to 2.6% with conservative assumptions; 1.5% to 5.7% with heroic assumptions

Comparison of weighting schemes

Does calibration weighting deal with the bias? Calculated three calibration weights: Standard calibration weight: design weights + modelled NR weights + calibration to age, gender and LA totals Design weights (+ modelled NR weights) + BQ A (conservative) + calibration weight Design weights (+ modelled NR weights) + BQ B (heroic) + calibration weight Bias estimates calculated as difference between BQ + calib. weighted estimates and standard calib. weighted estimates Findings: Similar pattern to before, but magnitude of estimated bias reduced Again bias estimates greater for heroic weight Bias remains substantial: 0.5% to 1.9% with conservative assumptions; 1.2% to 4.2% with heroic assumptions Calibration weighting reduces bias but leaves a lot uncorrected

Does calibration weighting deal with the bias?

Conclusions Conclusions: Basic question method has potential for being very useful method for estimating likely non-response bias for key survey variables At best, capable of delivering relatively direct non-response bias estimates Cheap and simple to implement Easily adapted to cases where key variables relate to individuals and not households and where key variables are constructed from multiple questions Caveats Teething problems likely while it is treated as “experimental” and interviewers are not completely used to it – eg missing cases for main survey respondents. Overcome these with suitable training Effectiveness of method depends on: Quality of answers to basic questions; need to collect further evidence on validity and reliability of BQ respondents’ answers Representativeness of BQ sample – maximise BQ RR; also wise to run checks using geodemographic and interviewers observation data