Nonresponse Bias in a Nationwide Dual-Mode Survey Matthew DeBell, Stanford University Natalya Maisel, Stanford University Ted Brader, University of Michigan Vanessa Meldener, Westat ITSEW 2018, Durham NC

American National Election Studies 2016 Time Series Study Survey of adult US citizens Seeks to explain voter turnout & candidate choice in presidential elections Dual mode ABS Face-to-face Mail-to-internet Clustered not clustered n=1180 n=3090 RR1=50% RR1=44%

This Talk What’s non-random about non-response? 1. Accuracy of estimates 2. “Easy to get” vs “hard to get” 3. Respondents vs non-respondents in a NRFU study What if we got more respondents from the Web sample? Conclusions or implications for… accuracy of estimates field effort field strategy & adaptive design

1. Accuracy of estimates Comparisons to benchmarks (Current Population Survey) Not necessarily non-response error, but indicative Age, gender, education, race/ethnicity, marital status, income, household size, home tenure, region, employment status, and nativity

Accuracy of estimates Characteristic Web error Face-to-face error Errors of 4 points or more (unweighted) Characteristic Web error Face-to-face error Voter turnout +18 +19 Edu: high school cr. -11 -7 Own home -6 -10 One-person HH +7 Age 18-29 -4 Income < $25K ns Married White +4

2. Hard to get? “Easy to get” Rs: make contact with few attempts & cooperate readily “Hard to get” Rs: require multiple contact attempts or refusal conversion Definitions of hard-to-get Web: if a refusal conversion letter was mailed or a refusal was recorded (39%) FtF: any refusal, >5 contacts, or late contact (41%) We assume non-respondents are more like hard-to-get Rs Differences between easy- and hard-to-get Rs would indicate field effort matters

How “hard to get” Rs differ from easier Rs Mail-to-web Younger Less: education, white, married More: Hispanic, renters Lower income More Southern, less NE Less likely to vote More Trump voters Face-to-face Younger No difference Higher income Less Southern, more NE No difference (ns) No difference (ns, opposite)

3. Non-Response Follow-Up (NRFU) Study Mail survey Both responding and non-responding households n=4,725 Study name & sponsorship differed from ANES One page paper questionnaire with 18 questions RR = 39%

NRFU Results: no differences Ethnicity (Hispanic) Talk to neighbors Like college professors Like news reporters Children under 18 Party ID Presidential vote choice

NRFU Results: differences Web Face-to-face Voter turnout (10 points) Like surveys (.11) Internet access (15 points) Worry personal privacy (.08) Interpersonal trust (.04) Free time (.02) Education Age (2.6 years) Voter turnout (10 points) Like surveys (.15) Like talking politics (.05)

4. What if we got NRFU Rs in the first place? Estimated effect on Web sample of adding 400 Rs by mail like NRFU Rs

4. What if we got NRFU Rs in the first place? Estimated effect on Web sample of adding 400 Rs by mail like NRFU Rs No effect (<1 percentage point) on observed demographics (age, sex, education, Hispanic ethnicity) No effect (<1 percentage point) on party ID

4. What if we got NRFU Rs in the first place? Estimated effect on Web sample of adding 400 Rs by mail like NRFU Rs No effect (<1 percentage point) on observed demographics (age, sex, education, Hispanic ethnicity) No effect (<1 percentage point) on party ID Trump vote error narrows from -2.1 to -1.7 Clinton vote error narrows from 1.2 to 1.0

4. What if we got NRFU Rs in the first place? Estimated effect on Web sample of adding 400 Rs by mail like NRFU Rs No effect (<1 percentage point) on observed demographics (age, sex, education, Hispanic ethnicity) No effect (<1 percentage point) on party ID Trump vote error narrows from -2.1 to -1.7 Clinton vote error narrows from 1.2 to 1.0 Turnout error narrows from 15.1 to 12.0

Conclusions: accuracy of estimates Both modes miss Non-voters* People who dislike surveys* Low education Home owners Younger respondents (18-29) Face-to-face misses Mid-higher income Married

Conclusions: field effort Full field effort Matters for web turnout & candidate choice Appears to matter less for FtF Sequential mail mode for Web sample would likely improve voter turnout accuracy

Conclusions: field strategy Finding Fieldwork implication NR bias for turnout and liking surveys De-emphasize voting and “surveys” in communication strategy

Conclusions: field strategy Finding Fieldwork implication NR bias for turnout and liking surveys De-emphasize voting and “surveys” in communication strategy Attitudes toward professors and news reporters not a source of NR bias References to these groups may be acceptable in our communications

Conclusions: field strategy Finding Fieldwork implication NR bias for turnout and liking surveys De-emphasize voting and “surveys” in communication strategy Attitudes toward professors and news reporters not a source of NR bias References to these groups may be acceptable in our communications Home-owners under-represented Target non-rental properties for greater recruitment effort (more contact attempts and rapidly escalated incentives)

Conclusions: field strategy Finding Fieldwork implication NR bias for turnout and liking surveys De-emphasize voting and “surveys” in communication strategy Attitudes toward professors and news reporters not a source of NR bias References to these groups may be acceptable in our communications Home-owners under-represented Target non-rental properties for greater recruitment effort (more contact attempts and rapidly escalated incentives) Younger and lower education (and married, in FTF) are under-represented Once HHs are screened, target these HHs for greater recruitment effort

in a Nationwide Dual-Mode Survey Thank you Nonresponse Bias in a Nationwide Dual-Mode Survey Matthew DeBell, Stanford University Natalya Maisel, Stanford University Ted Brader, University of Michigan Vanessa Meldener, Westat Contact: debell@stanford.edu