1. Nonresponse Bias in a Nationwide Dual-Mode Survey
Matthew DeBell, Stanford University
Natalya Maisel, Stanford University
Ted Brader, University of Michigan
Vanessa Meldener, Westat
ITSEW 2018, Durham NC

2. American National Election Studies 2016 Time Series Study
Survey of adult US citizens
Seeks to explain voter turnout
& candidate choice in presidential elections
Dual mode ABS
Face-to-face Mail-to-internet
Clustered not clustered
n= n=3090
RR1=50% RR1=44%

3. This Talk
What’s non-random about non-response? 1. Accuracy of estimates 2. “Easy to get” vs “hard to get” 3. Respondents vs non-respondents in a NRFU study What if we got more respondents from the Web sample? Conclusions or implications for… accuracy of estimates field effort field strategy & adaptive design

4. 1. Accuracy of estimates
Comparisons to benchmarks (Current Population Survey)
Not necessarily non-response error, but indicative
Age, gender, education, race/ethnicity, marital status, income, household size, home tenure, region, employment status, and nativity

5. Accuracy of estimates Characteristic Web error Face-to-face error
Errors of 4 points or more (unweighted)
Characteristic
Web error
Face-to-face error
Voter turnout
+18
+19
Edu: high school cr.
-11 -7
Own home -6
-10
One-person HH +7
Age 18-29 -4
Income < $25K ns
Married
White +4

6. 2. Hard to get?
“Easy to get” Rs: make contact with few attempts & cooperate readily
“Hard to get” Rs: require multiple contact attempts or refusal conversion
Definitions of hard-to-get
Web: if a refusal conversion letter was mailed or a refusal was recorded (39%)
FtF: any refusal, >5 contacts, or late contact (41%)
We assume non-respondents are more like hard-to-get Rs
Differences between easy- and hard-to-get Rs would indicate field effort matters

7. How “hard to get” Rs differ from easier Rs
Mail-to-web
Younger
Less: education, white, married
More: Hispanic, renters
Lower income
More Southern, less NE
Less likely to vote
More Trump voters
Face-to-face
Younger
No difference
Higher income
Less Southern, more NE
No difference (ns)
No difference (ns, opposite)

8. 3. Non-Response Follow-Up (NRFU) Study
Mail survey
Both responding and non-responding households
n=4,725
Study name & sponsorship differed from ANES
One page paper questionnaire with 18 questions
RR = 39%

9. NRFU Results: no differences
Ethnicity (Hispanic)
Talk to neighbors
Like college professors
Like news reporters
Children under 18
Party ID
Presidential vote choice

10. NRFU Results: differences
Web
Face-to-face
Voter turnout (10 points)
Like surveys (.11)
Internet access (15 points)
Worry personal privacy (.08)
Interpersonal trust (.04)
Free time (.02)
Education
Age (2.6 years)
Voter turnout (10 points)
Like surveys (.15)
Like talking politics (.05)

11. 4. What if we got NRFU Rs in the first place?
Estimated effect on Web sample of adding 400 Rs by mail like NRFU Rs

12. 4. What if we got NRFU Rs in the first place?
Estimated effect on Web sample of adding 400 Rs by mail like NRFU Rs
No effect (<1 percentage point) on observed demographics (age, sex, education, Hispanic ethnicity)
No effect (<1 percentage point) on party ID

13. 4. What if we got NRFU Rs in the first place?
Estimated effect on Web sample of adding 400 Rs by mail like NRFU Rs
No effect (<1 percentage point) on observed demographics (age, sex, education, Hispanic ethnicity)
No effect (<1 percentage point) on party ID
Trump vote error narrows from -2.1 to -1.7
Clinton vote error narrows from 1.2 to 1.0

14. 4. What if we got NRFU Rs in the first place?
Estimated effect on Web sample of adding 400 Rs by mail like NRFU Rs
No effect (<1 percentage point) on observed demographics (age, sex, education, Hispanic ethnicity)
No effect (<1 percentage point) on party ID
Trump vote error narrows from -2.1 to -1.7
Clinton vote error narrows from 1.2 to 1.0
Turnout error narrows from 15.1 to 12.0

15. Conclusions: accuracy of estimates
Both modes miss
Non-voters*
People who dislike surveys*
Low education
Home owners
Younger respondents (18-29)
Face-to-face misses
Mid-higher income
Married

16. Conclusions: field effort
Full field effort
Matters for web turnout & candidate choice
Appears to matter less for FtF
Sequential mail mode for Web sample would likely improve voter turnout accuracy

17. Conclusions: field strategy
Finding
Fieldwork implication
NR bias for turnout and liking surveys
De-emphasize voting and “surveys” in communication strategy

18. Conclusions: field strategy
Finding
Fieldwork implication
NR bias for turnout and liking surveys
De-emphasize voting and “surveys” in communication strategy
Attitudes toward professors and news reporters not a source of NR bias
References to these groups may be acceptable in our communications

19. Conclusions: field strategy
Finding
Fieldwork implication
NR bias for turnout and liking surveys
De-emphasize voting and “surveys” in communication strategy
Attitudes toward professors and news reporters not a source of NR bias
References to these groups may be acceptable in our communications
Home-owners under-represented
Target non-rental properties for greater recruitment effort (more contact attempts and rapidly escalated incentives)

20. Conclusions: field strategy
Finding
Fieldwork implication
NR bias for turnout and liking surveys
De-emphasize voting and “surveys” in communication strategy
Attitudes toward professors and news reporters not a source of NR bias
References to these groups may be acceptable in our communications
Home-owners under-represented
Target non-rental properties for greater recruitment effort (more contact attempts and rapidly escalated incentives)
Younger and lower education (and married, in FTF) are under-represented
Once HHs are screened, target these HHs for greater recruitment effort

21. in a Nationwide Dual-Mode Survey
Thank you
Nonresponse Bias
in a Nationwide Dual-Mode Survey
Matthew DeBell, Stanford University
Natalya Maisel, Stanford University
Ted Brader, University of Michigan
Vanessa Meldener, Westat
Contact: