1. Journalism 614: Sampling and Non-Response

2. Sampling Probability Sampling Non-probability sampling
Based on random selection
Non-probability sampling
Based on convenience

3. Sampling Miscues: Alf Landon for President (1936)
Literary Digest: post cards to voters in 6 states
Correctly predicting elections from
Names selected from telephone directories and automobile registrations
In 1936, they sent out 10 million post cards
Results pick Landon 57% to Roosevelt 43%
Election: Roosevelt in the largest landslide
Roosevelt 61% of the vote and in Elect. Col.
Why so inaccurate?: Poor sampling frame
Leads to selection of wealthy respondents

4. Sampling Miscues: Thomas E. Dewey for President (1948)
Gallup picks winner
Use quota sampling:
matches sample characteristics to population
Gallup quota samples on the basis of income
In 1948, Gallup picked Dewey to defeat Truman
Reasons:
1. Most pollsters quit polling in October
2. Undecided voters went for Truman
3. Unrepresentative samples—WWII changed society since census

5. Non-probability Sampling
In situations where sampling frame for randomization doesn’t exist
Types of non-probability samples:
1. Reliance on available subjects
convenience sampling
2. Purposive or judgmental sampling
3. Snowball sampling
4. Quota sampling

6. Reliance on Available Subjects
Person on the street, easily accessible
Examples:
Mall intercepts, college students, e-polls
Frequently used, but usually biased
Notoriously inaccurate
Especially in making inferences about larger population, even with many respondents

7. Purposive or Judgmental Sampling
Dictated by the purpose of the study
Situational judgments about what individuals should be surveyed to make for a useful or representative sample
E.g., Using college students to study third-person effects regarding rap and metal music
3pe: Others are more affected by exposure than self
Assessing effects on self and others
Using college students makes for homogeneity of self

8. Snowball Sampling
Used when population of interest is difficult to locate
E.g., homeless people, meth addicts
Research collects data from of few people in the targeted group
Initially surveyed individuals asked to name other people to contact
Good for exploration
Bad for generalizability

9. Quota Sampling
Begins with a table of relevant characteristics of the population
Proportions of Gender, Age, Education, Ethnicity from census data
Selecting a sample to match those proportions
Problems:
1. Quota frame must be accurate
2. Sample is not random, but can be representative

10. Probability Sampling Goal: Representativeness Random selection
Sample resembles larger population
Random selection
Enhancing likelihood of representative sample
Each unit of the population has an equal chance of being selected into the sample

11. Population Parameters
Parameter: Summary statistic for the population
E.g., Mean age of the population
Sample allows parameter estimates
E.g., Mean age of the sample
Used as an estimate of the population parameter

12. Sampling Error
Every time you draw a sample from the population, the parameter estimate will fluctuate slightly
E.g.:
Sample 1: Mean age = 37.2
Sample 2: Mean age = 36.4
Sample 3: Mean age = 38.1
If you draw lots of samples, you would get a normal curve of values

13. Normal Curve of Sample Estimates
Frequency of estimated means
from multiple samples
Likely population
parameter
Estimated Mean

14. Error and Sample Size As the sample size increases:
The error decreases
In other words, large sample estimate is likely to be closer to the population parameter
As the sample size increases, we get more confident in our parameter estimate

15. Confidence Interval
Interval width at which we are 95% confident the estimate contains the population parameter
For example, we predict that Candidate X will receive 45% of the vote with a 3% confidence interval
We are 95% sure the parameter will be between 42% and 48%
The “margin of error” in a poll
Confidence interval shrinks as:
Error is smaller
Sample size is larger

17. Sample Size & Confidence Interval
How precise does the estimate have to be?
More precise: larger sample size
Larger samples increase precision
But at a diminishing rate
Each unit you add to your sample contributes to the accuracy of your estimate
But the amount it adds shrinks with additional unit added

18. 95% Confidence Intervals
Sample Size
% split
N = 100
N = 200
N = 300
N = 400
N = 500
N = 700
N = 1000
N = 1500
50/50
10.0
7.1
5.8
5.0
4.5
3.8
3.2
2.6
70/30
9.2
6.5
5.3
4.6
4.1
3.5
2.9
2.4
90/10
6.8
4.2
3.0
2.7
2.3
1.9
1.5

19. Describe Sampling Frame
List of units from which sample is drawn
Defines your population
E.g., List of members of population
Ideally you’d like to list all members of your population as your sampling frame
Randomly select your sample from that list
Often impractical to list entire population

20. Sampling Frames for Surveys
Limitations of the telephone book:
Misses unlisted numbers/mobile numbers
SES and age bias:
Poor people may not have phone
Less likely to have multiple phone lines
Young people have mobile phone numbers
Most studies use a technique such as Random Digit Dialing as a way around this

21. Types of Sampling Designs
Simple Random Sampling
Systematic Sampling
Stratified Sampling
Multi-stage Cluster Sampling

22. Simple Random Sampling
Establish a sampling frame
A number is assigned to each element
Elements randomly selected into the sample
Use a random number generator to select every case you need for inclusion.

23. Systematic Sampling Establish sampling frame
Select every kth element with random start
E.g., 1000 on the list, choosing every 5th name yields a sample size of 200
Sampling interval: standard distance between units for the sampling frame
Sampling interval = pop. size / sample size
Sampling ratio: proportion of pop. selected
Sampling ratio = sample size / population size

24. Stratified Sampling
Modification used to reduce potential for sampling error
Research ensures that certain groups are represented proportionately in the sample
E.g., If the population is 60% female, stratified sample selects 60% females into the sample
E.g., Stratifying by region of the country to make sure that each region is proportionately represented

25. Cluster Sampling
Frequently, there is no convenient way of listing the population for sampling
E.g., Sample of Dane County or Wisconsin
Hard to get a list of the population members
Cluster sample
Sample of census blocks
List of census blocks, list people for selected blocks
Select sub-sample of people living on each block

26. Multi-stage Cluster Sample
Cluster sampling done in a series of stages:
List, then sample within
Example:
Stage 1: Listing zip codes
Randomly selecting zip codes
Stage 2: List census blocks within selected zip codes
Randomly select census blocks
Stage 3: List households on selected census blocks
Randomly select households
Stage 4: List residents of selected households
Randomly select person to interview

27. Nonresponse Declining contact and cooperation rates
Especially for “gold standard” RDD National Telephone Surveys
Early research suggests the issues are rather small, with little bias on results
Examined by comparing “easy to contact” individuals to “hard to contact”
More systematic version is to compare between standard 5-day and “rigorous” survey

28. Accelerating Problem
Survey firms reporting increasingly high rates of non-contact and non-cooperation
Americans leading increasingly busy lives
More and more unsolicited calls to home
Sophisticated technologies to avoid calls
Big drop offs in last years
Call screening (I only take known callers)
Cell phones (I pay for minutes during survey)

29. Hard to Gauge the Effect
Initial work conducted in late 90s
Curtain et al - Low effort “restricted call” design versus high effort “all call” design
See no difference in population estimates
Keeter et al – Two parallel surveys, one using standard 5 day vs. “rigorous”
On average, a two percentage point difference
Seem to suggest that lower response rate does not effect survey quality

30. Non-response in this Century
Lot has changed in last decade +
More legislative restrictions
More mobile technologies
More VOIP technologies
Re-ran the study and found similar results comparing 5-day and rigorous
5-day – 10 call backs, one refusal conversion
Rigorous – 21 weeks, advance letters, left messages, additional call backs, etc.
Little difference in findings

31. The Problem of Cell Phones
In 2006, 13% of cell phone only HHs
Increasing 1% every six months
Increasing 2% every six months after 2006
By 2015, 46% of U.S. Adults Live In Cellphone-Only HHs
64% of Millennials (born ) are Cellphone-Only
Bias in terms of who is missed is most prominent among young people.
“Serious coverage problem”
“Particular challenge”

32. Big differences in wireless only
HHs by Age and SES
Only 16% of those 65+
Nearly 70% among 25-29
Just over 40% among “not poor”
Over 50% for “near poor”
Nearly 60% for the “poor”
This creates systematic biases

34. Some substantial differences
Big differences between cell and non-cell respondents to a range of questions
Especially for issues that affect younger people, and behaviors such as voting
Register to vote?
Political knowledge?
Media usage?

35. Strive for Higher Response Rate

36. To Achieve a High Response Rate
Incentives: gifts or drawings for completion
prize drawing vs smaller incentives to alll
donating to a charity as an inducement
Run experiments to see which incentive works
Online: pre-invitation and landing page
Test different versions to see what encourages respondents to click on survey link (online)
Reminders and follow-ups to boost response
In general, you only want to send one or two reminders.
Make online survey friendly for all devices/browsers.
Make it usable on mobile or tablet. See if there is a high bounce rate from particular devices