1. The Implications of Misreporting for Longitudinal Studies of SNAP
Preliminary work. Do not cite without permission.
The Implications of Misreporting for Longitudinal Studies of SNAP
Brian Stacy
Erik Scherpf
The findings and conclusions in this preliminary publication have not been formally disseminated by the USDA and should not be construed to represent any agency determination or policy. This research was supported in part by the intramural research program of the USDA, Economic Research Service.

2. Introduction Researchers often rely on survey datasets to examine SNAP
Several studies have noted misreporting of SNAP participation
Many researchers use longitudinal estimators of SNAP
To study dynamics of SNAP participation
To possibly help with selection bias
However, longitudinal methods can exacerbate bias from misreporting

3. Objectives & Findings
This paper examines the misreporting of SNAP on causal effects
Particularly focused on longitudinal estimators
Link SNAP Administrative Data to CPS ASEC survey records
Create panel dataset for individuals
Three main findings:
Find evidence of severe errors in changes in reported SNAP participation
Of those switching onto SNAP from one period to next based on surveys, 86% are false
Application: Effects of SNAP on work effort
Find severe bias in longitudinal estimators
Find evidence that some methods to correct estimators for misreporting perform well
Conditional distribution method (CDM) works well for first-difference OLS estimator
CDM can be applied by researchers without access to admin records

4. Background
Meyer, Mittag, & Goerge (2018) find substantial under-reporting of SNAP in CPS ASEC survey in cross-section
Close to 50% of SNAP participants don’t report receipt in CPS ASEC
Little literature on errors in changes in reported SNAP participation over time, which identify many longitudinal estimators of SNAP effects
Paper is similar to Bound & Krueger (1991); Bound, Brown, Duncan, Rogers (1994); Freeman (1984)
These papers examine effects on longitudinal estimators of misreporting of earnings and union participation

5. Data
Combine SNAP administrative records on SNAP receipt to CPS ASEC survey
We assume admin reports are “true”, mismatch with survey is false report
Links based on PIK created by Census Bureau staff
Can create two-period panel datasets using longitudinal structure of CPS ASEC
Can directly assess errors in changes in reported SNAP participation
Create linkages for nine anonymous states from
Unfortunately, not all states have data available for all years.
From , states make up 32% of individual SNAP caseload
In 2005, when fewest states were available, made up 5% of caseload

6. Results on Misreporting in Cross-Section
Using our linked SNAP admin records and CPS ASEC records, examine misreporting
Cross-sectional Misreporting:
Find under-reporting rate of 43.9%
Find over-reporting rate of 2.9%
Find evidence that misreporting related to individual characteristics
Those who work are more likely to under-report SNAP
Females more likely to under-report
Those with more education more likely to under-report
Meyer et al. (2018) found higher income individuals more likely to under-report

7. Results on Misreporting Longitudinally
Using linked data, also examine changes in reported SNAP participation
Errors in changes in reported SNAP participation:
Four groups:
Switch off SNAP: 86% of survey based changes are false
Most of these people were on SNAP in both periods or off SNAP both periods
On SNAP – Both Periods: 27.5% of survey reports false
Even though over-reporting rare, off SNAP group makes up 88.5% of records
Off SNAP – Both Periods: 4.7% of survey reports false
Switch on SNAP: 87% of survey based changes are false
Again, most were either off SNAP or on SNAP both periods
Find errors in changes related to characteristics
Employment, age, race

8. Empirical Application: Effects of SNAP on Work

9. Cross Sectional w/ OLS: Reported SNAP -0.291*** -0.222*** -13.82***
Table 4. Regression Estimates of Effects of SNAP on Work and Hours Worked using Reported SNAP Measure from CPS ASEC versus Admin Verified SNAP Measure. Results from 16 Separate Regressions.
(1)
(2)
(3)
(4)
Estimated Effects
Worked Past Year
Hours Worked
Cross Sectional w/ OLS:
Reported SNAP
-0.291***
-0.222***
-13.82***
-9.912***
(0.007)
(0.257)
(0.267)
Admin Verified SNAP
-0.286***
-0.221***
-13.60***
-9.556***
(0.006)
(0.238)
(0.251)
First Differences w/ OLS:
-0.040***
-0.039***
-2.373***
-2.330***
(0.010)
(0.413)
(0.419)
0.050***
0.048***
1.769***
1.710***
(0.015)
(0.607)
(0.612)
Covariates NO
YES
Covariates include: age, gender, race, education, marital status, unemployment rate. Robust standard errors in parenthesis. *** p<0.01, ** p<0.05, * p<0.1

10. Solutions for Researchers
Find Conditional Distribution Methods work well (Mittag,2018)
Can be used by researchers using only public use data
No admin data link required
Two-Step approach:
Researchers with access to admin data publish estimates of conditional distribution of “true” SNAP participation conditional on covariates in public use data
Researchers using public use data then impute SNAP participation based on these covariates
Approach similar to multiple imputation
Key is that imputed SNAP participation has same covariance structure as “true” SNAP participation measure
Assumptions:
Conditional distribution must be same for linked data and public use data
Parametric model used to estimate conditional distribution must be properly specified

11. *** p<0.01, ** p<0.05, * p<0.1
Table 5. Regression Estimates of First Differences in Work and Hours Worked on First Differences in SNAP on using OLS with Reported SNAP Measure, OLS with Admin Verified Measure, and the Mittag (2018) Conditional Distribution Method
(1)
(2)
(3)
(4)
Estimated Effects
Worked Past Year
Hours Worked
First Differences w/ OLS:
Reported SNAP
-0.040***
-0.039***
-2.373***
-2.330***
(0.010)
(0.413)
(0.419)
Admin Verified SNAP
0.050***
0.048***
1.769***
1.710***
(0.015)
(0.607)
(0.612)
Conditional Distribution Method
0.054 ***
0.047 ***
1.947***
1.698***
(0.006)
(0.003)
(0.288)
(0.126)
Covariates NO
YES
Observations
36,000
Bootstrap standard errors in parenthesis from 100 bootstrap replications. In each bootstrap replication, both the multinomial logit estimation and regression estimation steps are performed. Covariates include controls for age, education, race, marital status, and the county unemployment rate.
*** p<0.01, ** p<0.05, * p<0.1

12. Conclusions
Many researcher make use of survey data and apply longitudinal methods
CPS ASEC supplement frequently used and has substantial misreporting
Find errors in changes in SNAP participation worse than cross-section
We characterize bias for cross-sectional and longitudinal estimators
Find severe bias in longitudinal estimator of effects of SNAP on Work
Solutions:
Conditional Distribution Methods seem promising
Researchers with access to admin linkages may wish to publish conditional distribution estimates

13. Questions?

14. Misreporting and Bias

15. Non-random Misreporting and Bias