A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar: Initial Results from the 2008 EHC “Paper” Test Jeff Moore Statistical Research Division Presentation to the ASA/SRM SIPP Working Group Alexandria, VA November 17, 2009

Paper prepared by: Jeff Moore, Jason Fields, Gary Benedetto, Martha Stinson, Anna Chan, & Jerry Maples For presentation at: American Association for Public Opinion Research (AAPOR) May 14-17, 2009

Overview Background: - SIPP “re-engineering” - event history calendar (EHC) methods Goals & Design of the 2008 EHC Paper Test Preliminary Results Summary / Conclusions / Next Steps

SIPP Survey of Income and Program Participation - income/wealth/poverty in the U.S.; program participation dynamics/effects - interviewer-administered; longitudinal - panel length = 3-4 years

SIPP Survey of Income and Program Participation - income/wealth/poverty in the U.S.; program participation dynamics/effects - interviewer-administered; longitudinal - panel length = 3-4 years Key Design Feature: - 3 interviews/year, 4-month reference pd.

SIPP Re-Engineering Implement Improvements to SIPP

SIPP Re-Engineering Implement Improvements to SIPP - reduce costs - reduce R burden - improve processing system - modernize instrument - expand/enhance use of admin records

SIPP Re-Engineering Implement Improvements to SIPP - reduce costs - reduce R burden - improve processing system - modernize instrument - expand/enhance use of admin records Key Design Change: - annual interview, 12-month reference pd., event history calendar methods

SIPP Re-Engineering Implement Improvements to SIPP - reduce costs - reduce R burden - improve processing system - modernize instrument - expand/enhance use of admin records Key Design Change: - annual interview, 12-month reference pd., event history calendar methods

EHC Interviewing Human Memory - structured/organized - links and associations

EHC Interviewing Human Memory - structured/organized - links and associations EHC Exploits Memory Structure - links between to-be-recalled events

EHC Interviewing Human Memory - structured/organized - links and associations EHC Exploits Memory Structure - links between to-be-recalled events EHC Encourages Active Assistance to Rs - flexible approach to help elicit an autobiographical “story”

Evaluations of EHC Methods Many EHC vs. “Q-List” Comparisons - various methods - in general: positive data quality results

Evaluations of EHC Methods Many EHC vs. “Q-List” Comparisons - various methods - in general: positive data quality results BUT, Important Research Gaps - data quality for need-based programs? - extended reference period?

Paper Test Goals & Design Basic Goal: Can an annual EHC interview collect data of comparable quality to standard SIPP?

Paper Test Goals & Design Basic Goal: Can an annual EHC interview collect data of comparable quality to standard SIPP? “Go/No-Go” signal for continued R&D

Paper Test Goals & Design Basic Goal: Can an annual EHC interview collect data of comparable quality to standard SIPP? “Go/No-Go” signal for continued R&D Basic Design: EHC re-interview of SIPP sample households

Design Details (1) Sample: SIPP 2004 panel interview cases - reported on CY-2007 in waves 10-12

Design Details (1) Sample: SIPP 2004 panel interview cases - reported on CY-2007 in waves EHC re-interview in 2008, about CY-2007

Design Details (2) SIPP Sample Cases in Two Sites - Illinois (all) - Texas (4 metro areas)

Design Details (2) SIPP Sample Cases in Two Sites - Illinois (all) - Texas (4 metro areas) Primary Sample Component: 1,096 Wave Addresses (cooperating wave 11 households) IL:487 TX:609

Design Details (3) EHC Questionnaire [handout]

Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period

Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period - start with landmark events

Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period - start with landmark events - subset of SIPP topics (“domains”) - month-level detail

Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period - start with landmark events - subset of SIPP topics (“domains”) - month-level detail Sample of Addresses, Not People - post-interview clerical match to SIPP

Design Details (4) 1096 initial sample addresses

Design Details (4) 1096 initial sample addresses Outcomes: household interviews (91%)

Design Details (4) 1096 initial sample addresses Outcomes: household interviews (91%) - 1,922 individual EHC interviews (99%)

Design Details (4) 1096 initial sample addresses Outcomes: household interviews (91%) - 1,922 individual EHC interviews (99%) - 1,658 EHC Rs matched to SIPP (86%)

Design Details (4) 1096 initial sample addresses Outcomes: household interviews (91%) - 1,922 individual EHC interviews (99%) - 1,658 EHC Rs matched to SIPP (86%) FINAL ANALYSIS SAMPLE: 1,620

Primary Evaluation Compare SIPP and EHC Survey Reports

Primary Evaluation Compare SIPP and EHC Survey Reports - same people - same time period - same characteristics

Primary Evaluation Compare SIPP and EHC Survey Reports - same people - same time period - same characteristics Differences Suggest Data Quality Effects

Primary Evaluation Compare SIPP and EHC Survey Reports - same people - same time period - same characteristics Differences Suggest Data Quality Effects (later: use administrative records for a more definitive data quality assessment)

Main Research Questions 1.Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)?

Main Research Questions 1.Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)? 2.Does the effect of interview method vary across calendar months (especially early in the year vs. late in the year)?

Main Research Questions 1.Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)? 2.Does the effect of interview method vary across calendar months (especially early in the year vs. late in the year)? 3.Bottom line: Go? Or no-go?

Initial Results 4 Government “Welfare” Programs: Food Stamps Supplemental Security Income (SSI) Temporary Aid to Needy Families (TANF) Women Infants & Children (WIC)

Initial Results 4 Government “Welfare” Programs: Food Stamps Supplemental Security Income (SSI) Temporary Aid to Needy Families (TANF) Women Infants & Children (WIC) 4 Other Characteristics: Medicare Social Security employment school enrollment

Results in Context

Almost All SIPP and EHC Reports Agree

Results in Context Almost All SIPP and EHC Reports Agree - all characteristics, all months

Results in Context Almost All SIPP and EHC Reports Agree - all characteristics, all months - in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree

Results in Context Almost All SIPP and EHC Reports Agree - all characteristics, all months - in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree - worst case (employment): 92-94%

Results in Context Almost All SIPP and EHC Reports Agree - all characteristics, all months - in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree - worst case (employment): 92-94% Disagreements are RARE EVENTS

Results Summary The details vary, but…

Results Summary 3 Patterns:

Results Summary 3 Patterns: 1. EHC = SIPP All Year

Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL)

Results Summary 3 Patterns: 1. EHC = SIPP All Year equivalent data quality

SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

% participation (% “yes”)

SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports % participation (% “yes”) months of CY 2007

SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports % participation (% “yes”) months of CY 2007 SIPP reports EHC reports

SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports % participation (% “yes”) months of CY 2007 SIPP reports EHC reports 10 percentage pts

SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports 100 percentage pts

SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary - no “main effect” for method (SIPP = EHC) - no significant method difference in any month

WIC (Illinois Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary - no “main effect” for method (SIPP = EHC) - no significant method difference in any month

Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL)

Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year

Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL)

Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year reduced EHC data quality, but not due to longer recall period

MEDICARE -- % Covered in Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary - significant “main effect” for method (SIPP > EHC) - method difference is constant across months

SOCIAL SECURITY -- % Covered in Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary - significant “main effect” for method (SIPP > EHC) - method difference is constant across months

WIC (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary - significant “main effect” for method (SIPP > EHC) - method difference is constant across months

FOOD STAMPS (Illinois Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary - significant “main effect” for method (SIPP > EHC) - method difference is essentially constant across months

Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL)

Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only

Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment

Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only EHC data quality may suffer due –to longer recall period

FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary - no significant “main effect” for method - BUT significant variation by month -- JAN-MAY: SIPP > EHC later months: no difference (reversal?)

TANF (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary - no significant “main effect” for method - BUT significant variation by month -- JAN-MAY: SIPP > EHC later months: no difference

EMPLOYMENT -- % Working for Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary - significant “main effect” for method (SIPP > EHC) - BUT significant variation by month -- JAN-AUG (SEP): SIPP > EHC later months: no difference

SCHOOL ENROLLMENT -- % Enrolled in Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary - no significant “main effect” for method - BUT significant variation by month JAN-APR: SIPP > EHC JUN-JUL: SIPP < EHC AUG-DEC: no difference

Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment

Field Test Overall Summary

Go! Successful “Proof of Concept”

Field Test Overall Summary Go! Successful “Proof of Concept” Overwhelming Finding: SIPP-EHC Agreement

Field Test Overall Summary Go! Successful “Proof of Concept” Overwhelming Finding: SIPP-EHC Agreement Valuable Lessons to Inform 2010 Test

Field Test Overall Summary Go! Successful “Proof of Concept” Overwhelming Finding: SIPP-EHC Agreement Valuable Lessons to Inform 2010 Test Specific Data Comparisons are Instructive

Results Implications Pattern 1. EHC = SIPP All Year SSI; WIC (IL)

Results Implications Pattern 1. EHC = SIPP All Year SSI; WIC (IL) No evident problems; no reason for concern about data quality in a 12-month EHC interview

Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL)

Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment,

Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment, but probably not due to EHC method or recall length

Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment, but probably not due to EHC method or recall length - less effective screening questions (no D.I.; fewer probes; no local labels)

Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment, but probably not due to EHC method or recall length - less effective screening questions (no D.I.; fewer probes; no local labels) - different definitions

Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment, but probably not due to EHC method or recall length - less effective screening questions (no D.I.; fewer probes; no local labels) - different definitions Likely fixes in CAPI

Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment

Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year

Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year Additional research: - why these characteristics?

Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year Additional research: - why these characteristics? - match to admin records

SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

SSI – Net Bias in the Monthly Estimates Compared to Administrative Records (Survey % “Yes” - Record % “Yes”)

SSI – Monthly Rates of Discrepancy With Administrative Records (Total Discrepancies as % of Non-Missing N)

FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports and ADRECS

Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year Additional research: - why these characteristics? - match to admin records (MORE TO DO)

Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year Additional research: - why these characteristics? - match to admin records - understand Paper Test time lag effects

. ????

Thanks very much! Any questions?