1. A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar: Initial Results from the 2008 EHC “Paper” Test Jeff Moore Statistical Research Division Presentation to the ASA/SRM SIPP Working Group Alexandria, VA November 17, 2009

2. Paper prepared by: Jeff Moore, Jason Fields, Gary Benedetto, Martha Stinson, Anna Chan, & Jerry Maples For presentation at: American Association for Public Opinion Research (AAPOR) May 14-17, 2009

3. Overview Background: - SIPP “re-engineering” - event history calendar (EHC) methods Goals & Design of the 2008 EHC Paper Test Preliminary Results Summary / Conclusions / Next Steps

4. SIPP Survey of Income and Program Participation - income/wealth/poverty in the U.S.; program participation dynamics/effects - interviewer-administered; longitudinal - panel length = 3-4 years

5. SIPP Survey of Income and Program Participation - income/wealth/poverty in the U.S.; program participation dynamics/effects - interviewer-administered; longitudinal - panel length = 3-4 years Key Design Feature: - 3 interviews/year, 4-month reference pd.

6. SIPP Re-Engineering Implement Improvements to SIPP

7. SIPP Re-Engineering Implement Improvements to SIPP - reduce costs - reduce R burden - improve processing system - modernize instrument - expand/enhance use of admin records

8. SIPP Re-Engineering Implement Improvements to SIPP - reduce costs - reduce R burden - improve processing system - modernize instrument - expand/enhance use of admin records Key Design Change: - annual interview, 12-month reference pd., event history calendar methods

9. SIPP Re-Engineering Implement Improvements to SIPP - reduce costs - reduce R burden - improve processing system - modernize instrument - expand/enhance use of admin records Key Design Change: - annual interview, 12-month reference pd., event history calendar methods

10. EHC Interviewing Human Memory - structured/organized - links and associations

11. EHC Interviewing Human Memory - structured/organized - links and associations EHC Exploits Memory Structure - links between to-be-recalled events

12. EHC Interviewing Human Memory - structured/organized - links and associations EHC Exploits Memory Structure - links between to-be-recalled events EHC Encourages Active Assistance to Rs - flexible approach to help elicit an autobiographical “story”

13. Evaluations of EHC Methods Many EHC vs. “Q-List” Comparisons - various methods - in general: positive data quality results

14. Evaluations of EHC Methods Many EHC vs. “Q-List” Comparisons - various methods - in general: positive data quality results BUT, Important Research Gaps - data quality for need-based programs? - extended reference period?

15. Paper Test Goals & Design Basic Goal: Can an annual EHC interview collect data of comparable quality to standard SIPP?

16. Paper Test Goals & Design Basic Goal: Can an annual EHC interview collect data of comparable quality to standard SIPP? “Go/No-Go” signal for continued R&D

17. Paper Test Goals & Design Basic Goal: Can an annual EHC interview collect data of comparable quality to standard SIPP? “Go/No-Go” signal for continued R&D Basic Design: EHC re-interview of SIPP sample households

18. Design Details (1) Sample: SIPP 2004 panel interview cases - reported on CY-2007 in waves 10-12

19. Design Details (1) Sample: SIPP 2004 panel interview cases - reported on CY-2007 in waves 10-12 EHC re-interview in 2008, about CY-2007

30. Design Details (2) SIPP Sample Cases in Two Sites - Illinois (all) - Texas (4 metro areas)

31. Design Details (2) SIPP Sample Cases in Two Sites - Illinois (all) - Texas (4 metro areas) Primary Sample Component: 1,096 Wave 10-11-12 Addresses (cooperating wave 11 households) IL:487 TX:609

32. Design Details (3) EHC Questionnaire [handout]

33. Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period

34. Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period - start with landmark events

35. Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period - start with landmark events - subset of SIPP topics (“domains”) - month-level detail

36. Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period - start with landmark events - subset of SIPP topics (“domains”) - month-level detail Sample of Addresses, Not People - post-interview clerical match to SIPP

37. Design Details (4) 1096 initial sample addresses

38. Design Details (4) 1096 initial sample addresses Outcomes: - 935 household interviews (91%)

39. Design Details (4) 1096 initial sample addresses Outcomes: - 935 household interviews (91%) - 1,922 individual EHC interviews (99%)

40. Design Details (4) 1096 initial sample addresses Outcomes: - 935 household interviews (91%) - 1,922 individual EHC interviews (99%) - 1,658 EHC Rs matched to SIPP (86%)

41. Design Details (4) 1096 initial sample addresses Outcomes: - 935 household interviews (91%) - 1,922 individual EHC interviews (99%) - 1,658 EHC Rs matched to SIPP (86%) FINAL ANALYSIS SAMPLE: 1,620

42. Primary Evaluation Compare SIPP and EHC Survey Reports

43. Primary Evaluation Compare SIPP and EHC Survey Reports - same people - same time period - same characteristics

44. Primary Evaluation Compare SIPP and EHC Survey Reports - same people - same time period - same characteristics Differences Suggest Data Quality Effects

45. Primary Evaluation Compare SIPP and EHC Survey Reports - same people - same time period - same characteristics Differences Suggest Data Quality Effects (later: use administrative records for a more definitive data quality assessment)

46. Main Research Questions 1.Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)?

47. Main Research Questions 1.Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)? 2.Does the effect of interview method vary across calendar months (especially early in the year vs. late in the year)?

48. Main Research Questions 1.Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)? 2.Does the effect of interview method vary across calendar months (especially early in the year vs. late in the year)? 3.Bottom line: Go? Or no-go?

49. Initial Results 4 Government “Welfare” Programs: Food Stamps Supplemental Security Income (SSI) Temporary Aid to Needy Families (TANF) Women Infants & Children (WIC)

50. Initial Results 4 Government “Welfare” Programs: Food Stamps Supplemental Security Income (SSI) Temporary Aid to Needy Families (TANF) Women Infants & Children (WIC) 4 Other Characteristics: Medicare Social Security employment school enrollment

51. Results in Context

52. Almost All SIPP and EHC Reports Agree

53. Results in Context Almost All SIPP and EHC Reports Agree - all characteristics, all months

54. Results in Context Almost All SIPP and EHC Reports Agree - all characteristics, all months - in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree

55. Results in Context Almost All SIPP and EHC Reports Agree - all characteristics, all months - in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree - worst case (employment): 92-94%

56. Results in Context Almost All SIPP and EHC Reports Agree - all characteristics, all months - in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree - worst case (employment): 92-94% Disagreements are RARE EVENTS

57. Results Summary The details vary, but…

58. Results Summary 3 Patterns:

59. Results Summary 3 Patterns: 1. EHC = SIPP All Year

60. Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL)

61. Results Summary 3 Patterns: 1. EHC = SIPP All Year equivalent data quality

62. SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

63. % participation (% “yes”)

64. SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports % participation (% “yes”) months of CY 2007

65. SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports % participation (% “yes”) months of CY 2007 SIPP reports EHC reports

66. SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports % participation (% “yes”) months of CY 2007 SIPP reports EHC reports 10 percentage pts

67. SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports 100 percentage pts

68. SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

69. Analysis Summary - no “main effect” for method (SIPP = EHC) - no significant method difference in any month

70. WIC (Illinois Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

71. Analysis Summary - no “main effect” for method (SIPP = EHC) - no significant method difference in any month

72. Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL)

73. Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year

74. Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL)

75. Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year reduced EHC data quality, but not due to longer recall period

76. MEDICARE -- % Covered in Each Month of CY2007 According to the SIPP and EHC Reports

77. Analysis Summary - significant “main effect” for method (SIPP > EHC) - method difference is constant across months

78. SOCIAL SECURITY -- % Covered in Each Month of CY2007 According to the SIPP and EHC Reports

79. Analysis Summary - significant “main effect” for method (SIPP > EHC) - method difference is constant across months

80. WIC (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

81. Analysis Summary - significant “main effect” for method (SIPP > EHC) - method difference is constant across months

82. FOOD STAMPS (Illinois Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

83. Analysis Summary - significant “main effect” for method (SIPP > EHC) - method difference is essentially constant across months

84. Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL)

85. Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only

86. Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment

87. Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only EHC data quality may suffer due –to longer recall period

88. FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

89. Analysis Summary - no significant “main effect” for method - BUT significant variation by month -- JAN-MAY: SIPP > EHC later months: no difference (reversal?)

90. TANF (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

91. Analysis Summary - no significant “main effect” for method - BUT significant variation by month -- JAN-MAY: SIPP > EHC later months: no difference

92. EMPLOYMENT -- % Working for Each Month of CY2007 According to the SIPP and EHC Reports

93. Analysis Summary - significant “main effect” for method (SIPP > EHC) - BUT significant variation by month -- JAN-AUG (SEP): SIPP > EHC later months: no difference

94. SCHOOL ENROLLMENT -- % Enrolled in Each Month of CY2007 According to the SIPP and EHC Reports

95. Analysis Summary - no significant “main effect” for method - BUT significant variation by month JAN-APR: SIPP > EHC JUN-JUL: SIPP < EHC AUG-DEC: no difference

96. Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment

97. Field Test Overall Summary

98. Go! Successful “Proof of Concept”

99. Field Test Overall Summary Go! Successful “Proof of Concept” Overwhelming Finding: SIPP-EHC Agreement

100. Field Test Overall Summary Go! Successful “Proof of Concept” Overwhelming Finding: SIPP-EHC Agreement Valuable Lessons to Inform 2010 Test

101. Field Test Overall Summary Go! Successful “Proof of Concept” Overwhelming Finding: SIPP-EHC Agreement Valuable Lessons to Inform 2010 Test Specific Data Comparisons are Instructive

102. Results Implications Pattern 1. EHC = SIPP All Year SSI; WIC (IL)

103. Results Implications Pattern 1. EHC = SIPP All Year SSI; WIC (IL) No evident problems; no reason for concern about data quality in a 12-month EHC interview

104. Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL)

105. Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment,

106. Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment, but probably not due to EHC method or recall length

107. Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment, but probably not due to EHC method or recall length - less effective screening questions (no D.I.; fewer probes; no local labels)

108. Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment, but probably not due to EHC method or recall length - less effective screening questions (no D.I.; fewer probes; no local labels) - different definitions

109. Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment, but probably not due to EHC method or recall length - less effective screening questions (no D.I.; fewer probes; no local labels) - different definitions Likely fixes in CAPI

110. Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment

111. Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year

112. Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year Additional research: - why these characteristics?

113. Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year Additional research: - why these characteristics? - match to admin records

114. SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

115. SSI – Net Bias in the Monthly Estimates Compared to Administrative Records (Survey % “Yes” - Record % “Yes”)

116. SSI – Monthly Rates of Discrepancy With Administrative Records (Total Discrepancies as % of Non-Missing N)

117. FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

118. FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports and ADRECS

119. Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year Additional research: - why these characteristics? - match to admin records (MORE TO DO)

120. Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year Additional research: - why these characteristics? - match to admin records - understand Paper Test time lag effects

125. . ????

126. Thanks very much! Any questions? contact:jeffrey.c.moore@census.gov 301-763-4975