1. Assessment of Bias When Field Operations Are Curtailed in a Mixed-Mode Telephone and Face-to-Face Survey of Persons with Disabilities May 13, 2011 Presentation to the AAPOR Annual Conference Eric Grau

2.  Multi-mode designs are used to boost response rates while controlling costs  Single mode: –Greater potential for nonresponse bias  Multi-mode: –Greater potential for measurement error bias  May be more problematic in certain populations (e.g., persons with disabilities) Introduction 2

3.  Telephone (CATI) and face-to-face (CAPI) interviews are often used together in mixed- mode studies  Similar: both involve an interviewer  Different: –CAPI interviews allow for greater ability to develop rapport and maintain the respondent’s interest –CAPI interviews tend to be easier to match pace and communication style Background 3

4.  Literature: –Response errors in CATI are different than those that occur in face-to-face interviews –Krosnick (2002): greater cognitive demand more shortcuts by respondents –Respondents who require more effort to obtain an interview provide poorer quality data, increasing measurement error Background (continued) 4

5.  CAPI is seen as necessary to adequately survey the population in studies of persons with disabilities  CATI is used to save money in studies with limited resources  Mixed-mode approaches (such as CATI/CAPI) can therefore be used in studies of persons with disabilities that have limited resources, where CAPI interviews are only used for some difficult-to-reach subpopulations Background (continued) 5

6.  This study: assume field operations reduce bias. Under this assumption: how much is bias reduced?  Is it possible to scale back or eliminate CAPI interviews without significant changes in the respondent population, or significant changes in outcomes?  Previous research: mode effects may not be trivial Research Question 6

7.  Ticket to Work (TTW) program provides participants with a coupon to obtain employment training and other employment- related services  Eligibility for TTW program: –Must be a beneficiary of one or both of two Social Security Administration programs for persons with disabilities, ages 18-64: Supplemental Security Income (SSI) Social Security Disability Insurance (SSDI) Evaluation of the Ticket to Work (TTW) Program 7

8.  Component surveys –Beneficiary Survey Target Population –All 10.4 million SSI/SSDI beneficiaries –Participant Survey Target Population –Among 10.4 million SSI/SSDI beneficiaries, subpopulation of TTW participants (108,000) Evaluation of the Ticket to Work (TTW) Program 8

9.  Collects health and well-being, service use, and employment information from SSI and SSDI beneficiaries  Dual-mode survey (CATI/CAPI)  Length: 45 minutes  In 2006 (third round of survey), 6,600 respondents –82% response rate –Approximately 5,100 by CATI and 1,500 by CAPI –23% of respondents by CAPI interview National Beneficiary Survey 9

10.  Multiple-Stage Cluster Sample –For the beneficiary sample and most subpopulations of the participant population Sample Design 10

11.  3,382 sample members –1,945 resolved by phone (completes, ineligibles, nonrespondents) –1,437 fielded (42%)  2,508 completes –1,645 completed by phone –863 fielded (34% of completes) 741 fielded completes with 60 days of field effort No fielded completes with no field efforts Attributes of Beneficiary Sample: Number Fielded and Number Complete 11

12.  Used paradata on respondents  Process details (in chronological order) –Codes describing each locating and/or interview attempt –The date that entries were recorded –Notes about the locating or interview attempt Current Study 12

13.  For sample members that were assigned to field operations at some point, we reassigned a disposition code 1.As if no field operations were conducted 2.As if field operations were limited to 60 days after the first field assignment  Once this was done, nonresponse adjustments were recalculated with new status codes Disposition Code Reassignment 13

14. Number of Standard Errors Respondent’s Disability Frame 5 Months of Field Effort 60 Days of Field Effort No Field Effort Deaf 0.9%0.1-0.4-1.2 Blind 2.4%1.6 Psychiatric Disability 30.4%-2.6-2.7-3.1 Intellectual Disability 13.5%0.70.6-0.6 Other Physical Disability 52.8%1.31.52.9 Comparison with Frame: Representative Beneficiary Sample 14

15. Number of Standard Errors Respondent’s Disability Frame 5 Months of Field Effort 60 Days of Field Effort No Field Effort Deaf 0.9%0.10.3-0.1 Blind 2.4%1.01.21.0 Psychiatric Disability 30.4%-1.2-1.1-0.7 Intellectual Disability 13.5%0.40.60.1 Other Physical Disability 52.8%0.50.20.3 Comparison with Frame: Beneficiary Sample After NR Adjustments 15

16.  Variables with Differences in –Item nonresponse More item nonresponse in CATI –Social desirability More likely to give socially desirable answers in CATI –Acquiescence CATI respondents “better informed” Comparison of Survey Variables: Measures 16

17. Maximum Number of Standard Errors Away from Original Estimate Survey Variable 60 Days of Field EffortNo Field Effort Race0.270.50 Father’s education0.200.98 Mother’s education0.380.72 Education level0.360.56 General health0.240.56 Health insurance0.141.79 Comparison of Survey Variables: Representative Beneficiary Sample (Item NR Variables) 17

18. Maximum Number of Standard Errors Away from Original Estimate Survey Variable 60 Days of Field EffortNo Field Effort Goals include moving up 0.370.40 Used drugs in past year 0.200.01 Work for pay next year 0.440.90 Work for pay next 5 years 0.320.71 Household income (median) 0.031.12 Comparison of Survey Variables: Representative Beneficiary Sample (Social Desirability Variables) 18

19. Maximum Number of Standard Errors Away from Original Estimate Survey Variable 60 Days of Field EffortNo Field Effort Heard of impairment- related work expenses exclusion 0.291.39 Heard of expedited reinstatement 0.161.66 Heard of BPAO 0.362.36 Heard of TTW 0.281.01 Comparison of Survey Variables: Representative Beneficiary Sample (Acquiescence Variables) 19

20.  Mode effects do exist for some key variables  Removing field effort is not recommended  Consistent with work by Sloan, Wright, and Barrett (2006)  Variables that are prone to “acquiescence” (that is, saying yes just to get through the interview) sensitive to reduced field effort Conclusions 20

21.  Reducing field efforts to 60 days did not have a major effect on estimates –Possible cost-saving measure  More research needed –Only three levels (no field effort, 60 days of field effort, and 5 months of field effort) –Need to determine how much field effort is enough to minimize bias Conclusions (continued) 21

22. Eric Grau egrau@mathematica-mpr.com Mathematica Policy Research 600 Alexander Park Princeton, NJ 08540 www.Mathematica-MPR.com Contact Information 22