1. Is There a Trade-Off Between Quality and Cost? An Experiment Comparing Telephone vs. Face-to-Face Responses to the National Beneficiary Survey May 13, 2011 Presented at the 66th Annual American Association for Public Opinion Research Conference (Phoenix, Arizona) Matt Sloan Debra Wright Kirsten Barrett

2. Multimode designs are used to boost response rates while controlling costs Measurement error may occur if the mode affects the interview process This problem may be exacerbated with certain populations Introduction 2

3. Telephone and face-to-face interviews are similar in that both involve an interviewer The two modes are different in that face-to- face interviews: –Allow the interviewer to more readily develop a rapport –Make it easier to match pace, communication style Mode Differences 3

4. Certain response errors may be more likely to occur in telephone interviews than in face-to- face interviews (Jordan et al. 1980; Jackle et al. 2010) Krosnick (1991) suggests that the greater the cognitive demand placed on respondents, the more likely they are to satisfice There is evidence that satisficing may be more pronounced among respondents with limited cognitive ability (Chang and Krosnick 2010) Mode Differences (contd.) 4

5. Are there differences in the quality of data collected by telephone versus face-to-face in a study of persons with disabilities? Research Question 5

6. Sponsored by the Social Security Administration (SSA) Collects information from SSA beneficiaries on their health and well-being, service use, and employment 45-minute, dual-mode survey (CATI/CAPI) In 2010, 8,038 cases were fielded, and 5,080 were completed National Beneficiary Survey 6

7. Randomized control design: –645 cases were randomly selected from the sample frame for CAPI-only treatment –The comparison group consisted of regular sample members who were interviewed by telephone Regular cases sent to CAPI were considered nonrespondents for this experiment –Yielded 1,574 completed cases: 1,229 completed by CATI, and 345 completed by CAPI 73% = CATI response rate 54% = CAPI response rate Experiment Design 7

8. Item nonresponse Social desirability Nondifferentiation Acquiescence Measures 8

9. Individual Items CATI Non- Response CAPI Non- Response Expects to work for pay next year 2.44%1.45% Expects to work for pay in the next five years 3.58%4.64% Race 5.37%3.77% Fathers education 35.39%32.17% Mothers education 22.13%19.42% Respondents weight 2.85%2.32% Household income* 30.68%43.48% * p<.05 Item Nonresponse 9

10. Series of Items CATI Non- Response CAPI Non- Response Awareness of SSA programs4.56%2.32% Health insurance*6.83%1.45% * p<.05 Item Nonresponse (contd.) 10

11. Questions CATI Response CAPI Response Goals include moving up (yes)* 51.43%43.32% Expects to work for pay next year (agree) 30.61%25.59% Expects to work for pay in the next five years (agree)* 48.35%37.99% Felt need to cut down on drinking (no) 84.60%79.35% Used drugs in past 12 months (no) 94.20%95.94% * p<.05 Social Desirability 11

12. Question CATI Response CAPI Response Household income$24,082$20,468 Social Desirability (contd.) 12

13. Nondifferentiation in ResponseCATICAPI Yes21.48%20.58% No78.52%81.09% Nondifferentiation 13

14. Question CATI Response (Yes) CAPI Response (Yes) Heard of Blind Work Expense7.62%5.56% Heard of expedited re-instatement *12.53%8.50% Heard of benefits specialist20.12%15.84% Heard of Ticket to Work *34.21%26.61% * p<.05 Acquiescence 14

15. Data collection mode appears to have a modest effect on data quality for this population Compared to CATI respondents, CAPI respondents tended to exhibit: –About the same level of item nonresponse and nondifferentiation –Fewer socially desirable responses –Less acquiescence Conclusions 15

16. Vague or demanding items in the telephone interviews showed the greatest drop in quality Results may reflect differences caused by the mode itself or characteristics of responders in each mode. Conclusions (contd.) 16

17. Please contact –Matt Sloan msloan@mathematica-mpr.com –Debra Wright dwright@mathematica-mpr.com –Kirsten Barrett kbarrett@mathematica-mpr.com Also visit www.DisabilityPolicyResearch.org For More Information