Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 RTI International is a trade name of Research Triangle Institute 3040 Cornwallis Road ■ P.O. Box 12194 ■ Research Triangle Park, North Carolina, USA.

Similar presentations


Presentation on theme: "1 RTI International is a trade name of Research Triangle Institute 3040 Cornwallis Road ■ P.O. Box 12194 ■ Research Triangle Park, North Carolina, USA."— Presentation transcript:

1 1 RTI International is a trade name of Research Triangle Institute 3040 Cornwallis Road ■ P.O. Box ■ Research Triangle Park, North Carolina, USA Phone Challenges of Designing and Implementing Multimode Instruments Jennifer Wine, Melissa Cominole, Ruth Heuer, and John Riccobono March 15, 2006

2 2 Why Multimode?  NCES postsecondary studies – primarily CATI in the past, with some CAPI  NCES required the addition of the web self-administered option to help offset the decline in survey response rates  Characteristics of our postsecondary populations  Technologically savvy  Unconventional schedules  Cell phones replacing land lines  Time and money

3 3 Data Collection Methodologies Used for the Postsecondary Studies

4 4 Typical Timeline for Data Collection Source: BPS FS March2006 March2006 April2006 June 2006 October2006 Send data collection announcement mailing Begin web,self-administered data collection Begin telephone interviewing (CATI) Begin field interviewing (CAPI) End data collection

5 5 Multimode Design Challenges and Adaptations

6 6 Interview Specifications -- All Modes Source: NPSAS FS

7 7 Selected Interview Screen – Web Mode Source: NPSAS FS

8 8 Selected Interview Screen – CATI Mode Source: NPSAS FS

9 9 Multimode Design Challenges and Adaptations

10 10 Interviews Completed during Early Response Incentive Period* *When evaluated in the FT studies, early response incentives were shown to increase the likelihood of a response (p<.05).

11 11 Design for Telephone Prompting Experiment Source: BPS FT

12 12 Interview Participation, by Prompting Status Type of prompting Response rates at end of early period Prompted21.6* Not prompted 10.4 *p<.01 Source: BPS FT

13 13 Prompting and Base Year Response Status Response rates at end of early period Base year respondents Prompted25.0 Base year nonrespondents Prompted20.8* Not prompted15.1* *p<.05 Source: BPS FT

14 14 Average Call Count per Completed Interview, by Mode p<.0001 Source: BPS FT

15 15 Multimode Design Adaptations

16 16 Proportion of Time in the Interview and in Transit for Web and CATI Respondents  Time on-screen  Time in transit *p<.0001 Source: B&B FS Web CATI Total On-screen * Transit *

17 17 Web Connection Speed

18 18 Multimode Design Adaptations

19 19 Help Desk Calls

20 20 On-Line Help Text Source: BPS FT

21 21 Rate of Help Text Access by Mode p<.0001 Source: NPSAS FS

22 22 Multimode Design Adaptations

23 23 Major/Field of Study Coder

24 24 Multimode Design Adaptations

25 25 Mode Differences in Item Reliability  Very few items with significant differences between modes, with no patterns evident  Some studies – NSOPF FT and BPS FT – showed no mode differences on any items

26 26 Multimode Design Adaptations

27 27 Percent of Indeterminate Responses, by Mode  38 interview questions, and over 26 percent of those in the finance section, had nonresponse rates of at least 10 percent  CATI/CAPI respondents provided an indeterminate response for nearly 2.5 percent of all responses.  Web respondents provided an indeterminate response for nearly 5 percent of all responses (p<.05). Source: B&B FT 5% 2.5%

28 28 Response Conversion Text for Selected Items Source: B&B FS

29 29 Use of Conversion Text  Percent converted to an explicit response ranged from 0 percent to 100 percent in the B&B FS; comparable rates (18 to 100 percent) were observed in the BPS FT  Items less likely to convert: dollar amounts related to income, monthly expenses, and overall debt (28 to 73 percent)  Web respondents in the BPS FT were more likely than interview respondents to convert on two items – race and undergraduate loan debt; no other mode differences were observed

30 30 Next Steps?  Different incentive amounts at different points in data collection  Prompting difficult subgroups  Evaluate the use of help text relative to item-level nonresponse and reliability  Continue efforts to assess differences in data quality across modes

31 31 Contact Information Melissa B. Cominole RTI International Slides and paper available at:


Download ppt "1 RTI International is a trade name of Research Triangle Institute 3040 Cornwallis Road ■ P.O. Box 12194 ■ Research Triangle Park, North Carolina, USA."

Similar presentations


Ads by Google