The European Statistical Training Programme (ESTP)

Chapter 14: Mixed-mode datacollection Introduction Mixed-mode designs Mixed-mode in practice: the Dutch re-design

Introduction Every mode has its weaknesses and its strengths CAPI: expensive, good quality data CATI: cheaper, but not every person has a telephone Mixing modes provides an opportunity to compensate for weaknesses of the individual modes Interviewer No interviewer Paper PAPI Mail-survey Laptop CAPI CASAQ Telephone CATI Voice response Internet CAWI Web-survey

Mixed-mode designs A mixed-mode design consists of a combination of two or more data collection modes Three possibilities: Concurrent Sequential Choice to the respondent

Concurrent mixed-mode design Sample Mode 1 Mode 2 … Mode m The sample is divided in groups that are approached by different modes, but at the same time

Sequential mixed-mode design Sample Mode 1 Mode 2 Response Nonresponse All sample elements are approached by the same mode, but a different mode is used to follow-up the nonrespondents

Examples of mixed-mode designs Safety Monitor 2006 and 2007 (n = 30,000, n = 3,600) LFS 2005 (n=18,000, n=1,000) Sample Phone F-to-F Phone F-to-F Web Response Nonresponse F-to-F Response Nonresponse Web Phone F-to-F

Examples of mixed-mode designs Informal Economy 2006 (n = 2,000, n = 2,000) Phone Web Response Nonresponse F-to-F

Examples: Response rates and composition Measure of representativity Strategy 1 Strategy 2 Safety Monitor 68% 67% 81% Informal Economy 57% 49% 77% 78% LFS 62% 76% 79%

Choosing a mixed-mode design Issues Questionnaire Trade-off between errors and costs: coverage, unit nonresponse and measurement errors Subject of the survey Constraints Time Costs Logistics

Mixed-mode datacollection in practice: Dutch re-design Project from 2007 - 2012 Aim: Reduce costs but maintain quality Main ingredients of re-design: Core questionnaire Use of register information Model based estimation Quality framework Parallel runs of old and new designs Mixed-mode datacollection

Mixed-mode datacollection in practice: Dutch re-design Web response nonresponse CAPI CATI

Mixed-mode datacollection in practice: Dutch re-design Issues: + Web response nonresponse CAPI CATI nonresponse response nonresponse response 14.13

Mixed-mode datacollection in practice: Dutch re-design Ideally, data collection strategy tailored to different groups based on: Costs Quality Logistics (systems) Quality is a mixture of nonresponse bias, coverage- and measurement errors in the different modes. Stability in the mixture of modes is important for planning of fieldwork, adjustment weighting and mode effects.

Mixed-mode datacollection in practice: Dutch re-design Coverage in different modes: CAPI covers entire population. For Web and CATI there is undercoverage for: Characteristic CATI Web Size of hh 1 person 1, 2 persons Mar. Status Unmarried Age 25 – 44 65+ Ethnicity Non-natives Urbanicity Strong degree of urbanisation Province plus large cities Large cities Income hh Low Low and middle Average house value 0 – 200.000 euro

Mixed-mode datacollection in practice: Dutch re-design Response in different modes (Health Survey 2010) Design Subgroup R-indicator CI Old 0.804 (0.780 – 0.828) New 0.796 (0.780 – 0.812) Internet 0.786 (0.772 – 0.800) Follow-up 0.820 (0.800 – 0.840) CATI 0.833 (0.807 – 0.859) CAPI 0.790 (0.758 – 0.822)

Mixed-mode datacollection in practice: Dutch re-design Response in different modes (Health Survey 2010) Conditional and unconditional partial R-indicators Old design (CAPI) vs new design (total Mixed-Mode)  CAPI Unconditional Conditional Age 56.7 43.5 Urbanicity 43.9 20.6 Type of hh 55.4 31.7 House value 50.9 25.9 Job 11.1 13.4 Ethnicity 51.0 31.8  Mixed-Mode Onconditional Conditional Age 60.7 44.6 Urbanicity 45.5 18.5 Type of hh 58.1 30.3 House value 54.0 25.0 Job 1.4 1.5 Ethnicity 54.3 37.2 Het verschil tussen de onconditionele en de conditionele waarden in Tabel 1 is groot. Dat betekent dat er veel onderlinge samenhang is tussen de achtergrondkenmerken.

Mixed-mode datacollection in practice: Dutch re-design For the new design: Internet response vs Internet nonresponse For the new design: Follow-up response vs follow-up nonresponse  Internet Onconditional Conditional Age 55.0 39.8 Urbanicity 30.2 10.0 Type of hh 64.0 43.6 House value 63.2 32.5 Job 23.7 17.2 Ethnicity 58.0 41.8  Follow-up Onconditional Conditional Age 70.5 52.1 Urbanicity 43.0 25.5 Type of hh 34.6 18.4 House value 29.3 Job 27.4 6.7 Ethnicity 33.1 21.2