Minimizing interviewer effects and reducing refusal rates at Statistics Iceland Anton Örn Karlsson

Survey errors Four categories: – Coverage error – Non response error – Sampling error – Measurement error 27/02/2016

Interviewer training to reduce errors Refusal aversion training – Increase response rates Reduce non response error Standardized interview techniques – Reduce interviewer variance Reduce measurement error 27/02/2016

Interviewers at Statistics Iceland Permanent – 13 CATI – 8 field interviewers Temporary – 40 CATI – For ICT and SILC 27/02/2016

Training for interviewers New training program started in 2009 Two goals: – Reduction of interviewer related error Standardized interviewing – Reading questions as worded – Probing inadequate or incomplete answers – Recording answers directly – Being neutral with regards to the respondent – Minimizing refusal rates Tailoring the participation request – Based on Groves and McGonagle (2001) 27/02/2016

Assessment of the training program Comparing data collected in 2008 with data collected in 2009 – For both ICT and SILC – Refusal rates Should be lower in 2009 compared to 2008 – Intraclass correlation coefficient (ICC), a measure of interviewer effects Should be lower in 2009 compared to /02/2016

Data collection The sequence of surveys: – 2008, January to the end of May: 1.ICT 2.Travel survey 3.SILC – 2009, February to the beginning of May 1.SILC 2.ICT All data for ICT and SILC was collected via CATI 27/02/2016

Calculations(1) Refusal rates: ICC – Few questions selected SILC: Amounts regarding the habitat of the household, debts and other economic issues of the household. ICT: Yes/no questions about the technical equipment in the household. – Three criteria: 1) At least 100 valid answers 2) At least 10 interviews per interviewer 3) Same wording in both years 27/02/2016

Calculations (2) A formula from Groves (1989): – ICC: Based on a one-way analysis of variance Va = Between mean square Vb = Within mean square M = Number of interviews per interviewer – A function was built in R in order to calculate ICC in that way 27/02/2016

Results(1) 27/02/2016

Results(2) 27/02/2016

Results(3) 27/02/2016

Results(4): Refusal rates for ICT and SILC from 2003 – /02/2016

Discussion(1) Interviewer variability – Some positive effects of the new training, but there is still more work to do – Compared to other studies: High variability – Mean coefficients lower in 2009 Majority of coefficients in ICT higher in 2009 – Effects of interviewer experience – The danger of high variability Variance inflation 27/02/2016

What can be done? Monitor interviewers – Give feedback about interviewer performance. Assess how interviewers read questions and how they probe for answers. Rewrite questions – Some will have to be rewritten so all respondents will understand them in the same way Giving interviewers standard phrases to use in interviews 27/02/2016

Discussion(2) Refusal rates – Seems that the refusal training has had positive effects on the refusal rates of both surveys. – SILC: A drop of four percentage points from 2008 to – ICT: The lowest recorded refusal rates are from 2009 and /02/2016

Thank you! 27/02/2016