1. Minimizing interviewer effects and reducing refusal rates at Statistics Iceland Anton Örn Karlsson

2. Survey errors Four categories: – Coverage error – Non response error – Sampling error – Measurement error 27/02/2016

3. Interviewer training to reduce errors Refusal aversion training – Increase response rates Reduce non response error Standardized interview techniques – Reduce interviewer variance Reduce measurement error 27/02/2016

4. Interviewers at Statistics Iceland Permanent – 13 CATI – 8 field interviewers Temporary – 40 CATI – For ICT and SILC 27/02/2016

5. Training for interviewers New training program started in 2009 Two goals: – Reduction of interviewer related error Standardized interviewing – Reading questions as worded – Probing inadequate or incomplete answers – Recording answers directly – Being neutral with regards to the respondent – Minimizing refusal rates Tailoring the participation request – Based on Groves and McGonagle (2001) 27/02/2016

6. Assessment of the training program Comparing data collected in 2008 with data collected in 2009 – For both ICT and SILC – Refusal rates Should be lower in 2009 compared to 2008 – Intraclass correlation coefficient (ICC), a measure of interviewer effects Should be lower in 2009 compared to 2008 27/02/2016

7. Data collection The sequence of surveys: – 2008, January to the end of May: 1.ICT 2.Travel survey 3.SILC – 2009, February to the beginning of May 1.SILC 2.ICT All data for ICT and SILC was collected via CATI 27/02/2016

8. Calculations(1) Refusal rates: ICC – Few questions selected SILC: Amounts regarding the habitat of the household, debts and other economic issues of the household. ICT: Yes/no questions about the technical equipment in the household. – Three criteria: 1) At least 100 valid answers 2) At least 10 interviews per interviewer 3) Same wording in both years 27/02/2016

9. Calculations (2) A formula from Groves (1989): – ICC: Based on a one-way analysis of variance Va = Between mean square Vb = Within mean square M = Number of interviews per interviewer – A function was built in R in order to calculate ICC in that way 27/02/2016

10. Results(1) 27/02/2016

11. Results(2) 27/02/2016

12. Results(3) 27/02/2016

13. Results(4): Refusal rates for ICT and SILC from 2003 – 2010 27/02/2016

14. Discussion(1) Interviewer variability – Some positive effects of the new training, but there is still more work to do – Compared to other studies: High variability – Mean coefficients lower in 2009 Majority of coefficients in ICT higher in 2009 – Effects of interviewer experience – The danger of high variability Variance inflation 27/02/2016

15. What can be done? Monitor interviewers – Give feedback about interviewer performance. Assess how interviewers read questions and how they probe for answers. Rewrite questions – Some will have to be rewritten so all respondents will understand them in the same way Giving interviewers standard phrases to use in interviews 27/02/2016

16. Discussion(2) Refusal rates – Seems that the refusal training has had positive effects on the refusal rates of both surveys. – SILC: A drop of four percentage points from 2008 to 2010. – ICT: The lowest recorded refusal rates are from 2009 and 2010. 27/02/2016

17. Thank you! 27/02/2016