1. 1© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 EU WORKSHOP ON RECENT DEVELOPMENT IN BUSINESS AND CONSUMER SURVEYS Brussels, 13 – 14 November 2014

2. 2© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Considerations around switching from telephone to online survey methodology Karsten Shaw and Donna Culverwell, GfK UK

3. 3© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 GfK have been conducting the Consumer research in the UK on behalf of the European Commission since 2006. Historically, fieldwork has been conducted as a telephone survey amongst all respondents using a RDD (Random Digit Dialling) sampling methodology But, we know that there are more and more considerations /issues with undertaking telephone fieldwork in the UK: This is a situation which we do not see any improvement on Taking this on board, we had to look at alternative ways of undertaking the fieldwork in the UK; minimising any impact to the long term data trend Caller ID in the home Respondents thinking that the interviewer is trying to sell them something The background Respondents becoming more savvy to research calls, meaning they are harder to reach: More and more respondents are now in mobile only households, which creates issues with reaching people via landlines Ever increasing cost of RDD sampling methodology

4. 4© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 The background How do you change a methodology that has been running for a number of years without creating shifts in the data? With the plan to move the UK Consumer survey to a 100% online survey from May 2015 onwards. Each month phasing out the number of telephone interviews, whilst increasing the number of online interviews Working closely with the EC we agreed on a mixed-mode telephone/online approach to the survey in the UK over a 10 month period (July 2014 – April 2015) 3 12 4

5. 5© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Sample Sizes

6. 6© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Changing the methodology of a tracking study needs to be done with great care Or is there a need to continue to track the data consistently over time? The answer to this question in this instance that we need as much consistency as possible, to provide a seamless track between results from the existing measures to the measures going forward Is the first wave of this ‘new’ study to be treated as a brand new survey, with no element of consistency going backwards? This has implications for how the new survey is set up, and what is done at the early stages to ensure a smooth transition

7. 7© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Extensive experience of migrating studies to create a seamless transition GfK has extensive experience in the art of moving studies from offline to online methodologies to negate the possibilities of movements in the data as a result GfK conducts expert statistical analysis to manage such changes We are aiming to ensure a smooth transition and to minimise any movement in data from before the move to after It is important that the key CCB measures are not affected by changes over the next 12 months. This means keeping all other aspects of the survey unchanged

8. 8© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Parallel run process The results of the parallel runs are used to identify the impact of design changes. If the results of the two surveys are the same, then we would conclude that there are no effects and the data from the old and new design can be integrated with no requirement for adjustment However, if differences in results are observed, then we would conduct a process of calibration to allow us to bring the two datasets into line. That is, to adjust the results to make them consistent and comparable Typically this calibration process would involve: Once this calibration process is complete, the two data streams can be fully integrated Identifying any structural differences in the two samples Adjusting for these differences through a process of calibration weighting. This will usually correct the majority of differences Developing an adjustment process to align any remaining differences

9. 9© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Integrating considerations Once we have a mechanism for adjusting differences between waves, we need to decide which of the data tracks we will adjust. That is, do we: adjust historic data to match the new data trend, or adjust the new data to match the historic data trend We usually recommend that we adjust history to match the new data trend/structures. However, in this instance we cannot re-adjust the historic data, so it will require constant re-adjustment of results going forward.

10. 10© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Weighting Methodology

11. 11© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Weighting Methodology In this instance, we want to treat the telephone sample as the ‘correct’ view of the world There are arguments for and against this, but the main reason is consistency Therefore the aim of the exercise is to apply additional weighting to the online data, over and above demographic data, to ensure it resembles the telephone sample as far as possible So in the first instance a complete parallel run was conducted in July 2014 of telephone and online

12. 12© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 We found some substantial differences in the questions between modes Profiles amongst key questions

13. 13© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Demographic Weighting Targets Applied equivalently to telephone and online samples

14. 14© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Demographic Differences Demographics not used in weighting scheme

15. 15© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 But demographic weighting does not correct the differences Even when weighting by employment, education and income, the differences in the CCB questions persist We attempted Propensity Modelling – using Logistic Regression to form propensity weights on the basis of the CCB questions and demographics However, this generated extreme weights which had a severe impact on the effective sample size

16. 16© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Propensity weighting had a severe impact on the effective sample size Propensity weighting is a common way used to adjust for samples collected from different methodologies Logistic regression models were run which generated propensity scores of being an online respondent (compared with a telephone respondent) The resulting propensity scores were then divided up into quintiles amongst the telephone respondents The cut-offs were then used to generate five groups amongst the online respondents and weights were generated so that the weighted groups were also equal in size

17. 17© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Propensity Weighting 20% Least likely to be online 20% Most likely to be online Logistic Regression modelling is applied to generate a likelihood of being online These scores are applied to the online sample so that five groups are created These five groups are transformed back to their original size using weighting 20%

18. 18© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 WeightsOriginal Demographic WeightsPropensity Weights Efficiency97%32% Minimum weight0.660.21 Maximum weight1.6112.9 Effect of propensity weighting This does remove most of the differences between the questions, but…

19. 19© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 So we used a modified weighting scheme The final weighting scheme used adjusts for the differences between the telephone and online samples by using questions which exhibit consistent differences between the modes Weights are then generated which minimise these differences and applied to the subsequent wave of online data These weights are updated every wave based on the additional information provided

20. 20© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 The questions were decided on using the main parallel run data in August These were the questions which exhibited the most substantial differences in some part of the distribution between telephone and online: How do you think the general economic situation in this country has changed over the LAST 12 MONTHS? How do you think consumer prices have developed over the last 12 months? In comparison with the past 12 months, how do you expect consumer prices will develop in the next 12 months? In view of the general economic situation, do you think now is the right time for people to make major purchases such as furniture or electrical goods? Over the NEXT 12 MONTHS, how likely will you be to save any money?

21. 21© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Weights were generated which removed these consistent differences

22. 22© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 These weights are then applied to the subsequent wave of data The weights which adjusted these differences were then applied to the subsequent wave of data The remaining differences were then examined to ensure that the profile of telephone and weighted online data were similar This way, we can ensure an objective weighting procedure which takes into account as much information as possible about previous waves and then applies that to the current wave of data The extra wave of data is then used as further information to refine the weights for the subsequent wave

23. 23© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Effect of weighting scheme used This is a more efficient weighting scheme but some efficiency is still lost * Most recent wave (October) WeightsOriginal Demographic WeightsWeighting Scheme* Efficiency97%74% Minimum weight0.660.26 Maximum weight1.613.94

24. 24© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Checklist Insert chart title here (optional) We will continue to monitor the weights at each wave and ensure they are fit for purpose without becoming too extreme and adversely impacting the effective sample size With each extra wave, we also check the weighting strategy to ensure it is still optimal There will also be annual waves of a full telephone survey to check and refine the weighting scheme This will ensure a smooth transition from telephone to online without any step change in the trend

25. 25© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 THANK YOU