We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byGiovanna Strudwick
Modified over 2 years ago
1© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 EU WORKSHOP ON RECENT DEVELOPMENT IN BUSINESS AND CONSUMER SURVEYS Brussels, 13 – 14 November 2014
2© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Considerations around switching from telephone to online survey methodology Karsten Shaw and Donna Culverwell, GfK UK
3© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 GfK have been conducting the Consumer research in the UK on behalf of the European Commission since Historically, fieldwork has been conducted as a telephone survey amongst all respondents using a RDD (Random Digit Dialling) sampling methodology But, we know that there are more and more considerations /issues with undertaking telephone fieldwork in the UK: This is a situation which we do not see any improvement on Taking this on board, we had to look at alternative ways of undertaking the fieldwork in the UK; minimising any impact to the long term data trend Caller ID in the home Respondents thinking that the interviewer is trying to sell them something The background Respondents becoming more savvy to research calls, meaning they are harder to reach: More and more respondents are now in mobile only households, which creates issues with reaching people via landlines Ever increasing cost of RDD sampling methodology
4© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 The background How do you change a methodology that has been running for a number of years without creating shifts in the data? With the plan to move the UK Consumer survey to a 100% online survey from May 2015 onwards. Each month phasing out the number of telephone interviews, whilst increasing the number of online interviews Working closely with the EC we agreed on a mixed-mode telephone/online approach to the survey in the UK over a 10 month period (July 2014 – April 2015)
5© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Sample Sizes
6© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Changing the methodology of a tracking study needs to be done with great care Or is there a need to continue to track the data consistently over time? The answer to this question in this instance that we need as much consistency as possible, to provide a seamless track between results from the existing measures to the measures going forward Is the first wave of this ‘new’ study to be treated as a brand new survey, with no element of consistency going backwards? This has implications for how the new survey is set up, and what is done at the early stages to ensure a smooth transition
7© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Extensive experience of migrating studies to create a seamless transition GfK has extensive experience in the art of moving studies from offline to online methodologies to negate the possibilities of movements in the data as a result GfK conducts expert statistical analysis to manage such changes We are aiming to ensure a smooth transition and to minimise any movement in data from before the move to after It is important that the key CCB measures are not affected by changes over the next 12 months. This means keeping all other aspects of the survey unchanged
8© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Parallel run process The results of the parallel runs are used to identify the impact of design changes. If the results of the two surveys are the same, then we would conclude that there are no effects and the data from the old and new design can be integrated with no requirement for adjustment However, if differences in results are observed, then we would conduct a process of calibration to allow us to bring the two datasets into line. That is, to adjust the results to make them consistent and comparable Typically this calibration process would involve: Once this calibration process is complete, the two data streams can be fully integrated Identifying any structural differences in the two samples Adjusting for these differences through a process of calibration weighting. This will usually correct the majority of differences Developing an adjustment process to align any remaining differences
9© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Integrating considerations Once we have a mechanism for adjusting differences between waves, we need to decide which of the data tracks we will adjust. That is, do we: adjust historic data to match the new data trend, or adjust the new data to match the historic data trend We usually recommend that we adjust history to match the new data trend/structures. However, in this instance we cannot re-adjust the historic data, so it will require constant re-adjustment of results going forward.
10© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Weighting Methodology
11© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Weighting Methodology In this instance, we want to treat the telephone sample as the ‘correct’ view of the world There are arguments for and against this, but the main reason is consistency Therefore the aim of the exercise is to apply additional weighting to the online data, over and above demographic data, to ensure it resembles the telephone sample as far as possible So in the first instance a complete parallel run was conducted in July 2014 of telephone and online
12© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 We found some substantial differences in the questions between modes Profiles amongst key questions
13© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Demographic Weighting Targets Applied equivalently to telephone and online samples
14© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Demographic Differences Demographics not used in weighting scheme
15© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 But demographic weighting does not correct the differences Even when weighting by employment, education and income, the differences in the CCB questions persist We attempted Propensity Modelling – using Logistic Regression to form propensity weights on the basis of the CCB questions and demographics However, this generated extreme weights which had a severe impact on the effective sample size
16© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Propensity weighting had a severe impact on the effective sample size Propensity weighting is a common way used to adjust for samples collected from different methodologies Logistic regression models were run which generated propensity scores of being an online respondent (compared with a telephone respondent) The resulting propensity scores were then divided up into quintiles amongst the telephone respondents The cut-offs were then used to generate five groups amongst the online respondents and weights were generated so that the weighted groups were also equal in size
17© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Propensity Weighting 20% Least likely to be online 20% Most likely to be online Logistic Regression modelling is applied to generate a likelihood of being online These scores are applied to the online sample so that five groups are created These five groups are transformed back to their original size using weighting 20%
18© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 WeightsOriginal Demographic WeightsPropensity Weights Efficiency97%32% Minimum weight Maximum weight Effect of propensity weighting This does remove most of the differences between the questions, but…
19© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 So we used a modified weighting scheme The final weighting scheme used adjusts for the differences between the telephone and online samples by using questions which exhibit consistent differences between the modes Weights are then generated which minimise these differences and applied to the subsequent wave of online data These weights are updated every wave based on the additional information provided
20© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 The questions were decided on using the main parallel run data in August These were the questions which exhibited the most substantial differences in some part of the distribution between telephone and online: How do you think the general economic situation in this country has changed over the LAST 12 MONTHS? How do you think consumer prices have developed over the last 12 months? In comparison with the past 12 months, how do you expect consumer prices will develop in the next 12 months? In view of the general economic situation, do you think now is the right time for people to make major purchases such as furniture or electrical goods? Over the NEXT 12 MONTHS, how likely will you be to save any money?
21© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Weights were generated which removed these consistent differences
22© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 These weights are then applied to the subsequent wave of data The weights which adjusted these differences were then applied to the subsequent wave of data The remaining differences were then examined to ensure that the profile of telephone and weighted online data were similar This way, we can ensure an objective weighting procedure which takes into account as much information as possible about previous waves and then applies that to the current wave of data The extra wave of data is then used as further information to refine the weights for the subsequent wave
23© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Effect of weighting scheme used This is a more efficient weighting scheme but some efficiency is still lost * Most recent wave (October) WeightsOriginal Demographic WeightsWeighting Scheme* Efficiency97%74% Minimum weight Maximum weight
24© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 Checklist Insert chart title here (optional) We will continue to monitor the weights at each wave and ensure they are fit for purpose without becoming too extreme and adversely impacting the effective sample size With each extra wave, we also check the weighting strategy to ensure it is still optimal There will also be annual waves of a full telephone survey to check and refine the weighting scheme This will ensure a smooth transition from telephone to online without any step change in the trend
25© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 THANK YOU
Jeopardy Topic 1Topic Q 1Q 6Q 11Q 16Q 21 Q 2Q 7Q 12Q 17Q 22 Q 3Q 8Q 13Q 18Q 23 Q 4Q 9Q 14Q 19Q 24 Q 5Q 10Q 15Q 20Q 25 Final Jeopardy.
25 seconds left….. 24 seconds left….. 23 seconds left…..
European Commission Directorate General Economic and Financial Affairs New question on capacity utilisation in services - state of play and way forward.
Internet Survey Method in the 2010 Census and Challenges to the 2015 Census in Japan Population Census Division, Statistics Bureau of Japan Hideki Koizumi.
Title Subtitle 1. A. B. C. C. * D. Click to try again! INCORRECT.
DIVIDING INTEGERS 1. IF THE SIGNS ARE THE SAME THE ANSWER IS POSITIVE 2. IF THE SIGNS ARE DIFFERENT THE ANSWER IS NEGATIVE.
1 Volume measures and Rebasing of National Accounts Training Workshop on System of National Accounts for ECO Member Countries October 2012, Tehran,
Family Resources Survey Proposals for the treatment of unlinked FRS data* Valerie Christian & Philip Clarke.
Conducted for: Conducted by: December Method Nationwide telephone survey of households Household members age 6+ enumerated Data on tennis participation.
McDonald’s calendar 2009 January
ABC Technology Project Mrs. Kiddle. ABCs of Technology Word 1 Word 2 Word 3 Word 4 Word 5 Word 6 Word 7 Word 8 Word 9 Word 19 Word 20 Word 21 Word 22.
We will resume in: 25 Minutes We will resume in: 24 Minutes.
1 Statistics Canadas Survey Methodology for the New Services Producer Price Index Surveys By:Saad Rais, Statistics Canada Zdenek Patak, Statistics Canada.
+21 Economic Expectations Europe December 2013 Indicator > +20 Indicator 0 to +20 Indicator 0 to -20 Indicator < -20 European Union total: +14 Indicator.
Januar 2005 S M T O T F L
EU-Regional Policy Structural actions 1 GROWING EVALUATION CAPACITY THE MID TERM EVALUATION IN OBJECTIVE 1 AND 2 REGIONS 8 OCTOBER 2004.
1 of 27 DA1241 Archive Companies Last updated: March-2004 DA1241 Archive Companies.
Results from a Mobile Finance Survey. 2 2 Second survey sponsored by CheckFree with fieldwork in April 2008; First survey completed in March ,007.
Graduate Admission System User Guide for Applicants 1 Last updated: April 2014.
Phase II/III Design: Case Study Case study: Seamless Phase II/III Design for Registration Program Jeff Maca, Ph.D. Senior Associate Director Novartis Pharmaceuticals.
European Commission Directorate General Economic and Financial Affairs Harmonisation of BCS: Aggregation methods Roberta Friz Business and consumer surveys.
BT Wholesale October Creating your own telephone network WHOLESALE CALLS LINE ASSOCIATED.
Who volunteers? Volunteering trends based on government data: A briefing from nfpSynergy February 2011 Telephone: (020)
© 2012 National Heart Foundation of Australia. Slide 2.
WEEK 1 You have 10 seconds to name…
Online Privacy Survey Results Conducted: December 2011.
2 Session Objectives Increase participant understanding of effective financial monitoring based upon risk assessments of sub-grantees Increase participant.
PULSE SURVEY The Incentive Industry Trends 2011 June 2011.
Addition 1’s to
CHAPTER 14 Expectations: The Basic Tools Expectations: The Basic Tools CHAPTER 14 Prepared by: Fernando Quijano and Yvonn Quijano Copyright © 2009 Pearson.
Year 6 mental test 5 second questions Multiplication and Division Tables knowledge.
You have been given a mission and a code. Use the code to complete the mission and you will save the world from obliteration…
1 ESTIMATION IN THE PRESENCE OF TAX DATA IN BUSINESS SURVEYS David Haziza, Gordon Kuromi and Joana Bérubé Université de Montréal & Statistics Canada ICESIII.
Weekly Attendance by Class w/e 6 th September 2013.
4/4/2015Slide 1 SOLVING THE PROBLEM A one-sample t-test of a population mean requires that the variable be quantitative. A one-sample test of a population.
Copyright Riello S.p.A. - confidential Andrea Mariottini Riello Gas Boilers Division THE SUCCESSFUL STORY OF CONDENSING BOILERS THE STRATEGY OF BOILERS.
A SMALL TRUTH TO MAKE LIFE 100%. If A B C D E F G H I J K L M N O P Q R S T U V W X Y Z is equal to
Improved Questionnaire Design Yields Better Data: Experiences from the UKs Annual Survey of Hours and Earnings Jacqui Jones, Pete Brodie, Sarah Williams.
Chapter fifteen Media Planning and Buying McGraw-Hill/Irwin Essentials of Contemporary Advertising Copyright © 2007 The McGraw-Hill Companies, Inc. All.
1 Recruitment in a Career-banding World Office of State Personnel July 2008.
Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs Tyler Lu and Craig Boutilier University of Toronto.
1 FY10 ACS Methods Panel Update Jennifer Guarino Tancreto Chief, ACS Data Collection Methods Staff Decennial Statistical Studies Division Presentation.
1 PART 1 ILLUSTRATION OF DOCUMENTS Brief introduction to the documents contained in the envelope Detailed clarification of the documents content.
21-1 Copyright 2010 McGraw-Hill Australia Pty Ltd PowerPoint slides to accompany Croucher, Introductory Mathematics and Statistics, 5e Chapter 21 Hypothesis.
1 4th International Trade Statistics Expert Meeting Methodological issues relating to partner countries Presentation by Poland Room document 2.
NTTS conference, February 18 – New Developments in Nonresponse Adjustment Methods Fannie Cobben Statistics Netherlands Department of Methodology.
© 2017 SlidePlayer.com Inc. All rights reserved.