Presentation on theme: "ESOMAR Panel Research 2006 Barcelona The Effects of Panel Recruitment and Management on Research Results Dutch Online Panel Study NOPVO 2006 A n initiative."— Presentation transcript:
ESOMAR Panel Research 2006 Barcelona The Effects of Panel Recruitment and Management on Research Results Dutch Online Panel Study NOPVO 2006 A n initiative of Pieter Willems - Millward Brown Robert van Ossenbruggen - ProCression Ted Vonk - Onderzoekpaleis with the support of the MOA
ESOMAR Panel Research 2006 – Barcelona - 2 COPYRIGHT This PowerPoint presentation contains the outcomes of the Dutch Online Panel Comparison study 2006 (NOPVO 2006). It is allowed to publish the outcomes of this study if you refer to: The name of the study (NOPVO 2006) The authors: Ted Vonk (Onderzoekpaleis), Robert van Ossenbruggen (ProCression) and Pieter Willems (Millward Brown) Contact:
ESOMAR Panel Research 2006 – Barcelona - 3 Agenda Introduction Different approaches, different results? Does the response rate affect the results? Does the type of respondent explain the differences between panels? Conclusions and recommendations
ESOMAR Panel Research 2006 Barcelona Introduction
ESOMAR Panel Research 2006 – Barcelona - 5 Some facts and figures about The Netherlands Demographics Population: 16,3 million years:10,5 million Internet penetration: 82% Broadband penetration: 63% (of all households, not internet users) Market Research Total market in 2005: 278 million ($ 346 – 12th largest market) Online data collection: 25% of quantitative revenue Over 25 commercial online access panels Majority of panels exist for longer than 5 years
ESOMAR Panel Research 2006 – Barcelona - 6 The Dutch Online Panel Study (NOPVO) Why? 1.To provide an answer to quality concerns 2.To provide input for guidelines for online panel innovation 3.A unique situation: large number of panels available Research questions - NOPVO 1.How do panels work? Panel overview 2.Who is the online panel member? 3.How do panellists respond? 4.To what extent is opinion and marketing data influenced by the choice of panel? No objective: to qualify good or bad panels
ESOMAR Panel Research 2006 – Barcelona - 7 Two parts of NOPVO 1. Inventory of all online panels in the Netherlands 1.Answers to the 25 questions of ESOMAR 2.Overview available at 2.Empirical study One survey Every panel draws sample of 1000 respondents Same fieldwork period of 7 days (no reminders) Omnibus questionnaire to avoid selective response 19 panels participated: Representing over panellists
ESOMAR Panel Research 2006 – Barcelona - 8 Extensive data available Per panel: -Type of incentive -Age of Panel -Invitation policy -etc. Per panel member: -Duration of panel membership -Method of recruitment -Number of invitations -Individual response rate -Socio demos, etc. Per respondent: -Response (time) -Membership of other panels - Data from the questionnaire
ESOMAR Panel Research 2006 – Barcelona - 9 Some facts about participating panels Almost all panels give incentives for cooperation Usage of different types of incentives Average 1,20 for 10 minutes Most panels have a restriction policy for inviting panel members to complete a survey million panel members Smallest panel: Largest panel: Several panels do not have full details available about interview history of panel members
ESOMAR Panel Research 2006 – Barcelona - 10 Observations about the samples Composition of samples to duration of panel membership differs a lot per panel Panels with mainly respondents recruited during last 6 months Panels with mainly panel members recruited during set up of panel Recruitment methods vary substantially across panels Most panels use a mixture of recruitment methods Some panels use one single method (online, by telephone)
ESOMAR Panel Research 2006 Barcelona Different approaches, different results?
ESOMAR Panel Research 2006 – Barcelona - 12 Differences in results between panels limited There will always be some fluctuations because of sampling (sampling error) Most of the differences between panels are about what one could expect because of confidence intervals There are huge differences between extreme scores Extreme scores lead to different conclusions Change of panel could lead to change in trends
ESOMAR Panel Research 2006 – Barcelona - 13 Substantial differences caused by a few panels with extreme scores How to read the graph Largest differences (N=19) Minimum and maximum excluded (N=17) 2 minima and maxima excluded (N=15 ) Extreme score are spread over all panels Lowest number of extreme scores: 1% Highest number of extreme scores 12%
ESOMAR Panel Research 2006 Barcelona Does the response rate affect the results?
ESOMAR Panel Research 2006 – Barcelona - 15 NOPVO response overall: 50% Level of response is a result of several factors Panel age, method of recruitment, incentives, panel care, etc It is hard to determine the effects of a single factor NOPVO response overall: 50% panel members invited… completed the questionnaire… Substantial differences in response per panel
ESOMAR Panel Research 2006 – Barcelona - 16 High/low response explained(1) Panels with a low response rate are panels that do not remove non- responding panellists from their panel database Panels invited panel members who did not complete a single questionnaire in over a year despite many invitations Number of panel members is an important communication topic to the market New panel members respond better than old panel members Effect of incentives Certainty about incentive is important Positive effect of incentive stops at 1 euro (10 minute questionnaire )
ESOMAR Panel Research 2006 – Barcelona - 17 High/low response explained(2) Method of recruitment: Recruitment by invitation (by telephone, traditional research) gives higher response than self selection (self registration, links/banners)
ESOMAR Panel Research 2006 – Barcelona - 18 Outcomes do not depend on response rate Number of … Panels with response <40% Panels with response 40-60% Panels with response >60% Words in open ended question24,725,725,9 Awareness beer brands (spont.)6,1 6,3 Awareness beer brands (aided)20,119,820,0 Glasses beer per week11,110,910,4 Advertisements seen5,8 Awareness TV brands (aided)13,813,513,8 No different means on survey questions between low/high responding panels
ESOMAR Panel Research 2006 – Barcelona - 19 No response effects explained Non response stages During recruitment Removal of bad/non responding panel members For a specific survey Non-response effects only occur if there are differences between respondents and non-respondents There is no distortion because of missing categories (they are missing in all panels) Members of online panels are a homogeneous group They are willing to participate in surveys They are heavy internet users Non-response effects take place while recruiting panels double opt-in explicit consent to participate in research
ESOMAR Panel Research 2006 – Barcelona - 20 Representative power of panels is more limited than assumed Missing groups in panels Ethnic minorities/immigrants are missing almost completely (10% of population, 35% in major cities) They are missing in all data collection methods Structural deviations in NOPVO study Too many heavy Internet users Too many voters (90% instead of 79%) Voting behaviour compared with last elections >Number of Christian Democrat voters (16% instead of 29%) >Too many Socialist Party voters (14% instead of 6%) People who go to church are underrepresented Large part of the population is not represented in the panels
ESOMAR Panel Research 2006 – Barcelona - 21 Response rate: panel strategy A high response rate does NOT indicate high sample quality Response rate is a panel strategy component, driven by rules of economics Panel setup and maintenance determine response rate Value of response rate is evaluated against recruitment costs Response rate can be influenced by Excluding hard to reach groups Selection of good i.e responding panellists The response problems of traditional research are not solved. Online research is dealing with it more efficient.
ESOMAR Panel Research 2006 – Barcelona - 22 Panel overlap diminishes panel management effects 62% of respondents are a member of multiple panels Average number of memberships of all respondents: 2,73 panels Panel overlap varies between 29% and 88% per panel Highest single overlap between two panels: 42% With this level of overlap a restriction policy for inviting panellists does not make much sense
ESOMAR Panel Research 2006 – Barcelona - 23 Highest overlap with addresses from a broker 81% of panellists recruited with bought addresses are members of multiple panels Recruitment by invitation (telephone, traditional research) smallest overlap Average number of panels varies between 2,0 recruitment by telephone and 4,3 by bought addresses
ESOMAR Panel Research 2006 – Barcelona %-80% rule is valid for online panel research Multiple panel members respond better and faster Limited group of panellists complete substantial part of all questionnaires
ESOMAR Panel Research 2006 Barcelona Does the type of respondent explain differences between panels?
ESOMAR Panel Research 2006 – Barcelona - 26 It's the perfect job for women who want to stay at home with their children, or anyone who needs extra money they would like to earn from home. By joining this site, you are joining a community of professional survey-takers with information on the best, paid survey sites in the UK, links to UK survey news, bulletin boards and advice on the business of online paid surveys. (source:
ESOMAR Panel Research 2006 – Barcelona - 27 Nice to meet you: the Professional Respondent Member of many panels Lots of questionnaires completed Operationalization NOPVO: index professionalism= 1.# panels membership (Z) + 2.# questionnaires completed last 4 weeks (Z) Selection for analysis: top 10%
ESOMAR Panel Research 2006 – Barcelona - 28 Nice to meet you: the Loyal Respondent The loyal respondent: takes time to complete questionnaire and puts effort in open ended questions Operationalization NOPVO: index loyalty= 1.duration of interview (Z) + 2.# words open ended questions (Z) Selection for analysis: top 10%
ESOMAR Panel Research 2006 – Barcelona - 29 Nice to meet you: the Inattentive Respondent The inattentive respondent: opposite of loyal respondent Thus: speeds through questionnaire and minimum input open ended questions Operationalization NOPVO: Index inattentiveness= index loyalty * -1 Selection for analysis: top 10%
ESOMAR Panel Research 2006 – Barcelona - 30 Present typology analogous to earlier studies 1 Smith, R. & Hofma Brown, H., (2005). Assessing the Quality of Data from Online Panels: Moving Forward with Confidence. White paper, Harris Interactive. 2 Downes-Le guin, T. (2005). Satisficing Behavior in Online Panelists. White paper, Doxus. also relevant in this discussion: Knapton, K. (2006). Catch Me If You Can! (How to Keep Cheaters Out of Your Next Online Survey). Presentation at The Market Research Event. Los Angeles, October 23, NOPVOSimilar to…Harris 1 Doxus 2 ProfessionalHyperactives LoyalOptimizers InattentiveInattentivesSatisficers FraudulentsGamers
ESOMAR Panel Research 2006 – Barcelona - 31 Respondent type & characteristics Professional respondent internet frequency fresh panels education unfit for work satisfaction income democracy perceived health Loyal respondent education Inattentive respondent older respondents
ESOMAR Panel Research 2006 – Barcelona - 32 Recruitment: loyals from traditional data collection methods inattentive: Self select loyal: By invitation professional: Self selection recruitment panellist (data available of 12 panels)
ESOMAR Panel Research 2006 – Barcelona - 33 Motivation: incentive or intrinsic inattentive: incentive driven loyal: intrinsic motivations professional: nice to do Reasons for participating in internet surveys. Which one is the most important to you? *Such as: Curiosity I like to give my opinion I like to see results I like to be part of an online community
ESOMAR Panel Research 2006 – Barcelona - 34 Political interest: large differences inattentive: weak political interest loyal: strong political interest professional: no effect To what extent are you interested in politics?
ESOMAR Panel Research 2006 – Barcelona - 35 Add familiarity: differences between types up to 15%... Largest effect for inattentive respondents Professional respondent just as atypical as loyal respondents (zero = total)
ESOMAR Panel Research 2006 – Barcelona - 36 Brand familiarity: differences between types up to 20%...
ESOMAR Panel Research 2006 – Barcelona - 37 Inattentive respondents: the easy way out
ESOMAR Panel Research 2006 – Barcelona - 38 Main concern: the inattentive respondent Q: Is the concern for the professional respondent justified? A: Partly, because this type of respondent is atypical, but the effects are not large Q: Is the loyal respondent also the ideal respondent? A: This is material for further investigation Serious effort (by definition) High data quality But: atypical… now who really is atypical? The loyal or the not so loyal? Q: Is the inattentive respondent a threat to online panel research? A: Absolutely: atypical group with large effects responsible for noise in data
ESOMAR Panel Research 2006 – Barcelona - 39 Respondent behaviour varies between panels relatively few inattentive respondents relatively many inattentive respondents
ESOMAR Panel Research 2006 Barcelona Conclusions
In general: the differences in outcomes between panels are limited despite differences in panel management and panel maintenance However, there are differences The differences are not a result of differences in response or sample composition. Respondent types, defined by questionnaire experience or effort, display more or less atypical and suspect response behaviour Differences between panels can substantially be explained by differences in distribution of these respondent types Panels comprise a specific group of respondents that differ on relevant criteria from the national population. The representative power of online panels is more limited than assumed so far Response percentage does not indicate sample or panel quality. The response rate is an indication of the level of efficiency of the panel provider. Online panels have a substantial overlap in terms of panellists. This reduces the impact of a panel management strategy.
ESOMAR Panel Research 2006 – Barcelona - 42 Recommendations Monitor suspect respondents and remove the really bad ones Deploy other, new recruitment methods to get the missing part of the population in the panels Use online panels for the right kind of research goals use panels for continuous research or trend measurement concept testing but not for incidence rating
ESOMAR Panel Research 2006 Barcelona Practice makes perfect Thank You