1. The Effects of Panel Recruitment and Management on Research Results Dutch Online Panel Study NOPVO An initiative of Pieter Willems - Millward Brown Robert van Ossenbruggen - ProCression Ted Vonk - Onderzoekpaleis with the support of the MOA

2. ESOMAR Panel Research 2006 – Barcelona - 2
COPYRIGHT
This PowerPoint presentation contains the outcomes of the Dutch Online Panel Comparison study 2006 (NOPVO 2006).
It is allowed to publish the outcomes of this study if you refer to:
The name of the study (NOPVO 2006)
The authors: Ted Vonk (Onderzoekpaleis), Robert van Ossenbruggen (ProCression) and Pieter Willems (Millward Brown)
Contact:
ESOMAR Panel Research 2006 – Barcelona - 2

3. ESOMAR Panel Research 2006 – Barcelona - 3
Agenda
Introduction
Different approaches, different results?
Does the response rate affect the results?
Does the type of respondent explain the differences between panels?
Conclusions and recommendations
ESOMAR Panel Research 2006 – Barcelona - 3

4. Introduction

5. Some facts and figures about The Netherlands
Demographics
Population: 16,3 million
18-65 years: 10,5 million
Internet penetration: 82%
Broadband penetration: 63% (of all households, not internet users)
Market Research
Total market in 2005: € 278 million ($ 346 – 12th largest market)
Online data collection: 25% of quantitative revenue
Over 25 commercial online access panels
Majority of panels exist for longer than 5 years
ESOMAR Panel Research 2006 – Barcelona - 5

6. The Dutch Online Panel Study (NOPVO)
Why?
To provide an answer to quality concerns
To provide input for guidelines for online panel innovation
A unique situation: large number of panels available
Research questions - NOPVO
How do panels work? Panel overview
Who is the online panel member?
How do panellists respond?
To what extent is opinion and marketing data influenced by the choice of panel?
No objective: to qualify good or bad panels
ESOMAR Panel Research 2006 – Barcelona - 6

7. ESOMAR Panel Research 2006 – Barcelona - 7
Two parts of NOPVO
Inventory of all online panels in the Netherlands
Answers to the 25 questions of ESOMAR
Overview available at
Empirical study
One survey
Every panel draws sample of 1000 respondents
Same fieldwork period of 7 days (no reminders)
Omnibus questionnaire to avoid selective response
19 panels participated:
Representing over panellists
ESOMAR Panel Research 2006 – Barcelona - 7

8. Extensive data available
Per panel:
Type of incentive
Age of Panel
Invitation policy
etc.
Per panel member:
Duration of panel membership
Method of recruitment
Number of invitations
Individual response rate
Socio demos, etc.
Per respondent:
Response (time)
Membership of other panels
Data from the questionnaire
ESOMAR Panel Research 2006 – Barcelona - 8

9. Some facts about participating panels
Almost all panels give incentives for cooperation
Usage of different types of incentives
Average € 1,20 for 10 minutes
Most panels have a restriction policy for inviting panel members to complete a survey
million panel members
Smallest panel:15.000
Largest panel:
Several panels do not have full details available about interview history of panel members
ESOMAR Panel Research 2006 – Barcelona - 9

10. Observations about the samples
Composition of samples to duration of panel membership differs a lot per panel
Panels with mainly respondents recruited during last 6 months
Panels with mainly panel members recruited during set up of panel
Recruitment methods vary substantially across panels
Most panels use a mixture of recruitment methods
Some panels use one single method (online, by telephone)
ESOMAR Panel Research 2006 – Barcelona - 10

11. Different approaches, different results?

12. Differences in results between panels limited
There will always be some fluctuations because of sampling (sampling error)
Most of the differences between panels are about what one could expect because of confidence intervals
There are huge differences between extreme scores
Extreme scores lead to different conclusions
Change of panel could lead to change in trends
ESOMAR Panel Research 2006 – Barcelona - 12

13. Substantial differences caused by a few panels with extreme scores
How to read the graph
Largest differences (N=19)
Minimum and maximum excluded (N=17)
2 minima and maxima excluded (N=15)
Extreme score are spread over all panels
Lowest number of extreme scores: 1%
Highest number of extreme scores 12%
ESOMAR Panel Research 2006 – Barcelona - 13

14. Does the response rate affect the results?

15. NOPVO response overall: 50%
Level of response is a result of several factors
Panel age, method of recruitment, incentives, panel care, etc
It is hard to determine the effects of a single factor
NOPVO response overall: 50%
panel members invited… completed the questionnaire…
Substantial differences in response per panel
ESOMAR Panel Research 2006 – Barcelona - 15

16. High/low response explained(1)
Panels with a low response rate are panels that do not remove non-responding panellists from their panel database
Panels invited panel members who did not complete a single questionnaire in over a year despite many invitations
Number of panel members is an important communication topic to the market
New panel members respond better than old panel members
Effect of incentives
Certainty about incentive is important
Positive effect of incentive stops at 1 euro (10 minute questionnaire)
ESOMAR Panel Research 2006 – Barcelona - 16

17. High/low response explained(2)
Method of recruitment:
Recruitment by invitation (by telephone, traditional research) gives higher response than self selection (self registration, links/banners)
ESOMAR Panel Research 2006 – Barcelona - 17

18. Outcomes do not depend on response rate
No different means on survey questions between low/high responding panels
Number of …
Panels with response <40%
Panels with response 40-60%
Panels with response
>60%
Words in open ended question
24,7
25,7
25,9
Awareness beer brands (spont.)
6,1
6,3
Awareness beer brands (aided)
20,1
19,8
20,0
Glasses beer per week
11,1
10,9
10,4
Advertisements seen
5,8
Awareness TV brands (aided)
13,8
13,5
ESOMAR Panel Research 2006 – Barcelona - 18

19. No response effects explained
Non response stages
During recruitment
Removal of bad/non responding panel members
For a specific survey
Non-response effects only occur if there are differences between respondents and non-respondents
There is no distortion because of missing categories (they are missing in all panels)
Members of online panels are a homogeneous group
They are willing to participate in surveys
They are heavy internet users
Non-response effects take place while recruiting panels
double opt-in
explicit consent to participate in research
ESOMAR Panel Research 2006 – Barcelona - 19

20. Representative power of panels is more limited than assumed
Missing groups in panels
Ethnic minorities/immigrants are missing almost completely (10% of population, 35% in major cities)
They are missing in all data collection methods
Structural deviations in NOPVO study
Too many heavy Internet users
Too many voters (90% instead of 79%)
Voting behaviour compared with last elections
Number of Christian Democrat voters (16% instead of 29%)
Too many Socialist Party voters (14% instead of 6%)
People who go to church are underrepresented
Large part of the population is not represented in the panels
ESOMAR Panel Research 2006 – Barcelona - 20

21. Response rate: panel strategy
A high response rate does NOT indicate high sample quality
Response rate is a panel strategy component, driven by rules of economics
Panel setup and maintenance determine response rate
Value of response rate is evaluated against recruitment costs
Response rate can be influenced by
Excluding hard to reach groups
Selection of good i.e responding panellists
The response problems of traditional research are not solved. Online research is dealing with it more efficient.
ESOMAR Panel Research 2006 – Barcelona - 21

22. Panel overlap diminishes panel management effects
62% of respondents are a member of multiple panels
Average number of memberships of all respondents: 2,73 panels
Panel overlap varies between 29% and 88% per panel
Highest single overlap between two panels: 42%
With this level of overlap a restriction policy for inviting panellists does not make much sense
ESOMAR Panel Research 2006 – Barcelona - 22

23. Highest overlap with addresses from a broker
81% of panellists recruited with bought addresses are members of multiple panels
Recruitment by invitation (telephone, traditional research) smallest overlap
Average number of panels varies between 2,0 recruitment by telephone and 4,3 by bought addresses
ESOMAR Panel Research 2006 – Barcelona - 23

24. 20%-80% rule is valid for online panel research
Multiple panel members respond better and faster
Limited group of panellists complete substantial part of all questionnaires
ESOMAR Panel Research 2006 – Barcelona - 24

25. Does the type of respondent explain differences between panels?

26. ESOMAR Panel Research 2006 – Barcelona - 26
“It's the perfect job for women who want to stay at home with their children, or anyone who needs extra money they would like to earn from home.”
“By joining this site, you are joining a community of professional survey-takers with information on the best, paid survey sites in the UK, links to UK survey news, bulletin boards and advice on the business of online paid surveys.”
(source:
ESOMAR Panel Research 2006 – Barcelona - 26

27. Nice to meet you: the Professional Respondent
Member of many panels
Lots of questionnaires completed
Operationalization NOPVO:
‘index professionalism’=
# panels membership (Z) +
# questionnaires completed last 4 weeks (Z)
Selection for analysis: top 10%
ESOMAR Panel Research 2006 – Barcelona - 27

28. Nice to meet you: the Loyal Respondent
The ‘loyal’ respondent: takes time to complete questionnaire and puts effort in open ended questions
Operationalization NOPVO:
‘index loyalty’=
duration of interview (Z) +
# words open ended questions (Z)
Selection for analysis: top 10%
ESOMAR Panel Research 2006 – Barcelona - 28

29. Nice to meet you: the Inattentive Respondent
The inattentive respondent: opposite of loyal respondent
Thus: speeds through questionnaire and minimum input open ended questions
Operationalization NOPVO:
‘Index inattentiveness’= index loyalty * -1
Selection for analysis: top 10%
ESOMAR Panel Research 2006 – Barcelona - 29

30. Present typology analogous to earlier studies
NOPVO
Similar to…
Harris1
Doxus2
Professional
Hyperactives
Loyal
Optimizers
Inattentive
Inattentives
Satisficers
Fraudulents
Gamers
1 Smith, R. & Hofma Brown, H., (2005). Assessing the Quality of Data from Online Panels: Moving Forward with Confidence. White paper, Harris Interactive.
2 Downes-Le guin, T. (2005). Satisficing Behavior in Online Panelists. White paper, Doxus.
also relevant in this discussion:
Knapton, K. (2006). Catch Me If You Can! (How to Keep Cheaters Out of Your Next Online Survey). Presentation at The Market Research Event. Los Angeles, October 23, 2006.
ESOMAR Panel Research 2006 – Barcelona - 30

31. Respondent type & characteristics
Professional respondent
internet frequency 
‘fresh’ panels 
education 
unfit for work 
satisfaction income 
democracy 
perceived health 
Loyal respondent
Inattentive respondent
older respondents 
ESOMAR Panel Research 2006 – Barcelona - 31

32. Recruitment: loyals from traditional data collection methods
recruitment panellist (data available of 12 panels)
inattentive:
Self select
loyal:
By invitation
professional:
Self selection
ESOMAR Panel Research 2006 – Barcelona - 32

33. Motivation: incentive or intrinsic
Reasons for participating in internet surveys. Which one is the most important to you?
mening kwijt
leuk om nieuwe ideeën en producten te zien
nieuwsgierigheid
onderwerpen interessant
leuk om de resultaten te zien
leuk om onderdeel uit te maken van online groep
*Such as:
Curiosity
I like to give my opinion
I like to see results
I like to be part of an online
community
inattentive:
incentive
driven
loyal:
intrinsic
motivations
professional:
nice to do
ESOMAR Panel Research 2006 – Barcelona - 33

34. Political interest: large differences
To what extent are you interested in politics?
inattentive:
weak
political
interest
loyal:
strong
professional: no
effect
ESOMAR Panel Research 2006 – Barcelona - 34

35. Add familiarity: differences between types up to 15%...
Largest effect for inattentive respondents
Professional respondent just as atypical as loyal respondents
(zero =
total)
ESOMAR Panel Research 2006 – Barcelona - 35

36. Brand familiarity: differences between types up to 20%...
ESOMAR Panel Research 2006 – Barcelona - 36

37. Inattentive respondents: the easy way out
ESOMAR Panel Research 2006 – Barcelona - 37

38. Main concern: the inattentive respondent
Q: Is the concern for the professional respondent justified?
A: Partly, because this type of respondent is atypical, but the effects are not large
Q: Is the loyal respondent also the ideal respondent?
A: This is material for further investigation
Serious effort (by definition)
High data quality
But: atypical… now who really is atypical? The loyal or the not so loyal?
Q: Is the inattentive respondent a threat to online panel research?
A: Absolutely: atypical group with large effects responsible for noise in data
ESOMAR Panel Research 2006 – Barcelona - 38

39. Respondent behaviour varies between panels
relatively few
inattentive
respondents
relatively many
inattentive
respondents
ESOMAR Panel Research 2006 – Barcelona - 39

40. Conclusions

41. Conclusions
In general: the differences in outcomes between panels are limited despite differences in panel management and panel maintenance
However, there are differences
The differences are not a result of differences in response or sample composition.
Respondent types, defined by questionnaire experience or effort, display more or less atypical and suspect response behaviour
Differences between panels can substantially be explained by differences in distribution of these respondent types
Panels comprise a specific group of respondents that differ on relevant criteria from the national population. The representative power of online panels is more limited than assumed so far
Response percentage does not indicate sample or panel quality. The response rate is an indication of the level of efficiency of the panel provider.
Online panels have a substantial overlap in terms of panellists. This reduces the impact of a panel management strategy.

42. ESOMAR Panel Research 2006 – Barcelona - 42
Recommendations
Monitor suspect respondents and remove the really bad ones
Deploy other, new recruitment methods to get the missing part of the population in the panels
Use online panels for the right kind of research goals
use panels for
continuous research or trend measurement
concept testing
but not for
incidence rating
ESOMAR Panel Research 2006 – Barcelona - 42

43. Practice makes perfect
Thank You