Presentation is loading. Please wait.

Presentation is loading. Please wait.

Survey Routers: A Primer Nancy Brigham, Ph.D., VP Global Operations, Ipsos Michael Fallig, Ph.D., Managing Director, Cogenti Applied Strategies April 20,

Similar presentations


Presentation on theme: "Survey Routers: A Primer Nancy Brigham, Ph.D., VP Global Operations, Ipsos Michael Fallig, Ph.D., Managing Director, Cogenti Applied Strategies April 20,"— Presentation transcript:

1 Survey Routers: A Primer Nancy Brigham, Ph.D., VP Global Operations, Ipsos Michael Fallig, Ph.D., Managing Director, Cogenti Applied Strategies April 20, 2012 © 2012 Ipsos. All rights reserved. Contains Ipsos' Confidential and Proprietary information and may not bedisclosed or reproduced without the prior written consent of Ipsos.

2 Agenda Online research today – why the move to routers? Major benefits & uses of routers Key terminology / concepts History / timeline of routers & common misconceptions How routing works: big picture Risks with routers Ensuring good sample quality Research-on-research Q&A

3 State of Online Research Today 3 We put people on panels and target them directly to a specific study InefficientLots of non- qualifiers Changing Market / Internet Environment Declining response rates Decline in panel joiners Changing Internet landscape and behaviors Rapid growth of online studies Traditional Online Research

4 Moving from Today to Tomorrow 4 Routers and Non-Panel Online Samples Key challenge facing the online industry today: Sample Capacity Consequences Inefficient use of sample – capacity not optimized Respondent engagement suffers Less reflectivity of the general population Where are we going?

5 History / Timeline of Routers Routers arent new – theyve been around since 1996! DMS creates the first router 1998 Early routers are created (e.g., OTX, Greenfield Online) Routers gain in popularity with clients and suppliers Quality and sampling considerations become more important Late 2000s Major suppliers start to create their own routers or more frequently buy routed sample 2010 Ipsos buys OTX – Ipsos panel is integrated into router 2011 ARF launches industry router quality initiative as part of FOQ2 Online survey research takes off

6 Common Misconceptions About Routers Routers mean sample blending Routers mean river sampling Routers are a black box 6 Routers river River sample needs routers, but routers dont need river. Routers sample blending Routers sample blending Routers are sample agnostic – they can use just one, or many sources. Routers are simply a technology that allows us to route respondents to a survey Sample sourcing is a separate issue Routers black box We can follow each respondent in the router and see exactly what happened to them

7 Routers: In Depth…. Routers are the technology that allow us to manage multiple surveys and potentially multiple sources within a closed, controlled system. Routers allow us to group surveys and manage them using a certain, consistent approach. Allocates sample across a set of studies, not just a single study Rules are used to assign respondents to one or more of a group of active surveys, based on research and sample capacity needs. Enable the efficient use of multiple sample sources (although multiple sources are not a requisite) Blending: the use of multiple sample sources within a single study Routers arent necessary for sample blending, but they make the process much more efficient and introduce better control over the entire survey system 7

8 Some Key Terminology/Concepts…. 8 Router Technology that lets us send respondents from one study to another Reallocation Respondents who dont qualify for a survey can attempt to qualify for other surveys that are open at that time. Blended sample A sample that contains multiple sources (multiple panels, multiple non-panels, or mixture of both). Non-panel sources Sample obtained directly from the Internet Social media, communities, reward/loyalty programs, ad networks, pop-ups, etc.

9 Some Key Terminology/Concepts…. 9 Parallel & serial router Serial: Respondents attempt to qualify for studies one at a time. Parallel: Respondents answer screening questions that will allow them to pre-qualify for multiple studies. Router selection algorithm How studies are selected by the router for a respondent Random, priority, hybrid Mini- routers Smaller routers that are set up for some specific purpose Usually done to manage risk & get greater efficiencies Screening process Potentially get screening questions from multiple studies

10 How Routing Works: Big Picture 10 SOURCE 1 SOURCE 2 SOURCE 6 SOURCE 5 SOURCE 4 SOURCE 3 SOURCE 7 Multiple Sample Sources – Panel & Non-Panel B A Screening for Survey… Survey D Survey E Sent to Survey Qualified For D C E Router – Survey Qualification Survey C Survey B Survey A The router is the engine that lets us manage all the sample sources for a study could be a single source, or blended sources

11 Major Benefits of Routers 11 Increased Sample Capacity & Population Reach Easier Survey Adaptation to New Online Realities Allows studies to be tailored to people accessing them using different devices (e.g., mobile) More efficient use of sample in general, and especially lower- response demographics Access to parts of the population that dont traditionally join panels Increased respondent engagement and satisfaction Even better control over operational quality aspects (e.g., automated) Increased Sample Capacity & Population Reach Higher Quality (Data & Respondents)

12 How Businesses are Using Routers How sample providers are using routers Maximize efficiency Increase productivity of each sample respondent Efficiently manage the sample coming from a variety of sources Increase the consistency of processes across studies More efficient method for monitoring multiple live projects Access and provide respondents Accommodate the needs of many different clients How research organizations are using routers Increase feasibility of conducting studies Provide higher quality of data & respondent satisfaction Maximize efficiency Manage category exclusions, quotas, lockouts, etc., more efficiently Increase productivity of each sample respondent Used primarily for own clients Targeted mini-routers (e.g., ethnic, client-based, study type) Increase consistency across studies 12

13 Managing Routers There are a variety of ways of managing routers The buyer needs to be observant of this Major aspects to consider 13 Sampling Degree of randomization in survey selection How screeners are presented to respondents How priority is managed How multiple sources are accessed and managed Environment Number & diversity of studies in the router Rules to ensure one study doesnt bias another Monitoring of the environment

14 Risks with Routers 14 Selection bias Incidence – ability to get and report incidence correctly Sampling/Routing Who has responsibility for router decisions Common screening environment Router rules and best practices Business pressures (such as very low cost) Environment Use proper sources/recruiting Inconsistent use of sources when consistency is needed Types of studies Sources

15 Ensuring Good Sample Quality with Routers With the right rules and controls in place, sample quality can be even higher with routers Much easier to monitor one environment with 300 studies than 300 environments with one study each Routers automate many of the controls and monitoring Benchmarking and parallel testing to measure sample consistency Keeping track of panelist experience in router Routers can record every single thing that happens to a respondent while in the router Also do quality control at back-end (through quotas at survey level) 15

16 Research-on-Research with Routers Route 66 The Long Road to Efficient and Effective Router Use (Miller, 2010) Presented at 15 th Annual CASRO Technology Conference, June 3,2010 Covers priority/randomization, taking multiple surveys in one sitting, effect of screener position Factorial Design on Survey Router Effects (Johnson & Fawson, 2010) Presented at the 2010 CASRO Panel Conference in Las Vegas Covers taking multiple surveys in one sitting, panelist fatigue Measuring Selection Bias Introduced by Routing (Porter, Scott, de Guademar & Kimura, 2010) Presented at the 2010 CASRO Panel Conference in Las Vegas Covers selection bias due to reallocation Sampling with Routers: Comparing Survey Results of Reallocated and Traditionally Sampled Respondents (Brigham, Porter, Markowitz & Fuller, 2011) Presented at the 2011 CASRO Technology Conference Covers priority/random assignment, selection bias due to reallocation, percent of sample reallocated in a single study, bias effect due associated with studies of low priority Survey Router Management: An Experimental Examination of the Impact on Survey Results (Brigham & Fuller, 2012) Presented at the 2012 CASRO Online Conference in Las Vegas Covers number of studies on router, correlation of studies on router, time spent in router being screened 16

17 FoQ 2 Router Initiative

18 Mission & Goals Team Mission Expand industry knowledge about routers by understanding the effects of practices on research outputs. Provide a level playing field in terms of transparency between clients and router providers. Team Goals » Document current router practices, key variables and terminology » Examine router optimization practices, evaluate router impact on sample survey results and overall quality; begin mapping out appropriate practices To best accomplish its mission and goals, the Router team has partitioned itself into three groups with distinct responsibilities… 18

19 Router Subgroups Subgroup Objectives » Group 1: Identify current practices including key variables that can impact sample, respondent and overall data quality. Design topic guides for Focus groups and compile participant lists to examine current practices Compile existing R-on-R and summarize Identify the universe of router variables around which practices may differ, including upstream and downstream variables that are not within the teams present scope to examine. Work with independent consultant to develop a report of findings regarding the primary research effort 19

20 Router Subgroups Subgroup Objectives (contd) » Group 2: Design and conduct primary research to assess router optimization practices, configurations and sample re-allocation practices. Study design includes a simulation component and a database that can be used for additional explorations. Independent consultant analyzes results » Group 3: Develop a set of generic router schematics to aid end users of the technology and those who buy routed sample – to aid the understanding of the router process. Establish a set of appropriate router use practices Develop a set of questions that buyers can ask when licensing routers or purchasing routed sample Compile a dictionary of router terms 20

21 Team Members, Group Affiliation & Oversight Leadership » Michael Fallig, Cogenti Applied Strategies (co-chair) (Group 2) » Chuck Miller, dm2corp (co-chair) (Group 3) » John Bremer, Toluna (Group 2) » Nancy Brigham, Ipsos (Group 2 leader) » Pete Cape, SSI (Group 3) » Mike Conklin, MarketTools (Group 2) » Steve Gittelman, MKTG Inc. Group 1) » Paul Johnson, Opinionology/SSI (Group 1) » Peter Milla, CASRO (Group 3) » Robin Murphy, Ipsos (Group 2) » Efrain Ribeiro, Kantar Lightspeed (Group 1) Oversight Leadership » Bill Cook, ARF » Don Gloeckler, ARF » Gian Fulgoni, comScore 21

22 FoQ 2 Research Priority 7: Frame current router practices, key variables & research implications Group 1, Core Activity: Two Focus Groups: » Practitioner Focus Group (in-person) Three areas of expertise, three participants from each from: IT, Operations and Sample Frame Questions prepared by the subgroup with full team input Group discussion completed: Comprised of 15+ members, convened at ARF headquarters, Reg Baker, MSI moderator, transcripts in production » End User Focus Group (in-person with virtual participation) Wider cross-section of participants by role in the research process, including client facing individuals Gather reactions and further guidance from users/buyers of router sample Objective: Obtain expert input and end-user key considerations. Learning advances committee deliverables, particularly with regard to existing practices, user needs and buying decisions. It will also provide information that could refine aspects of the existing research design and variables of exploration. 22

23 FoQ2 Research Priority 7 : Router optimization practices & router impact on sample & survey results Group 2, Core Activity: Quantitative Research Study Objectives: » Assess important variables, unique to routing, around which the industry can establish appropriate practices » Test different router configurations (e.g., total random, total priority) for impact on outcome metrics (e.g., efficiency, demographics, survey results – configurations and output metrics TBD) » Assess impact of various reallocation and sample re-use strategies on response rates, completion rates, satisfaction with the experience, level of engagement throughout survey, survey results » Examine variables within a controlled environment to reduce confounding and difficulty in assessing a particular variables impact » Provide results that can lead to a set of recommendations and appropriate practices 23

24 Group 2, Core Activity: Quantitative Research Study Design Elements: » Experimental design employing a mini-router, populated with projects of our choosing. » Router environment will simulate 15 studies, distribution of studies will reflect those found on existing routers today. » Live panelists sourced from four panels will complete 5 full studies on the router. Screening criteria for 10 other studies will also be completed. » Additional sample from the same sources will be assigned to complete 5 online surveys via traditional invites sent to inboxes. These will serve as parallel test control groups for each of the 5 studies on the router. » Certain independent variables under study will be experimentally manipulated, information about others will be produced via simulation. » Variables to study have not been finalized. Group 1 activities will help inform design. ARF membership is encouraged to offer suggestions regarding variables to explore » For this initial project, all sample will be sourced from four existing consumer access panels 24 FoQ2 Research Priority 7 : (contd)

25 Group 2, Core Activity: Quantitative Research Study Summary of Key Elements: # & type of surveys tested 5 – (a) customer satisfaction, (b) omnibus, (c) U&A, (d) concept test, and (e) TBD # of sample sources 4 sources per survey – 25% each. Proportion same for each survey. Quotas set on supplier. Sample source will be blinded. We anticipate only panel sample will be used, to remove panel vs. non-panel as a potential confounding variable. Completes per survey n=2500 for each full study on router n=1,000 for each of 5 parallel test control groups, N=17,500 in total Full Survey length20-30 minutes on average Additional screening criteria Need to develop screening questions for an additional 10 studies (e.g., trackers, copy) Sample frame – demo quotas Age, Gender, Region, Education (within supplier) # of samplesFor router: 20 samples in total – 5 surveys x 4 sources. 5 samples of 625 per supplier source For parallel test: 5 samples of 250 completes per supplier source. 25 FoQ2 Research Priority 7: (contd)

26 Group 2, Core Activity: Quantitative Research Study Controlled Mini-Router Environment / Router Simulation (MRE) + Parallel to Inbox Invites Control Groups (PCG) Potential Examinations: » Explore useful limits/parameters of several key router variables (MRE groups) » Evaluate appropriate practices range (MRE groups) » Test impact of different router configurations (MRE groups) » Examine & compare response rates & completion rates (MRE vs. PCG) » Calculate & compare incidence (MRE vs. PCG) » Examine & compare distributions of key demo and response measures (MRE vs. PCG) » Measure speeding & straight lining (MRE vs. PCG) » Measure respondent satisfaction (MRE vs. PCG) 26 FoQ2 Research Priority 7: (contd)

27 Potential Key Variables to Assess Independent variables (examined via manipulation or simulation) Priority vs. Random Assignment Number of studies in router Incidence rates of studies on router Correlation among screening questions Manipulate number of surveys someone can take in one sitting? Number of times someone could be reallocated? (Limit number of touches) Dependent variables Response rate, completion rate, over quotas, incidence calculations Efficiency metrics – capacity lift due to reallocation, etc. Sub-categories of questions – e.g., demos, attitudinal, behavioral, population benchmarks Router engagement - measures of respondent straight lining, speeding (Group 1 should say what these are) Respondent Satisfaction Group 2, Core Activity: Quantitative Research Study 27 FoQ2 Research Priority 7: (contd)

28 Thank you and Q&A 28 Nancy L. Brigham, Ph.D. VP Global Operations, Respondent Access & Engagement 160 Bloor St. East, Suite 300 Toronto, ON, Canada M4X 1A2 Phone: Michael A. Fallig, Ph.D. Managing Director/CTO, Cogenti Applied Strategies, Inc Avenue of the Americas New York, NY Phone:


Download ppt "Survey Routers: A Primer Nancy Brigham, Ph.D., VP Global Operations, Ipsos Michael Fallig, Ph.D., Managing Director, Cogenti Applied Strategies April 20,"

Similar presentations


Ads by Google