Presentation is loading. Please wait.

Presentation is loading. Please wait.

© 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.

Similar presentations

Presentation on theme: "© 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web."— Presentation transcript:

1 © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web Survey Methods, September 9 th 2009

2 What metrics for what panel Pre-recruited probability-based online panels Response rates can be calculated because the frame is known (AAPOR, 2006) Volunteer opt-in panels Response rates cannot be computed (AAPOR, 2007) However, other metrics can be calculated, e.g. completion rate 1

3 2 Current status Volunteer, non-probability (opt-in) panels, widely used in market research, outnumber probability-based Web panels More and more probability-based online panels being built 2007-2009 American National Election Studies (ANES) Panel Face-to-Face Recruited Internet Survey Platform (FFRISP, 2008) Dutch Long-term Internet Study for the Social Science (LISS) panel (2007) Still no officially agreed standard on how to compute response rates for online panels

4 Review of current standards Many efforts and proposals by different national and international organizations: European Society for Opinion and Marketing Research – ESOMAR European Federation of Associations of Market Research Orgs. –EFAMRO Interactive Marketing Research Organization – IMRO Advertising Research Association – ARF quality initiative Bob Lederer proposal endorsed by the American Marketing Association (AMA) Latest effort by ISO (standard #26362) touches on subject 3

5 Some journals are giving guidelines on how response rates should be computed specifically for online surveys (not necessarily online panels) Journals enforcing AAPOR standards: (e.g. POQ, IJPOR…) Journal of Medical Internet Research Journal of Medical Internet Research (Eysenbach, 2004): Journal recommendations 4 In online surveys, there is no single response rate. Rather, there are multiple potential methods for calculating a response rate, depending on what are chosen as the numerator and denominator. As there is no standard methodology, we suggest avoiding the term response rate and have defined how, at least in this journal, response metrics such as, what we call, the view rate, participation rate and completion rate should be calculated.

6 ESOMAR and IMRO examples ESOMAR (2005) metrics: Response based on the total amount of invites (% of full numbers) per sample drawn (country, questionnaire) % questionnaire opened % questionnaire completed (including screen-out) % in target group (based on quotas) % validated (the balance is cleaned out, if applicable) (p. 20). IMRO (2006) metrics: Response rate is based on the people who have accepted the invitation to the survey and started to complete the survey. Even if they are disqualified during screening, the attempt qualifies as a response (p. 13). Completion rate is calculated as the proportion of those who have started, qualified, and then completed the survey (p. 13). 5

7 AMA platform for data quality progress: 6 Platform for Data Quality Progress, Bob Lederer (under AMA umbrella, Nov 2008)

8 ISO 26362:2009 Participation rate: number of panel members who have provided a usable response divided by the total number of initial invitations requesting members to participate (p. 3) Usable response is one where the respondent has provided answers to all the questions required by the survey design The term response rate cannot be used to describe respondent cooperation for access panels 7

9 Necessary information to compute response metrics In order to compute response metrics for online panels we need to understand how panel members are recruited and what stages are used to build a panel Volunteer-opt-in design Probability-based design 8

10 Generalized volunteer opt-in panel design Stage 1: Encounter, discover, or seek out to join Stage 2: Provide profile information Stage 3: Get and do surveys 9

11 Volunteer opt-in panels: Stages 10 Postoaca, 2007

12 Stages for probability-based online panels Stage 1: Recruitment from frame Stage 2: Welcome and get profiled Stage 3: Active membership, ready for surveys, actual study 11

13 Common steps in building a probability- based panel 1.Recruitment Rate (RECR): the recruitment of potential panel members Recruitment rate calculation will depend on the recruitment mode: face -to-face, telephone, mail 2.Profile Rate (PROR): empanelling recruited persons This stage counts panel members that answered their profile survey, generally a questionnaire collecting background information and welcoming respondents to the panel The computation of the profile rate (a.k.a., connection rate) will depend on the data collection mode Profiled members are considered to be active members in the pool from which study samples can be drawn 12

14 Probability-based design features Implications for computing response rates 1.Single recruitment cohort (one-time effort) vs. multiple recruitment cohorts (on-going recruitment) 2.Within-household selection to recruit one person vs. whole household recruitment of all eligible persons 3.The data collection mode used for non-internet households (no access to online surveys at time of recruitment) 13

15 Methods of dealing with non-Internet households 14

16 Probability- based web panels: Recruitment 15

17 Probability- based web panels: Profile 16

18 Probability- based web panels: Actual study Same design for volunteer-opt-in panels 17

19 Active panel dynamics 18

20 Stage 1 of probability-based web panels IC = Initial consent R = Refusal UH and UO = Unknown if household or unknown other NC = Non-Contact O = Other non-interview e = Estimated proportion of unknown eligibility cases R Refusal (REFR) = Rate IC + (R + NC + O) + e(UH + UO) IC Recruitment (RECR) = Rate IC + (R + NC + O) + e(UH + UO) 19 Example: P_RECR =.4 x 100% = 40%

21 I = Profile survey complete P = Profile survey partial but acceptable Stage 2 (more likely for probability-based panel) * Opt-in panels may not know the denominator components. (I + P) Profile Rate (PROR) = (I + P) + (R + NC + O)* R Refusal to Profile (REFP) = (I + P) + (R + NC + O)* 20 Example: PROR =.6 x 100% = 60%

22 Stage 3 Specific Study Rates BF = Break-offs -- when the number of answers is below the definition of partial interview, it can be considered a break-off. R = Other than for the break-off rate, R includes break-offs as refusals (I + P) Completion Rate (COMR) = (I + P) + (R + NC + O) BF Break-off Rate (BFR) = (I + P) + BF Study R Refusal (SREF) = Rate (I + P) + (R + NC + O) 21 Example: COMR =.7 x 100% = 70%

23 Cumulative Response Rate Only for pre-recruited probability-based online panels P_RECR = Person recruitment rate PROR = Profile rate COMR = Completion rate for the single study RETR = Retention rate A multiplicative function Cumulative RR (CURR) = P_RECR x PROR x COMR 22 Cumulative RR2 (CURR) = P_RECR x PROR x RETR x COMR Example CURR=.4 x.6 x.7 =.168 x 100% = 16.8% Example CURR2=.4 x.6 x.8 x.7 =.134 x 100% = 13.4%

24 The computation of a CUMRR is straightforward when the panel is built with a single recruitment cohort Computing CUMRR with 1 cohort 23 Study Respondents RECR PRORRETR COMR

25 Unequal cohort contributions to a study sample selected from among all active members 24 Computing CUMRR with 3 cohorts

26 Formulas dealing with multiple cohorts (1.) RECR, PROR, RETR are calculated as the weighted average of the size contribution of each cohort Example to calculate RECR total 25 Where W cn = the number of cases contributed to the sample from cohort n

27 Example of RECR with 3 cohorts Cohort 1Cohort 2Cohort 3 Size in the final sample 20010050 Recruitment rate (RECR).35.27.15 26

28 Formulas dealing with multiple cohorts (2.) 27

29 Full example with 3 cohorts Cohort 1Cohort 2Cohort 3___R total Size20010050 RECR. PROR. RETR. 28 Assume a survey completion rate (COMR) of.713

30 Computing completion rate (COMR) when multiple data collection modes are used Completion rates need to be computed separately for each mode Web survey Mail, phone or IVR These rates should also be combined as a weighted average 29

31 Technical condition in order to compute response metrics In order to compute response metrics each panel organization must keep an historical database with rates for each member More specifically for probability-based online panels it is necessary that: Each panel member ever recruited must have a record of his/her: –Recruitment rate cohort value –Profile rate cohort value –Retention rate cohort value 30

32 Which formula for which panel? MetricProbability- based Volunteer opt-in RecruitmentYesN/A Refusal to be recruitedYesN/A ProfileYesMaybe Refusal to profileYesMaybe ScreeningYes EligibilityYes CompletionYes Break-offYes RefusalYes Cumulative ResponseYesN/A 31

33 Which formula for which panel? II MetricPre-recruitedVolunteer Attrition cross sectionalYes Attrition longitudinalYes ReinterviewYes 32

34 Dutch study (Vonk, van Ossenbruggen, & Willems, Esomar 2006) Panel Management or Manipulation? 33

35 Some factors affecting each rate Recruitment rate Recruitment methods Incentives Profile rate Incentives Panel management efforts Retention rate Time elapsed since recruitment Incentives Panel management efforts Survey completion rate Field time Incentives Reminders 34

36 References Callegaro, M. and DiSogra, C. (2008). Computing response metrics for online panels. Public Opinion Quarterly, 72, pp. 1008-1032. DiSogra, C. and Callegaro, M. (forthcoming). Computing response rates for probability based web panels. In American Statistical Association (Ed.). Proceedings of the joint statistical meetings: section on survey research methods [Cd-Rom]. Alexandria, VA: American Statistical Association. 35

37 Future work Recruitment level computed at a household or at a person level (when recruiting multiple members per household) Attrition rates for cross sectional design Attrition rates for longitudinal designs Response rates for longitudinal designs 36

Download ppt "© 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web."

Similar presentations

Ads by Google