1. Understanding the Rates
LONDON | AMSTERDAM | PARIS | FRANKFURT | MUNICH | MADRID | TIMISOARA | HAIFA | WILTON
NORTH ANDOVER | SHERMAN OAKS | TORONTO | SYDNEY | GURGAON | TOKYO | HONG KONG

2. What is the Response Rate?
Response Rate (RR) is the percentage of respondents attempting to take part in the survey through the invitations received (the percentage of how many people clicked on the link from the invitation)
1000 respondents were sent an invite
100 respondents clicked on the link
100 out of 1000 is 10%
Response Rate is 10%
In order to calculate the RR, one must first know the number of invitations sent, which act as the denominator in the calculation.
Hence it can only be calculated for Direct Invites (Manual, TAI and IP)
For all the other sampling channels you will only see the starters and hence won’t be able to calculate a response rate

3. Response Rates Factors
The Response Rate of a survey depends on a series of factors that can be categorized into
technical
Although invites have been sent, the survey is still on Testing/ On Hold
non technical
Invites sent at a wrong time
Wrong / non-appropriate subject lines / invite template
Unattractive subject line
Seasonal and special events
Below is a list of factors that influence the low Response Rate and action plan on tackling this
Wrong timing for sending invites
Check local time zone of fieldwork and schedule/send targeted sample accordingly
Seasonal & Special events ( holidays period, Sports events.. )
Modify subject line
Recalculate Project feasibility
Unattractive subject line
Change Subject line by default for specific subject line
Non appropriate subject line
Check & revise subject line used
Wrong Invite template used
Check & revise invite template used

4. Why is it important to monitor your RR?
The RR has a direct impact on the overall feasibility of the survey/project: indeed, the number of invitations needed to fulfill a survey is calculated based on :
# completes/ Response rate/ Incidence rate
If the RR in field is lower than the RR projected, you are at risk to not deliver all required interviews, as you will need more invitations to complete the survey and you may not have more supply available.
How to identify a low RR
Go to Codebook / General Feasibility tab
Enter Country + Age groups &Gender + authorized channels + LOI + DIF. Go to “Details”
Compare In Field Response Rate average to Codebook Response Rate average
If RR in field < Codebook RR => IMPACT on my fieldwork
Keep in mind that the younger age group (16-24) and the older age groups (55+) have a significant lower RR than the other age groups

5. What is the Incidence Rate?
The Incidence Rate is the frequency of something occurring within the survey/target group
It is the penetration rate of the target group in the survey, the frequency of respondents who meet a specific criteria
The IR is usually applied to the population of defined a target group or population of interest:
- The frequency of respondents who use or purchase a specific product/brand.
Ex: 75% of the population in Germany own/lease a car
- It can also be applied to the demographics for a specific target group
Ex: 80% of the people aged are consumers of Coca-Cola compared to the
people aged where the IR is 15%

6. Definition of Key Terms
Non-target Terminates/ Screened outs, or non target quota fulls: terminates or quota fulls that cannot be targeted from our panel. The following demographic groups can be targeted 100% through UPS and would not count towards the non-target terminates or quota full group
Gender
Age
Region
Terminated or screened outs: Respondents who were terminated out because they did not qualify in the screener
Status of screened shown in Confirmit or termed in UPS.
Incompletes or drop outs: Respondents who have dropped out of the study without reaching the final screening question.
Abandon: If, after 12 hours the incomplete respondents did not returned to the survey they will be counted as an Abandon.
Qualified Incompletes: Respondents who qualified for the study but then dropped out after moving past the final screening question (within the survey)
Completes: all respondents who have qualified for a study and completed the questionnaire (saw our complete end page)
Status of complete will be shown in Confirmit or qualified in UPS

7. How do we calculate incidence at Toluna?
Every company activating in the field of market research has its own formula for calculating the IR. Incidence is calculated in 2 ways:
Net Incidence
The % of respondents who meet all qualifying or eligibility criteria and who have completed the survey.
Formula
Completes / Non-targetable Terminates + Non-targetable quota fulls + Completes
Net Effective Incidence
The % of respondents who meet all qualifying or eligibility criteria and who may or may not complete the survey.
Qualified Incompletes + Completes / Qualified Incompletes + Non-Targetable Term + Non-targetable quota fulls + Completes
Refer to the following article for more details on how the IR and CCR are calculated in UPS

8. What incidence measure is used by PMs?
Net Incidence
Sample Only – report Net incidence to clients 24 hours after soft launch.
If there is a variance of +/- 20% the PM should enter a Project Change in UPS.
PM is also required to notify clients of any change of incidence and advise them that there may be cost and timing implications
Net Effective Incidence
Full Service – report Net Effective Incidence to clients 24 hours after soft launch and at end of study.

9. Why isn’t calculating IR as simple as it seems?
It is not always possible to acquire counts for qualified incompletes, targetable quota fulls and non-targetable quota fulls.  This is mainly due to limitations on the details available on Sample-only projects, given that we don’t have direct access to the response data.
Even on FS projects, a PM will need to export data or have hidden tallies programmed into the script in order to properly differentiate the two varieties of quota fulls and incompletes. 
There may be confusion surrounding how to distinguish targetable and non-targetable quota fulls or targetable and non targetable terminates. Further, this can be an unnecessarily time-consuming process since breaking down quota fulls and abandons into separate categories (targetable versus not targetable & qualified or not qualified) generally affects a very small portion of the total respondent base and does not weigh that heavily into the final IR calculation.
If you are confused on which criteria to include in your calculation, and as long as you don’t have an unexpectedly high number of quota fulls or incompletes, just use the simple calculation:
Completes / (Completes + Terminates)
The following slides are giving more details on what we call targetable versus non tragetable terminates or quota fulls

10. Targetable or Non Targetable Terminates
While Calculating incidence the PM should only include non Targetable (behavioral) Terminates and Quota Fulls in denominator. The important step is to segregate targetable and non targetable criteria
Below are some question that will help the PM decide to which category a terminate or a quota full belongs:
Targetable or Non Targetable Terminate
Did the client ask to send invites to general population
Is the terminate/ QF on demographic
Did the client ask to target a behavior for e.g. Beer drinker

11. Targetable or Non Targetable Terminates
Client’s Target Audience
General Population
When the target audience of client is general population, the idea is to see what % of respondents get terminated at different stages of the survey. Basis this data client might want to gauge the penetration of a behavior or awareness amongst general population.
In such scenarios the PM might see some terminates which might be targetable through UPS, for e.g. Primary decision makers, car owners, Beer drinkers etc. However they should be included in incidence calculation as non targetable terminates.
Tip- The incidence that is bid on general population is generally lower.

12. Targetable or Non Targetable Terminates
Age / Gender
Income /
Region / Language
Ethnicity/Education
If the terminate report has terminates on age, gender, region, ethnicity, income, education or any other demographic which is targetable through UPS, that should not be included in incidence calculation. Since these are basic demographic which are expected to be targeted by UPS if need be.
These are categorized as Targetable terminates.
PMs should take proper measures to ensure the right audience is being invited and there are no targetable terminates.

13. Targetable or Non Targetable Terminates
In some cases the client may assume that we will be prescreening for certain behaviors and incidence is bid accordingly . Say the incidence of a specific brand of beer is bid to be at 10% among beer drinkers. In such case the client will not count terminates happening at the beer drinking question ( 2nd layer) into incidence. They will take that as a targetable terminate.
General population -100%
Beer Drinkers – 50%
Regular Beer Drinkers -30% of Beer Drinkers
Regular consumers of a specific brand of Beer- 10% of regular beer drinker

14. Targetable or Non Targetable Terminates
Let’s now focus on 3 specific cases:
Sensitive industry question
Technical requirements
Past participation

15. Sensitive Industry Question
Following is an example of a sensitive industry question. Though this client terminates respondents basis the industry they or their immediate family member works in. Most of the clients will expect from any panel provider to be able to target respondents who are not working in market research Industry and invite them to the survey.
Are you or any member of your immediate family employed in any of the following occupations? (Please select all that apply.)
Advertising/Public Relations
Journalism
Information Technology
Market Research
Marketing
Pharmaceutical
Telecommunications
Transportation
None of these
[ Programmers Note :TERMINATE IF ‘ADVERTISING/PUBLIC RELATIONS’, ‘JOURNALISM’, ‘MARKET RESEARCH’, ‘MARKETING’ OR ‘TELECOMMUNICATIONS’ SELECTED; OTHERWISE, CONTINUE.]

16. Sensitive Industry Question Terminates
I work in Market Research
I am in IT industry
My family member works in Market Research
I am in education industry
Although industry sensitive question is non targetable it should ideally not be included in non targetable terminates for incidence calculation.
However in an exceptional case where sensitive industry terminates are making a remarkable impact on Incidence rate, it is advised to revisit the question along with the client and either re-frame it or include the terminates in incidence calculation.

17. Technical Question Terminates
When the respondents subscribe for online surveys, they are expected to have a proper laptop/Personal computer with internet connection and liaised with the latest and important software like Internet Explorer, Adobe, Media Player, proper audio video functioning etc.
If respondent happens not to have the required feature in his laptop/PC then such terminates also should not be considered as non targetable terminates.
If the survey demands for a software or function that is not common and needs some technical knowledge to be able to use it, such terminates should be counted in exceptional cases and should be included in Incidence calculation as non targetable terminates.

18. Past Participation
Past Participation question is implemented to check when was the last time the respondent has participated in a survey. This could be categorized into:
General Past Participation- General Past Participation is when a respondent has participated in any survey regardless of the topic or Category.
Category Past Participation-When the past participation question is about a specific category or topic as in the following example.
Have you participated in a telecommunications research study in the past month? (This would include local or long distance telephone, cell phone or wireless phone studies of any type.)
Yes No
[Programmer Note : IF ‘YES’, TERMINATE; OTHERWISE, CONTINUE.]

19. Past Participation
Some standards in regards to Past participation question:
General Past Participation- In any survey the general past participation should not be permissible, and we should steer the client toward a category exclusion instead. The PM should explain to the client a panel company such as Toluna exists for the purpose of having its members take surveys, and that eliminating recent survey participants from the sample can adversely impact feasibility.
Category Past Participation-Category past participation should not be for more than 3 months and should be specific to the type of product or service being asked about in the survey. For example, an exclusion based on any past-3-month survey about food products would be too broad. An exclusion based on any past-3-month survey about frozen meals would be acceptable.
If the client is unwilling to follow the above mentioned standards then the PM must include the past participation terminates in the incidence calculation and let the stakeholders know about this, as this will impact the sample size and feasibility for the survey.

20. Incidence & OSS Vendors
Often, the incidence at OSS vendor and Toluna’s side will differ, mainly due to DRD activation. Below is an example assuming 200 OSS respondents got terminated due to DRD before entering the UPS. Hence they will not show up on the UPS tracking page. Another 200 respondents got terminated on TQS. Incidence in this case will be different for vendor, UPS and client as shown in the tables below.
OSS Vendor
Started
1200
Incomplete
200
Complete
300
Terminates
700
Incidence at Vendor Side
30%
In UPS
Started
800
Incomplete
200
Complete
300
Terminates at Client Side
TQS Terminates (Behavioral) 
 200
Incidence in UPS
38%
IR at Client side
Started
800
Incomplete
200
Complete
300
Terminates at Client Side
Incidence at client Side
50%
Note: DRD rejections can be seen in DSS under OSS rejection report.

21. When to calculate the IR?
The “real IR” will be determined at the soft launch stage
The reason we do the soft launch is to confirm all project specs. It gives us the possibility to anticipate any issues and to make a forecast on the rest of the fieldwork
It also gives us the opportunity to “fine-tune” the feasibility and sampling approach
Key is to soft launch exactly within the population of interest of our client (or as defined in the WO), to make sure we calculate the Incidence Rate amongst the correct population
Are we going to send out targeted sample at the Soft Launch stage?
For B2B/geotargeting/Parents/pet owners / Healthcare
YES
If my client wants it
I can target but project has been sold on Gen pop NO
I can't target

22. Where does the confusion comes from when calculating the IR?
We may enter in long discussions with clients when we get back to them questioning the actual IR in field versus IR quoted in the work order – WHY?
Issue 1: We may not calculate the IR within the same population of interest as our client
Solution: re-confirm the population of interest with your client and make sure this is how the price was built in the WO
Issue 2: We may not use the same formula as our client
Solution: re-confirm the formula we are using with our client and find a common agreement on calculation
Issue 3: We may not include the correct criteria in the calculation, because we don’t have access to the data
Solution: ask your client to send you a detailed term report

23. Why is it important to monitor the IR?
Any decrease of the forecasted IR will affect the fieldwork and potentially impact:
Feasibility: more people than estimated are terminated (hence less complete the survey). As we have a limited number of potential respondents in our supply, there is a risk we may not be able to deliver the requested number of completes.
Always reassess feasibility if IR drops by more than 10%.
Timelines: if more people than estimated are terminated, we will need more time than expected to gather all interviews
Budget: Incidence Rate (along LOI) is the main factor
to calculate the price. Any drop of 20% or more in
incidence will result in a re-cost to the client (and hence
need to be communicated to sales via a PCF)
WO IR
Actual IR – Project change applies
100%
80% or lower
80%
64% or lower
60%
48% or lower
40%
32% or lower
20%
16% or lower
15%
12% or lower
10%
8% or lower 5%
4% or lower 3%
2% or lower 1%
0,8% or lower
In case you issue a PCF, make sure you always communicate to both sales and clients how you calculated the IR, as clients may have a different way to compile it and disagree with the re-cost.

24. Troubleshooting the IR
A lower IR may come from:
Hidden screening questions
Behavioral quota full
Incorrect targeting/setup
Respondent quality checks
Wrong calculation of the IR by the client
How to check from where it is coming from?
Technical set up & Sampling criteria
Client / ConfirmIT & UPS quotas set up comparison
End links used
Screening criteria
Re-test the questionnaire to identify behavioral Quota fulls, or QF on brands
Term report
Re-check questionnaire and ask for support (if applicable) from TL/PST

25. Boosting the IR Improve the efficiency of Profiles & Targets
In the situation where the client has approved for targeted sample to be sent out, please use all the available targets for the specific topic.
Use the codebook to search for available targets that approximate your survey’s requirements. Even if not precise, using approximate target profiles yields a more responsive and more-likely-to-qualify sample
Put in place a Pre-Screeners / TQS
Borrow from the codebook or a client’s screener to obtain properly formatted questions and language translations.
If the IR doesn’t improve, amend the TQS accordingly

26. What is CCR?
The Completion Conversion rate is another rate used at Toluna to measure fieldwork progress.
In order better understand the differences between CCR and IR, let’s look at them in comparison
Completion Conversion Rate (CCR) is the measure of respondents converting into completes in a survey taking into consideration all types of terminates and quota fulls
IR defines the measure of respondents qualifying in a survey taking into consideration their profile and the target audience for the study
Because the CCR takes into consideration the quota fulls, it will always be equal or lower than the IR
PM can use CCR to calculate the feasibility while in production
The formula for CCR is:
CCR = completes/(completes + terminates + quota fulls)

27. RR,IR and CCR Relation 10,000 10,000 respondents invited in the survey
The number of invites sent for a survey does not determine the number of completes yielded in a survey. Response Rate, Incidence Rate(IR) and thereafter Completion Conversation Rate also play an important role here. As shown in the below screenshot.
10,000
10,000 respondents invited in the survey
800
800 Respondents attempted the survey.
Response Rate=8%
320
Out of 800 starts, 300 terminated, 100 quota full , 80 incomplete and 320 respondent completed the survey.
IR=52%
CCR=44%

28. Dropout Rate
The Dropout Rate (DR) is the % of respondents that have left the survey before completing it. They are also referred to as incompletes.
Why may people drop out of the survey?
The survey is too long and respondents get bored
There are technical requirements the respondent does not meet
There is a technical mistake in the survey, or an interruption of service at a given time
The respondent does not click on the last page to confirm his/ her participation (in this case he/ she is considered as qualified incomplete)
The DR has a direct impact on the feasibility of the fieldwork. If higher than normal, it is a good indicator that something is not going well in your survey.

29. Troubleshooting the DR
The DR may be influenced by different factors. Please find below a list of factors and their resolutions:
Incompatible Web browser system
Ask Scripting team / clients for check
Indicate Browser constraints in Invite Template
Run projects with Direct Invites preferably
Special technical requirements (downloads..)
Indicate Technical Requirements in Invite Template
LOI in field over 20% higher than sold one
Inform Sales & Clients
Change LOI information in Survey tab
Request for higher incentive matching real LOI
Scripting errors
if targeted sample sent, Re-Invite panelists with Re-Invite template
Monotonous questionnaire
Alert client & offer alternative display solutions when existing
Server slowness / failure
If targeted sample sent, Re-Invite panelists with Re-Invite template

30. Field Reports In UPS
Saving You Time!

31. Terminate Report In UPS – FS projects
The terminate report is available in UPS.
SP uses a bridgecode for Full Service studies so that there is always a description for each terminate.
The goal is to have UPS provide incidence calculations for you, and for your client.
You will still need to calculate incidence for studies with more than one incidence group (ie, natural fallout for OQs).
You can export the terminate report from DSS.
Make sure you liaise with the scripting team to ensure they program non-targetable and targetable terminates correctly (see following slides).  Not scripting them correctly will result in incorrect calculations not only in UPS, but also in the Fieldwork report available to the client in Toluna Analytics. 

32. Targetable Terminates for UPS
To enable UPS the ability to determine targetable terms vs. non-targetable terms SP uses Terminates 7 through 14 for all standard terminates which are targetable. These include:
Terminate 7 - Gender 7
Terminate 8 - Age 8
Terminate 9 - Income 9
Terminate 10 - Ethnicity 10
Terminate 11 - Race 11
Terminate 12 - Region 12
Terminate 13 - Education 13
Terminate 14 - Language 14
Terminates are open for other targetable terms on a study by study basis.
(i.e. age/gender nested, products purchased, products owned etc.)

33. Non-Targetable Terminates for UPS
Non-Targetable terminates will be listed under codes 21 though 50
Terminate 21 – Non-User of Fast Food Restaurants
Terminate 22 – Eats vegetarian meals at restaurants
Terminate 23 – Likes pizza
Terminate 24 – Does not like pepperoni
Terminate 25 – etc.
SP includes the terminate description for Full Service studies. You can work with your programmer to ensure the description is adequately descriptive.
Descriptions are not available for Sample Only jobs with the exception of pre-screeners programmed by our SP team. SO clients will have the ability to pass back variables to enable UPS to capture where respondents are terminating.

34. Other Terminates for UPS
The remaining terminates including SO client terminates and status changes will use codes 70+.
Terminate 79 – Invalid SRC for survey
Terminate 83 – Terminate – integrity based
Terminate 99 – Status Change – former completes who had a status change
Terminate 70 – DP Removed based on rejections
Terminate 71 – DP removed based on quota selection
Terminate 72 – DP removed based on cleaning
Terminate 91 – SO Client Incomplete
Terminate 93 – SO Client Quota Full
Terminate 94 – SO Client Terminate
Terminate 98 – SO Client Time Terminate

35. To View the Field Report in DSS
In the Intranet, open the UPS reports
Then select Survey Reporting
Then select Field Status Report

36. Inputs to view the Field Status Report in DSS
Project ID
Survey ID
Wave
Start Date, End Date (default is NULL which will provide all touches)
Show Client’s Excluded Fields (Default is No)
Select a format

37. The DSS Field Status Report: FS Show Client’s Excluded Fields = No
Status Summary
Total Terminates
IR Results
LOI
Definitions

38. The DSS Field Status Report: FS Show Client’s Excluded Fields = Yes
Incompletes and Terminates are broken out by type: Qualified vs non qualified for Incompletes Targetable vs. non-targetable vs other terminates for Terminates.

39. The DSS Field Status Report: FS Show Client’s Excluded Fields = Yes
The WO incidence rate, CCR and timestamp will display if “Show client excluded field” is checked.

40. Field Status Reports
The Field Status Report calculates the overall incidence for your study, but is not built to calculate incidence for several different incidence groups within the same Survey ID.
If you have such a study you will still need to calculate the incidence for
each individual group as you have always done

41. Non-Standard Requests
Exercises
Non-Standard Requests

42. Example 1
Q. How do I calculate net effective incidence for qualification up to a specific screening question? A. Consider the following example, your client wants to find out incidence of respondents who own a vehicle that is more than 3 years old and who intend to purchase a car, van, truck or other kind of vehicle within the next 2 years QS1. Is your current vehicle...? (Terminate if less than 3 years old). Less than 3 years old n=300 More than 3 years old n=700 QS2. How soon will you purchase a new vehicle? (Terminate if more than 2 years from now). Less than 2 years from now n=300 More than 2 years from now n=400 QS3. Which type of vehicle do you intend to purchase? (Terminate if “None”) Car n=100 Truck n=100 Other n=25 None n=25 Note, 50 people abandoned

43. Solution 1
The formula to use is: qualified/qualified + non-targetable terminates. Qualified: All 225 respondents who did not terminate at Q3 Non-targetable terminates: All 700 respondents who were screened at QS1 (300) and QS2 (400) and at QS3 (25) Hence Net Effective incidence of qualifying up to QS3 is: 225/( )=24% Incidence of respondents who own a vehicle that is more than 3 years old and who intend to purchase a car, van, truck or other kind of vehicle within the next two years is 24%

44. Example 2
A. Here is another example from the same study. Your client wants to find out incidence of respondents who own a vehicle that is more than 3 years old and who intend to purchase a car within the next 2 years QS1. Is your current vehicle...? (Terminate if less than 3 years old). Less than 3 years old n=300 More than 3 years old n=700 QS2. How soon will you purchase a new vehicle? (Terminate if more than 2 years from now). Less than 2 years from now n=300 More than 2 years from now n=400 QS3. Which type of vehicle do you intend to purchase? (Terminate if “None”) Car n=100 Truck n=100 Other n=25 None n=25 Note, 50 people abandoned

45. Solution 2
The formula to use is: qualified/qualified + non-targetable terminates.
Qualified: All 100 respondents who intend to purchase a car at QS3
Non-targetable terminates: All 850 respondents who were screened at QS1 (300) and QS2 (400) and the respondents who did not intend to purchase a car at QS3 (150).
For the purposes of this analysis respondents who do not intend to purchase a car would not qualify for the study.
Hence net effective incidence of qualifying as a car purchaser is:
100/( )=11%
Incidence of respondents who own a vehicle that is more than 3 years old and who intend to purchase a car within the next two years is 11%