1. Assessing Quality of Paradata to Better Understand the Data Collection Process for CAPI Social Surveys
François Laflamme
Milana Karaganis
European Conference on Quality and Methodology in Official Statistics
Helsinki, May 2010
Statistics Canada • Statistique Canada
02/01/2019

2. Statistics Canada • Statistique Canada
Outline
Introduction
Paradata Sources
CAPI Environment
Quality
6 dimensions
Paradata Quality
Initial Research on Data Collection Process
Reaching Respondents
Productivity and Cost Relationship
Summary and Next Steps
Statistics Canada • Statistique Canada
02/01/2019

3. Statistics Canada • Statistique Canada
Introduction
Data collection organization
Statistics Canada has both CATI/CAPI interviewers
Responsible for data collection - no sub-contracting
Face-to-face interviews (CAPI)
About 2-3 concurrent CAPI surveys each month
~100,000 monthly attempts
January CAPI concurrent surveys
Initial research objectives
Assess the quality of paradata and its impact on the scope of possible research and resulting conclusions
Better understand data collection process and practices
Statistics Canada • Statistique Canada
02/01/2019

4. Statistics Canada • Statistique Canada
Paradata Sources
Attempts and contact information for Computer Assisted Personal Interview (CAPI) surveys
Attempt = visit or call
Administrative and payroll information
Extracted from Statistics Canada paradata database
Historical information since 2003
Updated on daily basis
Targeted CAPI surveys
Canadian Community Health Survey (CCHS), Survey of Household Spending (SHS), Labour Force Survey (LFS)
Statistics Canada • Statistique Canada
02/01/2019

5. Statistics Canada • Statistique Canada
CAPI Environment
CAPI interviewers
Work independently from their homes
Assigned a set of cases to complete within a specified period of time
Requested to record all attempts at the time they were made
Most paradata captured automatically, except
Attempts outcome - tel. call/personal visit flag
Pay information (hours worked, km, fees)
Daily transmission of production and cost data
Every day worked
Statistics Canada • Statistique Canada
02/01/2019

6. Quality - 6 Dimensions (QAF)
Relevance
Describe collection processes
Meet research needs (e.g. number and scope of research)
Timeliness
Day after paradata received
Accessibility
Owner of information
Sensitive and confidential information controlled (e.g. interviewer ID)
Accuracy
Coherence
Interpretability
Focus on the last 3 dimensions
Statistics Canada • Statistique Canada
02/01/2019

7. Statistics Canada • Statistique Canada
Paradata Quality
Attempt with long duration - potential outliers
Application left open by interviewers
Cap at 2.5 hours - less than 0.5% of attempts
Short interviews
Less than 0.5%-0.9% of interviews
Pattern of attempts
Lag between logged attempts was in line with the expected time required to move between cases
Under coverage of attempts
Number of attempts recorded by type of cases (i.e. respondents, non-respondents, voids) is comparable to US CAPI survey
Personal visits vs telephone calls (production only)
Proportion of tel. calls vary from 25%-40% depending on the survey
Impact on any type of geographic, productivity or cost analysis
More investigations required
Statistics Canada • Statistique Canada
02/01/2019

8. Statistics Canada • Statistique Canada
Paradata Quality
Production and Payroll data consistency (at interviewer-day level)
About 70% of records on both files
About 80%-85% on both files and production
Representing over 90% of the system time (production)
Over 90% on both files and payroll
Representing over 95% of payroll hours
Personal/telephone information on production and payroll
About 75% coherence
Traveling code on Payroll data (at interviewer-day level)
In general, about 85%-90% of interviewers reported travel with CAPI interview
Vary by RO
Statistics Canada • Statistique Canada
02/01/2019

9. Initial Research on Data Collection Process
Reaching respondents
Contact rate
Best interview time
Contact vs interview
Production and cost relationship
Survey Productivity indicators
Interaction between surveys
Statistics Canada • Statistique Canada
02/01/2019

10. Contact Rates for First Attempt
Best time to contact: early evening but…
Consistent with information from CATI social surveys pattern
Surprisingly, the shape of this graph varies by survey and even by survey cycle – not stable
Depending on the interaction between surveys and between telephone and personal attempts?
Statistics Canada • Statistique Canada
02/01/2019

11. Statistics Canada • Statistique Canada
Interview
When interview are conducted
Peak period: 10:00-11:00, 13:00-15:00, 18:00-20:00
Very similar by survey and survey-cycle
Statistics Canada • Statistique Canada
02/01/2019

12. First Contact and First Appointment versus Interview
~ 38% respondents reached at the first attempt
~ 45% respondents reached on the day of the first attempt
~ 53% respondents required at least one appointment prior to interview
~ 60% interviews completed within 2 attempts
Note that the distribution of lag of days is much more uniform suggesting that interviewers are likely to distribute appointments during collection period
Statistics Canada • Statistique Canada
02/01/2019

13. Relationship between Production and Cost
Good relationship between production (system time) and payroll hours throughout survey cycle
Statistics Canada • Statistique Canada
02/01/2019

14. Survey Productivity Indicators
Daily Productivity Indicators
Productivity ratios are relatively stable during collection period - except at the end; different for CATI surveys
Total System Time / Total Payroll Hours
20%-30% CAPI vs % CATI
Complete Interview System Time / Total System Time
60-80% CAPI vs. 30%-60% for CATI
These ratios are affected by interview length and response rate
Statistics Canada • Statistique Canada
02/01/2019

15. Interaction Between Surveys
The proportion of interviewers that work on more than one survey on a given day varies over time
Proportion affected by interview workload distribution and field collection process and practices
Sample coordination initiative
Statistics Canada • Statistique Canada
02/01/2019

16. Statistics Canada • Statistique Canada
Summary
CAPI paradata
Good quality but more ‘noise’ than for CATI
Good relationship between production and cost indicators
Effort is evenly distributed throughout the day and collection period - different for CATI surveys
Productivity stable throughout collection period
Interaction between surveys varies over time
Next Steps
Continue to assess data limitations and its impact
Interaction between personal and telephone attempts
Include geography workload characteristics
Evaluate new initiatives: sample coordination
Identify ‘viable’ operational efficiency opportunities
Statistics Canada • Statistique Canada
02/01/2019

17. For more information, please contact
Pour plus d’information, veuillez contacter
François Laflamme
Statistics Canada • Statistique Canada
02/01/2019

18. Average Number of Attempts by Final Status of Cases
Statistics Canada surveys are comparable in terms of number of attempts required to resolved cases – except for LFS
Comparison with US survey suggests no (or low) under coverage in terms of attempts recorded
Statistics Canada • Statistique Canada
02/01/2019

19. Statistics Canada • Statistique Canada
Distribution of Cases, Respondents, Attempts and System Time by Total Number of Attempts
The proportion of cases and system time for cases that required 6 attempts or more is about the same
The ratio %respondents / %cases is still high for cases with 6 attempts or more - but lower than other type of cases
Very different for CATI surveys
Statistics Canada • Statistique Canada
02/01/2019

20. Production and Cost Concepts
Production (System Time)
Complete Interview System Time: System time to complete interviews
Total System Time: Total system time includes all type of attempts
Costs (Payroll Hours)
Direct Collection Payroll Hours: Time charged to conduct direct collection activities (including travel time)
Total Payroll Hours : Total time charged
Includes administration, data transmission time, etc.
Statistics Canada • Statistique Canada
02/01/2019

21. Effort and Productivity - by Period of the Day
Effort (system time) is relatively evenly distributed throughout the period of day - no peak in evening
Productivity seems to be stable throughout the day
Statistics Canada • Statistique Canada
02/01/2019