Assessing Quality of Paradata to Better Understand the Data Collection Process for CAPI Social Surveys François Laflamme Milana Karaganis European Conference.

Slides:



Advertisements
Similar presentations
Optimizing CATI Call Scheduling International Total Survey Error Workshop Hidiroglou, M.A., with Choudhry, G.H., Laflamme, F. Statistics Canada 1 Statistics.
Advertisements

Kevin Deardorff Assistant Division Chief, Decennial Management Division U.S. Census Bureau 2014 SDC / CIC Conference April 2, Census Updates.
08/08/2015 Statistics Canada Statistique Canada Paradata Collection Research for Social Surveys at Statistics Canada François Laflamme International Total.
18/08/2015 Statistics Canada Statistique Canada Responsive Collection Design (RCD) for CATI Surveys and Total Survey Error (TSE) François Laflamme International.
The 2006 National Health Interview Survey (NHIS) Paradata File: Overview And Applications Beth L. Taylor 2008 NCHS Data User’s Conference August 13 th,
The Future of Administrative Data ICES III End Panel Discussion Don Royce Statistics Canada June 2007.
Short-Term Economic Statistics Working PartyJune Short Term Economic Statistics Timeliness Framework Richard McKenzie OECD.
Stop the Madness: Use Quality Targets Laurie Reedman.
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output quality Menno Cuppen Paul van der Laan Wim van Nunspeet.
A Theoretical Framework for Adaptive Collection Designs Jean-François Beaumont, Statistics Canada David Haziza, Université de Montréal International Total.
Statistics Canada Statistique Canada Cost-Efficient Framework for Data Collection for CATI Surveys Social Surveys Collection Research Steering Committee.
© Federal Statistical Office Germany, Division IB, Institute for Research and Development in Federal Statistics Sheet 1 Surveys, administrative data or.
Modeling and Simulation of Survey Collection Using Paradata Presented by: Kristen Couture Co-authored by: Yves Bélanger Elisabeth Neusy.
United Nations Oslo City Group on Energy Statistics OG7, Helsinki, Finland October 2012 ESCM Chapter 8: Data Quality and Meta Data 1.
Administrative Data at Statistics Canada – Current Uses and the Way Forward Wesley Yung and Peter Lys, Statistics Canada.
WHO The World Health Survey HOUSEHOLD QUESTIONNAIRE
Miami, Florida, 9 – 12 November 2016
Understanding the RUC Survey Instrument
A FRUIT AND VEGETABLE PRESCRIPTION PROGRAM
Social Research Methods
Rachel Vis-Visschers & Vivian Meertens QDET2, 11 November 2016
At a glance Health access and utilization survey among non-camp refugees in Lebanon UNHCR November 2015.
Microsimulation: An Update
State Coordinator Intervention
9 Selling Your Product Section 9.1 Principles of Successful Selling
Health Statistics Division
Telehealth Survey Update.
Pierre Montagnier, OECD
Reducing Survey Burden Through Third-Party Data Sources
Modifying Interviewer Strategies to Reduce Cost of Data Collection
Refusal Conversions: When to Call It Quits
Trena M. Ezzati-Rice, Frederick Rohde, Robert Baskin
Panel on Indian Economic Development, Labor and Population Data
COMPILATION OF DISTRIBUTIVE TRADE STATISTICS IN UGANDA
Post Enumeration Survey Census
LISA, Anticipating the Next Generation of Longitudinal Data
MEASURING HOUSEHOLD LABOR ON TANZANIAN FARMS
LIVESTOCK PRODUCTION AND PRODUCTIVITY
Operational Agility in the American Community Survey: The Promise of Administrative Records Victoria Velkoff and Jennifer Ortman American Community Survey.
Étienne Saint-Pierre and Serge Godbout, Statistics Canada
Chapter 2: The nonresponse problem
LISA, Anticipating the Next Generation of Longitudinal Data
An Active Collection using Intermediate Estimates to Manage Follow-Up of Non-Response and Measurement Errors Jeannine Claveau, Serge Godbout and Claude.
The European Statistical Training Programme (ESTP)
Survey phases, survey errors and quality control system
9 Selling Your Product Section 9.1 Principles of Successful Selling
Multi-Mode Data Collection Approach
Survey phases, survey errors and quality control system
Electronic Data Collection at Statistics Canada
Organization of efficient Economic Surveys
The European Statistical Training Programme (ESTP)
9 Selling Your Product Section 9.1 Principles of Successful Selling
The European Social Model and Quality of Life
UNODC-UNECE Manual on Victimization Surveys: Content
Capacity building on the use of Geospatial Data and Technologies
Duke Energy Carolinas Stakeholder Meeting
The change of data sources in the Spanish SILC
Telling Canada’s story in numbers Marie-Josée Major
Introduction to geospatial data management and technologies for PHDs
Multi-Mode Data Collection Approach
Karin Blix, Statistics Denmark,
Changes in the Canadian Census of Population Program
ISSUE MANAGEMENT PROCESS MONTH DAY, YEAR
Barış DULKADİR TURKSTAT Expert
Multi-Mode Data Collection
The Application of Statistical Matching to the 2010 ESF Leavers Survey
Chapter 2: The nonresponse problem
Chapter 5: The analysis of nonresponse
Working towards a central Register : Simple, Complete and Widely Accessible September 29, 2010 Session no 5 - Register quality as a common task : Cooperation.
Impact Evaluation and Administrative Data Systems
Presentation transcript:

Assessing Quality of Paradata to Better Understand the Data Collection Process for CAPI Social Surveys François Laflamme Milana Karaganis European Conference on Quality and Methodology in Official Statistics Helsinki, May 2010 Statistics Canada • Statistique Canada 02/01/2019

Statistics Canada • Statistique Canada Outline Introduction Paradata Sources CAPI Environment Quality 6 dimensions Paradata Quality Initial Research on Data Collection Process Reaching Respondents Productivity and Cost Relationship Summary and Next Steps Statistics Canada • Statistique Canada 02/01/2019

Statistics Canada • Statistique Canada Introduction Data collection organization Statistics Canada has both CATI/CAPI interviewers Responsible for data collection - no sub-contracting Face-to-face interviews (CAPI) About 2-3 concurrent CAPI surveys each month ~100,000 monthly attempts January 2009 - 7 CAPI concurrent surveys Initial research objectives Assess the quality of paradata and its impact on the scope of possible research and resulting conclusions Better understand data collection process and practices Statistics Canada • Statistique Canada 02/01/2019

Statistics Canada • Statistique Canada Paradata Sources Attempts and contact information for Computer Assisted Personal Interview (CAPI) surveys Attempt = visit or call Administrative and payroll information Extracted from Statistics Canada paradata database Historical information since 2003 Updated on daily basis Targeted CAPI surveys Canadian Community Health Survey (CCHS), Survey of Household Spending (SHS), Labour Force Survey (LFS) Statistics Canada • Statistique Canada 02/01/2019

Statistics Canada • Statistique Canada CAPI Environment CAPI interviewers Work independently from their homes Assigned a set of cases to complete within a specified period of time Requested to record all attempts at the time they were made Most paradata captured automatically, except Attempts outcome - tel. call/personal visit flag Pay information (hours worked, km, fees) Daily transmission of production and cost data Every day worked Statistics Canada • Statistique Canada 02/01/2019

Quality - 6 Dimensions (QAF) Relevance Describe collection processes Meet research needs (e.g. number and scope of research) Timeliness Day after paradata received Accessibility Owner of information Sensitive and confidential information controlled (e.g. interviewer ID) Accuracy Coherence Interpretability Focus on the last 3 dimensions Statistics Canada • Statistique Canada 02/01/2019

Statistics Canada • Statistique Canada Paradata Quality Attempt with long duration - potential outliers Application left open by interviewers Cap at 2.5 hours - less than 0.5% of attempts Short interviews Less than 0.5%-0.9% of interviews Pattern of attempts Lag between logged attempts was in line with the expected time required to move between cases Under coverage of attempts Number of attempts recorded by type of cases (i.e. respondents, non-respondents, voids) is comparable to US CAPI survey Personal visits vs telephone calls (production only) Proportion of tel. calls vary from 25%-40% depending on the survey Impact on any type of geographic, productivity or cost analysis More investigations required Statistics Canada • Statistique Canada 02/01/2019

Statistics Canada • Statistique Canada Paradata Quality Production and Payroll data consistency (at interviewer-day level) About 70% of records on both files About 80%-85% on both files and production Representing over 90% of the system time (production) Over 90% on both files and payroll Representing over 95% of payroll hours Personal/telephone information on production and payroll About 75% coherence Traveling code on Payroll data (at interviewer-day level) In general, about 85%-90% of interviewers reported travel with CAPI interview Vary by RO Statistics Canada • Statistique Canada 02/01/2019

Initial Research on Data Collection Process Reaching respondents Contact rate Best interview time Contact vs interview Production and cost relationship Survey Productivity indicators Interaction between surveys Statistics Canada • Statistique Canada 02/01/2019

Contact Rates for First Attempt Best time to contact: early evening but… Consistent with information from CATI social surveys pattern Surprisingly, the shape of this graph varies by survey and even by survey cycle – not stable Depending on the interaction between surveys and between telephone and personal attempts? Statistics Canada • Statistique Canada 02/01/2019

Statistics Canada • Statistique Canada Interview When interview are conducted Peak period: 10:00-11:00, 13:00-15:00, 18:00-20:00 Very similar by survey and survey-cycle Statistics Canada • Statistique Canada 02/01/2019

First Contact and First Appointment versus Interview ~ 38% respondents reached at the first attempt ~ 45% respondents reached on the day of the first attempt ~ 53% respondents required at least one appointment prior to interview ~ 60% interviews completed within 2 attempts Note that the distribution of lag of days is much more uniform suggesting that interviewers are likely to distribute appointments during collection period Statistics Canada • Statistique Canada 02/01/2019

Relationship between Production and Cost Good relationship between production (system time) and payroll hours throughout survey cycle Statistics Canada • Statistique Canada 02/01/2019

Survey Productivity Indicators Daily Productivity Indicators Productivity ratios are relatively stable during collection period - except at the end; different for CATI surveys Total System Time / Total Payroll Hours 20%-30% CAPI vs. 60-70% CATI Complete Interview System Time / Total System Time 60-80% CAPI vs. 30%-60% for CATI These ratios are affected by interview length and response rate Statistics Canada • Statistique Canada 02/01/2019

Interaction Between Surveys The proportion of interviewers that work on more than one survey on a given day varies over time Proportion affected by interview workload distribution and field collection process and practices Sample coordination initiative Statistics Canada • Statistique Canada 02/01/2019

Statistics Canada • Statistique Canada Summary CAPI paradata Good quality but more ‘noise’ than for CATI Good relationship between production and cost indicators Effort is evenly distributed throughout the day and collection period - different for CATI surveys Productivity stable throughout collection period Interaction between surveys varies over time Next Steps Continue to assess data limitations and its impact Interaction between personal and telephone attempts Include geography workload characteristics Evaluate new initiatives: sample coordination Identify ‘viable’ operational efficiency opportunities Statistics Canada • Statistique Canada 02/01/2019

For more information, please contact Pour plus d’information, veuillez contacter François Laflamme francois.laflamme@statcan.gc.ca Statistics Canada • Statistique Canada 02/01/2019

Average Number of Attempts by Final Status of Cases Statistics Canada surveys are comparable in terms of number of attempts required to resolved cases – except for LFS Comparison with US survey suggests no (or low) under coverage in terms of attempts recorded Statistics Canada • Statistique Canada 02/01/2019

Statistics Canada • Statistique Canada Distribution of Cases, Respondents, Attempts and System Time by Total Number of Attempts The proportion of cases and system time for cases that required 6 attempts or more is about the same The ratio %respondents / %cases is still high for cases with 6 attempts or more - but lower than other type of cases Very different for CATI surveys Statistics Canada • Statistique Canada 02/01/2019

Production and Cost Concepts Production (System Time) Complete Interview System Time: System time to complete interviews Total System Time: Total system time includes all type of attempts Costs (Payroll Hours) Direct Collection Payroll Hours: Time charged to conduct direct collection activities (including travel time) Total Payroll Hours : Total time charged Includes administration, data transmission time, etc. Statistics Canada • Statistique Canada 02/01/2019

Effort and Productivity - by Period of the Day Effort (system time) is relatively evenly distributed throughout the period of day - no peak in evening Productivity seems to be stable throughout the day Statistics Canada • Statistique Canada 02/01/2019