Stop the Madness: Use Quality Targets Laurie Reedman

Scope  Aspects of quality Timeliness and accuracy  Mechanisms to manage quality Indicators and pre-set targets  Survey processes Computer assisted interviewing Collection follow-up Manual processing 25/10/2015 Statistics Canada Statistique Canada 2

Statistics Canada’s dimensions of quality  Relevance  Accuracy  Timeliness  Accessibility  Interpretability  Coherence 25/10/2015 Statistics Canada Statistique Canada 3

Statistics Canada’s dimensions of quality  Relevance  Accuracy  Timeliness  Accessibility  Interpretability  Coherence 25/10/2015 Statistics Canada Statistique Canada 4

Statistics Canada’s dimensions of quality  Relevance  Accuracy  Timeliness  Accessibility  Interpretability  Coherence 25/10/2015 Statistics Canada Statistique Canada 5 How can a survey manager manage both process and product quality of data collection and manual processing activities?

Interviewer Monitoring  Computer assisted interviewing  Monitor observes and grades samples of interviewer work  Frequency of monitoring sessions geared to attain desired average outgoing quality level 25/10/2015 Statistics Canada Statistique Canada 6

Responsive Collection Design (RCD)  An adaptive approach to survey data collection  Uses information prior to and during data collection to adjust the strategy for the remaining in-progress cases (Groves and Herringa, JRSS 2006)  Can use RCD to: Control quality (response rate, representativeness) Control cost (time and resources spent) 25/10/2015 Statistics Canada Statistique Canada 7

Responsive Collection Design (RCD)  RCD was piloted on 2 surveys at Statistics Canada (Laflamme and St-Jean, JSM 2011).  Three distinct phases during data collection Early in collection – attempt all cases Mid collection – increase response rates Late collection – reduce variability of response rates between domains of interest  Key to success is changing from one phase to the next at the optimal time 25/10/2015 Statistics Canada Statistique Canada 8

Responsive Collection Design (RCD)  The turning point decisions are based on the comparison of quality indicators to pre-set target levels  Indicators are derived from paradata from current and previous collection activity If targets are too high or too low the turning points will not be effective at improving quality Targets need to reflect the priorities, for example to reduce costs, improve response rates, or optimize both simultaneously 25/10/2015 Statistics Canada Statistique Canada 9

Selective Editing and Top-Down Approach  Data editing is a quality assurance activity, not a data correction activity (John Kovar, 199?)  Goals: Make data fit for use (not perfect) - effectiveness Use as few resources as necessary - efficiency  Human resources to do telephone follow-up calls and manual analysis and data modification are costly  Managers need a mechanism to improve efficiency and effectiveness of manual processes without significantly impacting accuracy 25/10/2015 Statistics Canada Statistique Canada 10

Selective Editing and Top-Down Approach  Focus effort where it will do the most good (Hedlin, UNECE 2008)  Tackle efficiency and effectiveness from two angles: Choose certain units or domains of units for further processing, cease processing of the rest Arrange the units requiring further processing in priority order  Pro-actively control the impact on quality by basing turning points and priorities on comparisons of quality indicators to pre-set targets 25/10/2015 Statistics Canada Statistique Canada 11

Selective Editing  When to stop processing Quality indicator could be mean squared error, coefficient of variation, response rate, calculated for key variables at cell or domain level  Targets need to be set carefully If too high, might never be reached, end of processing will never be triggered, costs will not be reduced If too low, resulting data might not be fit for use 25/10/2015 Statistics Canada Statistique Canada 12

Top-Down Approach  How to prioritize units needing more processing Score function – to get a single rank incorporating several different criteria simultaneously “Biggest” based on some size measure (prior knowledge) “Biggest” based on a measure of impact (relative to what has already been collected) Most outrageous errors (outlier detection) 25/10/2015 Statistics Canada Statistique Canada 13

An Example: Edit process  Canadian Census of Population edit and imputation process  110 modules grouped into 43 processes  Underwent a “Quality Assurance Review” in 2013 (Reedman and Julien, FCSM 2013).  65-70% of time was spent on manual data verification  Recommended increased use of automation, and pre-set quality thresholds to limit activity that amounts to “polishing the apple” 25/10/2015 Statistics Canada Statistique Canada 14

An Example: Edit process  Pro-active quality management could include: Derive quality indicators for key variables, compare to pre-set targets, and direct satisfactory records onwards to the next processing step, while only retaining unsatisfactory records for appropriate intervention Use a top-down prioritization method to further restrict manual intervention to only records having a significant impact  The effect on data accuracy and potential time (cost) savings could be estimated using Census 2011 data 25/10/2015 Statistics Canada Statistique Canada 15

Conclusions  Many sources of error in statistical processes  We looked at four ways to manage accuracy and timeliness in data collection and manual processing Interviewer monitoring Responsive Collection Design Selective editing Top-down prioritization Using paradata Feasibility and effectiveness demonstrated Can be used separately or together 25/10/2015 Statistics Canada Statistique Canada 16

Thank-you! For more information, please contact: Laurie Reedman Statistics Canada 25/10/2015 Statistics Canada Statistique Canada 17