1. A Theoretical Framework for Adaptive Collection Designs Jean-François Beaumont, Statistics Canada David Haziza, Université de Montréal International Total Survey Error Workshop Québec, June 19-22, 2011

2. Overview  Selected literature review  Framework Definition of the problem Choice of quality indicator and cost function Mathematical formulation of the problem  Solution and discussion  Conclusion 2

3. Literature review: Groves & Heeringa (2006, JRSS, Series A)  Responsive designs: Use paradata to guide changes in the features of data collection in order to achieve higher quality estimates per unit cost Paradata: Data about data collection process Examples of features: mode of data collection, use of incentives, … Need to define quality and determine quality indicators Two main concepts: phase and phase capacity 3

4. Literature review: Groves & Heeringa (2006, JRSS, Series A)  Phase: Period of data collection during which the same set of methods is used Phase 1: gather information about design features Phases 2+: alter features (e.g., subsampling of nonrespondents, larger incentives, …)  A phase is continued until its phase capacity is reached Judged by the stability of an indicator as the phase matures 4

5. Literature review: Schouten, Cobben & Bethlehem (2009, SM)  Goal: determine an indicator of nonresponse bias as an alternative to response rates  Proposed a quality indicator, called R-indicator: Population standard deviation must be estimated Response probabilities,, must be estimated using some model  An issue: indicator depends on the proper choice of model (choice of auxiliary variables) 5

6. Literature review: Schouten, Cobben & Bethlehem (2009, SM)  Another issue: indicator does not depend on the variables of interest but nonresponse bias does  Maximal bias of :  is the unadjusted estimator of the population mean:  Two limitations of maximal bias (and R-indicator): unadjusted estimator is rarely used in practice depends on proper specification of 6

7. Literature review: Peytchev, Riley, Rosen, Murphy & Lindblad (2010, SRM)  Goal: Reduce nonresponse bias through case prioritization  Suggest targeting individuals with lower estimated response probabilities For instance, give them larger incentives or give interviewer incentives Their approach is basically equivalent to trying to increase the R-indicator (or achieving a more balanced sample)  Recommend using auxiliary variables that are associated with the variables of interest 7

8. Literature review: Laflamme & Karaganis (2010, ECQ)  Development and implementation of responsive designs for CATI surveys at Statistics Canada  Planning phase: before data collection starts (determination of strategies, analyses of previous data, …)  Initial collection phase: evaluate different indicators to determine when the next phase should start  Two Responsive Designs (RD) phases 8

9. Literature review: Laflamme & Karaganis (2010, EQC)  RD phase 1: prioritize cases (based on paradata or other information) with the objective of improving response rates increase the number of respondents (desirable)  RD phase 2: prioritize cases with the objective of reducing the variability of response rates between domains of interest (increasing R-indicator) likely reduce the variability of weight adjustments (desirable) 9

10. Literature review: Schouten, Calinescu & Luiten (2011, Stat. Netherlands)  First paper to propose a theoretical framework for adaptive survey designs  Suggest: Maximizing quality for a given cost; or Minimizing cost for a given quality  Requires a quality indicator (e.g., overall response rate, R-indicator, Maximal bias, …) Which one to use? 10

11. Definition of the problem  Adaptive collection design: Any procedure of calls prioritization or resources allocation that is dynamic as data collection progresses Use paradata (or other information) to adapt itself to what is observed during data collection Focus on calls prioritization  Our objective: Maximize quality for a given cost  Context: CATI surveys 11

12. Choice of quality indicator  Focus of the literature: Find collection designs that reduce nonresponse bias (or maximize R- indicator) of an unadjusted estimator  We think the focus should not be on nonresponse bias. Why? Any bias that can be removed at the collection stage can also be removed at the estimation stage  We suggest reducing nonresponse variance of an estimator adjusted for nonresponse 12

13. Quality indicator  Suppose we want to estimate the total:  Assuming that nonresponse is uniform within cells, an asymptotically unbiased estimator is:  Quality indicator: The nonresponse variance 13

14. Overall cost  Overall cost: 14

15. Expected overall cost  Expected overall cost: 15

16. Mathematical formulation  Objective: Find that minimizes the nonresponse variance subject to a fixed expected overall cost,  Solution:  Note:Equivalent to maximizing the R-indicator only in a very special scenario 16

17. Implementation  Find the effort (number of attempts) necessary to achieve the target response probability  Procedure: Select cases to be interviewed with probability proportional to the effort  Issues:1) Avoid small estimated to avoid an unduly large effort 2) Might want to ensure that a certain time has elapsed between two consecutive calls 17

18. Graph of variance vs cost Minimum nonresponse variance Expected overall cost 18

19. Revised solution  Solution of the optimization problem is found before data collection starts  May be a good idea to revise the solution periodically (e.g., daily) Some parameters might need to be modified Update remaining budget and expected overall cost The revised optimization problem is similar to the initial one 19

20. Revised solution  Solution (same as before):  Revised target response probability:  Effort: 20 Could be negative

21. Conclusion  Next steps: Simulation study Adapt the theory for practical applications Test in a real production environment  Which quality indicator? Nonresponse variance? Others?  Reduction of nonresponse bias: subsampling of nonrespondents Our approach could be used within the subsample 21

22. Thanks - Merci  For more information, please contact:  Pour plus d’information, veuillez contacter : Jean-François Beaumont (Jean-Francois.Beaumont@statcan.gc.ca)Jean-Francois.Beaumont@statcan.gc.ca David Haziza (David.Haziza@umontreal.ca)David.Haziza@umontreal.ca 22