Download presentation

Presentation is loading. Please wait.

Published byGabrielle Romero Modified about 1 year ago

1
The estimation strategy of the National Household Survey (NHS) François Verret, Mike Bankier, Wesley Benjamin & Lisa Hayden Statistics Canada Presentation at the ITSEW 2011 June 21, 2011

2
2 2 Outline of the presentation 1.Introduction 2.Handling non-response error 3.Simulation set-up 4.Results 5.Limits of the study 6.Conclusion 7.Future work

3
3 1. Introduction 2006 Census: 20% long form, 80% short form 2011: 100% Census mandatory short form 30% sampled to voluntarily complete the NHS long form Objectives of the long form: get data to plan, deliver and support government programs directed at target populations 2011 common topics to both forms: demography, family structure, language Additional 2011 long form topics: education, ethnicity, income, immigration, mobility… NHS sample size is 4.5 million dwellings (f = 30%)

4
4 1. Introduction Non-response error in the NHS: Survey now voluntary => expect significant non-response To minimize the impact, after a fixed date restrict the collection efforts to a Non-Response Follow-Up (NRFU) random sub-sample Set-up developed by Hansen & Hurwitz (1946) 1.Select 1 st phase sample s from population U 2.Non-response s nr observed in s 3.NRFU selected from s nr 4.Response NRFU r and non-response NRFU nr observed in the NRFU (HH assumed 100% resp. rate) U s srsr s nr NRFU NRFU r NRFU nr

5
5 1. Introduction When 100% of the NRFU responds (as in Hansen and Hurwitz original setting), the NRFU can be used to estimate without non-response bias the total in s nr This is not the case in the NHS. However focusing the collection efforts on the NRFU converts part of the non-response bias (that would be observed in the full s nr ) into sub-sampling error U s srsr s nr NRFU NRFU r NRFU nr

6
6 2. Handling non-response error The estimation method chosen to minimize the remaining non-response bias should have the following properties: As few bias assumptions as possible should be made The method should be simple to explain and to implement in production Available micro-level auxiliary data to adjust for non- response: 2011 Census short form Tax data Calibration: Agreement with Census totals is desirable from a user’s perspective

7
7 2. Handling non-response error First class of contenders: Reweighting Usual method used to compensate for total non-response in social surveys The Hansen & Hurwitz estimator of a total is unbiased if 100% of the NRFU answers When the assumption does not hold, we must model the last non-response mechanism/phase and reweight accordingly…

8
8 2. Handling non-response error Scores method: Model the probability of response with a logistic regression Form Response Homogeneity Groups (RHG) of respondents and non-respondents with similar predicted response probabilities Calculate the response rate in each RHG and assign these new predicted response probabilities to respondents Divide the NRFU r weights by this probability:

9
9 2. Handling non-response error Second class of contenders: Imputation Usual method to compensate for item non-response We will consider nearest-neighbour imputation using the CANadian Census Edit & Imputation System (CANCEIS) only 1.Partial imputation: Impute only non-respondents to the subsample (NRFU nr ) and use reweighting to take sampling into account 2.Mass imputation: Impute all non-respondents ( s nr /NRFU r )

10
10 2. Handling non-response error Some pros & cons Method ScoresPartial imputation Mass imputation Preserves micro-level information of non-respondents √√√ Does not create synthetic information√√√ Uses less heavy non-response hypotheses √√ Fully takes sub-sampling design into account √√ Census systems available√√ More calibration to known Census totals can be done √√√

11
11 3. Simulation set-up Use 2006 Census 20% long form sample data Restricted to Census Metropolitan Area (CMA) of Toronto Simulation aimed at preserving the properties of the NHS (except for the f = 30%): Non-response to the 1 st phase was simulated by deterministically blanking out the data of the 63% of respondents who answered last in 2006 Of these non-respondents, the 78% who answered first will have their response restored if they are selected in the NRFU sub-sample NRFU sub-sampling was simulated by selecting a stratified random sample of 41% of s nr

12
12 3. Simulation set-up Estimators calculated As points of reference, unbiased estimators: As contenders:

13
13 3. Simulation set-up The scores method A single logistic regression was done for the whole CMA of Toronto Household response probability was predicted Considered for stepwise selection: household-level variables, our best attempt at summarizing the person-level information and one paradata variable R-square of 26% 13 RHG formed with predicted probabilities ranging from 29% to 95%

14
14 3. Simulation set-up Imputation methods Nearest-neighbour imputation done with CANCEIS RHG is defined by household size The distance between non-respondents and donors (respondents) is defined by weighting each household-level, person-level and paradata characteristics in the distance function Preference is given to donors who are geographically close For each non-respondents, a list of donors is made and one is randomly selected with probability proportional to a measure of size (1 st phase weight for mass imputation, score method weights for partial imputation)

15
15 3. Simulation set-up M=84 non short form characteristics over the various topics Average relative difference: Calculated at the CMA level: At the Weighting area (953 WA in total) level within the CMA:

16
16 4. Results Errors at the CMA and WA levels for Toronto CMAWA Point of comparison Full first- phase Hansen & Hurwitz Full first- phase Hansen & Hurwitz Hansen & Hurwitz estimator Mass imputation 2.97N/A24.56N/A Partial imputation Scores method

17
17 5. Limits of the study Results: The simulation only includes one replication of the sub- sampling and non-response mechanisms Non-response bias is the measure of interest, but errors were presented Non-response mechanisms were generated deterministically. Should they be generated probabilistically? The 2011 sampling, non-response and available data (ex: paradata) cannot be replicated exactly Only totals studied. What about other parameters such as correlations?

18
18 5. Limits of the study Possible confounding effects: Logistic regression was done at the aggregated level of the CMA and no WA effect or interaction were considered Paradata for imputation is more closely related to non- response mechanism (give preference to late respondents in the distance) Weighting of donors in imputation has an impact Calibration done from sample to U; calibration at inner levels/phases could help scores and partial imputation

19
19 With these preliminary results, it seems scores method is doing well at aggregate levels, while partial imputation is doing better than scores at finer levels Mass imputation: Can you override the known sub-sample design with an imputation model? Partial imputation: Can include more information (person- level, paradata) than scores, but weighting of each component in the distance is partially data driven and not straightforward Scores method: More difficult to include the information, but variable selection to explain non-response is direct 6. Conclusion

20
20 Possible: Replicate sub-sampling and imputation more than once to isolate bias components Consider other levels of calibration in the comparisons Hybrid of scores and partial imputation Definite: Implement a method into NHS production Estimate the errors and variances (multi-phase, large sampling fractions, errors due to modeling,…) and educate data users Important to get a good model for the last non- response mechanism. Whatever the method, quality of the results is a function of the auxiliary data available. 7. Future Work

21
21 For more information, please contact: François Verret - SSMD/DMES (613)

Similar presentations

© 2016 SlidePlayer.com Inc.

All rights reserved.

Ads by Google