Copyright 2010, The World Bank Group. All Rights Reserved. Estimation and Weighting Part II

Copyright 2010, The World Bank Group. All Rights Reserved. Nonresponse Adjustment (NR) Not all sampled HHs respond to surveys Ignoring nonresponse results in serious bias Simple weighted estimate x of a total X will fall short if baseweights are used in a simple weighted estimate only for respondents

Copyright 2010, The World Bank Group. All Rights Reserved. Nonresponse Adjustment (NR) It is a HH level adjustment Adjust weights to account for in-scope (eligible) households not interviewed –Eligible HH should be interviewed –Some nonresponses are eligible –Some nonresponses are NOT eligible Destroyed Housing Unit Vacant Housing Unit HH moved or everyone is deceased Survey procedures should identify 100% of sampled HH that are NOT eligible

Copyright 2010, The World Bank Group. All Rights Reserved. Simple Nonresponse Adjustment Simple Random Sample n from N of households with baseweight bw = N/n –r household responses –nr e eligible HH nonresponses –nr ie nonresponses that are NOT eligible HH N includes NOT eligibles also so naive nonresponse adjustment of n/r is wrong Use (r + nr e )/r

Copyright 2010, The World Bank Group. All Rights Reserved. Simple Nonresponse Adjustment All persons have same basic weight w i = bw = N/n New weights adjusted for nonresponse w i  = bw*(r + nr e )/r = (N/n)*(r + nr e )/r Use w i  for a simple weighted estimate

Copyright 2010, The World Bank Group. All Rights Reserved. Nonresponse Weighting Classes The simple adjustment assumes that nonresponse is the same everywhere Instead divide into weighting classes –N and n must be known for each weighting class –eligibility of nonresponses must be known also Make the same type of nonresponse adjustment restricted to a weighting class

Copyright 2010, The World Bank Group. All Rights Reserved. Nonresponse Weighting Classes Possibilities for defining weighting classes Each major geographic stratum Do cities, villages, and rural areas have different nonresponse rates? Do wealthy areas have different nonresponse rates than poor areas?

Copyright 2010, The World Bank Group. All Rights Reserved. Ex: Weighting Class NR Adjustment State A: N =500,000, n=2000, bw=N/n=250 r=1,800 nr e =150 nr ei =50 nra A = (r + nr e )/r = (1,950/1,800) = w i  = bw * nra A = 250*(1.083) = State B: N =175,000, n=1750, bw=N/n=100 r=1000 nr e =700 nr ei =50 nra B = (r + nr e )/r = (1,700/1,000) = 1.7 w i  = bw * nra B = 100*(1.7) = 170

Copyright 2010, The World Bank Group. All Rights Reserved. Ex: Weighting Class NR Adjustment For State A the simple weighted estimate of employment becomes Without nonresponse adjustment, the estimate of 750,000 was probably too low

Copyright 2010, The World Bank Group. All Rights Reserved. NR Adjustment & Bias Weighting class adjustment for nonresponse does not eliminate nonresponse bias It assumes that respondents in a weighting class resemble the eligible nonrespondents –never exactly true Note the extreme difference in the example of the nonresponse adjustments – State A, State B, 1.7 –A combined adjustment would suit neither state

Copyright 2010, The World Bank Group. All Rights Reserved. Population Controls Most major household surveys benchmark to independent population controls –Population Controlling The controls are typically derived from census information updated to the present The “population” can be of HHs, of adults, of working-age adults, or of all persons One population control per geographic stratum is the simplest case –Some countries have complex multi-step “raking”

Copyright 2010, The World Bank Group. All Rights Reserved. Population Controlling Purpose: Adjusts for undercoverage/overcoverage –perhaps due to frame deficiencies –perhaps due to nonresponse Lowers variances of data items correlated with the population controls –Ex: Employment is “large” and is highly correlated with the adult population

Copyright 2010, The World Bank Group. All Rights Reserved. Population Controlling This is a person-level adjustment to weights Persons in the same HH get different adjustments to weights Person weights after nonresponse adjustment are used to estimate populations Estimates of populations are forced to equal “known” population controls Coverage bias is les- sened but not eliminated

Copyright 2010, The World Bank Group. All Rights Reserved. Example of Population Controlling Continue the state A example where, through nonresponse adjustment: –all persons have w i  = –3,000 persons responding as employed (EMP) –simple weighted estimate of employed 812,500 Expand the example –response for adult men: 2,100 with 1,700 EMP –Response for adult women 2,200 with 1,300 EMP

Copyright 2010, The World Bank Group. All Rights Reserved. Example of Population Controlling Simple weighted estimates through nonresponse adjustment –adult men 2,100 * = 568,750 –EMP men 1,700 * = 460,417 –adult women 2,200 * = 595,833 –EMP women 1,300 * = 352,083 Suppose the population controls are: –adult men: 580,000 –adult women: 585,000

Copyright 2010, The World Bank Group. All Rights Reserved. Example of Population Controlling The assumption is that the population controls are batter than the estimates made from the sample Men are somewhat undercovered either due to frame deficiencies, failure of households to properly report some men, or to nonresponse properties of households with men and nonresponse adjustment process. Women are are somewhat overcovered We want to change the sample weights so that sample estimates of the number of adult men and adult women match the controls

Copyright 2010, The World Bank Group. All Rights Reserved. Example of Population Controlling Simple ratio adjustment to weights for men –adult men: adj = = 580,000/568,750 –adult women adj = = 585,000/595,833 Adjusted weights are different by gender –adult men: w i   = = * –adult women: w i   = = *.98182

Copyright 2010, The World Bank Group. All Rights Reserved. Example of Population Controlling The populations “change” and so do the estimates of employed –adult men are increased –Adult women are decreased Simple weighted estimates after and before population controlling after before –adult men 2,100 * = 580, ,750 –EMP men 1,700 * = 469, ,417 –adult women 2,200 * = 585, ,833 –EMP women 1,300 * = 345, ,083 –EMP total 815, ,500

Copyright 2010, The World Bank Group. All Rights Reserved. Review of Typical Estimation Steps To recap, household surveys have four estimation steps Editing and Imputation are aimed at controlling response errors Basic Weighting using probabilities of selection would produce essentially unbiased estimates if 100% response with no response error was possible Weighting Class Nonresponse Adjustment helps avoid some obvious biases that arise when nonrespondents are ignored Population Controls help minimize some coverage problems and reduce variances