Presentation on theme: "Measuring the Effects of Unit Nonresponse in Establishment Surveys"— Presentation transcript:
1 Measuring the Effects of Unit Nonresponse in Establishment Surveys Clyde Tucker and John DixonU.S. Bureau of Labor StatisticsDavid CantorWestat
2 AcknowledgementWe would like to thank Bob Groves and Mike Brick for the use of their materials from their short course “Practical Tools for Nonresponse Bias Studies.”We also thank Bob for the use of materials from his 2006 POQ article “Nonresponse Rates and Nonresponse Bias in Household Surveys,” Public Opinion Quarterly, 70,
3 Factors Affecting Nonresponse Outside Survey Organization Control Three clusters of factors identified as being outside the control of the survey organization (Willimack et al., 2002)External environmental attributes (“climate”)Characteristics of the sample unitCharacteristics of the establishment employee(s) who decide whether to join a survey, priority of responding, and length of participationEffects attributable to all three factors are widespread and substantially affect nonresponseEven if nonresponse not increasing, these factors make it hard to maintain the status quo
4 Some Examples of These Factors Downsizing decreases staff available to provide dataIncreased firm size due to mergers and acquisitions increases complexity reporting burden“Gatekeeping” poses significant barriersAttitudes of owners or key managers toward government and data confidentialityWhether or not anyone in the firm actually uses data products from the surveyStaff turnoverThe growing accounting practice of using third parties such as payroll processing or accounting firms
5 Nonresponse Error for Sample Mean In simplest termsORRespondent Mean = Full Sample Mean +(Nonresponse Rate)*(Respondent Mean – Nonrespondent Mean)
6 Thinking Causally About Nonresponse Rates and Nonresponse Error Key scientific question concerns mechanisms of response propensity that create covariance with survey variablewhere is the covariance between the surveyvariable, y, and the response propensity, pWhat mechanisms produce the covariance?
7 Reporting BiasThe relative bias provides a measure of the magnitude of the bias. Interpreted similar to a percent, it is useful in comparing bias from survey measures which are in different scales.:Where:Rel B ( ) = the relative bias with respect to the estimate, .
8 Reporting BiasThe bias ratio provides an indication of how confidence intervals are affected by bias:Where: = the standard error.
9 What does the Stochastic View Imply? Key issue is whether what influences survey participation also influences the survey variablesIncreased nonresponse rates do not necessarily imply increased nonresponse error. Although lower propensity will tend to increase error.Hence, investigations are necessary to discover whether the estimates of interest might be subject to nonresponse errors because of a correlation between p and y
10 Alternative Causal Models for Studies of Nonresponse Rates and Nonresponse Bias
11 A More Specific Theory Relating Nonresponse to Bias Levels of bias will differ by subpopulationsDifferences between estimates from the total sample and just respondents will be greatest on either end of the nonresponse continuum, but potential bias greatest when response rates are lowFor example: Bias in a business survey may be greatest in the Services sector because it often has the lowest response rates
14 Nonresponse Bias Study Techniques Comparison to other estimates (benchmarking)Nonresponse bias for estimates based on variables available on sampleStudying variation within the respondent setAltering the weighting adjustments
15 Weights and Response Rates A base or selection weight is the inverse of the probability of selection of the unit. The sum of all the sampled units’ base weights estimates the population total.When units are sampled using a complex sample design, suggest using (base) weights to compute response rates that reflect the percentage of the sampled population that respond. Unweighted rates are useful for other purposes, such as describing the effectiveness of the effort.Weighted response rates are computed by summing the units’ base weights by disposition code rather than summing the unweighted counts of units.In establishment surveys, it is useful to include a measure of size (e.g., number of employees or students) to account for the units relative importance. The weight for computing response rates is the base weight times the measure of size.
16 Weights and Nonresponse Analysis A general rule is that weights should be used in nonresponse analysis studies so that relationships at the population level can be examined. Guides for choosing the specific weights to use are:Use base weights for nonresponse bias studies that compare all sampled respondents and nonrespondents. Weights adjusted for nonresponse may be misleading in this situation.Use fully adjusted weights for nonresponse bias studies that compare survey estimates with data from external sources. One important exception is when the survey weights are poststratified. In this case, weights prior to poststratification are generally more appropriate.
17 1. Comparison to Other Estimates -- Benchmarking Data or estimates from another source that are closely related to respondent estimates used to evaluate bias due to nonresponse in the survey estimatesAssume that alternative data source has different sources of measurement error and/or is a superior measure to target survey.
18 1. Benchmarking Survey Estimates to those from Another Data Source Another survey or administrative record system may contain estimates of variables similar to those being produced from the surveyDifference between estimates from survey and other data source is an indicator of bias (both nonresponse and other)
19 1. How to Conduct a Nonresponse Bias Benchmark Study Identify comparison estimatessurveys with very high response ratesadministrative systems with different measurement error propertiesAssess major reasons why the survey estimates and the estimates from the comparison sources differCompute estimates from the survey (using final weights) and from the comparison source to be as comparable as possible (often requires estimates for domains)The difference is an estimate of the direction, or perhaps the magnitude, of the bias
20 Pro’s and Con’s of Benchmark Comparison to Estimate NR Bias Relatively simple to do and often inexpensiveEstimates from survey use final weights and are thus relevantGives an estimate of bias that may be important to analystsCon’sEstimated bias contains errors from the comparison source as well as from the survey; this is why it is very important that the comparison source be highly accurateMeasurement properties are generally not consistent for survey and comparison source; often is largest source of errorItem nonresponse in both data sets reduces comparabilityHard to find comparable data for establishment surveys (IRS records?)More common in household surveys
21 2. Using Variables on Respondents and Non-respondents Compare statistics available on both respondents and non-respondentsThe extent there is a difference is an indication of the bias
22 Possible Sources of Data on Respondents and Non-respondents Sampling frame variablesMatched variables from other data-setsScreener information
23 Pro’s and Con’s of Using Data on both Respondents and non-respondents Measurement properties for the variables are consistent for respondents and nonrespondentsBias is strictly due to nonresponseProvides data on correlation between propensity to respond and the variablesCon’sBias estimates are for the variables; only variables highly correlated with the key survey statistics are relevantThe method assumes no nonresponse adjustments are made in producing the survey estimates; if variables are highly correlated, then they could be used in adjustment
24 The CES StudyJ. Dixon and C. Tucker (ICES3), “Assessing Bias in Estimates of Employment”Collects employment, hours and earnings monthly from a current sample of over 300,000 establishmentsTracks the gains and losses in jobs in various sectors of the economyIn this paper, nonresponse bias work on this survey focuses on estimating bias for establishment subpopulations with different patterns of nonresponse using data from the 2003 CES and QCEW (state UI files)
25 Link relative estimate of employment (Y) Let Yt be the estimate for a primary cell for month t, then Yt = Rt,t-1 * Yt-1where Rt,t-1 is the ratio of the total sample employment in month t to the total sample employment in month t-1 for all sample units reporting data for both months.
26 Estimate of BiasUsing the most recent employment reports in the QCEW (not CES) for both responders and nonrespondersCompare the link relative for respondents to that for nonrespondentsResults presented are not weighted by probability of selection, but weighted results show similar patternsAt this point, not comparing the link relative of responders to the entire sample
27 Quantile RegressionBias analysis performed at the establishment level on subpopulations defined by size and industryTesting for the difference in employment between CES responders and nonresponders. Y=a+Bx+e where x is an indicator of nonresponse (essentially a t-test).Since size of firm is theorized to relate to nonresponse, the coefficients relating nonresponse to employment is likely to be different for different size firms.Quantile regression examines the coefficients for different quantiles of the distribution of the sizes of firms.Since industries can be expected to have different patterns, the quantile regressions are done by industry group.
28 Distribution of size and the quantile regression curve
30 MSA percent bias predicted by response rate for Mining
31 MSA percent bias predicted by response rate for Food Manufacturing
32 MSA percent bias predicted by response rate for Retail Trade
33 MSA percent bias predicted by response rate for Accommodation and Food Services
34 Hing (1987). Nonresponse bias in expense data from the 1985 national nursing home survey. Proceedings of the Survey Research Methods Section, American Statistical Association,Purpose: Estimate cost of care in nursing homesTarget population: Nursing home facilities in U.S.Sample design: Stratified list sample of facilities, facilities sampled with probabilities proportionate to estimated number of beds, second stage sample of residents and staffMode of data collection: In-person interview of facility administrator, with drop-off self-administered Expense questionnaire for accountantResponse rate: Facility q’naire: 93%; Expense q’naire: 68% of those responding to Facility interviewTarget estimate: Estimated cost of careNonresponse error measure: Comparison of Facility questionnaire items for respondents and nonrespondents of Expense questionnaire
35 Using the Facility Questionnaire to Estimate Nonresponse Bias based on Participation in Expense QuestionnaireTables 2 and 5 from Hing (1987)
36 ConclusionsSmaller nursing homes underrepresented; thus, respondent estimates overestimate averages on size-related attributesAnalysis suggested poststratification by ownership type would significantly reduce biasesLimitation:Nonresponse bias estimate does not reflect nonresponse on Facility questionnaire
37 3. Weighting Adjustments Alter estimation weights and compare the estimates using the various weights to evaluate nonresponse bias. Weighting methods may include poststratification, raking, calibration, logistic regression, or even imputation.
38 Adjust Weights Using Model of Characteristics Weighting can reduce nonresponse bias if the weights are correlated with the estimate. Auxiliary data in weighting that are good predictors of the characteristic may give alternative weights that have less bias. If the estimates using the alternative weights do not differ from the original estimates, then either the nonresponse is not resulting in bias or the auxiliary data does not reduce the bias.If the estimates vary by the weighting scheme, then the weighting approach should be carefully examined and the one most likely to have lower nonresponse bias should be used.
39 How to Conduct Nonresponse Bias Analysis Using Weights From Modeling Characteristics Using weighting method such as calibration estimation with these variables and produce alternative weights.Compute the difference between the estimates using the alternative weights and the estimates from the regular weights as a measure of nonresponse bias for the estimate.
40 Pro’s and Con’s of Comparing Alternative Estimates Based on Modeling the Estimate If good predictors are available, then it is likely that the use of these in the weighting will reduce the bias in the statistics being evaluatedIf the differences in the estimates are small, it is evidence that nonresponse bias may not be largeCon’sRecomputing weights may be expensiveIf good correlates are not available then lack of differences may be indicator of poor relationships rather than the absence of biasThe approach is limited to statistics that have high correlation with auxiliary data
41 4. STUDYING VARIATION WITHIN THE RESPONDENTS: Level of Effort Some nonresponse models assume that those units that require more effort to respond (more callbacks, incentives, refusal conversion) are similar to the units that do not respondCharacteristics are estimated for respondents by level of effort (e.g., response propensity scores)Models fitted to see if it fits and can be used to estimate characteristics of nonrespondents
42 Analyze Level of Effort Associate level of effort data to respondents (e.g., number of callbacks, ever refused, early or late responder)Compute statistics for each level of effort separately (usually unweighted or base weights only)If there is a (linear) relationship between level of effort and the statistic, then may decide to extrapolate to estimate statistic for those that did not respondOften more appropriate to do the analysis separately for major reasons for nonresponse
43 Pro’s and Con’s of Using Level of Effort Analysis to Estimate Bias Simple to do, provided data collection systems capture the pertinent informationIn some surveys may provide a reasonable indicator of the magnitude and direction of nonresponse biasCon’sHighly dependent on model assumptions that have not been validated in many applicationsDifficult to extrapolate to produce estimates of nonresponse bias without other data
44 Brick et al (2007) “Relationship between length of data collection period, field costs, and data quality” Paper presented at ICES III, Montreal, CanadaPurpose: Estimate amount and condition of research space for science and engineeringTarget population: Colleges and universities.Sample design: CensusMode of data collection: Web and mailResponse rate: 94%Target estimate: Square feetNonresponse error measure: Relative bias when level of effort is cut by 25%
45 Relative bias for Facilities Survey estimates for academic institutions, by field and response level
46 Summary The patterns are not consistent The 75% response level generally exhibits larger bias than the other response levels, but generally not statistically significantNo significant bias if data collection were terminated at the 88% response level
47 5. Followup of Nonrespondents Use of respondent data obtained through extra-ordinary efforts as comparison to respondent data obtained with traditional efforts“Effort” may include callbacks, incentives, change of mode, use of elite corps of interviewers
48 How to Do a Nonrespondent Followup Study Define a set of recruitment techniques judged to be superior to those in the ongoing effortDetermine whether budget permits use of those techniques on all remaining active casesIf not, implement 2nd phase sample (described later)Implement enhanced recruitment protocolCompare respondents obtained in enhanced protocol with those in the initial protocol
49 Pro’s and Con’s of Nonresponse Followup Study Direct measures are obtained from previously nonrespondent casesSame measurements are usedNonresponse bias on all variables can be estimatedCon’sRarely are followup response rates 100%Requires extended data collection period
50 Marker, et al (2005).Terrorism Risk Insurance Program: Policyholders Survey. Final report prepared for the Department of Treasury. Westat: Rockville, MDPurpose: Estimate use of Terrorism Insurance by BusinessesTarget population: Businesses and state/local government offices in the U.S..Sample design: Stratified sample.Mode of data collection: Web and mailResponse rate:17%Target estimate: Estimated percent that have insuranceNonresponse error measure: Indicators of use of terrorism insurance
51 Follow-up ProceduresContacted follow of 1000 non-resopndents to the survey.A shortened instrument was used to collect critical measuresInterviews conducted by telephone
52 Selected comparisons from non-response follow-up SurveyFollow-upDiff% with Commercial Property Insurance92.389.33.0% with Terrorism Risk under TRIA through Workers Compensation42.616.825.7*% with Terrorism risk under TRIA through umbrella Policy35.731.74.0* p<.000
53 Summary Out of 8 estimates, one was statistically significant Some indication that non-response leads to overestimate of certain types of insuranceThe significant difference may have been due to measurement error on the follow-up instrument
54 Five Things You Should Remember from this Lecture The three principal types of nonresponse bias studies are:- Comparing surveys to external data- Studying internal variation within the data collection, and- Contrasting alternative postsurvey adjusted estimatesAll three have strengths and weaknesses; using multiple approaches simultaneously provides greater understandingNonresponse bias is specific to a statistic, so separate assessments may be needed for different estimatesAuxiliary variables correlated with both the likelihood of responding and key survey variables are important for evaluationThinking about nonresponse before the survey is important because different modes, frames, and survey designs permit different types of studies