Nonresponse Bias Correction in Telephone Surveys Using Census Geocoding: An Evaluation of Error Properties Paul Biemer RTI International and University of North Carolina Andy Peytchev RTI International

2 Estimating the Population Mean in an RDD Survey Respondents {R}Nonrespondents TOTAL SAMPLE Nonrespondents {NR} mean of NRs is unknown

3 Methods for Adjusting and Evaluating for RDD Nonresponse  Very limited information on NRs in RDD surveys  Post-stratification adjustments are the norm  Effectiveness at reducing bias is questionable at best  Bias is sometimes evaluated using  Maximum followup effort approaches  Only evaluates reduction in bias due to slight elevations in response rates  Comparison to external gold standard estimates  Limited in scope to a few characteristics  Census block group geocoding  Error properties are largely unknown The focus of this research

4 Census Geocoding (CG) Method  Obtain the addresses for nonrespondents (50-60% “success” rate)  Geocode addresses  Link to census aggregate data  Address matched: link unit to census block group (CBG) via geocode  Exchange matched: link to census tract (CT) via telephone number  Substitute the corresponding CBG or CT mean for the nonrespondent’s characteristic

5 Estimating the Population Mean in an RDD Survey Respondents {R}Nonrespondents TOTAL SAMPLE Nonrespondents {NR} mean of NRs is unknown

6 Impute Nonrespondent Characteristics from Census Aggregate Data Respondents {R}Nonrespondents TOTAL SAMPLE Nonrespondents {NR} Obtained from NRs CBG or CT

7 Questions Addressed by this Research  What is the bias in ?  Is a valid estimate of the bias in the unadjusted (or post-stratified) estimator of the mean?  Does the CG method provide useful data for modeling response propensity?  The first two questions will be addressed in today’s presentation.

8 Decomposition of the Bias in RespondentsNonrespondents Correctly matched addresses Incorrectly matched addresses Correctly matched exchanges Incorrectly matched exchanges TOTAL SAMPLE Size Expected Difference

9

10 Components of the Bias in where

11 Components of the Bias in where correct CBG match incorrect CBG match correct CT match incorrect CT match

12 Estimation of the Bias Components  National Comorbidity Survey Replication (NCS-R)  National probability sample of 18+ in households  Face to face survey with 71% response rate  All addresses were geocoded  CG was applied to 8,178 responding hh’s that provided a telephone number (88% of NCS sample)  CG bias components estimated based on 41% response rate (response after 3 callbacks)  Sensitivity analysis based on three response rates:  2 callbacks  26% response rate  3 callbacks  40% response rate  5 callbacks  60% response rate

13 Why is it reasonable to use a face to face survey to evaluate the CG bias in an RDD survey?  The nonresponse mechanism is not a critical factor in the assessment of the CG bias.  A survey with a relatively high response rate is needed to evaluate the bias.  Addresses are known for all sample members and can therefore be geocoded to their correct CGs.  Sensitivity analysis can be performed to assess the effect on CG bias of increasing response rates.

14 Weighted Respondent Mean, True Mean, and CG Imputed Mean for Available Characteristics

15 Weighted Respondent Mean, True Mean, and CG Imputed Mean for Available Characteristics

16 by Response Rate

RTI International 17 Average Estimates of for { s } = {CA}, {IA}, {CE}, and {IE} Bias Component (Percentage points)

RTI International 18 Average Relative Size of the Bias Components

19 Conclusions  Bias in the CG estimates of NR bias is unacceptably large  race, age, and income were the most biased  Major source of bias  {IE} followed by {CA} (surprisingly)  Approximately 75% of the cases fall into these subsets  Correctly matching to CBGs reduces the bias, but minimally  Biases tend to build across components rather than netting out.  Increasing the survey response rate reduces bias in the CG approach; relative importance of each component is stable

20 Next Steps  Further characterize the CG bias by its components  Consider the use of CBG and CT information obtain from the CG method for:  modeling of response propensities  adjusting for nonresponse bias

21 ME TO REQUEST FULL REPORT