Assessing Nonresponse Bias and Measurement Error Using Statistical Matching John Dixon U.S. Bureau of Labor Statistics June 15, 2010 The opinions expressed are those of the author, not the BLS

Nonresponse Bias and Measurement Error Nonresponse bias and measurement error both refer to difficult to measure errors in surveys. Nonresponse bias refers to unmeasured persons, and measurement error refers to an unmeasured construct (which is mis-measured by the survey). Contact history has the potential to describe the concerns of potential respondents as well as contactability. Those concerns have been found to relate to nonresponse, but little relationship to measurement error. 2

Consumer Expenditure Quarterly Survey Household survey provided part of the “market basket” for CPI and other indices. 5 quarterly interviews for each household. Typically 6160 households a month. Nonresponse rate: 15% Refusal rate: 13% More probing than the similar Diary survey

National Health Interview Survey Household survey provides health information. Household, Family, Child, Adult modules. Typically 35,000 households a year. Nonresponse rate: 12% Refusal rate: 6%

Contact History Instrument (CHI) Data

Mean Rates of CHI concerns

7 Predicting Nonresponse Logistic model Coefficients

8

Factor Pattern for Contact History Concerns

Statistical Matching X 01 Z 11 Y 10 10

Refusal effect for matched data.

Noncontact effect for matched data.

Summary The Chi data was useful in modeling the relationship between concerns expressed by respondents and refusal/noncontact. The CHI data showed factor patterns which could describe broad areas of concern. They related well in predicting nonresponse. The nonresponse bias looked similar for the CE and NHIS for both refusal and noncontact The NHIS had higher estimates of health expenditures, but the reason for those differences need investigation. The matching method was sensitive to binning method, but a regression approach may have more robustness and better diagnostics.

Limitations and Future Research The CHI data is limited in that it only reflects the concerns expressed by respondents. Some of the most common concerns may mask the real reasons, for example, “busy” may hide concerns about privacy, which weren’t expressed to the interviewer. Statistical matching may be useful for suggesting further research, but the accuracy of the estimated differences needs other data sources to separate out method/survey differences.

Contact Information Assessing Nonresponse Bias and Measurement Error Using Statistical Matching John Dixon