1. Building Wave Response Rates in a Longitudinal Survey: Essential for Nonsampling Error Reduction or Last In - First Out? Steven B. Cohen Fred Rohde and William Yu Agency for Healthcare Research and Quality

2. Advancing Excellence in Health Care Purpose of Discussion Need for essential longitudinal data on health care coverage, use and expenditures to inform health care policy and practice Need for essential longitudinal data on health care coverage, use and expenditures to inform health care policy and practice Description of the Medical Expenditure Panel Survey (MEPS): purpose, longitudinal design and analytical capacity Description of the Medical Expenditure Panel Survey (MEPS): purpose, longitudinal design and analytical capacity Focus on field efforts to achieve target response rates Focus on field efforts to achieve target response rates

3. Advancing Excellence in Health Care Purpose of Discussion Evaluations of the quality of the MEPS nonresponse adjustment strategies Evaluations of the quality of the MEPS nonresponse adjustment strategies Determination of characteristics for cases fielded at end of field period (EOF) & conversion of temporary refusals (TNR). Determination of characteristics for cases fielded at end of field period (EOF) & conversion of temporary refusals (TNR). Examine ROI for inclusion of these cases. Examine ROI for inclusion of these cases.

4. Advancing Excellence in Health Care Purpose of Discussion Examine impact on annual and longitudinal response rates; completion of self administered questionnaires (SAQ) Examine impact on annual and longitudinal response rates; completion of self administered questionnaires (SAQ) Impact on key survey estimates of health insurance coverage and expenditures Impact on key survey estimates of health insurance coverage and expenditures Implications of alternative field strategies Implications of alternative field strategies

5. Advancing Excellence in Health Care Medical Expenditure Panel Survey (MEPS) Annual Survey of 15,000 households: provides national estimates of health care use, expenditures, insurance coverage, sources of payment, access to care and health care quality Permits studies of: Distribution of expenditures and sources of payment Distribution of expenditures and sources of payment Role of demographics, family structure, insurance Role of demographics, family structure, insurance Expenditures for specific conditions Expenditures for specific conditions Trends over time Trends over time

6. Advancing Excellence in Health Care Key Features of MEPS-HC Survey of U.S. civilian noninstitutionalized population Survey of U.S. civilian noninstitutionalized population Sub-sample of respondents to the National Health Interview Survey (NHIS) Sub-sample of respondents to the National Health Interview Survey (NHIS) – Linkage to NHIS Oversample of minorities and other target groups Oversample of minorities and other target groups Panel Survey – new panel introduced each year Panel Survey – new panel introduced each year – Continuous data collection over 2 ½ year period – 5 in-person interviews (CAPI) – Data from 1st year of new panel combined with data from 2nd year of previous panel

7. Advancing Excellence in Health Care MEPS Overlapping Panels (Panels 8 and 9) MEPS Household Component MEPS Panel 8 2003-2004 Round 2Round 3Round 4Round 5 Round 1Round 2Round 3 MEPS Panel 9 2004-2005 1/1/20031/1/2004 Round 1 NHIS2002 NHIS2003 Round 4Round 5

8. Advancing Excellence in Health Care MEPS Household Component Sample Design Oversampling of policy relevant domains 1996 Minorities (Blacks & Hispanics) 1997 Minorities Low income Children with activity limitations Children with activity limitations Adults with functional limitations Adults with functional limitations Predicted high expenditure cases Predicted high expenditure cases Elderly Elderly 1998-2001 Minorities 2002+Minorities, Asians, Low Income 15,000 households; ~35,000 persons

9. Advancing Excellence in Health Care Design Specifications Target Precision Specifications for national and regional estimates; policy relevant subgroups Target Precision Specifications for national and regional estimates; policy relevant subgroups Overall Design effect of 1.6 Overall Design effect of 1.6 200 PSU design (Max) 200 PSU design (Max) Overall/round specific survey response rate requirements Overall/round specific survey response rate requirements Linkage to NHIS Linkage to NHIS Multistage design Multistage design Disproportional sampling Disproportional sampling Longitudinal design Longitudinal design Minimize survey cost for fixed precision Minimize survey cost for fixed precision

10. MEPS, 1996–2006: Number of uninsured, under age 65 Number in millions Source: Center for Financing, Access, and Cost Trends, AHRQ, Household Component of the Medical Expenditure Panel Survey, 1996–2005 Full-Year and 1996–2006 Point-in-Time Files

11. Advancing Excellence in Health Care Trends in Concentration Percentage of expenditures Source: National Medical Care Expenditure Survey, 1977; National Medical Expenditure Survey, 1987; Medical Expenditure Panel Survey, 1996 and 2005.

12. Advancing Excellence in Health Care MEPS Field Force Westat is data collection organization Westat is data collection organization 500 interviewers 500 interviewers Sample allocated in ~ 200 PSUs, spread across all 50 states Sample allocated in ~ 200 PSUs, spread across all 50 states extensive training modules extensive training modules Information on socio-demographic characteristics of households available based on linkage with NHIS Information on socio-demographic characteristics of households available based on linkage with NHIS Remuneration for survey participation Remuneration for survey participation

13. Advancing Excellence in Health Care Tool Chest of Methods to Maximize Survey Response Recruitment of experienced and bilingual interviewer Recruitment of experienced and bilingual interviewer 10+ days training (including procedures for obtaining signed consents) 10+ days training (including procedures for obtaining signed consents) Uses of MEPS data as reference materials for interviewers Uses of MEPS data as reference materials for interviewers Periodic retraining and special trainings (e.g. methods to improve response rates) Periodic retraining and special trainings (e.g. methods to improve response rates) Respondent remuneration Respondent remuneration Advance mailings from co-sponsors of survey Advance mailings from co-sponsors of survey Monthly planning calendar and MEPS DVD Monthly planning calendar and MEPS DVD Daily emails to interviewers regarding interviewing progress Daily emails to interviewers regarding interviewing progress Multiple contacts for refusal conversions Multiple contacts for refusal conversions

14. Advancing Excellence in Health Care MEPS Target Response Rates by Round and Overall Response Rate NHIS90%* Round 180% Round 295% Round 396% Year 1 of Panel66% Round 497% Round 598% Pooled Response Rate-Two Panels65% *NHIS response rate among households designated for MEPS. Note: Year 1 and the Overall response rate include the NHIS response rate.

15. Advancing Excellence in Health Care Person Level (survey attrition) Nonresponse Adjustment Covariates Factors associated with survey attrition (after R1) Factors associated with survey attrition (after R1) – Indicator for initial refusal to R1 interview – Family size – Age – MSA, census region – Marital status (family reference person) – Race/ethnicity – Education of reference person – Employment status – Health insurance status – Total expenditures (in yr 1 for yr 2 adj.) – # doctor visits (in yr 1) – Self reported health status

16. Advancing Excellence in Health Care Person Level Adjustments: Annual Estimates Each MEPS panel weighted separately Each MEPS panel weighted separately Nonresponse adjustment for complete nonresponse and for survey attrition Nonresponse adjustment for complete nonresponse and for survey attrition Final Poststratification adjustment –CPS 12/31: Final Poststratification adjustment –CPS 12/31: age, race/ethnicity, sex, region, MSA status, poverty status age, race/ethnicity, sex, region, MSA status, poverty status

17. Advancing Excellence in Health Care Testing for Panel Effect

18. Advancing Excellence in Health Care Capacity of MEPS to Produce Comparable NHIS Estimates of Health Insurance Coverage

19. Advancing Excellence in Health Care Characteristics of Respondents Fielded at End of First Round or Temporary Refusal Initial Refusals: Higher likelihood: MSA residence; Northeast region; white Non- Hispanic; elderly; excellent health; some high school; family size 2+; Attrite in future waves of data collection MSA residence; Northeast region; white Non- Hispanic; elderly; excellent health; some high school; family size 2+; Attrite in future waves of data collection End of Field Period Cases Higher likelihood: Race: Asian or Black Race: Asian or Black in excellent health in excellent health Attrite in future waves of data collection Attrite in future waves of data collection

20. Advancing Excellence in Health Care Testing for Reluctant Response Effect on Coverage Estimates ------------------------------------------------------- DF Wald F P-Value DF Wald F P-Value------------------------------------------------------- OVERALL MODEL 22 107.98 0.0000 JULY INTERVIEW 1 2.38 0.1244 TEMP. REFUSALS 1 0.92 0.3393 SEX 1 98.02 <0.0001 RACE/ETHNICITY 3 58.42 <0.0001 MARITAL STATUS 4 16.90 <0.0001 EDUCATION 4 10.94 <0.0001 POVERTY STATUS 4 43.97 <0.0001 MSA STATUS 1 4.34 0.0382 INDIVIDUAL INCOME 1 35.52 <0.0001 MEDICAL $ 1 35.79 <0.0001 ------------------------------------------------------- -2 * Normalized Log-Likelihood Full Model: 13037.78 Pseudo R 2: :0.194167

21. Advancing Excellence in Health Care Mean Number of Contacts by Month

22. Advancing Excellence in Health Care Mean Number of Temporary Refusals by Month

23. Advancing Excellence in Health Care Conditional Response Rates by Month of Round 1 Response: Panel 9

24. Advancing Excellence in Health Care Panel/temp. refusal Interviewdate N persons N persons Mean R1 contacts Temp.refusals full year Cond.responseLongitudCond.response Overall P9 Overall 18,250 18,2506.170.3089.90%85.40% P10 18,170 18,1706.670.3588.80%. P9 < Jul 17,537 17,5375.640.1990.30%85.90% P10 17,380 17,3806.160.2389.30%. P9Jul 713 71319.212.9479.20%71.10% P10 790 79017.782.9476.70%. 0 P9 < Jul 16,631 16,6315.420.0091.00%86.80% 0 P10 16,377 16,3775.900.0089.90%. 1+ P9 Jul 272 27217.267.7177.20%62.50% 1+ P10 328 32816.557.0873.20%. ExclusionsJuly RR reduction.041.959 June refusal July.062.938

25. Measure Standard Field Operation50% sample33% sample25% sampleExclusion Difficult Cases Mean Expenditures3479.013477.903478.243521.023543.21 Standard Error91.0392.0292.9394.1794.94 Responding sample3158930769304953035929948 July cases+June temp refusals1641821547411 RSE0.026 0.027 Bias000064.20 MSE8287.108467.378635.938868.8613135.24 RMSE91.0392.0292.9394.17114.61 Ratio of MSE to Standard11.0221.0421.0701.586 Mean no. hours per complete14.222 12.5 Add’l sample to meet precision goal 670 1,284 2,132 Impact on MSE of Mean Medical Expenditure Estimates for alternative field strategies

26. Measure Standard Field Operation50% sample33% sample25% sampleExclusion Difficult Cases Proportion Standard Error Responding sample July cases+June temp refusals RSE Bias MSE RMSE Ratio of MSE to Standard Impact on MSE of Proportion with Medical Expenses in excess of $10,000 for alternative field strategies 0.0756 0.07570.0770.0774 0.00210.0022 0.00230.0022 3158930769304953035929948 1641821547411 0.0280.029 0.0300.028 00000.002 0.000004410.00000484 0.000005298.08E-06 0.00210.0022 0.00230.0028 11.098 1.2001.832

27. Advancing Excellence in Health Care Summary Need for accurate and reliable national data on health insurance coverage to inform policy and practice Need for accurate and reliable national data on health insurance coverage to inform policy and practice MEPS design features and analytical capacity MEPS design features and analytical capacity Statistical, methodological and operational design features to adjust for nonresponse and attrition Statistical, methodological and operational design features to adjust for nonresponse and attrition Evaluation of estimation strategies to correct for survey attrition Evaluation of estimation strategies to correct for survey attrition Examination of ROI for inclusion of difficult cases Examination of ROI for inclusion of difficult cases Options identified for more efficient and effective field strategies Options identified for more efficient and effective field strategies

28. Agency for Healthcare Research and Quality Advancing Excellence in Health Care www.ahrq.gov