Minimizing Differences through Cross-Agency Development of Measures Julie D. Weeks, Ph.D. National Center for Health Statistics 2012 FCSM Statistical Policy Seminar Washington, D.C. December 4-5, 2012

The ACS Disability Measure: A Case Study The development of the six-question ACS disability measure can serve as a model for future collaborations. Starting early in the data collection life-cycle; Involving a broad spectrum of stakeholders; Understanding the disparate needs for the data; Using existing models and conceptualizations as guides; Joint efforts in testing, including cognitive testing and analyses using other data sources, to inform development; Coordinated implementation and research using resulting data; and Promotion to wider federal and data user communities.

Origin of the ACS Disability Subcommittee The ACS disability questions were the same as those administered as part of the 2000 Census long form. In 2002, the Interagency Committee for the ACS created a Disability Subcommittee to ensure that the ACS disability questions would meet the needs of data users. Subcommittee charge: Review the existing disability question set; Recommend changes or improvements to provide a more accurate estimate of the population with disabilities. Resulting recommendations had to be made in time for a) 2006 Survey Content Test, and b) implementation in the 2008 ACS.

The ACS Disability Subcommittee NCHS Census OMB BLS OASPE AoA DoE OSERS NSF VA HUD NIDDR NIMH NCD SSA HIS AHRQ FCC DoJ

Legislative Mandates for Disability Data The ACS must provide information on disability in order to satisfy agencies with statutory or programmatic requirements for disability data. A review of legislative mandates associated with a wide range of agencies revealed that data on disability was used by agencies: in the distribution of benefits or the provision of opportunities, for programmatic needs regarding the overall welfare of the disabled population. Agencies and organizations with legislative mandates included: VA, HUD, SSA, AoA, DOC, DoEd, NIH, CMS, OMH, HIS, DOJ, DOT, EEC, FCC, NIDRR, NSF, NCD, ACF.

Measurement Objectives Based on a legislative review, programmatic needs for data and the constraints of the ACS format, the subcommittee determined that: “equalization of opportunity” would be the primary measurement objective for the disability questions questions would be designed to identify persons who, because of their limitations in functioning, are at risk for discrimination or lack appropriate opportunity for participation in all aspects of social life one measure of severe disability would be included to identify those in need of assistance to maintain independence (for purposes of service provision planning).

Equalization of Opportunities Adopted as Purpose of Measurement Seeks to identify all those at greater risk than the general population for limitations in activity or participation Disability as a demographic Percent of persons with HS diploma

Functioning Domains Question development began by focusing on the functional areas most commonly associated with limitations in participation. Based on analyses conducted on large national datasets, including NHIS, MEPS and SIPP, the functioning domains that were found to best predict the total population estimate of disability, including those identified by mental health problems: Vision Hearing Mobility Cognitive functioning Self care Independence getting outside the home

Testing the New ACS Questions Testing of the new question set was required to understand if the new questions: a) would capture the concepts chosen, b) were more reliable than the old ones, and c) would perform equally well across administration modes. Cognitive Testing: June 2004 – February 2005, 5 rounds of interviews conducted (n=69) at NCHS and Census, 3 data collection modes (face-to-face, telephone, PAPI self-admin) Field Testing: 2006 ACS Content Test 62,900 residences, 48 states split ballot: ½ sample is “control” with ACS production version and ½ sample is “test” with modified content initial mail-out December 2005, mail returned January 2006, non-response follow-up February 2006

Content Test Results Based on the criteria set out before the test, the revised questions used in the test interviews performed better than the control or original questions. Reliability in the test version was equal to or better for all disability items and the summary “with a disability / no disability” measure. Nonresponse in the test version was lower for all disability items and the summary “with a disability / no disability” measure.

ACS Disability Questions as a Basic Standard for Federal Surveys Surveys conducted by other Federal agencies that have adopted the rationale and specific domains of the ACS instrument: CPS SIPP NHIS NCVS FCC AHS Education Increases the standardization of disability questions and data across surveys. Provides an opportunity for crosswalk analyses standard disability items across surveys.

Same Questions: Similar or Different Numbers Using the same questions across surveys does not guaranteed that the same prevalence estimates will be generated. Issues that account for different numbers: Questions asked at the family level or person level Focus of survey - health, employment, crime, or housing Mode of data collection A coordinated approach is needed to understand the differences and appropriately present data to the public

Prevalence of Disability by Survey: 2009 Disability ItemAges NHIS CPS ASEC † (%) Family- Style (%) Person- Style (%) ACS (%) Vision difficulty Hearing difficulty Ambulatory difficulty Cognitive difficulty Self-care difficulty Independent living diff Any disability † NOTE: CPS data are for population 16 years and over.

Prevalence of Any Disability by Survey and Age: 2009 Age NHIS CPS ASEC † (%) Family- Style (%) Person-Style (%) ACS (%) † NOTE: CPS data are for population 16 years and over.

Prevalence of Disability by Survey and Race/Ethnicity: 2009 Race/Ethnicity NHIS CPS ASEC † (%) Family- Style (%) Person-Style (%) ACS (%) Hispanic Non-Hispanic white Non-Hispanic black Non-Hispanic Asian Non-Hispanic Other † NOTE: CPS data are for population 16 years and over.

* CPS and ACS cover persons 16-64; NHIS covers persons

Development of the ACS Disability Measure: Pros and Cons Reduced data collection flexibility (lengthy process); Resource-intensive process (staff, time, testing costs, survey costs); Multiple stakeholders increase costs (logistics); Negotiation is difficult; Compromises in each step of measure development; Comparability (inter- versus intra-agency); Post-data collection coordination prolongs the demands and needs for resources (including data release, production of methodological and analytic documentation, research, and outreach).

The New ACS Disability Measures For sample persons 1 year of age and older: 1. Is this person deaf or does he/she have serious difficulty hearing? 2. Is this person blind or does he/she have serious difficulty seeing? For sample persons 5 years of age and older: 3. Because of a physical, mental, or emotional condition, does this person have serious difficulty concentrating, remembering or making decisions? 4. Does this person have serious difficulty walking or climbing stairs? 5. Does this person have difficulty dressing or bathing? For sample persons 15 years of age and older: 6. Because of a physical, mental, or emotional condition, does this person have difficulty doing errands alone such as visiting a doctor’s office or shopping?