Continuous Surveys: Statistical Challenges and Opportunities Carl Schmertmann Center for Demography & Population Health Florida State University

Outline  CHALLENGES (long) Increased Temporal Complexity Increased Sampling Error New Weighting Problems  OPPORTUNITIES (brief, but important)

Sample Size Comparison  US CENSUS LONG FORM: % / decade  ACS ROLLING SURVEY: 2 per 1000 Households / month 24per 1000 Households / year 240per 1000 Households / decade % / decade

Sampling Differences over Decade Long FormACS Sample Size≈ 17%≈ 24% Taken on…1 day3650 days Released as…1 dataset10+ datasets Simultaneous 100% count? YESNO

1. Temporal Complexity Long FormACS Sample Size≈ 17%≈ 24% Taken on…1 day3650 days Released as…1 dataset10+ datasets Simultaneous 100% count? YESNO 1. Temporal Complexity

What is the Population?  1-Day Census Population membership is binary: {0,1} Each individual is IN or OUT  Continuous Survey Population membership is fuzzy: Individuals can be MORE IN (more person-days of residence) or MORE OUT (fewer) 1. Temporal Complexity

JFMAMJJASOND● Type A Type B ● Residents (in 000s)

1. Temporal Complexity JFMAMJJASOND● Type A Type B ● Residents (in 000s) Census Population = (83% Type A)

1. Temporal Complexity JFMAMJJASOND● Type A Type B ● Residents (in 000s) An ACS ‘Data Sandwich’ includes samples from all months

1. Temporal Complexity JFMAMJJASOND● Type A Type B ● Residents (in 000s) ACS samples from person-months Avg Population: (65% Type A)

Characteristics change over the Sampling Period  Persons Age Marital Status Employment Education  Housing Units Vacancy Number of Occupants $ Value 1. Temporal Complexity

Rolling ‘Population’ Population formed by sandwiching monthly samples is the average frame of a film, not a snapshot Individuals and housing units with changing characteristics are sampled and caught ‘in motion’. 1. Temporal Complexity

Reference Period Problems Many ‘long-form’ questions refer to retrospective periods:  Income in last 12 months  Place of residence 1 year ago  Child born in last 12 months?  Etc. 1. Temporal Complexity

Time Reference Example  ‘2004’ data from 12 monthly samples taken in Jan04…Dec04  Question on fertility in the 12 months prior to the survey, so there are 12 overlapping periods in ‘2004’ data ‘Jan04’ question covers Jan03-Jan04 ‘Feb04’ question covers Feb03-Feb04 etc. 1. Temporal Complexity

Jan 2004 x x x x x x x x x x x x ● Jan 03Jan 04 Feb x x x x x x x x x x x x ● Mar x x x x x x x x x x x x ● Apr x x x x x x x x x x x x ● May x x x x x x x x x x x x ● Jun x x x x x x x x x x x x ● Jul x x x x x x x x x x x x ●..... Aug x x x x x x x x x x x x ●.... Sep x x x x x x x x x x x x ●... Oct x x x x x x x x x x x x ●.. Nov x x x x x x x x x x x x ●. Dec x x x x x x x x x x x x ● Jan Temporal Complexity

Reference Periods for ‘Last 12 Month’ Questions in 1-year ACS Datasets

Temporal Issues Summarized ‘Data Sandwiches’ contain:  New meaning of ‘population’  Units that change over sampling period (moving targets)  Multiple reference periods for retrospective questions 1. Temporal Complexity

2. Sampling Error Long FormACS Sample Size≈ 17%≈ 24% Taken on…1 day3650 days Released as…1 dataset10+ datasets Simultaneous 100% count? YESNO 2. Sampling Error

Small Samples More overall data from continuous sampling, but… 1-, 3-, or 5-Year Sandwiches have smaller samples than the single, decennial long form survey  more sampling error in published data 2. Sampling Error

Small Samples The problem is especially acute for small areas narrow age groups rare subpopulations e.g., How many unmarried teen births per year in Sevier County, Tennessee? ACS says 0 ± Sampling Error

St. Johns County, FL Year ACS Data for Males BELOW POVERTYABOVE POVERTYPOVERTY RATE AGEEstimateMOEEstimateMOEPercentMOE* /-5623,495+/ / / /-4670+/ /-3635,401+/ / /-2922,787+/ / /-3001,342+/-4600+/ /-3001,995+/-4170+/ ,235+/-6555,387+/ / /-37110,192+/ / /-19411,558+/ / /-39912,794+/ / /-45210,679+/ / /-2005,825+/ /-3.3 *Denominators have MOE≈0 under ACS sampling and weighting design

2. Sampling Error C SEX BY OCCUPATION – Key West, Florida Data Set: American Community Survey 3-Year Estimates ( …etc

Temporal Instability Teenage Birth Rate in a County

Unfortunate Result Aggregating over 1+ years of surveys produces datasets that are often  Unfamiliar and difficult to understand  Still too noisy to be useful for planners and researchers 2. Sampling Error

3. Weighting for Non-Response Long FormACS Sample Size≈ 17%≈ 24% Taken on…1 day3650 days Released as…1 dataset10+ datasets Simultaneous 100% count? YESNO 3. Weighting Problems

Weighting Weighting from Respondents  Total Population requires Population Control Totals: (Place x Age x Sex x Race x Ethnicity x …) 3. Weighting Problems

Decennial Long Form Sample  Control Totals Measured from a simultaneous enumeration of the population (Sample & Census on same day) Only 1 set needed Sample and Population defined identically (resid. on Census Day) 3. Weighting Problems

Continuous Survey  Control Totals Must be estimated (no simultaneous census) Many sets needed (2006, 2007, , , , …) Sample and Population defined differently 3. Weighting Problems

ACS Control Totals (Persons) 3. Weighting Problems ACS responses are weighted to match official intercensal estimates by Year (1 July midpoint snapshot) County (sometimes city) Age Race Sex Hispanic Origin (yes/no)

ACS Control Totals (Persons) 3. Weighting Problems Potential Errors  Estimates are Wrong: Unanticipated internal migration Unanticipated international migration etc  Population Definition don’t match Seasonal fluctuations Different race/ethnic categories

3. Weighting Problems JFMAMJJASOND● Type A Type B ● Census Pop = (83% Type A) Average Pop= (65% Type A) If every year looks like this… Intercensal Estim= (83% Type A)

Weighting Error Example ACS weighting to estimates produces:  Popn too small (Census < Avg Pop)  Popn too “A” (seasonal Bs missed)  Overestimates of vars + correl. with A (e.g., % with college education)  Underestimates of vars - correl. with A (e.g., % single-parent families) 3. Weighting Problems

Opportunities Census Survey Continuous Survey Frequency Recency Sample Error Familiarity 4. Opportunities

Statistical models that exploit likely cell relationships (over times, ages, sexes, places, variables …) could, in principle Opportunities ACS table cells = millions of “seemingly unrelated” maximum likelihood estimates 4. Opportunities Retain frequency & recency Reduce variance of estimates Recover familiar measures

Conclusion 5. Conclusion CONTINUOUS SURVEYS like ACS create  Big Problems for producers and users Unfamiliar, temporally complex data Potentially high sample error Technical problems with weighting  Big Opportunities, IF we can develop appropriate statistical models and practices

5. Conclusion Thanks! ¡Gracias! Obrigado!