Who Needs RDD? Combining Directory Listings with Cell Phone Exchanges for an Alternative Sampling Frame Presented at AAPOR 2008 New Orleans, LA May 16, 2008

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 2... A unit of the Weldon Cooper Center for Public Service Thomas M. Guterbock James M. Ellis Abdoulaye Diop Kien Le John Lee Holmes CSR—University of Virginia

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 3 The Research Problem: RDD under threat Are there good alternatives?

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 4 RDD under threat Random Digit Dialing involves a certain degree of inefficiency Costs of this extra effort justified by completeness of coverage (at least until recently) Recent trends raising these costs: –decreasing density of working numbers, –increasing rates of non-contact, –and rising rates of refusal Advent of cellular phone only households diminishes completeness of its coverage

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 5 A “New Norm?” Dual-frame “RDD+Cell” has arisen in response to these challenges –traditional list-assisted RDD sample with RDD of working cellphone exchanges. To screen or not to screen? –And, if not, how to weight? But other dual frames may also be worth exploring...

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 6 Proposed Alternative: EWP+Cell EWP+Cell = –“Electronic White Pages” + Cell Phone RDD Promises considerably greater efficiency and cost savings over RDD+Cell –especially for specific, small geographic regions –or areas not co-extensive with any set of telephone Area Codes. EWP+Cell fails to cover: unlisted landline households that have no cell phone –We will examine: How big a problem is that?

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 7 Data Source: 2006 National Health Interview Survey permits estimations of the size of... the non-covered segment demographic characteristics health characteristics degree of coverage bias

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 8 What did we find? A Preview Surprisingly little coverage bias to be expected from EWP+Cell Potential cost savings from EWP+Cell compared to RDD+Cell

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 9 A brief review of the research Not much literature or research compares directory- listed samples with list-assisted, landline RDD samples Consequently, the degree and nature of the differences between listed and unlisted households is not established.

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 10 Older studies (before 2002) Most found only slight differences in substantive results between EWP and RDD frame samples Some efficiency gains in smaller geographic areas Some differences in demographics noticed And in 2007, Zogby announced plans to rely on EWP over RDD phone samples, citing lack of substantive differences in results.

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 11 Recent studies on EWP vs RDD: Substantial differences shown Unlisted rates are higher for: Blacks, Hispanics Lower income Renters Single people See: Guterbock, Diop and Holian (2007)

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 12 From 3 segments to 5

The universe of U.S. telephone households, 2006

RDD samples cover all landline households, listed or not RDD Cell-phone- only households are excluded

Cell phone samples include some that are also in the RDD frame Cell phones Landline- only households are excluded

RDD and Cell samples overlap, yield complete coverage Cell phones RDD CELL ONLY 16.6% CELL + LANDLINE 52.0% LANDLINE ONLY 31.4% All percentages are from 2006 NHIS data.

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 17 We need also to consider: listedness Some landlines are listed in the residential directory or Electronic White Pages [EWP] –LLL = Listed Landline Some landline households are unlisted –ULL = unlisted landline LLL and ULL may or may not also have a cell phone in the household Cell phones are unlisted by definition Result: five segments of the telephone universe...

Five telephone segments 1 CELL ONLY 16.6% 2 CELL + ULL 17.7% 4 CELL + LLL 34.3% 3 ULL ONLY 14.2% 5 LLL ONLY 17.2% All percentages are from 2006 NHIS data. See table I

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 19 Five segments differ, sometimes sharply

Segments differ on key demographics 1 Cell Only 2 Cell + ULL 3 ULL Only 4 Cell + LLL 5 LLL onlyAll Percent African/ American Percent 18 – All percentages are from 2006 NHIS data. See table II

Segments differ on key health questions 1 Cell Only 2 Cell + ULL 3 ULL Only 4 Cell + LLL 5 LLL onlyAll Smokers (yes) Diabetes (yes) All percentages are from 2006 NHIS data. See table II

Our analysis deals with three sampling frames: 1)EWP 2)List-assisted (landline) RDD 3)Cell phone RDD

We examine 4 sampling designs: 2 Single frame designs: –EWP only –Landline RDD

Center for Survey Research University of Virginia Center for Survey Research University of Virginia We examine 4 sampling designs 2 Dual Frame designs: –EWP+Cell –RDD+Cell

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 25 Three design contrasts RDD+Cell is the base for all comparisons –It includes the full universe of phone HH We will compute coverage bias for each contrast: EWP vs. RDD+Cell RDD vs. RDD+Cell EWP+Cell vs. RDD+Cell

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 26 Formula for coverage bias Ῡ = mean for full population Ῡ C = mean for covered cases Ῡ U = mean for cases not covered U = cases not covered N = all cases

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 27 Contrast I: EWP vs. RDD+Cell telephone samples

The universe of U.S. telephone households Ῡ

EWP sample excludes unlisted landline and cell-only 2 EXCLUDED CELL + ULL 17.7% 4 CELL + LLL 34.3% 3 EXCLUDED ULL ONLY 14.2% 5 LLL ONLY 17.2% EWP All listed landline phones ῩCῩC ῩUῩU U/N = EXCLUDED CELL ONLY 16.6%

Coverage bias table: EWP vs. RDD+Cell U/N ῩCῩC ῩUῩU Ῡ C - Ῡ U ῩῩ C - Ῡ Percent African/ American Percent All percentages are from 2006 NHIS data. See table IV

Coverage bias table: EWP vs. RDD+Cell U/N ῩCῩC ῩUῩU Ῡ C - Ῡ U ῩῩ C - Ῡ Smokers (yes) Diabetes (yes) All percentages are from 2006 NHIS data. See table IV

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 32 Contrast II: RDD vs. RDD+Cell only telephone samples

RDD samples cover all landline households, listed or not RDD ῩCῩC ῩUῩU

RDD fails to cover 16.6% Cell phones RDD CELL ONLY 16.6% CELL + LANDLINE 52.0% LANDLINE ONLY 31.4% All percentages are from 2006 NHIS data. U/N =.166

Coverage bias table: RDD vs. RDD+Cell U/N ῩCῩC ῩUῩU Ῡ C - Ῡ U ῩῩ C - Ῡ Percent African/ American Percent All percentages are from 2006 NHIS data. See table V

Coverage bias table: RDD vs. RDD+Cell U/N ῩCῩC ῩUῩU Ῡ C - Ῡ U ῩῩ C - Ῡ Smokers (yes) Diabetes (yes) All percentages are from 2006 NHIS data. See table V

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 37 Contrast III: EWP+Cell vs. RDD+Cell telephone samples

RDD+Cell covers all phone households Cell phones RDD CELL ONLY 16.6% CELL + LANDLINE 52.0% LANDLINE ONLY 31.4%

EWP + Cell Sample Design EXCLUDES ULL- ONLY Cell EWP All listed landline phones

EWP + Cell excludes ULL-only households 1 CELL ONLY 16.6% 2 CELL + ULL 17.7% 4 CELL + LLL 34.3% EXCLUDED: 3 ULL ONLY 14.2% 5 LLL ONLY 17.2% ῩUῩU ῩCῩC U/N =.142

Coverage bias table: EWP+Cell vs. RDD+Cell U/N ῩCῩC ῩUῩU Ῡ C - Ῡ U ῩῩ C - Ῡ Percent African/ American Percent All percentages are from 2006 NHIS data. See table VI

Coverage bias table: EWP+Cell vs. RDD+Cell U/N ῩCῩC ῩUῩU Ῡ C - Ῡ U ῩῩ C - Ῡ Smokers (yes) Diabetes (yes) All percentages are from 2006 NHIS data. See table VI

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 43 Summary of 3 contrasts

3 contrasts: 2006 estimates ( Ῡ C ) Variables RDD+Cell EWPRDD EWP+Cell Demographics African American11.6%9.1%11.3%11.0% %10.0%11.8%15.3% Health Related Questions Smokers (yes)20.5%18.6%19.3%20.3% Diabetes (yes)7.9%9.0%8.6%7.7% See table VII

3 contrasts: 2006 raw bias ( Ῡ C - Ῡ ) Variables RDD+Cell EWPRDD EWP+Cell Demographics African American---2.5%-0.3%-0.6% %-3.2%0.3% Health Related Questions Smokers (yes)---1.9%-1.2%-0.2% Diabetes (yes)--1.1%0.7%-0.2% See table VIII

3 contrasts: 2006 percent bias Variables RDD+Cell EWPRDD EWP+Cell Demographics African American %-2.6%-5.2% %-21.5%1.8% Health Related Questions Smokers (yes)---9.3%-5.9%-1.0% Diabetes (yes)--13.9%8.9%-2.5% See table IX

Center for Survey Research University of Virginia Center for Survey Research University of Virginia Changes in Coverage NHIS

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 48 Changes in telephone status over time unlisted landline only cell phone only

49 Changes in percent bias over time: RDD vs. RDD+Cell (NHIS data)

Changes in percent bias over time: EWP+Cell vs. RDD+Cell (NHIS data)

Changes in percent bias over time: RDD vs. RDD+Cell (NHIS data)

52 Changes in percent bias over time: EWP+Cell vs. RDD+Cell (NHIS data)

Center for Survey Research University of Virginia Center for Survey Research University of Virginia Cost comparisons

ABC Traditional Design"New Norm"Proposed RDD onlyRDD + CellEWP + Cell RDD CellEWPCell target N per hour cost$32 CPH * * CPH= completions per hour

Cost comparisons (data collection only) ABC Traditional Design"New Norm"Proposed RDD onlyRDD + CellEWP + Cell RDD CellEWPCell target N cost$29,091$23,273$11,636$18,286$11,636 Total$29,091$34,909$29,922

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 56 Conclusions EWP+Cell omits ULL–onlies, but: –These aren’t particularly untypical –They are not numerous –Their numbers are declining EWP+Cell includes unlisteds (who have cell phones) thus avoiding some bias from EWP EWP+Cell includes cell–onlies, offsetting bias from omitting unlisted HH –Unlisted are somewhat similar to the cell–onlies.

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 57 Conclusions We propose EWP+Cell as a cost effective sampling solution that appears to offer good coverage across age, race, most demographics, and key health indicators It offers a significant cost advantage over the ‘new norm’ (RDD+Cell), especially if: –geographic area to be studied is small –target population is hard to find (requiring screener calls)

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 58 Cautions & caveats We have not offered a direct, experimental field test of EWP+Cell sampling in contrast with RDD+Cell –But we have several such experiments planned in our 2008 local surveys Nobody knows the proper weights for combining the two sample frames – but these are also unknown for local studies that use “the new norm” –RDD+Cell 2008 phone segments could differ from 2006 more than we think

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 59 More cautions... Our analysis assumes that non-response and measurement errors are the same in realized samples from each telephone segment –That is, we have considered coverage error only Even if the proportion of excluded cases (ULL- onlies) is declining, coverage error may not decrease, because: as changes, so can.

Center for Survey Research University of Virginia Center for Survey Research University of Virginia 60 Planned tests We are currently conducting a county-wide citizen survey in Prince William County, VA, that features –A 10% cell phone component (unscreened) –The balance of completions split: 45% RDD sample 45% EWP sample We will be able to compare directly: –EWP+Cell vs. RDD+Cell –Coverage bias, productivity, and costs

Who Needs RDD? Combining Directory Listings with Cell Phone Exchanges for an Alternative Sampling Frame Presented at AAPOR 2008 New Orleans, LA May 16,