Presentation is loading. Please wait.

Presentation is loading. Please wait.

Survey Nonresponse: A Decision- Making Approach Roger Tourangeau Joint Program in Survey Methodology, University of Maryland Survey Research Center, University.

Similar presentations

Presentation on theme: "Survey Nonresponse: A Decision- Making Approach Roger Tourangeau Joint Program in Survey Methodology, University of Maryland Survey Research Center, University."— Presentation transcript:

1 Survey Nonresponse: A Decision- Making Approach Roger Tourangeau Joint Program in Survey Methodology, University of Maryland Survey Research Center, University of Michigan

2 1 Outline Falling Response Rates –Many surveys taking countermeasures –Costs are rising, but response rates still falling Whats Behind It? –Variety of theories –Lack of civic engagement; value of polls and related activities –Decrease in discretionary time –Fending off unwanted intrusions now routinized At the individual level, how do people decide? –The Salience-leverage model –Belief-sampling as a model for quick judgments –The heuristics approach

3 2 Outline (Contd) Does the decline matter? The relation between nonresponse rates and nonresponse error The impact on surveys –Rising use of incentives: Is there a cost? –Surveys as opportunity/obligation vs. survey as transaction Many thanks to Bob Groves, from whom I stole many slides (and many ideas)!!

4 Falling Response Rates

5 4 What is Nonresponse? Unit nonresponse is the failure to obtain survey measures on a sample unit It occurs after the sampling step of survey It reflects total failure to obtain survey data (wont talk about item nonresponse, the failure to obtain an answer to a given item)

6 5 Total Nonresponse and Refusal Rates Increasing over Time for BLS Consumer Expenditure Survey Census Bureau

7 6 Until Recently, Current Population Surveys Rates Had Been Stable Census Bureau

8 7 National Health Interview Survey Nonresponse Trends

9 8 Also Apparent for Telephone Surveys: Survey of Consumer Attitudes Thanks to Rich Curtin for the data

10 9 Multi-Country Studies of Cooperation (de Leeuw & de Heer, 2002) 16 countries (Western Europe and U.S.) As many as 10 ongoing surveys (mostly central government sponsored), mid-1980s-late 1990s –Labor force –Consumer expenditure –Health –Travel On average, 3 percentage point decline per year in cooperation rate

11 10 Expected Proportion Noncontacted and Refused Under deLeeuw & deHeer Model by Year

12 11 Impact on Costs per Case Costs have risen as surveys take countermeasures More extensive use of advance letters (especially in telephone surveys where they werent used before) More extensive use of incentives (revisit issue of cost impact) More callbacks –Steeh et al. (2001) present evidence that Michigans Survey of Consumer Attitudes used to take around 6 calls per complete (mid-1990s) –By 1999, it took 12 calls on average Seems clear in U.S. that survey costs rising far faster than inflation

13 Theories of Response and Nonresponse: The Sociology of Nonresponse

14 13 Three Forms of Nonresponse Noncontact Noncooperation Inability Unfavorable societal developments on all three fronts

15 14 Noncontact: Face-to-Face Surveys Rise of doormen buildings, locked condos, and gated communities More than eight million American now live in gated communities and nearly 40 percent of newly built residential developments are gated (Blakely and Snyder, 1997) New residential arrangements featuring gatekeepers (assisted living, nursing homes, etc)

16 15 Noncontact: Telephone Surveys Rise of answering machines, Caller-ID, cell phones By 1995, most U.S. households had answering machines and roughly 40 percent reported they used them to screen their calls (Tuckel and ONeill, 1995) By 1996, about 10 percent of all households nationally had Caller-ID.

17 16 Inability To Provide Data Reflects both physical/mental limitations and language barriers –Rising proportion of the population is 65 or older –Concomitant increase in hearing problems, other disabilities Increase in immigrant populations –2002: 11.5 percent of the U.S. population was foreign-born –According to Long Form data from Census 2000, 8.1 percent of the population over age five reported that they speak English less than very well. –Many surveys now field both Spanish and English questionnaires, but only two-thirds of those who are less than completely fluent in English are Spanish speakers.

18 17 General Theories Regarding (Non)- Cooperation Still, the big problem is non-cooperation Some theories are couched in terms of societal trends, others based on person-level characteristics Nonetheless, although level of analysis is different, the ultimate causal mechanisms in these theories are similar Three accounts widely cited –People are too busy –People are too self-absorbed –People are erecting barriers to unwanted intrusions Both noncontact and non-cooperation may be the result

19 18 Too Busy More people are labor force participants (e.g., 66.0% of all civilians, 16+ in U.S. were in the labor force in 2004 vs. 60.2% in 1970) The change is particularly dramatic for women (from whom respondents are disproportionately drawn): 59.2% of all women 16+ in the U.S. were in labor force in 2004 vs. 43.3% in 1970 60% of women who worked at all during 2003 were full- time (vs. 41% in 1970) Societal trend with individual-level impact: Opportunity costs of survey participation too high

20 19 But Are People Really any Busier? Fewer adults are married than 25 years ago; also, fewer are parents More and more people are retired and they are retiring at younger ages; according to Robinson and Godbey, Americans aged 55-64 gained an average of ~10 hours of free time per week since 1965 Again, according to Robinson and Godbeys time diary studies, Americans have gained about 5 hours of free time per week on average since the 1960s Nonetheless, people feel busier, in part because of relentless multitasking Based on their perception, they may be more reluctant to give up free time

21 20 Too Self-Absorbed Many survey researchers subscribe to one version or other of the social capital hypothesis Response rates falling for the same reason as declines in voting, other forms of civic participation; people feel less obligated, less interested in helping others

22 21 Groves, Singer, and Corning (2000) Groves, Singer, and Corning (2000) assessed community participation in face-to-face survey: five items on joining organization to solve some community problem, writing to officials, doing volunteer work, etc. Civic duty: A feeling of obligation to provide help … in the belief that the common good is thereby served. Apparently independent mail survey request of those completing a face to face survey, with reported community involvement attributes $5 incentive experiment, prepaid

23 22 Results Overall, about 15% difference in response rates: 58.0% (262) vs. 43.1% (116); even bigger diff. with no incentive

24 23 Implications Could explain why election polls, which traditionally get low response rates, nonetheless generally give accurate results Those most likely to vote overrepresented in polls; both surveys and elections overrepresent those high in involvement, social capital

25 24 Too Many Unwanted Intrusions Modern life often seems to consist of continuous bombardment with unwanted information, intrusions via every medium –Junk mail –Telemarketing –Spam –Panhandling in big cities In response, people take countermeasures to limit access –Spam filters –Do Not Call lists, Caller-ID, answering machines –Crackdown on panhandling in NYC and elsewhere –Gated communities, locked apartment buildings, etc.

26 25 Unwanted IntrusionsII May reflect diminished community involvement, sense of busyness Whatever the cause, contactability and willingness to cooperate may not be distinct phenomena but reflect effects to fend off unwanted contacts

27 Theories of Response and Nonresponse: The Psychology of Nonresponse

28 27 Leverage-Salience Theory of Survey Cooperation How do these societal trends play out at the individual level? Leverage-salience offers one important account –Persons vary in the magnitude and direction (positive and negative) of influence of various psychological predispositions toward survey participation in general and toward various design features (topic, sponsor): Leverage –The information about the survey request processed by the person varies due to interviewer variation in introductory scripts and their cognitive associations with information provided (Salience)

29 28 Burden Incentive Authority of Sponsor Topic Person 1 Person 2 Authority of Sponsor

30 29 Burden Incentive Authority of Sponsor Topic Person 1 Person 2

31 30 Implication of Leverage-Salience Theory for Nonresponse Bias People make decisions to cooperate or refuse on different bases If the salience of design attributes or focus of interviewer behavior systematically varies over contacts, then decisions can be based on different weightings of attributes over contacts Bias results when same survey attribute related to survey variable and survey participation decision (common cause model)

32 31 Groves, Presser, and Dipko I Sample from five frames (four list samples plus RDD) teachers, parents of children under 6 months, people 65+, contributors to fringe candidates Two IVs: Topic (intro. Mentions topic twice) and letter plus incentive (half get letter plus $5) Done by Maryland SRC plus 12 Practicum students Response rate ~63.0

33 32 Groves, Presser, and Dipko II Key result: Outcome of first contact in which topic mentioned Incentives diminish topic effects for teachers and seniors, but increase them for parents

34 33 The Belief-Sampling Model Model of attitude judgments made on the fly; four key components; salience-leverage a special case 1.Determine the issue (the pool of beliefs, values, impressions, existing judgments) from which sample will be drawn 2.Sample some of these considerations (i.e., think about the issue)probability of retrieving any given consideration related to its accessibility (salience may determine accessibility) 3.Scale (leverage) and integrate Anchor-and-adjustment Average/weighted average 4.Map onto response scale

35 34 Formal model Integration Phase Reliability over two occasions n1: Number of considerations sampled at time 1 n2: Number of considerations sampled at time 2 ρ1: Consistency in assigning scale values (scaling consistency) ρ2: Homogeniety in pool of considerations (homogeniety) q: Overlap between samples, expressed as proportion of n2 (overlap)

36 35 An Alternative: Rules/Policies for Dealing with Unwanted Intrusions Both leverage-salience and belief- sampling assume that people make a decision at the time survey request is made Maybe their past decisions have hardened into a policya set of unconscious/semi-conscious procedures for dealing with unwanted intrusions

37 36 Example Policies Throw out all mail, unless –Its clearly a bill –Its clearly a personal letter –Its from a familiar organization to which you are sympathetic Hang up/dont answer telephone, unless –Its a personal call from someone you know –Its from a familiar organization to which you are sympathetic –Its a business call that you are expecting

38 37 Implications for Nonresponse Bias Rules could vary by subgroup, changing composition of samples in predictable ways Consistent with studies of telephone response, which indicate nonresponse happens very quickly Important to discover rules and learn 1.How to avoid quick judgment (by making policy seem inapplicable; Im not selling anything) 2.How to get people to suspend rule; people do make exceptions to policies or alter them Advance letters may help (moving call into different categoryexpected business call)

39 Does Nonresponse Matter? Empirical Estimates of the Impact of Nonresponse

40 39 Nonresponse Bias Classic formula for bias in uncorrected mean:

41 40 Nonresponse Bias II Assumes nonresponse deterministic; two types of people –Those who never respond (W N ) –Those who always respond (1- W N ) –Sampling error in mix, so nonresponse rate (and error) is not fixed Alternative: –People have a response propensity (probability that theyll respond); –Response propensity: Probability that theyll be contacted, agree to cooperate, etc. –As a result, even if same people in the sample, outcome might be different

42 41 Nonresponse Bias III Bias now depends on correlation between p (response propensity) and y (substantive variable)

43 42 Empirical Estimates of Bias Three papers, taken together, suggest that as an empirical matter, nonresponse may not be that big of a problem –Keeter et al. compare two surveys that vary in response rates –Merkle and Edelman examine within-precinct error as a function of precinct response rate –Curtin et al.: Look at what would have been result if SCA had stopped interviewing sooner

44 43 Keeter, Miller, Kohut, Groves, & Presser Compared two surveysstandard vs. rigorouswith same questionnaire but different field procedures designed to yield different response rates

45 44 Keeter et al.II Big weaknessconfounds many variables (R rule, advance letter, race and experience of interviewers, etc.) Big strengths –Examine 91 items from a variety of domains –Big differences in response rates 36.0% for standard vs. 60.6% for rigorous Mostly due to differences in contact rates (68.5% vs. 92.0%) rather than cooperation rates (58.1% vs. 73.7% cooperation)

46 45 Keeter et al.III Key result: Differences in estimates –14/91 significant –Mean over 91 items around 2% (Is this big or small? What if this were unemployment rate?) –Largest difference (9 percent) involves interviewer rating of R interest (rigorous adds less interested cases) Other findings –Standard survey doesnt seem to underrepresent Republicans or conservatives relative to rigorous –Reluctant Rs (those classified as refusals) have higher item nonresponse rates

47 46 Merkle and Edelman I Examine relation between within precinct error (relative to actual vote) and precinct response rate Strength: Know the truth and can a direct measure of bias Weakness: Purely correlational Key results: Little relation between nonresponse and error

48 47 Merkle and Edelman II

49 48 Curtin, Presser, and Singer Simulate effect of changing procedures in Survey of Consumer Attitudes (e.g., reducing number of callbacks) How would estimates change if we drop cases interviewed after call x? Converted cases? Percent Significant Differences

50 49 Nonresponse Rates and Nonresponse Bias Overall picture: Little nonresponse bias –Within a certain range (~25 to ~65%) –Not looking for small effects (change in percentage of 0.2%) Two papers lead to different conclusions –Groves, Presser, & Dipko (2004) –Teitler, Reichman, & Sprachman (2003)

51 50 Groves, Presser, and Dipko Samples teachers, parents of children under 6 months, people 65+, contributors to fringe candidates, general pop. Topic (intro. Mentions topic twice) and letter plus incentive (half get letter plus $5) Response rate ~63.0 Impact on estimates: 8 of 12 tests in line with hypotheses, 4 significantly so Part of the effect may be framing: people interpret questions differently based on stated topic

52 51 Teitler, Reichman, and Sprachman I Fragile Families and Child Wellbeing Study –Longitudinal study of new parents and their children –Baseline survey: Births sampled from 75 hospitals in 20 large (200,000+) cities –Total baseline sample size: 4,898 families; follow-ups at age 1 and 3 –90% of mothers complete baseline; 70% of fathers (78% of those where mother completed baseline) –Can characterize nonresponding fathers, since mothers provide some data about them

53 52 Teitler, Reichman, and Sprachman II Baseline data collection –Two survey houses (NORC, MPR); results based on 13 MPR cities –Mothers interviewed in hospital –Fathers in hospital (68%), by phone (10%), or face- to-face at home (2%); 80% interviewed in 13 cities –Costs vary dramatically: Telephone cases 2x hospital cases, FTF cases 6x hospital cases

54 53 Teitler, Reichman, and Sprachman III Estimates by cumulative rate, Respondents vs. All fathers

55 Conclusions

56 55 Transformation of Surveys Response rates lower Survey data collection costs are higher Incentives more widespread –Small prepaid incentives most effective –Less bang for the buck in telephone/face-to-face surveys: lower percentage point gains per dollar expended (Singer, 2002, page 165) –Money more effective than gifts

57 56 Transformation of Surveys II Image of surveys may be being fundamentally altered –Formerly, surveys seen as an opportunity (to be heard) or an obligation –Polls seen as an important and legitimate input to policy discussions –Now, an economic transaction –Incentives as payments rather than tokens invoking norm of reciprocity

Download ppt "Survey Nonresponse: A Decision- Making Approach Roger Tourangeau Joint Program in Survey Methodology, University of Maryland Survey Research Center, University."

Similar presentations

Ads by Google