1. Data Collection Mode Effects Controlling for Sample Origins in a Panel Survey: Telephone and Internet Presented at the 2005 Annual Meeting fo the American Association of Public Opinion Research Mike Dennis, Knowledge Networks Cindy Chatt, University of Nebraska-Lincoln Alicia Motta-Stanko, Knowledge Networks Paul Pulliam, RTI International

2. 2 Acknowledgements The survey designed by: RTI International The Odum Institute at the University of North Carolina The authors wish to thank RTI International for permission to analyze the data used in this paper and disseminate the findings.

3. 3 The Purpose of the Research What matters more in accounting for variance in survey responses? Mode of Data Collection (Telephone versus Internet) or Sample Origin (KN panel sample versus KN panel rejecters/non-cooperators) This is a contest of data collection mode versus sample quality. The main purpose of this research is to help further our understanding of the respective contributions of mode of data collection (phone vs. web) and sample origin (cooperators vs. non- cooperators) in explaining variance in survey response data.

4. 4 Background of Study The analysis is based on the Survey on Civic Attitudes and Behaviors After 9/11. Study was created with a non-response follow-up study and implemented by Knowledge Networks from January to March 2002. The study is important because: It is the first mode-effects study using KN data that effectively controls for sample type (persons who joined the KN panel and participate in KN studies, versus those who refused to join the panel or who refused to participate in the panel study).

5. 5 Questionnaire: Public Policy & Civic Attitudes & Behaviors The survey questionnaire had several modules of interests: ratings of Bush and Gore (early 2002!), attitudes toward terrorism, the adequacy of governmental response to bioterrorism and terrorism more generally, attitudes toward what the government should be doing and would do in the event of a terrorist attack, civic participation and civic values questions, and background questions on religious faith and other aspects. The survey was approximately 25 minutes in length.

6. 6 Main Finding The mode of data collection is more often a significant factor in accounting for variance in survey responses than is the origin of the sample. When controlling for panel experience and demographics, for every one question where sample origin was a significant factor, mode of data collection was significant for six questions. MODE Sample

7. 7 Sample Design for the Study RDD Sample Panel Acceptors Internet Sample Telephone Sample (Controls) Panel Rejecters Non-Response Follow-Up Sample (NRFUS) Random sample for each of three groups.

8. 8 Sample Composition, Mode, & Sample Size Sample Group/Mode Sample Size Fielded Completed Interviews Completion Rate KN Panel by Web3627297982% KN Panel by Telephone (Controls) 47730063% Nonresponse 1 (NRFUS) by Telephone 273060022% 1 Nonresponse sample consists of ‘panel rejecters’ and those panelists that did not respond to the Internet survey.

9. 9 Sample Representativeness: Demographic Comparison Each interviewed group was compared to the January 2002 Current Population Survey (Census) Main Findings:  Mean Error (Unweighted): –Panel by Web: 2.8 percentage points –Panel by Phone: 4.1 percentage points –NRFUS by Phone: 3.6 percentage points

10. 10 Summary of Multivariate Analyses: Count of Statistically Significant Predictors of Answers by Mode and NRFUS (p <.05) N items N significant for mode N significant for NRFUS Grade Bush's performance (attitudinal)2 1 Worried about terrorism (attitudinal)22 Information expected during bioterrorist event (attitudinal)65 Sources from which anthrax information sought (behavioral)662 Trusted source during bioterrorism event (attitudinal)1 Feeling thermometers for Bush and Gore (attitudinal)22 Important issues, politics, current events (behavioral)33 Neighborhood statements (combination - attitudinal and behavioral)1391 Self-perception statements (attitudinal)552 Volunteerism and/or donating behavior (behavioral)42 44346 100%77%14%

11. 11 Results of Multivariate Analyses Phone respondents more likely to:  Disagree that bioterrorism is the most important problem  Seek info on anthrax from …web, hotlines, national TV, own doctor, local government, other  Rate Pres. Bush and Al Gore higher on feeling thermometer  Discuss politics  Discuss community issues  Help neighbors  Be happy about their neighborhood  Have pride in their neighborhood  Have a sense of belonging in neighborhood  Have a neighborhood that gives them pleasure  Rely on neighbors  Trust others  Enjoy mixing socially with others

12. 12 Freq Distributions by Group (Unweighted) QuestionPanel by Internet Panel by Telephone NRFUS by Telephone Rate George Bush for job done as President (% ‘A’ grade) 4240 Grade Bush on dealing with terrorism (% ‘A’ grade) 57 51 No large differences across the groups for grading political job performance.

13. 13 Freq Distributions by Group (Unweighted) Question: When a bioterrorism event happens, what info do you expect CDC to provide the public? Panel by Internet Panel by Telephone NRFUS by Telephone Type of GERM (% Yes)939496 Name CRIMINAL Suspects (% Yes)3860 No. of people dead, ill, etc. (% Yes)718588 Explanation of steps people should take to minimize risk (% Yes) 919798 Recommendations for treating illness caused by germ (% Yes) 8897 Travel advisories/restrictions (% Yes)7089 Telephone mode: Higher expectations for information from CDC.

14. 14 Freq Distributions by Group (Unweighted) Question: Sought info on ANTHRAX or other SEVERE ILLNESSES from… Panel by Internet Panel by Telephone NRFUS by Telephone Local radio and TV464340 WEB health and news sites122617 Phone HOTLINES156 Cable 24-hour News and Network News 264643 Own Physician or Health Care Professional 41012 Local or State Health Department31011 Telephone mode: Higher report of information-seeking behavior.

15. 15 Freq Distributions by Group (Unweighted) Question: Within the past year, how often have you… Panel by Internet Panel by Telephone NRFUS by Telephone Borrowed, loaned, etc, small things with neighbors like sugar (% Sometimes & Often) 394346 Helped people in this neighborhood or they helped you with small favors (% Sometimes & Often) 607170 Telephone mode: Higher report of ‘good neighbor’ behavior.

16. 16 Freq Distributions by Group (Unweighted) Civic Self-Perception Statements (% Completely Agree) Panel by Internet Panel by Telephone NRFUS by Telephone I am happy to live in this neighborhood354957 See myself as part of neighborhood213740 Feel sense of belonging to neighborhood 223842 Being in neighborhood gives me a lot of pleasure 233943 Would get my neighbors to fix problems92322 Telephone mode: Higher report of positive feelings toward own neighborhood.

17. 17 Freq Distributions by Group (No weight) Sociability Self-Perception Statements (% Completely Agree) Panel by Internet Panel by Telephone NRFUS by Telephone I am trusting of others92322 I easily fit into groups92832 I like to mix with others133139 I tend to be a happy person133139 I enjoy helping others234146 Telephone mode: Higher report of positive sociability self-perceptions.

18. Discussion

19. 19 Major Differences in Response Caused by Mode? Major differences existing between Internet and telephone modes of data collection include:  Presence or absence of interviewer  Dependence on visual or aural communication  Interviewer or respondent control of interview pace or information sequence Each difference can lead to multiple mode effects seen in the data. (Dillman, Sangster, Tarnai, and Rockwood, 1996)

20. 20 Presence or Absence of Interviewer Can lead to:  Tendency to answer on extreme ends of response scales  Non-differentiation  Social desirability / self-presentation bias

21. 21 1. I am happy to live in this neighborhood. 2. I really see myself as a part of this neighborhood 3. I feel a sense of belonging to this neighborhood. 4. Being in this neighborhood gives me a lot of pleasure. 5. If there are things in my neighborhood that need to be fixed or improved, I would be able to get my neighbors to do something about it. Presence or Absence of Interviewer: Neighborhood Statements

22. 22 Very similar results for self-perception items (e.g., “I am trusting.”)

23. 23 Evidence of Non-Differentiation and Recency Effects on Phone (Neighborhood Statements) Response: Completely Agree (+5) InternetTelephoneNRFUS All same Answer4.9%16%*13.8%* 4 out 5 Same10.2%14.8%*17.3%* Response: Neutral (0) InternetTelephoneNRFUS All same Answer2.1%0.1%1.4% 4 out 5 Same2.5%1.0%2.4% Response: Completely Disagree (-5) InternetTelephoneNRFUS All same Answer2.0%1.1%1.0% 4 out 5 Same1.6%0.4%0.9% *Significant differences at p<.05 between Internet and Telephone and Internet and NRFUS for Completely Agree

24. 24 Conclusions Sample origins do not appear to be having much of an effect on results compared to mode of data collection Differences in the data are consistent with inherent differences between Internet and telephone data collection, leading to patterns in the phone-collected data reflecting:  Social desirability bias  Extreme positive responses on answer scales  Answer distribution differences due to communication type differences  Item Non-differentiation Study limitation/Buyer Beware: Questionnaire potentially laden with more questions (compared to many surveys) susceptible to social desirability bias.

25. Thank you! mdennis@knowledgenetworks.com (650) 289-2160