1. Internet Surveys and Political Attitudes: Evidence from the 2005 British Election Study David Sanders, Harold Clarke, Paul Whiteley and Marianne Stewart University of Essex University of Texas at Dallas Co-funding from ESRC, Electoral Commission and Newsnight Preliminary Findings – not for citation or quotation

2. Four sections 1. Study design – comparing a probability face-to- face sample (response rate 62%) with an internet sample 2. Comparison of marginal distributions on key variables 3. False causal inferences? Comparison of models of turnout and vote using face and internet modes 4. Internet measurement experiments – feedback to respondents

3. Wave 1 Pre-election Probability Sample, Face-to-Face N=3500 128 PSUs Wave 2 Post-election Probability Sample, Face-to-Face N=3220 Including top-up, mail-back; 128 PSUs Wave 3 One Year Out Internet users from Wave 2 Probability Sample, Internet Survey method N=c2000 BES 2005 CORE FACE-TO-FACE PANEL SURVEY: BES 2005 INTERNET CAMPAIGN PANEL SURVEY: Wave 1 Pre-campaign Baseline Survey N=c7000 Wave 2 Campaign survey 200 interviews per Day for 30 days N=c7000 Wave 3 Post-election Interview N=6117 Wave 4 One Year Out Interview N=c4000 Probability Internet sample versus traditional Internet sample Sampling Experiment: Face-to-face vs Internet sampling experiment (2) Face-to-face vs Internet sampling experiment (1) Pre Face-to-Face Survey released June 1 st 2005; Post released August 10 th 2005 Internet Panel (Pre/Campaign/Post) released June 1 st 2005

4. 2. Comparisons of NatCen and YouGov marginals on key variables Party vote shares, partisanship and reported turnout Assessments of Government competence Liking of party leaders Position of respondent and parties on tax/spend scales Position of respondent and parties on authoritarian/liberal scales Interest in politics and personal political influence Attitudes to voting and sense of civic duty

5. Comparison of Voting and Party Identification marginals….

6. Comparison of Turnout marginals….

7. Assessments of Government Policy Competence…

8. Likeability of Party Leaders…

9. Comparison of positions on (left-right) raise taxes and improve services scale…

10. Comparison of positions on authoritarian/liberal scale…

11. Comparison of interest in politics and political efficacy…

12. Comparison of attitudes to voting and civic duty…

13. Conclusions on Comparisons These are two different sets of measures in terms of the overall pattern of marginal distributions. YouGov sample is more politically interested, which may help to explain why YouGov vote share is more accurate and its turnout share less accurate. YouGov sample is less Labour and less left generally in its attitudes than the NatCen sample. But the electorate seems less Labour and less left than the NatCen sample implies. Because of survey non-response in the probability sample, there is no telling which is correct.

14. 3. Do the two samples produce different causal inferences? 1.Turnout model: based on discounted benefits minus costs; civic duty; exposure to personal campaigning and demographics 2.Model of (Labour) vote: based on party identification; policy competence/valence; Downsian ideological distance on tax/spend and authoritarian/liberal scales; leader ratings; economic perceptions; and demographics

15. Turnout Model Effects virtually identical, though civic duty and owner have slightly smaller effects in YouGov sample

16. Vote Model No significant differences in coefficients across the two samples. Only exception is LibDem identification – slightly smaller effect in YouGov sample

17. Conclusions on Causal Inferences Even though the marginals on key variables are significantly different across the two samples, the inter-correlations among variables are similar. As a result, the estimates of the two typical models are very similar across the two surveys. Even in the turnout model, where the marginal distributions are very different, almost exactly the same patterns of effect are observed across the two modes. Crucially, if the probability sample gets close to reality, we would not draw false causal inferences by using the internet sample.

18. 4. Internet Survey Experiments Ask respondent to self-locate on two 0-10 scales: tax/spend and liberal/authoritarian Logic: improve measurement of multidimensional scale responses by inviting respondent to adjust her/his assigned position in a 2-d space [WRONG] Later in survey show respondent where s/he is located in 2-d space defined by earlier responses Ask if respondent wishes to re-locate self Split sample on cues provided (eight experimental groups plus control)

19. Earlier in the survey we asked you for your views about taxes and public spending. We also asked you for your views about the importance of reducing crime versus the importance of protecting the rights of the accused. The graph below indicates the point that we think best summarises your position. The graph also shows the positions of the Labour Party, the Conservative Party and the Liberal Democrat party: Reducing crime more important Rights of accused more important Raise taxes, more spent on services Cut taxes, less spent on services Did we locate you in the right place? Yes: please click to submit No: Please point and click your mouse to indicate the point on the graph that you think best summarises your position Dont know Conservative Party Labour Party Liberal Democrat Party YOU

20. Experimental treatments Treatment 1:CONTROL = Respondents position only Treatment 2:Respondent plus average voter Treatment 3:Respondent plus party supporters Treatment 4:Respondent plus party leaders Treatment 5:Respondent plus leaders with party labels Treatment 6:Respondent plus parties 1983 scenario Treatment 7:Respondent plus parties 1964 scenario Treatment 8:Respondent plus parties 2005 (LD left) Treatment 9:Respondent plus party supporters/leaders

21. Raw Changes in taxes/services and authoritarian/liberal scores, before and after experimental stimuli Are the changes significant? See next slide…

22. Effects of experimental manipulations on respondents self-placements on tax/services and auth/liberal scales Some become more pro-taxes; all except control become more liberal

23. Regardless of the specific stimulus, Conservative and other party supporters respond differently to feedback from Labour and LibDem supporters….

24. Do the pre-stimulus or post-stimulus measures best explain party choice? Similar effects observed when further controls applied for party identification; policy competence; leader ratings; econ perceptions. The model:

25. The pre-stimulus measures clearly explain party choice better than the post-stimulus measures…. Note how post-experiment R2 values are all lower than their pre- experiment counterparts. Same pattern with coeffs, sigs and dps.

26. Summary of Experimental Results Regardless of treatment, Labour and Lib-Dem voters wish to re- position themselves as being more in favour of higher taxes/better services (i.e. they move themselves left). Control group respondents do not wish to change their position on the liberal-authoritarian axis. But all treatment groups wish to re-position themselves as being more liberal on the liberal-authoritarian axis than they first indicated. The effects are most pronounced for Labour and Liberal Democrat voters: when they see where they have located themselves in relation to the parties, they wish to re-describe themselves as more liberal. Pre-treatment measures are better predictors of vote preference than post-treatment measures. Conclusion: additional party-cue information encourages a more politically correct (liberal) response. It doesnt improve measures.

27. Conclusions 1. We conducted the comparison between internet and face probability modes because the low/declining response rate problem means there is a real need systematically to explore the potential and limitations of internet polling. 2. The marginal distributions of our probability and internet polls were typically significantly different from one another. 3. The key difference is that internet respondents are more politically interested than probability sample respondents. 4. The relationship between key dependent variables and big beast predictor variables does not vary significantly across the two modes. 5. There is clearly a big role for internet polling in future academic research. Further studies are needed to assess inter- mode similarities and differences.