How media cover election polls in the U.S. Presentation to Instituto Nacional Electoral December 5, 2016 Scott Clement Polling Manager The Washington Post

Is the U.S. poll-crazy? 38 national polls conducted during the final week before U.S. presidential election 219 battleground state polls conducted during the final week Wide range in methodologies Random digit dial telephone Including/excluding mobile phones Live interviewer/Automated interviews Web-based methods (opt-in panel, river sample) No face-to-face interview surveys

The ways media cover pre-election polls Horserace – who is ahead, who is behind, and how has this changed Including poll averages How are key demographic and political groups voting Gender, age, race/ethnicity, partisan affiliation Views of candidates’ strengths and weaknesses on issues and personal qualities (i.e. Who will do a better job handling the economy? Which candidate is more honest? Support for key policy proposals during campaign and their connection with vote preferences Any other ways polls are covered?

Little regulation of how polls are conducted and published Federal regulation on dialing of cellular phones without automated dialers Local laws on distance a person must be from voting places impact exit pollsters No legal restrictions on… Type of sampling, interviewing, weighting or likely voter modeling When polls can be released (permitted all the way through Election Day) What details of a poll’s methodology must be disclosed upon release

What influences how polls are reported? Standards for disclosure from professional polling organizations AAPOR Standards and Transparency Initiative National Council on Public Polls (NCPP)

National Council on Public Polls disclosure requirements Sponsorship of the survey Fieldwork provider (if applicable) Dates of interviewing Sampling method employed (for example, random-digit dialed telephone sample, list-based telephone sample, area probability sample, opt-in internet panel, non-probability convenience sample) Population that was sampled (for example, general population; registered voters; likely voters; or any specific population group defined by gender, race, age, occupation or any other characteristic) Size of the sample that serves as the primary basis of the survey report Size and description of the subsample, if the survey report relies primarily on less than the total sample Margin of sampling error (if a probability sample) Survey mode (for example, telephone/interviewer, telephone/automated, mail, internet, fax, e-mail) Complete wording and ordering of questions mentioned in or upon which the release is based Percentage results of all questions reported

AAPOR Transparency Initiative requirements 1. Survey sponsor and who conducted it, as well as all sources of funding. 2. The exact wording and presentation of questions and response options whose results are reported. 3. A definition of the population under study and its geographic location. 4. Dates of data collection. 5. A description of the sampling frame(s) and its coverage of the target population, including mention of any segment of the target population that is not covered by the design (Truncated) 6. The name of the sample supplier, if the sampling frame and/or the sample itself was provided by a third party. 7. The methods used to recruit the panel or participants, if the sample was drawn from a pre-recruited panel or pool of respondents. 8. A description of the sample design, giving a clear indication of the method by which the respondents were selected, recruited, intercepted or otherwise contacted or encountered, along with any eligibility requirements and/or oversampling (truncated). 9. Method(s) and mode(s) used to administer the survey (e.g., CATI, CAPI, ACASI, IVR, mail survey, Web survey) and the language(s) offered. 10. Sample sizes (by sampling frame if more than one was used) and a discussion of the precision of the findings. (Truncated). 11. A description of how the weights were calculated, including the variables used and the sources of weighting parameters.

What influences how polls are reported? Standards for disclosure from professional polling organizations AAPOR Standards and Transparency Initiative National Council on Public Polls Marketing Research Association Media vetting of survey quality Conducted at individual news organizations, standards made public at some large national organizations (Associated Press, ABC News) Can influence whether surveys are covered or not, or how surveys are covered Often behind-the-scenes, but media sometimes cover polls in a critical light (USC/LA Times poll, Gallup 2012 polling) Marketing Research Association – limited requirements The method of data collection used The date(s) of data collection The sampling frame The sampling method The sample size The calculated margin of sampling error

How many election polls abide by professional disclosure standards? Among 38 national polls during final week before U.S. presidential election, 12 were conducted by organizations that are members of AAPOR’s Transparency Initiative or the National Council on Public Polls Among the 219 battleground state polls conducted during the final week, 77 (35%) were conducted by organizations that participate in AAPOR’s Transparency Initiative or the National Council on Public Polls Nearly all polls report some basic information Interview mode, sample size, margin of error and field dates. Less common is disclosure of sample methodology details, full question wording, weighting and likely voter modeling is less common.

How The Washington Post vets polls Researcher is contacted to obtain detailed description of sample design, interviewing, question wording, weighting, and likely voter modeling. While most firms to not publish these details, they are willing to provide them on request. Existence of AAPOR/NCPP standards helps to encourage disclosure. Vetting decisions based on several factors Sample design – probability or non-probability sample? What is sample’s coverage of target population? Mode of interviewing – Live professional interviewers, automated Question wording – Are questions clear and balanced? With vote choice, is there potential for order effects? Weighting approach – to known, reliable and relevant population parameters? Accuracy of polling firm’s previous pre-election estimates? Level of methodological disclosure Weighting/Likely voter modeling – were reliable population benchmarks used? Was likely voter screen tight/loose given the election?

Considerations about restrictions on poll publication during final week(s) of campaign In U.S., restrictions may be unconstitutional given 1st amendment right for freedom of speech and the press Polls are most reliable source for standing of the election and why voters support candidates, which is of very high news interest at the end of a campaign Barring polls’ publication may encourage reliance on less reliable sources for this Private polls will be conducted by campaigns and others, making reliable data on vote support available only to these sponsors

Considerations about restrictions on poll publication during final week(s) of campaign Impact of polls on candidate support and turnout 2012 U.S. experiment finds exposure to local presidential polling during final two weeks of campaign increased turnout, but did not impact support for candidates (Vanette and Westwood) Mixed effects on attitudes and intention to vote in earlier studies

Questions after 2016 presidential election Depending on causes of errors that are identified, are additional disclosure requirements needed to evaluate poll quality? Can media or professional organizations motivate broader disclosure of survey methodological details and membership in disclosure programs? What lessons will news organizations take from 2016 experience, and how does this inform coverage of polls in future?

Thank you! Contact information: Scott.clement@washpost.com