1. How media cover election polls in the U.S.
Presentation to Instituto Nacional Electoral
December 5, 2016
Scott Clement
Polling Manager
The Washington Post

2. Is the U.S. poll-crazy?
38 national polls conducted during the final week before U.S. presidential election
219 battleground state polls conducted during the final week
Wide range in methodologies
Random digit dial telephone
Including/excluding mobile phones
Live interviewer/Automated interviews
Web-based methods (opt-in panel, river sample)
No face-to-face interview surveys

3. The ways media cover pre-election polls
Horserace – who is ahead, who is behind, and how has this changed
Including poll averages
How are key demographic and political groups voting
Gender, age, race/ethnicity, partisan affiliation
Views of candidates’ strengths and weaknesses on issues and personal qualities (i.e. Who will do a better job handling the economy? Which candidate is more honest?
Support for key policy proposals during campaign and their connection with vote preferences
Any other ways polls are covered?

4. Little regulation of how polls are conducted and published
Federal regulation on dialing of cellular phones without automated dialers
Local laws on distance a person must be from voting places impact exit pollsters
No legal restrictions on…
Type of sampling, interviewing, weighting or likely voter modeling
When polls can be released (permitted all the way through Election Day)
What details of a poll’s methodology must be disclosed upon release

5. What influences how polls are reported?
Standards for disclosure from professional polling organizations
AAPOR Standards and Transparency Initiative
National Council on Public Polls (NCPP)

6. National Council on Public Polls disclosure requirements
Sponsorship of the survey
Fieldwork provider (if applicable)
Dates of interviewing
Sampling method employed (for example, random-digit dialed telephone sample, list-based telephone sample, area probability sample, opt-in internet panel, non-probability convenience sample)
Population that was sampled (for example, general population; registered voters; likely voters; or any specific population group defined by gender, race, age, occupation or any other characteristic)
Size of the sample that serves as the primary basis of the survey report
Size and description of the subsample, if the survey report relies primarily on less than the total sample
Margin of sampling error (if a probability sample)
Survey mode (for example, telephone/interviewer, telephone/automated, mail, internet, fax, )
Complete wording and ordering of questions mentioned in or upon which the release is based
Percentage results of all questions reported

7. AAPOR Transparency Initiative requirements
1. Survey sponsor and who conducted it, as well as all sources of funding. 2. The exact wording and presentation of questions and response options whose results are reported. 3. A definition of the population under study and its geographic location. 4. Dates of data collection. 5. A description of the sampling frame(s) and its coverage of the target population, including mention of any segment of the target population that is not covered by the design (Truncated) 6. The name of the sample supplier, if the sampling frame and/or the sample itself was provided by a third party. 7. The methods used to recruit the panel or participants, if the sample was drawn from a pre-recruited panel or pool of respondents. 8. A description of the sample design, giving a clear indication of the method by which the respondents were selected, recruited, intercepted or otherwise contacted or encountered, along with any eligibility requirements and/or oversampling (truncated). 9. Method(s) and mode(s) used to administer the survey (e.g., CATI, CAPI, ACASI, IVR, mail survey, Web survey) and the language(s) offered. 10. Sample sizes (by sampling frame if more than one was used) and a discussion of the precision of the findings. (Truncated). 11. A description of how the weights were calculated, including the variables used and the sources of weighting parameters.

8. What influences how polls are reported?
Standards for disclosure from professional polling organizations
AAPOR Standards and Transparency Initiative
National Council on Public Polls
Marketing Research Association
Media vetting of survey quality
Conducted at individual news organizations, standards made public at some large national organizations (Associated Press, ABC News)
Can influence whether surveys are covered or not, or how surveys are covered
Often behind-the-scenes, but media sometimes cover polls in a critical light (USC/LA Times poll, Gallup 2012 polling)
Marketing Research Association – limited requirements
The method of data collection used
The date(s) of data collection
The sampling frame
The sampling method
The sample size
The calculated margin of sampling error

9. How many election polls abide by professional disclosure standards?
Among 38 national polls during final week before U.S. presidential election, 12 were conducted by organizations that are members of AAPOR’s Transparency Initiative or the National Council on Public Polls
Among the 219 battleground state polls conducted during the final week, 77 (35%) were conducted by organizations that participate in AAPOR’s Transparency Initiative or the National Council on Public Polls
Nearly all polls report some basic information
Interview mode, sample size, margin of error and field dates.
Less common is disclosure of sample methodology details, full question wording, weighting and likely voter modeling is less common.

10. How The Washington Post vets polls
Researcher is contacted to obtain detailed description of sample design, interviewing, question wording, weighting, and likely voter modeling.
While most firms to not publish these details, they are willing to provide them on request. Existence of AAPOR/NCPP standards helps to encourage disclosure.
Vetting decisions based on several factors
Sample design – probability or non-probability sample? What is sample’s coverage of target population?
Mode of interviewing – Live professional interviewers, automated
Question wording – Are questions clear and balanced? With vote choice, is there potential for order effects?
Weighting approach – to known, reliable and relevant population parameters?
Accuracy of polling firm’s previous pre-election estimates?
Level of methodological disclosure
Weighting/Likely voter modeling – were reliable population benchmarks used? Was likely voter screen tight/loose given the election?

11. Considerations about restrictions on poll publication during final week(s) of campaign
In U.S., restrictions may be unconstitutional given 1st amendment right for freedom of speech and the press
Polls are most reliable source for standing of the election and why voters support candidates, which is of very high news interest at the end of a campaign
Barring polls’ publication may encourage reliance on less reliable sources for this
Private polls will be conducted by campaigns and others, making reliable data on vote support available only to these sponsors

12. Considerations about restrictions on poll publication during final week(s) of campaign
Impact of polls on candidate support and turnout
2012 U.S. experiment finds exposure to local presidential polling during final two weeks of campaign increased turnout, but did not impact support for candidates (Vanette and Westwood)
Mixed effects on attitudes and intention to vote in earlier studies

13. Questions after 2016 presidential election
Depending on causes of errors that are identified, are additional disclosure requirements needed to evaluate poll quality?
Can media or professional organizations motivate broader disclosure of survey methodological details and membership in disclosure programs?
What lessons will news organizations take from 2016 experience, and how does this inform coverage of polls in future?

14. Thank you!
Contact information: