All the Best Polls Agree with Me: Bias in Evaluations of Political Polling
Do Americans consider polling results an objective source of information? Experts tend to evaluate the credibility of polls based on the survey methods used, vendor track record, and data transparency, but it is unclear if the public does the same. In two different experimental studies—one focusing on candidate evaluations in the 2016 U.S. election and one on a policy issue—we find a significant factor in respondent assessments of polling credibility to be the poll results themselves. Respondents viewed polls as more credible when majority opinion matched their opinion. Moreover, we find evidence of attitude polarization after viewing polling results, suggesting motivated reasoning in the evaluations of political polls. These findings indicate that evaluations of polls are biased by motivated reasoning and suggest that such biases could constrain the possible impact of polls on political decision making.
KeywordsPolling Poll evaluation Public opinion Motivated reasoning Cognitive bias
- Atkeson, L. R., & Alvarez, R. M. (2018). Introduction to polling and survey methods. In The Oxford handbook of polling and survey methods (Vol. 1).Google Scholar
- Bartels, L. M. (1988). Presidential primaries and the dynamics of public choice. Princeton: Princeton University Press.Google Scholar
- Blais, A., Gidengil, E., & Nevitte, N. (2006). “Do polls influence the vote?” Capturing campaign effects (pp. 263–279). Ann Arbor: The University of Michigan Press.Google Scholar
- Blumenthal, M. 2016. “Polling: Crisis or Not, We’re in a New Era.” The Huffington Post. Retrieved June 6, 2016 from https://www.huffingtonpost.com/mark-blumenthal/polling-crisis-or-not-wer_b_10328648.html.
- Blumenthal, M., Clement, S., Clinton, J. D., Durand, C., Franklin, C., Miringoff, L., Olson, K., Rivers, D., Saad, Y. L., & Witt, G. E. (2017). An evaluation of 2016 election polls in the US.Google Scholar
- Crespi, I. (1988). Pre-election polling: Sources of accuracy and error. New York: Russell Sage Foundation.Google Scholar
- Guess, A., & Coppock, A. (2015). Back to bayes: Confronting the evidence on attitude polarization. Unpublished Paper, Yale University. Google Scholar
- Hillygus, S. D., & Guay, B. (2016). The virtues and limitations of election polling in the United States. Seminar Magazine (September).Google Scholar
- Jackson, N. (2018). The rise of poll aggregation and election forecasting. In L. R. Atkinson & R. M. Alvarez (Eds.), The Oxford Handbook of polling and survey methods, 2018. Oxford: Oxford University Press.Google Scholar
- Kahan, D. M. (2016a). The politically motivated reasoning paradigm, Part 1: What politically motivated reasoning is and how to measure it. Emerging Trends in Social & Behavioral Sciences. https://doi.org/10.1002/9781118900772.etrds0417/pdf.CrossRefGoogle Scholar
- Kennedy, C., Mercer, A., Keeter, S., Hatley, N., McGeeney, K., & Gimenez, A. (2016). Evaluating online nonprobability surveys. Washington, DC: Pew Research Center.Google Scholar
- Landy, D., Guay, B., & Marghetis, T. (2017). Bias and ignorance in demographic perception. Psychonomic Bulletin & Review, 6, 1–13.Google Scholar
- Langer, G. (2016). Clinton, trump all but tied as enthusiasm dips for democratic candidate. ABC News. Accessed November 01, 2018. Retrieved November 01, 2016, from https://abcnews.go.com/Politics/clinton-trump-tied-democratic-enthusiasm-dips/story?id=43199459.
- Lavrakas, P. J., Presser, S., Price, V., & Traugott, M. (1998). Them but not me: The perceived impact of election polls. In Paper Presented at the Annual Meeting of the American Association for Public Opinion Research, St. Louis, MO, USA.Google Scholar
- Panagopoulos, C., Gueorguieva, V., Slotnick, A., Gulati, G., & Williams, C. (2009). Politicking online: The transformation of election campaign communications. New Brunswick: Rutgers University Press.Google Scholar
- Ryan, T. J. (2018). Data contamination on MTurk. Blog post. Published August 12, 2019. Available online at timryan.web.unc.edu.
- Stonecash, J. M. (2008). Political polling: Strategic information in campaigns. Lanham: Rowman & Littlefield.Google Scholar
- Vannette, D., & Westwood, S. (2013). Voter mobilization effects of poll reports during the 2012 presidential campaign. In Paper Presented at the 68th Annual AAPOR Conference, May 17.Google Scholar
- Wood, T., & Porter, E. (2016). The elusive backfire effect: Mass attitudes’ steadfast factual adherence. Political Behavior, 65, 1–29.Google Scholar
- Zukin, C. (2015). What’s the matter with polling. New York Times, 20.Google Scholar