Advertisement

All the Best Polls Agree with Me: Bias in Evaluations of Political Polling

  • Gabriel J. MadsonEmail author
  • D. Sunshine Hillygus
Original Paper

Abstract

Do Americans consider polling results an objective source of information? Experts tend to evaluate the credibility of polls based on the survey methods used, vendor track record, and data transparency, but it is unclear if the public does the same. In two different experimental studies—one focusing on candidate evaluations in the 2016 U.S. election and one on a policy issue—we find a significant factor in respondent assessments of polling credibility to be the poll results themselves. Respondents viewed polls as more credible when majority opinion matched their opinion. Moreover, we find evidence of attitude polarization after viewing polling results, suggesting motivated reasoning in the evaluations of political polls. These findings indicate that evaluations of polls are biased by motivated reasoning and suggest that such biases could constrain the possible impact of polls on political decision making.

Keywords

Polling Poll evaluation Public opinion Motivated reasoning Cognitive bias 

Notes

Supplementary material

11109_2019_9532_MOESM1_ESM.docx (14 kb)
Supplementary material 1 (DOCX 13 kb)
11109_2019_9532_MOESM2_ESM.pdf (465 kb)
Supplementary material 2 (PDF 466 kb)

References

  1. Ampofo, L., Anstead, N., & O’Loughlin, B. (2011). Trust, confidence, and credibility: Citizen responses on twitter to opinion polls during the 2010 UK general election. Information, Communication & Society, 14(6), 850–871.CrossRefGoogle Scholar
  2. Ansolabehere, S., & Iyengar, S. (1994). Of horseshoes and horse races: Experimental studies of the impact of poll results on electoral behavior. Political Communication, 11(4), 413–430.CrossRefGoogle Scholar
  3. Ansolabehere, S., & Rivers, D. (2013). Cooperative survey research. Annual Review of Political Science, 16, 307–329.CrossRefGoogle Scholar
  4. Atkeson, L. R., & Alvarez, R. M. (2018). Introduction to polling and survey methods. In The Oxford handbook of polling and survey methods (Vol. 1).Google Scholar
  5. Bartels, L. M. (1988). Presidential primaries and the dynamics of public choice. Princeton: Princeton University Press.Google Scholar
  6. Bartels, L. M. (2002). Beyond the running tally: Partisan bias in political perceptions. Political Behavior, 24(2), 117–150.CrossRefGoogle Scholar
  7. Blais, A., Gidengil, E., & Nevitte, N. (2006). “Do polls influence the vote?” Capturing campaign effects (pp. 263–279). Ann Arbor: The University of Michigan Press.Google Scholar
  8. Blumenthal, M. 2016. “Polling: Crisis or Not, We’re in a New Era.” The Huffington Post. Retrieved June 6, 2016 from https://www.huffingtonpost.com/mark-blumenthal/polling-crisis-or-not-wer_b_10328648.html.
  9. Blumenthal, M., Clement, S., Clinton, J. D., Durand, C., Franklin, C., Miringoff, L., Olson, K., Rivers, D., Saad, Y. L., & Witt, G. E. (2017). An evaluation of 2016 election polls in the US.Google Scholar
  10. Boudreau, C., & McCubbins, M. D. (2010). The blind leading the blind: Who gets polling information and does it improve decisions? The Journal of Politics, 72(2), 513–527.CrossRefGoogle Scholar
  11. Bullock, J. G. (2009). Partisan bias and the Bayesian ideal in the study of public opinion. The Journal of Politics, 71(3), 1109–1124.CrossRefGoogle Scholar
  12. Clifford, S., Jewell, R. M., & Waggoner, P. D. (2015). Are samples drawn from Mechanical Turk valid for research on political ideology? Research & Politics, 2(4), 2053168015622072.CrossRefGoogle Scholar
  13. Crespi, I. (1988). Pre-election polling: Sources of accuracy and error. New York: Russell Sage Foundation.Google Scholar
  14. Gerber, A., & Green, D. (1999). Misperceptions about perceptual bias. Annual Review of Political Science, 2(1), 189–210.CrossRefGoogle Scholar
  15. Großer, J., & Schram, A. (2010). Public opinion polls, voter turnout, and welfare: An experimental study. American Journal of Political Science, 54(3), 700–717.CrossRefGoogle Scholar
  16. Guess, A., & Coppock, A. (2015). Back to bayes: Confronting the evidence on attitude polarization. Unpublished Paper, Yale University. Google Scholar
  17. Hill, S. J. (2017). Learning together slowly: Bayesian learning about political facts. The Journal of Politics, 79(4), 1403–1418.CrossRefGoogle Scholar
  18. Hillygus, S. D., & Guay, B. (2016). The virtues and limitations of election polling in the United States. Seminar Magazine (September).Google Scholar
  19. Iyengar, S., & Westwood, S. J. (2015). Fear and loathing across party lines: New evidence on group polarization. American Journal of Political Science, 59(3), 690–707.CrossRefGoogle Scholar
  20. Jackson, N. (2018). The rise of poll aggregation and election forecasting. In L. R. Atkinson & R. M. Alvarez (Eds.), The Oxford Handbook of polling and survey methods, 2018. Oxford: Oxford University Press.Google Scholar
  21. Jacobs, L. R., & Shapiro, R. Y. (2005). Polling politics, media, and election campaigns: Introduction. The Public Opinion Quarterly, 69(5), 635–641.CrossRefGoogle Scholar
  22. Jerit, J., & Barabas, J. (2012). Partisan perceptual bias and the information environment. The Journal of Politics, 74(3), 672–684.CrossRefGoogle Scholar
  23. Kahan, D. M. (2016a). The politically motivated reasoning paradigm, Part 1: What politically motivated reasoning is and how to measure it. Emerging Trends in Social & Behavioral Sciences.  https://doi.org/10.1002/9781118900772.etrds0417/pdf.CrossRefGoogle Scholar
  24. Kahan, D. (2016b). The politically motivated reasoning paradigm, P2: Unanswered questions. Emerging Trends in Social & Behavioral Sciences.  https://doi.org/10.1002/9781118900772.etrds0418.CrossRefGoogle Scholar
  25. Kennedy, C., Blumenthal, M., Clement, S., Clinton, J. D., Durand, C., Franklin, C., et al. (2018). An evaluation of the 2016 election polls in the United States. Public Opinion Quarterly, 82(1), 1–33.CrossRefGoogle Scholar
  26. Kennedy, C., Mercer, A., Keeter, S., Hatley, N., McGeeney, K., & Gimenez, A. (2016). Evaluating online nonprobability surveys. Washington, DC: Pew Research Center.Google Scholar
  27. Kim, S. T., Weaver, D., & Willnat, L. (2000). Media reporting and perceived credibility of online polls. Journalism & Mass Communication Quarterly, 77(4), 846–864.CrossRefGoogle Scholar
  28. Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108(3), 480.CrossRefGoogle Scholar
  29. Kuru, O., Pasek, J., & Traugott, M. W. (2017). Motivated reasoning in the perceived credibility of public opinion polls. Public Opinion Quarterly, 81(2), 422–446.CrossRefGoogle Scholar
  30. Landy, D., Guay, B., & Marghetis, T. (2017). Bias and ignorance in demographic perception. Psychonomic Bulletin & Review, 6, 1–13.Google Scholar
  31. Langer, G. (2016). Clinton, trump all but tied as enthusiasm dips for democratic candidate. ABC News. Accessed November 01, 2018. Retrieved November 01, 2016, from https://abcnews.go.com/Politics/clinton-trump-tied-democratic-enthusiasm-dips/story?id=43199459.
  32. Lau, R. R., & Redlawsk, D. P. (2006). How voters decide: Information processing in election campaigns (p. 2006). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  33. Lavrakas, P. J., Presser, S., Price, V., & Traugott, M. (1998). Them but not me: The perceived impact of election polls. In Paper Presented at the Annual Meeting of the American Association for Public Opinion Research, St. Louis, MO, USA.Google Scholar
  34. Lelkes, Y., Sood, G., & Iyengar, S. (2017). The hostile audience: The effect of access to broadband internet on partisan affect. American Journal of Political Science, 61(1), 5–20.CrossRefGoogle Scholar
  35. Lodge, M., & Taber, C. S. (2013). The rationalizing voter. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  36. Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37(11), 2098.CrossRefGoogle Scholar
  37. Marsh, C. (1985). Back on the bandwagon: The effect of opinion polls on public opinion. British Journal of Political Science, 15(1), 51–74.CrossRefGoogle Scholar
  38. Mosier, N. R., & Ahlgren, A. (1981). Credibility of precision journalism. Journalism Quarterly, 58(3), 375–518.CrossRefGoogle Scholar
  39. Nyhan, B., Porter, E., Reifler, J., & Wood, T. (2017). Taking corrections literally but not seriously?. The effects of information on factual beliefs and candidate favorability: Political Behavior.  https://doi.org/10.2139/ssrn.2995128.CrossRefGoogle Scholar
  40. Nyhan, B., & Reifler, J. (2010). When corrections fail: The persistence of political misperceptions. Political Behavior, 32(2), 303–330.CrossRefGoogle Scholar
  41. Panagopoulos, C., Endres, K., & Weinschenk, A. C. (2018). Preelection poll accuracy and bias in the 2016 US general elections. Journal of Elections, Public Opinion and Parties, 28(2), 157–172.CrossRefGoogle Scholar
  42. Panagopoulos, C., Gueorguieva, V., Slotnick, A., Gulati, G., & Williams, C. (2009). Politicking online: The transformation of election campaign communications. New Brunswick: Rutgers University Press.Google Scholar
  43. Price, V., & Stroud, N. J. (2005). Public attitudes toward polls: Evidence from the 2000 U.S. presidential election. International Journal of Public Opinion Research, 18(4), 393–421.CrossRefGoogle Scholar
  44. Redlawsk, David P. (2002). Hot cognition or cool consideration? Testing the effects of motivated reasoning on political decision making. Journal of Politics, 64(4), 1021–1044.CrossRefGoogle Scholar
  45. Redlawsk, D. P. (2006). Motivated reasoning, affect, and the role of memory in voter decision making. In D. P. Redlawsk (Ed.), Feeling politics (pp. 87–107). New York: Palgrave Macmillan.CrossRefGoogle Scholar
  46. Rothschild, D., & Malhotra, N. (2014). Are public opinion polls self-fulfilling prophecies? Research & Politics.  https://doi.org/10.1177/2053168014547667.CrossRefGoogle Scholar
  47. Ryan, T. J. (2018). Data contamination on MTurk. Blog post. Published August 12, 2019. Available online at timryan.web.unc.edu.
  48. Salwen, M. B. (1987). Credibility of newspaper opinion polls: Source, source intent and precision. Journalism Quarterly, 64(4), 813–819.CrossRefGoogle Scholar
  49. Searles, K., Smith, G., & Sui, M. (2018). Partisan media, electoral predictions, and wishful thinking. Public Opinion Quarterly, 82(S1), 302–324.CrossRefGoogle Scholar
  50. Stonecash, J. M. (2008). Political polling: Strategic information in campaigns. Lanham: Rowman & Littlefield.Google Scholar
  51. Taber, C. S., & Lodge, M. (2006). Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science, 50(3), 755–769.CrossRefGoogle Scholar
  52. Tourangeau, R., Steiger, D. M., & Wilson, D. (2002). Self-administered questions by telephone: Evaluating interactive voice response. The Public Opinion Quarterly, 66(2), 265–278.CrossRefGoogle Scholar
  53. Traugott, M. W. (2005). The accuracy of the national preelection polls in the 2004 presidential election. Public Opinion Quarterly, 69(5), 642–654.CrossRefGoogle Scholar
  54. Tsfati, Y. (2001). Why do people trust media pre-election polls? Evidence from the Israeli 1996 elections. International Journal of Public Opinion Research, 13(4), 433–441.CrossRefGoogle Scholar
  55. Utych, S. M., & Kam, C. D. (2013). Viability, information seeking, and vote choice. The Journal of Politics, 76(1), 152–166.CrossRefGoogle Scholar
  56. Valentino, N. A., Banks, A. J., Hutchings, V. L., & Davis, A. K. (2009). Selective exposure in the Internet age: The interaction between anxiety and information utility. Political Psychology, 30(4), 591–613.CrossRefGoogle Scholar
  57. Valentino, N. A., King, J. L., & Hill, W. W. (2017). Polling and prediction in the 2016 presidential election. Computer, 50(5), 110–115.CrossRefGoogle Scholar
  58. Vannette, D., & Westwood, S. (2013). Voter mobilization effects of poll reports during the 2012 presidential campaign. In Paper Presented at the 68th Annual AAPOR Conference, May 17.Google Scholar
  59. Wlezien, C., & Erikson, R. (2002). The timeline of presidential election campaigns. The Journal of Politics, 64(4), 969–993.CrossRefGoogle Scholar
  60. Wood, T., & Porter, E. (2016). The elusive backfire effect: Mass attitudes’ steadfast factual adherence. Political Behavior, 65, 1–29.Google Scholar
  61. Zukin, C. (2015). What’s the matter with polling. New York Times, 20.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Duke UniversityDurhamUSA

Personalised recommendations