Political Behavior

, Volume 38, Issue 3, pp 713–746 | Cite as

Unresponsive and Unpersuaded: The Unintended Consequences of a Voter Persuasion Effort

  • Michael A. Bailey
  • Daniel J. Hopkins
  • Todd Rogers
Original Paper

Abstract

To date, field experiments on campaign tactics have focused overwhelmingly on mobilization and voter turnout, with far more limited attention to persuasion and vote choice. In this paper, we analyze a field experiment with 56,000 Wisconsin voters designed to measure the persuasive effects of canvassing, phone calls, and mailings during the 2008 presidential election. Focusing on the canvassing treatment, we find that persuasive appeals had two unintended consequences. First, they reduced responsiveness to a follow-up survey among infrequent voters, a substantively meaningful behavioral response that has the potential to induce bias in estimates of persuasion effects as well. Second, the persuasive appeals possibly reduced candidate support and almost certainly did not increase it. This counterintuitive finding is reinforced by multiple statistical methods and suggests that contact by a political campaign may engender a backlash.

Keywords

Field experiment Political campaigns Political persuasion Non-random attrition Survey response 

Notes

Acknowledgments

This paper has benefitted from comments by David Broockman, Kevin Collins, Eitan Hersh, Seth Hill, Michael Kellermann, Gary King, Marc Meredith, David Nickerson, Maya Sen, and Elizabeth Stuart. For research assistance, the authors gratefully acknowledge Julia Christensen, Zoe Dobkin, Katherine Foley, Andrew Schilling, and Amelia Whitehead. David Dutwin, Alexander Horowitz, and John Ternovski provided helpful replies to various queries.Earlier versions of this manuscript were presented at the 30th Annual Summer Meeting of the Society for Political Methodology at the University of Virginia, July 18th, 2013 and at Vanderbilt University’s Center for the Study of Democratic Institutions, October 18th, 2013.

References

  1. Adams, W. C., & Smith, D. J. (1980). Effects of telephone canvassing on turnout and preferences: A field experiment. Public Opinion Quarterly, 44(3), 389–395.CrossRefGoogle Scholar
  2. Albertson, B., & Busby, J. W. (2015). Hearts or minds? Identifying persuasive messages on climate change. Research & Politics.Google Scholar
  3. Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables (with discussion). Journal of the American Statistical Association, 91, 444–455.CrossRefGoogle Scholar
  4. Ansolabehere, S., & Hersh, E. (2011). Who really votes? In P. M. Sniderman & B. Highton (Eds.), Facing the challenge of democracy: Explorations in the analysis of public opinion and political participation. Princeton University Press.Google Scholar
  5. Ansolabehere, S., & Hersh, E. (2012). Validation: What big data reveal about survey misreporting and the real electorate. Political Analysis, 20(4):437–459.Google Scholar
  6. Arceneaux, K. (2005). Using cluster randomized field experiments to study voting behavior. The Annals of the American Academy of Political and Social Science, 601(1), 169–179.CrossRefGoogle Scholar
  7. Arceneaux, K. (2007). I’m asking for your support: The effects of personally delivered campaign messages on voting decisions and opinion formation. Quarterly Journal of Political Science, 2(1), 43–65.CrossRefGoogle Scholar
  8. Arceneaux, K., & Kolodny, R. (2009). Educating the least informed: Group endorsements in a grassroots campaign. American Journal of Political Science, 53(4), 755–770.CrossRefGoogle Scholar
  9. Arceneaux, K., & Nickerson, D. W. (2009). Who is mobilized to vote? A re-analysis of 11 field experiments. American Journal of Political Science, 53(1), 1–16.CrossRefGoogle Scholar
  10. Bechtel, M. M., Hainmueller, J., Hangartner, D., & Helbling, M. (2014). Reality bites: The limits of framing effects for salient and contested policy issues. Political Science Research and Methods (forthcoming).Google Scholar
  11. Broockman, D. E., & Green, D. P. (2014). Do online advertisements increase political candidates’ name recognition or favorability? evidence from randomized field experiments. Political Behavior, 36, 263–289.CrossRefGoogle Scholar
  12. Cardy, E. A. (2005). An experimental field study of the GOTV and persuasion effects of partisan direct mail and phone calls. The Annals of the American Academy of Political and Social Science, 601(1), 28–40.CrossRefGoogle Scholar
  13. Cranmer, S. J., & Gill, J. (2013). We have to be discrete about this: A non-parametric imputation technique for missing categorical data. British Journal of Political Science, 43(2), 425–449.CrossRefGoogle Scholar
  14. Das, M., Newey, W. K., & Vella, F. (2003). Nonparametric estimation of sample selection models. The Review of Economic Studies, 70(1), 33–58.CrossRefGoogle Scholar
  15. Demirtas, H., Arguelles, L. M., Chung, H., & Hedeker, D. (2007). On the performance of bias-reduction techniques for variance estimation in approximate Bayesian bootstrap imputation. Computational Statistics & Data Analysis, 51(8), 4064–4068.CrossRefGoogle Scholar
  16. Enos, R. D., Fowler, A., & Vavreck, L. (2014). Increasing inequality: The effect of GOTV mobilization on the composition of the electorate. The Journal of Politics, 76(1), 273–288.CrossRefGoogle Scholar
  17. Enos, R. D., & Hersh, E. D. (2015). Party activists as campaign advertisers: The ground campaign as a principal-agent problem. American Political Science Review, 109(02), 252–278.CrossRefGoogle Scholar
  18. Gerber, A., Karlan, D., & Bergan, D. (2009). Does the media matter? A field experiment measuring the effect of newspapers on voting behavior and political opinions. American Economic Journal: Applied Economics, 1(2), 35–52.Google Scholar
  19. Gerber, A., & Green, D. (2000). The effects of canvassing, telephone calls, and direct mail on voter turnout: A field experiment. American Political Science Review, 94(3), 653–663.CrossRefGoogle Scholar
  20. Gerber, A. S., Kessler, D. P., & Meredith, M. (2011). The persuasive effects of direct mail: A regression discontinuity based approach. Journal of Politics, 73(1), 140–155.CrossRefGoogle Scholar
  21. Gerber, A. S., & Green, D. P. (2012). Field experiments: Design, analysis, and interpretation. New York, NY: W.W. Norton and Company.Google Scholar
  22. Gerber, A. S., Huber, G. A., Doherty, D., Dowling, C. M., & Hill, S. J. (2013). Who wants to discuss vote choices with others? Polarization in preferences for deliberation. Public Opinion Quarterly, 77(2), 474–496.CrossRefGoogle Scholar
  23. Gerber, A. S., Huber, G. A., & Washington, E. (2010). Party affiliation, partisanship, and political beliefs: A field experiment. American Political Science Review, 104(04), 720–744.CrossRefGoogle Scholar
  24. Gerber, A. S., Gimpel, J. G., Green, D. P., & Shaw, D. R. (2011). How large and long-lasting are the persuasive effects of televised campaign ads? Results from a randomized field experiment. American Political Science Review, 105(01), 135–150.CrossRefGoogle Scholar
  25. Glynn, A. N., & Quinn, K. M. (2010). An introduction to the augmented inverse propensity weighted estimator. Political Analysis, 18(1), 36–56.CrossRefGoogle Scholar
  26. Green, D. P., & Gerber, A. S. (2008). Get out the vote: How to increase voter turnout. Washington, DC: Brookings Institution Press.Google Scholar
  27. Heckman, J. (1976). The common structure of statistical models of truncation, sample selectionand limited dependent variables, and simple estimator for such models. Annals of Economic and Social Measurement, 5, 475–492.Google Scholar
  28. Hersh, E. D. (2015). Hacking the electorate: How campaigns perceive voters. New York, NY: Cambridge University Press.CrossRefGoogle Scholar
  29. Hersh, E. D., & Schaffner, B. F. (2013). Targeted campaign appeals and the value of ambiguity The Journal of Politics, 75(02), 520–534.CrossRefGoogle Scholar
  30. Hopkins, D. J. (2009). No more wilder effect, never a Whitman effect: When and why polls mislead about black and female candidates. The Journal of Politics, 71(3), 769–781.CrossRefGoogle Scholar
  31. Huber, G. A., & Arceneaux, K. (2007). Identifying the persuasive effects of presidential advertising. American Journal of Political Science, 51(4), 957–977.CrossRefGoogle Scholar
  32. Imai, K., King, G., & Stuart, E. A. (2008). Misunderstandings between experimentalists and observation lists about causal inference. Journal of the Royal Statistical Society: Series A, 171(2), 481–502.CrossRefGoogle Scholar
  33. Issenberg, S. (2012). Obama Does It Better. Slate.Google Scholar
  34. King, G., Honaker, J., Joseph, A., & Scheve, K. (2001). Analyzing incomplete political science data: An alternative algorithm for multiple imputation. American Political Science Review, 95(1), 49–69.Google Scholar
  35. Ladd, J. M., & Lenz, G. S. (2009). Exploiting a rare communication shift to document the persuasive power of the news media. American Journal of Political Science, 53(2), 394–410.CrossRefGoogle Scholar
  36. Little, R. J., D’Agostino, R., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., et al. (2012). The prevention and treatment of missing data in clinical trials. New England Journal of Medicine, 367(14), 1355–1360.CrossRefGoogle Scholar
  37. Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data (2nd ed.). New York, NY: Wiley.Google Scholar
  38. Matland, R. E., & Murray, G. R. (2013). An experimental test for backlash against social pressure techniques used to mobilize voters. American Politics Research, 41(3), 359–386.CrossRefGoogle Scholar
  39. Michelson, M. R. (2014). Memory and voter mobilization. Polity, 46, 591–610.CrossRefGoogle Scholar
  40. Moore, R. T. (2012). Multivariate continuous blocking to improve political science experiments. Political Analysis, 20(4), 460–479.CrossRefGoogle Scholar
  41. Nicholson, S. P. (2012). Polarizing cues. American Journal of Political Science, 56(1), 52–66.CrossRefGoogle Scholar
  42. Nickerson, D. W. (2005a). Partisan mobilization using volunteer phone banks and door hangers. The Annals of the American Academy of Political and Social Science, 601(1), 10–27.CrossRefGoogle Scholar
  43. Nickerson, D. W. (2005b). Scalable protocols offer efficient design for field experiements Political Analysis, 13, 233–252.CrossRefGoogle Scholar
  44. Nickerson, D. W. (2008). Is voting contagious? Evidence from two field experiments. American Political Science Review, 102(1), 49.CrossRefGoogle Scholar
  45. Nickerson, D. W., & Rogers, T. (2010). Do you have a voting plan? Implementation intentions, voter turnout, and organic plan making. Psychological Science, 21(2), 194–199.CrossRefGoogle Scholar
  46. Panagopoulos, C., & Green, D. P. (2008). Field experiments testing the impact of radio advertisements on electoral competition. American Journal of Political Science, 52(1), 156–168.CrossRefGoogle Scholar
  47. Rogers, T., & Nickerson, D. (2013). Can inaccurate beliefs about incumbents be changed? And can reframing change votes? HKS Faculty Research Working Paper Series RWP13-018.Google Scholar
  48. Rogers, T., & Middleton, J. A. (2015). Are ballot initiative outcomes influenced by the campaigns of independent groups? A precinct-randomized field experiment showing that they are. Political Behavior, 37, 567–593.Google Scholar
  49. Rogers, T., & Aida, M. (2014). Vote self-prediction hardly predicts who will vote, and is (misleadingly) unbiased. American Politics Research, 42(3), 503–528.CrossRefGoogle Scholar
  50. Rubin, D. B. (2008). For objective causal inference, design trumps analysis. The Annals of Applied Statistics, 2, 808–840.Google Scholar
  51. Rubin, D. B., & Schenker, N. (1991). Multiple imputation in health-care databases: An overview and some applications. Statistics in Medicine, 10(4), 585–598.CrossRefGoogle Scholar
  52. Rubin, D., & Schenker, N. (1986). Multiple imputation for interval estimation for simple random samples with ignorable nonresponse. Journal of the American Statistical Association, 81(394), 366–374.CrossRefGoogle Scholar
  53. Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall.CrossRefGoogle Scholar
  54. Shi, Y. (2015). Cross-cutting messages and voter turnout: Evidence from a same-sex marriage amendment. Political Communication. (forthcoming).Google Scholar
  55. Siddique, J., & Belin, T. R. (2008a). Multiple imputation using an iterative hot-deck with distance-based donor selection. Statistics in Medicine, 27(1), 83–102.CrossRefGoogle Scholar
  56. Siddique, J., & Belin, T. R. (2008b). Using an approximate Bayesian bootstrap to multiply impute nonignorable missing data. Computational Statistics & Data Analysis, 53(2), 405–415.CrossRefGoogle Scholar
  57. Simon, H. A., & Stern, F. (1955). The effect of television upon voting behavior in Iowa in the 1952 presidential election. American Political Science Review, 49(2), 470–477.CrossRefGoogle Scholar
  58. Sinclair, B. (2012). The social citizen. Chicago, IL: University of Chicago Press.CrossRefGoogle Scholar
  59. Sinclair, B., McConnell, M., & Green, D. P. (2012). Detecting spillover effects: Design and analysis of multilevel experiments. American Journal of Political Science, 56(4), 1055–1069.CrossRefGoogle Scholar
  60. Taber, C. S., & Lodge, M. (2006). Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science, 50(3), 755–769.CrossRefGoogle Scholar
  61. Van Buuren, S., Brand, J. P. L., Groothuis-Oudshoorn, C. G. M., & Rubin, D. B. (2006). Fully conditional specification in multivariate imputation. Journal of Statistical Computation and Simulation, 76(12), 1049–1064.CrossRefGoogle Scholar
  62. Vavreck, L., et al. (2007). The exaggerated effects of advertising on turnout: The dangers of self-reports Quarterly Journal of Political Science, 2(4), 325–343.CrossRefGoogle Scholar
  63. Westfall, P. H., & Young, S. S. (1993). Resampling-based multiple testing: Examples and methods for p-value adjustment. New York, NY: Wiley.Google Scholar
  64. Zaller, J. R. (1992). The nature and origins of mass opinion. New York, NY: Cambridge University Press.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Michael A. Bailey
    • 1
  • Daniel J. Hopkins
    • 2
  • Todd Rogers
    • 3
  1. 1.Colonel William J.Walsh Professor of American Government, Department of Government and McCourt School of Public PolicyGeorgetown University WashingtonUSA
  2. 2.Department of Political ScienceUniversity of PennsylvaniaPhiladelphia USA
  3. 3.Center for Public Leadership, John F. Kennedy School of GovernmentHarvard UniversityCambridge USA

Personalised recommendations