Abstract
Survey methodologists are familiar with the use of experiments to show how variations in question wording and survey design can affect people’s expressed opinions, but what does it mean when respondents’ answers do not significantly differ across experimental groups? In this chapter, we draw upon theory from sociology and political science to argue that null results can tell us just as much about public opinion as statistically significant findings. We analyze data from a survey experiment that tested the effect of exposure to news images of police-civilian interactions, depicting a range of interactions from benign to hostile in nature, on public opinion about the police. We find that image exposure did not significantly affect opinion. We interpret this null finding as evidence that people’s opinions about the police tap into fundamental beliefs and anxieties about the state of society, and due to their basis in emotion and values, these opinions are easier for people to form than opinions about more abstract or complicated public policies. According to framing theorists, “easy issues” are relatively immune to manipulation by the media, political elites, or pollsters.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The treatment images may be viewed on the following websites, which are the sources from which we copied the pictures for use in our survey experiment: militarized policing (Topaz 2014), community policing (Mirko 2013), stop and frisk (Post Editorial Board 2015). The original stop and frisk image actually depicted an officer training exercise, and the officers were carrying bright blue model side arms. We digitally altered the picture in order to color the handgun hilts black so that they would look like regular guns, thereby depicting a typical stop and frisk.
- 2.
All questions in this survey included a “prefer not to answer” response option. We dropped these responses prior to the present analyses. Only between 31 and 48 respondents chose this option in each of the dependent variable questions.
- 3.
Two additional pieces of evidence strengthen our conclusion of null findings. First, we also conducted a series of two one-sided tests of equivalence for each of the dependent variables following the recommendations of Lakens (2017). Equivalence tests provide an additional check against the possibility that non-significant treatment effects are a Type II error; essentially, they test whether a difference between two groups is large enough to be meaningful according to a standard set by the researcher. This validity check confirmed that the differences in police attitudes across treatment groups are statistically equivalent; these results are available from the first author upon request. Second, in a separate analysis, we did find that the treatment caused significant differences in respondents’ expressed presidential vote preference (Wozniak et al. 2019). This finding indicates that image exposure did affect some of the respondents’ expressed preferences, just not their attitudes toward the police.
- 4.
Moule et al. (2019) present evidence that supports this interpretation. They found that nearly 65% of respondents to their national survey expressed support for the use of police SWAT teams to respond to “instances of civil unrest.” On the other hand, only about 7% of respondents supported the use of SWAT teams to respond to “peaceful protests.” Their finding suggests that people judge the appropriateness of police tactics differently under different circumstances, but the nature of those circumstances may be in the eye of the beholder. See also Lockwood et al. (2018) for complimentary findings.
References
Alcindor, Y. (2016, August 16). Trump, rallying white crowd for police, accuses democrats of exploiting blacks (p. 9). The New York Times.
Arora, M., Phoenix, D. L., & Delshad, A. (2019). Framing police and protesters: Assessing volume and framing of news coverage post-Ferguson, and corresponding impacts on legislative activity. Politics, Groups, and Identities, 7(1), 151–164. https://doi.org/10.1080/21565503.2018.1518782.
Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., et al. (2010). AAPOR report on online panels. Public Opinion Quarterly, 74(4), 711–781. https://doi.org/10.1093/poq/nfq048.
Baumgartner, F. R., & Jones, B. D. (2009). Agendas and instability in American politics (2nd ed.). Chicago: University of Chicago Press.
Benford, R. D., & Snow, D. A. (2000). Framing processes and social movements: An overview and assessment. Annual Review of Sociology, 26(1), 611–639. https://doi.org/10.1146/annurev.soc.26.1.611.
Bonilla-Silva, E. (2017). Racism without racists: Color-blind racism and the persistence of racial inequality in America (5th ed.). Lanham, MD: Rowman & Littlefield Publishers.
Brenner, P. S., & DeLamater, J. (2016). Lies, damned lies, and survey self-reports? Identity as a cause of measurement bias. Social Psychology Quarterly, 79(4), 333–354. https://doi.org/10.1177/0190272516628298.
Callanan, V. J., & Rosenberger, J. S. (2011). Media and public perceptions of the police: Examining the impact of race and personal experience. Policing and Society, 21(2), 167–189. https://doi.org/10.1080/10439463.2010.540655.
Carmines, E. G., & Stimson, J. A. (1980). The two faces of issue voting. American Political Science Review, 74(1), 78–91.
Chong, D., & Druckman, J. N. (2007). Framing theory. Annual Review of Political Science, 10, 103–126.
Cobbina, J. E. (2019). Hands up, Don’t shoot: Why the protests in Ferguson and Baltimore matter, and how they changed America. New York: New York University Press.
Converse, P. E. (1964). The nature of belief systems in mass publics. In D. E. Apter (Ed.), Ideology and discontent (pp. 206–261). New York: Free Press.
Dillman, D. A., Phelps, G., Tortora, R., Swift, K., Kohrell, J., Berck, J., & Messer, B. L. (2009). Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the internet. Social Science Research, 38(1), 1–18. https://doi.org/10.1016/j.ssresearch.2008.03.007.
Dowler, K., & Zawilski, V. (2007). Public perceptions of police misconduct and discrimination: Examining the impact of media consumption. Journal of Criminal Justice, 35(2), 193–203. https://doi.org/10.1016/j.jcrimjus.2007.01.006.
Drakulich, K., Hagan, J., Johnson, D., & Wozniak, K. H. (2017). Race, justice, policing, and the 2016 American presidential election. Du Bois Review: Social Science Research on Race, 14(1), 7–33. https://doi.org/10.1017/S1742058X1600031X.
Druckman, J. N., Green, D. P., Kuklinski, J. H., & Lupia, A. (2006). The growth and development of experimental research in political science. American Political Science Review, 100(4), 627–635. https://doi.org/10.1017/S0003055406062514.
Druckman, J. N., & Nelson, K. R. (2003). Framing and deliberation: How citizens’ conversations limit elite influence. American Journal of Political Science, 47(4), 729–745.
Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484.
Gately, G., & Stolberg, S. G. (2015, November 16). Baltimore police assailed for response after Freddie Gray’s death. The New York Times. Retrieved from https://www.nytimes.com/2015/11/17/us/baltimore-police-assailed-for-response-after-freddie-grays-death.html
Gerber, A., & Malhotra, N. (2008a). Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science, 3(3), 313–326. https://doi.org/10.1561/100.00008024.
Gerber, A., & Malhotra, N. (2008b). Publication bias in empirical sociological research: Do arbitrary significance levels distort published results? Sociological Methods & Research, 37(1), 3–30. https://doi.org/10.1177/0049124108318973.
Goffman, E. (2009). Stigma: Notes on the management of spoiled identity. Simon and Schuster.
Gorden, R. L. (1952). Interaction between attitude and the definition of the situation in the expression of opinion. American Sociological Review, 17, 50–58. https://doi.org/10.2307/2088359.
Graziano, L. M., & Gauthier, J. F. (2019). Examining the racial-ethnic continuum and perceptions of police misconduct. Policing and Society, 29(6), 657–672. https://doi.org/10.1080/10439463.2017.1310859.
Groves, R. M. (2004). Survey errors and survey costs. Hoboken, NJ: Wiley.
Groves, R. M., Fowler, F. J., Jr., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Wiley.
Haider-Markel, D. P., & Joslyn, M. R. (2001). Gun policy, tragedy, and blame attribution: The conditional influence of issue frames. Journal of Politics, 63(2), 520–543.
Haidt, J. (2012). The righteous mind: Why good people are divided by politics and religion. Knopf Doubleday Publishing Group.
Holley, P. (2016, July 18). Wisconsin sheriff: Black lives Matter’s ‘hateful ideology’ caused police killings. Washington Post.. Retrieved from https://www.washingtonpost.com/news/post-nation/wp/2016/07/18/sheriff-black-lives-matters-hateful-ideology-caused-police-killings/.
Howe, L. C., & Krosnick, J. A. (2017). Attitude strength. Annual Review of Psychology, 68(1), 327–351. https://doi.org/10.1146/annurev-psych-122414-033600.
Jackson, J., & Bradford, B. (2009). Crime, policing and social order: On the expressive nature of public confidence in policing. The British Journal of Sociology, 60(3), 493–521. https://doi.org/10.1111/j.1468-4446.2009.01253.x.
Jackson, J., & Sunshine, J. (2007). Public confidence in policing: A neo-Durkheimian perspective. The British Journal of Criminology, 47(2), 214–233. https://doi.org/10.1093/bjc/azl031.
Kahneman, D. (2011). Thinking, fast and slow. New York: Farrar, Straus and Giroux.
Kennedy, R., & Schuessler, J. (2014, August 14). Ferguson images evoke civil rights era and changing visual perceptions (p. A14). The New York Times.
Krosnick, J. A., & Schuman, H. (1988). Attitude intensity, importance, and certainty and susceptibility to response effects. Journal of Personality and Social Psychology, 54(6), 940–952. https://doi.org/10.1037/0022-3514.54.6.940.
Lakens, D. (2017). Equivalence tests: A practical primer for t-tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362. https://doi.org/10.1177/1948550617697177.
Lasley, J. R. (1994). The impact of the Rodney king incident on citizen attitudes toward police. Policing and Society, 3(4), 245–255. https://doi.org/10.1080/10439463.1994.9964673.
Lecheler, S., de Vreese, C., & Slothuus, R. (2009). Issue importance as a moderator of framing effects. Communication Research, 36(3), 400–425.
Lepkowski, J. M., Tucker, N. C., Brick, J. M., de Leeuw, E. D., Japec, L., Lavrakas, P. J., et al. (2007). Advances in telephone survey methodology. Hoboken, NJ: Wiley.
Lockwood, B., Doyle, M. D., & Comiskey, J. G. (2018). Armed, but too dangerous? Factors associated with citizen support for the militarization of the police. Criminal Justice Studies, 31(2), 113–127. https://doi.org/10.1080/1478601X.2017.1420652.
Melley, B. (2014, August 31). Ferguson’s flashpoint sparks national outrage. Detroit News. Retrieved from https://www.detroitnews.com/story/news/local/metro-state/2014/08/31/fergusons-flashpoint-sparks-national-outrage/14900625/
Mirko, M. (2013, July 3). Pictures: New Haven community policing. Retrieved October 1, 2019, from Courant.com website.: https://www.courant.com/news/connecticut/hc-new-haven-community-policing-20130703-photogallery.html
Moule, R. K., Parry, M. M., & Fox, B. (2019). Public support for police use of SWAT: Examining the relevance of legitimacy. Journal of Crime and Justice, 42(1), 45–59. https://doi.org/10.1080/0735648X.2018.1556862.
Mutz, D. C. (2011). Population-based survey experiments. Princeton, N.J: Princeton University Press.
National Advisory Commission on Civil Disorders, & Kerner, O. (1968). Report of the National Advisory Commission on Civil Disorders. Washington, DC: US Government Printing Office.
Olson, W. (2014, August 13). Police militarization in Ferguson—And your town. Retrieved October 17, 2019, from Cato institute website.: https://www.cato.org/blog/police-militarization-ferguson-nationwide
Paul, R. (2014, August 14). We must demilitarize the police. Time Magazine. Retrieved from https://time.com/3111474/rand-paul-ferguson-police/
Peck, J. H. (2015). Minority perceptions of the police: A state-of-the-art review. Policing: An International Journal of Police Strategies & Management, 38(1), 173–203. https://doi.org/10.1108/PIJPSM-01-2015-0001.
Peffley, M., & Hurwitz, J. (2010). Justice in America: The separate realities of blacks and whites. New York: Cambridge University Press.
Post Editorial Board. (2015, June 1). How many New Yorkers must die before the mayor brings back stop-and-frisk? Retrieved October 1, 2019, from https://nypost.com/2015/06/01/how-many-new-yorkers-must-die-before-the-mayor-brings-back-stop-and-frisk/
Sappenfield, M. (2014, November 30). Can Ferguson spark new civil rights movement? How times have changed. Christian Science Monitor. Retrieved from https://www.csmonitor.com/USA/Society/2014/1130/Can-Ferguson-spark-new-civil-rights-movement-How-times-have-changed
Schneider, A., & Ingram, H. (1993). Social construction of target populations: Implications for politics and policy. American Political Science Review, 87(2), 334–347. https://doi.org/10.2307/2939044.
Steiner, I. D. (1954). Primary group influences on public opinion. American Sociological Review, 19(3), 260–267. https://doi.org/10.2307/2087755.
Stryker, S. (1980). Symbolic interactionism: A social structural version. Caldwell, NJ: Blackburn.
Thorsen, L., & Giegerich, S. (2014, August 10). Shooting of teen by Ferguson officer spurs angry backlash (p. A1). St. Louis Post-Dispatch.
Topaz, J. (2014, August 14). Critics slam “militarization” of police. Retrieved October 1, 2019, from POLITICO website.: https://www.politico.com/story/2014/08/ferguson-critics-police-militarization-110017
Tourangeau, R., Couper, M. P., & Conrad, F. (2004). Spacing, position, and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68(3), 368–393.
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. New York: Cambridge University Press.
Tourangeau, R., & Smith, T. W. (1996). Asking sensitive questions: The impact of data collection mode, question format, and question context. Public Opinion Quarterly, 60(2), 275–304.
Tyler, T. R. (2006). Why people obey the law (2nd ed.). Princeton, N.J: Princeton University Press.
Vitale, A. S. (2016, July 19). Giuliani’s convention speech got everything wrong about policing. The Nation. Retrieved from https://www.thenation.com/article/giulianis-convention-speech-got-everything-wrong-about-policing/
Weinberg, J., Freese, J., & McElhattan, D. (2014). Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociological Science, 1, 292–310. http://dx.doi.org.ezproxy.lib.umb.edu/10.15195/v1.a19.
Weisburd, D., & Eck, J. E. (2004). What can police do to reduce crime, disorder, and fear? The Annals of the American Academy of Political and Social Science, 593(1), 42–65.
Weitzer, R. (2002). Incidents of police misconduct and public opinion. Journal of Criminal Justice, 30(5), 397–408. https://doi.org/10.1016/S0047-2352(02)00150-2.
Weitzer, R., & Tuch, S. A. (2005). Racially biased policing: Determinants of citizen perceptions. Social Forces, 83(3), 1009–1030. https://doi.org/10.1353/sof.2005.0050.
Wozniak, K. H. (2016). Ontological insecurity, racial tension, and confidence in the police in the shadow of urban unrest. Sociological Forum, 31(4), 1063–1082. https://doi.org/10.1111/socf.12296.
Wozniak, K. H., Calfano, B. R., & Drakulich, K. M. (2019). A “Ferguson effect” on 2016 presidential vote preference? Findings from a framing experiment examining “shy voters” and cues related to policing and social unrest. Social Science Quarterly, 100(4), 1023–1038. https://doi.org/10.1111/ssqu.12622.
Yeager, D. S., & Krosnick, J. A. (2012). Does mentioning “some people” and “other people” in an opinion question improve measurement quality? Public Opinion Quarterly, 76(1), 131–141. https://doi.org/10.1093/poq/nfr066.
Yokley, E. (2015, January 18). In Ferguson, push for criminal justice reform draws comparisons to ‘60s fight for civil rights (p. A11). The New York Times.
Zaller, J. (1992). The nature and origins of mass opinion. New York: Cambridge University Press.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Wozniak, K.H., Drakulich, K.M., Calfano, B.R. (2020). Theories of Public Opinion Change Versus Stability and their Implications for Null Findings. In: Brenner, P.S. (eds) Understanding Survey Methodology. Frontiers in Sociology and Social Research, vol 4. Springer, Cham. https://doi.org/10.1007/978-3-030-47256-6_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-47256-6_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-47255-9
Online ISBN: 978-3-030-47256-6
eBook Packages: Social SciencesSocial Sciences (R0)