This work describes an analysis of political associations in 27 million diachronic (1975–2019) news and opinion articles from 47 news media outlets popular in the United States. We use embedding models trained on individual outlets content to quantify outlet-specific latent associations between positive/negative sentiment words and terms loaded with political connotations such as those describing political orientation, party affiliation, names of influential politicians, and ideologically aligned public figures. We observe that both left- and right-leaning news media tend to associate positive sentiment words with terms used to refer to members of their own political in-group and negative sentiment words with terms used to denote members of their ideological outgroup. Outlets rated as centrist by humans display political associations that are often milder but similar in orientation to those of left-leaning news organizations. A weighted average of political associations by outlets’ readership volume hints that political associations embedded in left of center news outlets might have larger societal reach. A chronological analysis of political associations through time suggests that political sentiment polarization is increasing in both left- and right-leaning news media contents. Our approach for measuring sentiment associations of words denoting political orientation in outlet-specific embedding models correlates substantially with external human ratings of outlet ideological bias (r > 0.7). Yet, specific sentiment associations are sometimes multifaceted and challenging to interpret. Overall, our work signals the potential of machine learning models derived from news media language usage to quantify the ideological bias embedded in news outlet content.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Neuendorf, K. A. (2001). The Content Analysis Guidebook (1st ed.). SAGE Publications Inc.
Rozado, D. (2019). Using word embeddings to analyze how universities conceptualize ‘diversity’ in their online institutional presence. Society, 56(3), 256–266. https://doi.org/10.1007/s12115-019-00362-9
Setting the Agenda: Mass Media and Public Opinion, 2 edition. Polity, 2014
Rozado, D. (2020). Prejudice and victimization themes in New York times discourse: a chronological analysis. Academic Questions, 33(1), 89–100. https://doi.org/10.1007/s12129-019-09857-7
Gorp, B. V. (2007). The constructionist approach to framing: bringing culture back in. The Journal of Communication, 57(1), 60–78. https://doi.org/10.1111/j.0021-9916.2007.00329.x
Gadarian, S. K. (2010). The politics of threat: how terrorism news shapes foreign policy attitudes. The Journal of Political, 72(2), 469–483. https://doi.org/10.1017/S0022381609990910
“Media Bias (Real and Perceived) and the Rise of Partisan Media,” Niskanen Center, Nov. 06, 2017. https://www.niskanencenter.org/media-bias-real-perceived-rise-partisan-media/ (accessed May 03, 2021)
Grabe, M. E., Bucy, E. P. Image Bite Politics: News and the Visual Framing of Elections. Oxford University Press. Accessed: May 03, 2021. [Online]. https://www.oxford.universitypressscholarship.com/view/10.1093/acprof:oso/9780195372076.001.0001/acprof-9780195372076
Groeling, T. (2008). Who’s the Fairest of them All? An empirical test for partisan bias on ABC, CBS, NBC, and Fox News. Presidential Studies Quarterly, 38(4), 631–657. https://doi.org/10.1111/j.1741-5705.2008.02668.x
Schiffer, A. J. (2006). Assessing partisan bias in political news: the case(s) of local senate election coverage. Political Communication, 23(1), 23–39. https://doi.org/10.1080/10584600500476981
Groseclose, T., & Milyo, J. (2005). A measure of media bias. The Quarterly Journal of Economics, 120(4), 1191–1237.
Gasper, J. (2011). Shifting ideologies? Re-examining media bias. Quarterly Journal of Political Science, 6, 357–370. https://doi.org/10.1561/100.00010006
Martin, G. J., & Yurukoglu, A. (2017). Bias in cable news: persuasion and polarization. The American Economic Review, 107(9), 2565–2599. https://doi.org/10.1257/aer.20160812
Larcinese, V., Puglisi, R., & Snyder, J. M., Jr. (2007). Partisan bias in economic news: evidence on the agenda-setting behavior of US newspapers. National Bureau of Economic Research. https://doi.org/10.3386/w13378
Waldman, P., & Devitt, J. (1998). Newspaper photographs and the 1996 presidential election: the question of bias. Journalism & Mass Communication Quarterly, 75(2), 302–311. https://doi.org/10.1177/107769909807500206
Barrett, A. W., & Barrington, L. W. (2005). Bias in newspaper photograph selection. Political Research Quarterly, 58(4), 609–618. https://doi.org/10.1177/106591290505800408
Kahn, K. F., & Kenney, P. (2002). The slant of the news: how editorial endorsements influence campaign coverage and citizens’ views of candidates. American Political Science Review., 96(2), 381–394. https://doi.org/10.1017/S0003055402000230
Media Bias in Presidential Election Coverage 1948–2008: Evaluation via Formal Measurement. Accessed: May 03, 2021. [Online]. https://rowman.com/isbn/0739164740
Niven, D. (2003). Objective evidence on media bias: newspaper coverage of congressional party switchers. Journalism & Mass Communication Quarterly, 80(2), 311–326. https://doi.org/10.1177/107769900308000206
Dalal, S., Adlim, B., & Lesk, M. (2019). How to measure relative bias in media coverage? Significance, 16(5), 18–23. https://doi.org/10.1111/j.1740-9713.2019.01316.x
Mitchell, A., Gottfried, J., Kiley, J., Matsa, K. E. “Political Polarization & Media Habits,” Pew Research Center’s Journalism Project, Oct. 21, 2014. https://www.journalism.org/2014/10/21/political-polarization-media-habits/ (accessed May 06, 2021)
Eveland, W. P., & Shah, D. V. (2003). The impact of individual and interpersonal factors on perceived news media bias. Political Psychology, 24(1), 101–117. https://doi.org/10.1111/0162-895X.00318
Baum, M. A., & Gussin, P. (2008). In the eye of the beholder: how information shortcuts shape individual perceptions of bias in the media. Quarterly Journal of Political Science, 3(1), 1–31. https://doi.org/10.1561/100.00007010
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., Dean, J. “Distributed Representations of Words and Phrases and their Compositionality,” in Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2013, pp. 3111–3119. Accessed: May 11, 2017. [Online]. http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf
Mikolov, T., Yih, W., Zweig, G. “Linguistic Regularities in Continuous Space Word Representations,” in Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, Georgia, 2013, pp. 746–751. Accessed: Mar. 19, 2019. [Online]. http://aclweb.org/anthology/N13-1090
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183–186. https://doi.org/10.1126/science.aal4230
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018). Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16), E3635–E3644. https://doi.org/10.1073/pnas.1720347115
Kozlowski, A. C., Taddy, M., & Evans, J. A. (2019). The geometry of culture: analyzing the meanings of class through word embeddings. American Sociological Review, 84(5), 905–949. https://doi.org/10.1177/0003122419877135
“AllSides Media Bias Ratings,” AllSides. https://www.allsides.com/blog/updated-allsides-media-bias-chart-version-11 (accessed May 10, 2020)
Rozado, D. (2020). Wide range screening of algorithmic bias in word embedding models using large sentiment lexicons reveals underreported bias types. PLoS ONE, 15(4), e0231189. https://doi.org/10.1371/journal.pone.0231189
Notess, G. R. “The Wayback Machine: The Web’s Archive.,” Online, vol. 26, no. 2, 2002, Accessed: Mar. 14, 2021. [Online]. https://www.elibrary.ru/item.asp?id=4370136
Mehmood, M. A., Shafiq, H. M., Waheed, A. “Understanding regional context of World Wide Web using common crawl corpus,” in 2017 IEEE 13th Malaysia International Conference on Communications (MICC), 2017, pp. 164–169. https://doi.org/10.1109/MICC.2017.8311752
R. Řehůřek and P. Sojka, “Software Framework for Topic Modelling with Large Corpora,” in Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, Valletta, Malta, May 2010, pp. 45–50.
Gladkova, A., Drozd, A., Matsuoka, S. “Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t.,” in Proceedings of the NAACL Student Research Workshop, San Diego, California, Jun. 2016, pp. 8–15. Accessed: Mar. 25, 2019. [Online]. http://www.aclweb.org/anthology/N16-2002
Byers, D. “Twitter’s most influential political journalists,” POLITICO. https://www.politico.com/blogs/media/2015/04/twitters-most-influential-political-journalists-205510.html (accessed Aug. 02, 2020)
Harnden, T. “The most influential US liberals: 20–1,” Jan. 15, 2010. Accessed: Aug. 02, 2020. [Online]. https://www.telegraph.co.uk/news/worldnews/northamerica/usa/6991000/The-most-influential-US-liberals-20-1.html
Harnden, T. “The most influential US conservatives: 20–1,” Jan. 15, 2010. Accessed: Aug. 02, 2020. [Online]. https://www.telegraph.co.uk/news/worldnews/northamerica/usa/6990965/The-most-influential-US-conservatives-20-1.html
Gunkel, P. “Ideonomy Primary Personality Traits.” http://ideonomy.mit.edu/essays/traits.html (accessed Aug. 08, 2020)
“Similarweb.com—Digital World Market Intelligence Platform,” SimilarWeb.com. https://www.similarweb.com/ (accessed Nov. 19, 2020)
Stone, P. J., Dunphy, D. C., Smith, M. S. (1966). The general inquirer: A computer approach to content analysis. M.I.T. Press, 1966. [Online]. http://wjh.harvard.edu/~inquirer/
Enten, H. “How Roger Ailes Polarized TV News,” FiveThirtyEight, May 19, 2017. https://fivethirtyeight.com/features/how-roger-ailes-polarized-tv-news/ (accessed May 20, 2021)
Mann, T. E. “Asymmetrical Polarization Undermined? Thoughts on the New Pew Research Center’s Report on Political Polarization,” Brookings, Nov. 30, 1AD. https://www.brookings.edu/blog/fixgov/2014/06/13/asymmetrical-polarization-undermined-thoughts-on-the-new-pew-research-centers-report-on-political-polarization/ (accessed May 20, 2021)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. https://doi.org/10.18653/v1/N19-1423
G. Inc, “Americans’ Trust in Mass Media Sinks to New Low,” Gallup.com, Sep. 14, 2016. https://news.gallup.com/poll/195542/americans-trust-mass-media-sinks-new-low.aspx (accessed May 06, 2021)
G. Inc, “Americans Remain Distrustful of Mass Media,” Gallup.com, Sep. 30, 2020. https://news.gallup.com/poll/321116/americans-remain-distrustful-mass-media.aspx (accessed May 06, 2021)
al-Gharbi, M. “The New York Times’ obsession with Trump, quantified,” Columbia Journalism Review. https://www.cjr.org/covering_the_election/new-york-times-trump.php (accessed May 09, 2020)
al-Gharbi, M. “Cable news profits from its obsession with Trump. Viewers are the only victims.,” Columbia Journalism Review. https://www.cjr.org/politics/cable-news-trump-obsession.php (accessed Mar. 22, 2021)
Conflict of interests
The authors declare that they have no conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Below is the link to the electronic supplementary material.
About this article
Cite this article
Rozado, D., al-Gharbi, M. Using word embeddings to probe sentiment associations of politically loaded terms in news and opinion articles from news media outlets. J Comput Soc Sc 5, 427–448 (2022). https://doi.org/10.1007/s42001-021-00130-y