Skip to main content

Using word embeddings to probe sentiment associations of politically loaded terms in news and opinion articles from news media outlets


This work describes an analysis of political associations in 27 million diachronic (1975–2019) news and opinion articles from 47 news media outlets popular in the United States. We use embedding models trained on individual outlets content to quantify outlet-specific latent associations between positive/negative sentiment words and terms loaded with political connotations such as those describing political orientation, party affiliation, names of influential politicians, and ideologically aligned public figures. We observe that both left- and right-leaning news media tend to associate positive sentiment words with terms used to refer to members of their own political in-group and negative sentiment words with terms used to denote members of their ideological outgroup. Outlets rated as centrist by humans display political associations that are often milder but similar in orientation to those of left-leaning news organizations. A weighted average of political associations by outlets’ readership volume hints that political associations embedded in left of center news outlets might have larger societal reach. A chronological analysis of political associations through time suggests that political sentiment polarization is increasing in both left- and right-leaning news media contents. Our approach for measuring sentiment associations of words denoting political orientation in outlet-specific embedding models correlates substantially with external human ratings of outlet ideological bias (r > 0.7). Yet, specific sentiment associations are sometimes multifaceted and challenging to interpret. Overall, our work signals the potential of machine learning models derived from news media language usage to quantify the ideological bias embedded in news outlet content.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7


  1. Neuendorf, K. A. (2001). The Content Analysis Guidebook (1st ed.). SAGE Publications Inc.

    Google Scholar 

  2. Rozado, D. (2019). Using word embeddings to analyze how universities conceptualize ‘diversity’ in their online institutional presence. Society, 56(3), 256–266.

    Article  Google Scholar 

  3. Setting the Agenda: Mass Media and Public Opinion, 2 edition. Polity, 2014

  4. Rozado, D. (2020). Prejudice and victimization themes in New York times discourse: a chronological analysis. Academic Questions, 33(1), 89–100.

    Article  Google Scholar 

  5. Gorp, B. V. (2007). The constructionist approach to framing: bringing culture back in. The Journal of Communication, 57(1), 60–78.

    Article  Google Scholar 

  6. Gadarian, S. K. (2010). The politics of threat: how terrorism news shapes foreign policy attitudes. The Journal of Political, 72(2), 469–483.

    Article  Google Scholar 

  7. “Media Bias (Real and Perceived) and the Rise of Partisan Media,” Niskanen Center, Nov. 06, 2017. (accessed May 03, 2021)

  8. Grabe, M. E., Bucy, E. P. Image Bite Politics: News and the Visual Framing of Elections. Oxford University Press. Accessed: May 03, 2021. [Online].

  9. Groeling, T. (2008). Who’s the Fairest of them All? An empirical test for partisan bias on ABC, CBS, NBC, and Fox News. Presidential Studies Quarterly, 38(4), 631–657.

    Article  Google Scholar 

  10. Schiffer, A. J. (2006). Assessing partisan bias in political news: the case(s) of local senate election coverage. Political Communication, 23(1), 23–39.

    Article  Google Scholar 

  11. Groseclose, T., & Milyo, J. (2005). A measure of media bias. The Quarterly Journal of Economics, 120(4), 1191–1237.

    Article  Google Scholar 

  12. Gasper, J. (2011). Shifting ideologies? Re-examining media bias. Quarterly Journal of Political Science, 6, 357–370.

    Article  Google Scholar 

  13. Martin, G. J., & Yurukoglu, A. (2017). Bias in cable news: persuasion and polarization. The American Economic Review, 107(9), 2565–2599.

    Article  Google Scholar 

  14. Larcinese, V., Puglisi, R., & Snyder, J. M., Jr. (2007). Partisan bias in economic news: evidence on the agenda-setting behavior of US newspapers. National Bureau of Economic Research.

    Article  Google Scholar 

  15. Waldman, P., & Devitt, J. (1998). Newspaper photographs and the 1996 presidential election: the question of bias. Journalism & Mass Communication Quarterly, 75(2), 302–311.

    Article  Google Scholar 

  16. Barrett, A. W., & Barrington, L. W. (2005). Bias in newspaper photograph selection. Political Research Quarterly, 58(4), 609–618.

    Article  Google Scholar 

  17. Kahn, K. F., & Kenney, P. (2002). The slant of the news: how editorial endorsements influence campaign coverage and citizens’ views of candidates. American Political Science Review., 96(2), 381–394.

    Article  Google Scholar 

  18. Media Bias in Presidential Election Coverage 1948–2008: Evaluation via Formal Measurement. Accessed: May 03, 2021. [Online].

  19. Niven, D. (2003). Objective evidence on media bias: newspaper coverage of congressional party switchers. Journalism & Mass Communication Quarterly, 80(2), 311–326.

    Article  Google Scholar 

  20. Dalal, S., Adlim, B., & Lesk, M. (2019). How to measure relative bias in media coverage? Significance, 16(5), 18–23.

    Article  Google Scholar 

  21. Mitchell, A., Gottfried, J., Kiley, J., Matsa, K. E. “Political Polarization & Media Habits,” Pew Research Center’s Journalism Project, Oct. 21, 2014. (accessed May 06, 2021)

  22. Eveland, W. P., & Shah, D. V. (2003). The impact of individual and interpersonal factors on perceived news media bias. Political Psychology, 24(1), 101–117.

    Article  Google Scholar 

  23. Baum, M. A., & Gussin, P. (2008). In the eye of the beholder: how information shortcuts shape individual perceptions of bias in the media. Quarterly Journal of Political Science, 3(1), 1–31.

    Article  Google Scholar 

  24. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., Dean, J. “Distributed Representations of Words and Phrases and their Compositionality,” in Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2013, pp. 3111–3119. Accessed: May 11, 2017. [Online].

  25. Mikolov, T., Yih, W., Zweig, G. “Linguistic Regularities in Continuous Space Word Representations,” in Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, Georgia, 2013, pp. 746–751. Accessed: Mar. 19, 2019. [Online].

  26. Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183–186.

    Article  Google Scholar 

  27. Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018). Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16), E3635–E3644.

    Article  Google Scholar 

  28. Kozlowski, A. C., Taddy, M., & Evans, J. A. (2019). The geometry of culture: analyzing the meanings of class through word embeddings. American Sociological Review, 84(5), 905–949.

    Article  Google Scholar 

  29. “AllSides Media Bias Ratings,” AllSides. (accessed May 10, 2020)

  30. Rozado, D. (2020). Wide range screening of algorithmic bias in word embedding models using large sentiment lexicons reveals underreported bias types. PLoS ONE, 15(4), e0231189.

    Article  Google Scholar 

  31. Notess, G. R. “The Wayback Machine: The Web’s Archive.,” Online, vol. 26, no. 2, 2002, Accessed: Mar. 14, 2021. [Online].

  32. Mehmood, M. A., Shafiq, H. M., Waheed, A. “Understanding regional context of World Wide Web using common crawl corpus,” in 2017 IEEE 13th Malaysia International Conference on Communications (MICC), 2017, pp. 164–169.

  33. R. Řehůřek and P. Sojka, “Software Framework for Topic Modelling with Large Corpora,” in Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, Valletta, Malta, May 2010, pp. 45–50.

  34. Gladkova, A., Drozd, A., Matsuoka, S. “Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t.,” in Proceedings of the NAACL Student Research Workshop, San Diego, California, Jun. 2016, pp. 8–15. Accessed: Mar. 25, 2019. [Online].

  35. Byers, D. “Twitter’s most influential political journalists,” POLITICO. (accessed Aug. 02, 2020)

  36. Harnden, T. “The most influential US liberals: 20–1,” Jan. 15, 2010. Accessed: Aug. 02, 2020. [Online].

  37. Harnden, T. “The most influential US conservatives: 20–1,” Jan. 15, 2010. Accessed: Aug. 02, 2020. [Online].

  38. Gunkel, P. “Ideonomy Primary Personality Traits.” (accessed Aug. 08, 2020)

  39. “—Digital World Market Intelligence Platform,” (accessed Nov. 19, 2020)

  40. Stone, P. J., Dunphy, D. C., Smith, M. S. (1966). The general inquirer: A computer approach to content analysis. M.I.T. Press, 1966. [Online].

  41. Enten, H. “How Roger Ailes Polarized TV News,” FiveThirtyEight, May 19, 2017. (accessed May 20, 2021)

  42. Mann, T. E. “Asymmetrical Polarization Undermined? Thoughts on the New Pew Research Center’s Report on Political Polarization,” Brookings, Nov. 30, 1AD. (accessed May 20, 2021)

  43. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding.

  44. G. Inc, “Americans’ Trust in Mass Media Sinks to New Low,”, Sep. 14, 2016. (accessed May 06, 2021)

  45. G. Inc, “Americans Remain Distrustful of Mass Media,”, Sep. 30, 2020. (accessed May 06, 2021)

  46. al-Gharbi, M. “The New York Times’ obsession with Trump, quantified,” Columbia Journalism Review. (accessed May 09, 2020)

  47. al-Gharbi, M. “Cable news profits from its obsession with Trump. Viewers are the only victims.,” Columbia Journalism Review. (accessed Mar. 22, 2021)

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to David Rozado.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 1329 KB)

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rozado, D., al-Gharbi, M. Using word embeddings to probe sentiment associations of politically loaded terms in news and opinion articles from news media outlets. J Comput Soc Sc 5, 427–448 (2022).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: