In recent years, there has been a growing interest in research on hate speech on social media. However, researchers face many challenges in producing meaningful results and must develop innovative methods to keep pace with the rapidly evolving nature of this field. How can we effectively determine the prevalence of specific types of hate speech on a given platform? What approaches can we employ to assess whether there has been an increase or decrease in content that can be classified as hate speech? Using the context of Elon Musk's acquisition of Twitter, a period characterized by media reports suggesting a surge in antisemitism on the platform, we explore a range of qualitative and quantitative computational methods. Our analysis reveals that, starting from October 9, 2022, the usage of the term “Jews” on Twitter nearly doubled compared to the preceding period. Additionally, there was a sudden spike in the use of the term “K***s.” However, the question arises: how indicative are these trends of a rise in antisemitism on that platform? We demonstrate that relying solely on keyword-based timelines can be misleading. Nevertheless, when utilized alongside corroborating timelines incorporating additional keywords identified through word frequency analysis, they can serve as powerful tools for content estimation. Nonetheless, it is crucial to supplement these approaches with interpretative methods to validate assumptions based on timelines. By employing a triangulation of methods encompassing descriptive analysis, such as timelines, word and retweet frequency analysis, and manual interpretation and labeling of representative samples, we uncover that discussions about Jews on Twitter during a turbulent 5-week period were predominantly centered around antisemitism. However, these discussions took various forms, including expressing concerns about the increase in antisemitism, denouncing antisemitism, remembering the Holocaust, refuting accusations of antisemitism, and even promoting antisemitic ideologies. We observe a significant escalation in both the volume and the proportion of antisemitic tropes within these conversations, particularly evident in late October 2022. This increase can be attributed to three triggering events that might have been overlooked when drawing conclusions solely from a simplistic timeline analysis of the term “Jews” or targeted slurs. In conclusion, we advocate for a mixed-methods approach where quantitative computational tools are complemented by qualitative discourse analysis. This combination is essential for comprehensively examining trends in complex content such as hate speech. Additionally, the integration of “manual” observation and labeling of representative data samples proves particularly valuable for distinguishing between instances of hate speech and the act of calling out such speech.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
The dataset used for this study is not publicly available and was obtained through the Twitter API/Developer Account. Information on how to obtain it and reproduce the analysis is available from the corresponding author on request.
They annotated 1,100 tweets altogether (before and after Musk’s takeover) and found 264 anti-Jewish tweets, that is, 24% anti-Jewish tweets; https://twitter.com/orensegal/status/1593715384267702272.
His tweet on the night of October 8 read “I’m a bit sleepy tonight but when I wake up I’m going death con 3 On JEWISH PEOPLE The funny thing is I actually can’t be Anti Semitic because black people are actually Jew also You guys have toyed with me and tried to black ball anyone whoever opposes your agenda.”
He posted the message “No President has done more for Israel than I have. Somewhat surprisingly, however, our wonderful Evangelicals are far more appreciative of this than the people of the Jewish faith, especially those living in the U.S. Those living in Israel, though, are a different story – Highest approval rating in the World, could easily be P.M.! U.S. Jews have to get their act together and appreciate what they have in Israel—Before it is too late!”
The tweet was posted on October 15, 2022. At the time of writing, it was still online at https://twitter.com/ramzpaul/status/1581278263187451906.
We wanted a representative sample of all messages on that day and therefore included retweets.
The screenshot shows the date of the original tweet, sent out on October 29. However, this tweet was among the tweets in our sample from October 30 because it was retweeted on that day.
Finklea, K. (2022). Law enforcement and technology: using social media. Congressional Research Service. https://crsreports.congress.gov/product/pdf/R/R47008. Accessed 6 Jul 2023.
Macdonald, S., Correia, S. G., & Watkin, A.-L. (2019). Regulating terrorist content on social media: Automation and the rule of law. International Journal of Law in Context, 15, 183–197. https://doi.org/10.1017/S1744552319000119
Krishnan, N., Gu, J., Tromble, R., & Abroms, L. C. (2021). Research note: Examining how various social media platforms have responded to COVID-19 misinformation. Harvard Kennedy School Misinformation Review. https://doi.org/10.37016/mr-2020-85
Koltay, A. (2019). New media and freedom of expression: Rethinking the constitutional foundations of the public sphere. Hart.
Klonick, K., & Kadri, T. (2019). Facebook v. Sullivan: Building constitutional law for online speech. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3332530
Yin, W., & Zubiaga, A. (2021). Towards generalisable hate speech detection: A review on obstacles and solutions. PeerJ Computer Science, 7, e598. https://doi.org/10.7717/peerj-cs.598
MacAvaney, S., Yao, H.-R., Yang, E., Russell, K., Goharian, N., & Frieder, O. (2019). Hate speech detection: Challenges and solutions. PLoS ONE, 14, e0221152. https://doi.org/10.1371/journal.pone.0221152
Becker, M. J., Troschke, H., & Allington, D. (2021). Decoding antisemitism. An AI-driven study on hate speech and imagery online. First discourse report. Center for Research on Antisemitism (Technical University Berlin) and King’s College London.
Jikeli, G., Axelrod, D., Fischer, R. K., Forouzesh, E., Jeong, W., Miehling, D., & Soemer, K. (2022). Differences between antisemitic and non-antisemitic English language tweets. Computational and Mathematical Organization Theory. https://doi.org/10.1007/s10588-022-09363-2
Hübscher, M., & Von Mering, S. (2022). Antisemitism on social media. Routledge Taylor and Francis Group.
DataReportal. (2023). Digital 2023: Global Overview Report. In: DataReportal – Glob. Digit. Insights. https://datareportal.com/reports/digital-2023-global-overview-report. Accessed 8 May 2023
Meta Stories Ads. (2023). In: Meta Bus. https://www.facebook.com/business/ads/stories-ad-format. Accessed 8 May 2023
Twitter Support [@TwitterSupport]. (2015). 500 million Tweets are sent every day! Learn how to search through all that content: https://support.twitter.com/articles/71577
Zamith, R., & Lewis, S. C. (2015). Content analysis and the algorithmic coder: What computational social science means for traditional modes of media analysis. The Annals of the American Academy of Political and Social Science, 659, 307–318. https://doi.org/10.1177/0002716215570576
Elia-Shalev, A. (2022). “Now that Elon is taking over”: Online antisemites see opening as Twitter is handed over to Musk. In: Jew. Telegr. Agency. https://www.jta.org/2022/10/28/united-states/now-that-elon-is-taking-over-online-antisemites-see-opening-as-twitter-is-handed-over-to-musk. Accessed 17 Nov 2022
Thompson, S. A. (2022). Antisemitic campaign tries to capitalize on Elon Musk’s Twitter takeover. N. Y. Times.
Finkelstein, J., Donohue, J., Goldenberg, A., Jussim, L., Ross, A. R., Vasko, C., Ramos, C., Glover, T., Jagdeep, A., Boyle, A., & Modi, J. (2023). #Twittertakeover: How the Musk acquisition became a launchpad for Gen-Z Neo-Nazis, Ye, and widespread antisemitism. Network Contagion Research Institute.
Miller, C., Weir, D., Ring, S., Marsh, O., Inskip, C., & Chavana, N. P. (2023). Antisemitism on twitter before and after Elon Musk’s acquisition. Institute for Strategic Dialogue.
Jikeli, G., Karali, S., Miehling, D., & Soemer, K. (2023). Antisemitic messages? A guide to high-quality annotation and a labeled dataset of tweets. Arxivpreprint. https://doi.org/10.48550/ARXIV.2304.14599
Jikeli, G., Cavar, D., & Miehling, D. (2019). Annotating antisemitic online content. Towards an applicable definition of antisemitism. ArXiv191001214 CsCY. https://doi.org/10.5967/3r3m-na89
Jikeli, G., Cavar, D., Joeng, W., Miehling, D., Wagh, P., & Pak, D. (2022). Towards an AI definition of antisemitism? In M. Hübscher & S. Von Mering (Eds.), Antisemitism on social media (pp. 193–212). Routledge Taylor & Francis Group.
McClain, C., Widjaya, R., Rivero, G., & Smith, A. (2021). The behaviors and attitudes of U.S. adults on twitter. In: Pew Res. Cent. Internet Sci. Tech. https://www.pewresearch.org/internet/2021/11/15/the-behaviors-and-attitudes-of-u-s-adults-on-twitter/. Accessed 3 Apr 2023
Cramer, P. (2022). “Kanye is right about the Jews”, antisemitic group says on Los Angeles highway banner. Jewish Telegraphic Agency.
Anglesey, A. (2022). Antisemitic Kanye message projected on to stadium at Florida football game. In: Newsweek. https://www.newsweek.com/antisemitic-kanye-west-message-beamed-stadium-florida-football-game-1755640. Accessed 20 Nov 2022
Gerken, T., & Sardarizadeh, S. (2023). Elon Musk: I will say what I want even if it costs me. BBC News.
Rosenberg, Y. (2023). Elon Musk among the anti-semites. In: The Atlantic. https://www.theatlantic.com/ideas/archive/2023/05/elon-musk-george-soros-anti-semites/674072/. Accessed 22 Jun 2023
This work used Jetstream2 at Indiana University through allocation HUM200003 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. We are grateful for the support of Indiana University’s Observatory on Social Media (OSoMe), Sameer Karali for his help with data collection, and other members of the Social Media & Hate Research Lab at Indiana University’s Institute for the Study of Contemporary Antisemitism, Elisha S. Breton, Kathryn Rose Cooper, Robin Forstenhäusler, Sophie von Máriássy, Daniel Miehling, Mabel Pointdexter, Jenna Solomon, Clara Schilling, and Victor Tschiskale.
Conflict of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Jikeli, G., Soemer, K. The value of manual annotation in assessing trends of hate speech on social media: was antisemitism on the rise during the tumultuous weeks of Elon Musk’s Twitter takeover?. J Comput Soc Sc 6, 943–971 (2023). https://doi.org/10.1007/s42001-023-00219-6