Skip to main content

The value of manual annotation in assessing trends of hate speech on social media: was antisemitism on the rise during the tumultuous weeks of Elon Musk’s Twitter takeover?


In recent years, there has been a growing interest in research on hate speech on social media. However, researchers face many challenges in producing meaningful results and must develop innovative methods to keep pace with the rapidly evolving nature of this field. How can we effectively determine the prevalence of specific types of hate speech on a given platform? What approaches can we employ to assess whether there has been an increase or decrease in content that can be classified as hate speech? Using the context of Elon Musk's acquisition of Twitter, a period characterized by media reports suggesting a surge in antisemitism on the platform, we explore a range of qualitative and quantitative computational methods. Our analysis reveals that, starting from October 9, 2022, the usage of the term “Jews” on Twitter nearly doubled compared to the preceding period. Additionally, there was a sudden spike in the use of the term “K***s.” However, the question arises: how indicative are these trends of a rise in antisemitism on that platform? We demonstrate that relying solely on keyword-based timelines can be misleading. Nevertheless, when utilized alongside corroborating timelines incorporating additional keywords identified through word frequency analysis, they can serve as powerful tools for content estimation. Nonetheless, it is crucial to supplement these approaches with interpretative methods to validate assumptions based on timelines. By employing a triangulation of methods encompassing descriptive analysis, such as timelines, word and retweet frequency analysis, and manual interpretation and labeling of representative samples, we uncover that discussions about Jews on Twitter during a turbulent 5-week period were predominantly centered around antisemitism. However, these discussions took various forms, including expressing concerns about the increase in antisemitism, denouncing antisemitism, remembering the Holocaust, refuting accusations of antisemitism, and even promoting antisemitic ideologies. We observe a significant escalation in both the volume and the proportion of antisemitic tropes within these conversations, particularly evident in late October 2022. This increase can be attributed to three triggering events that might have been overlooked when drawing conclusions solely from a simplistic timeline analysis of the term “Jews” or targeted slurs. In conclusion, we advocate for a mixed-methods approach where quantitative computational tools are complemented by qualitative discourse analysis. This combination is essential for comprehensively examining trends in complex content such as hate speech. Additionally, the integration of “manual” observation and labeling of representative data samples proves particularly valuable for distinguishing between instances of hate speech and the act of calling out such speech.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Data availability

The dataset used for this study is not publicly available and was obtained through the Twitter API/Developer Account. Information on how to obtain it and reproduce the analysis is available from the corresponding author on request.




  3. They annotated 1,100 tweets altogether (before and after Musk’s takeover) and found 264 anti-Jewish tweets, that is, 24% anti-Jewish tweets;


  5. His tweet on the night of October 8 read “I’m a bit sleepy tonight but when I wake up I’m going death con 3 On JEWISH PEOPLE The funny thing is I actually can’t be Anti Semitic because black people are actually Jew also You guys have toyed with me and tried to black ball anyone whoever opposes your agenda.”

  6. He posted the message “No President has done more for Israel than I have. Somewhat surprisingly, however, our wonderful Evangelicals are far more appreciative of this than the people of the Jewish faith, especially those living in the U.S. Those living in Israel, though, are a different story – Highest approval rating in the World, could easily be P.M.! U.S. Jews have to get their act together and appreciate what they have in Israel—Before it is too late!



  9. The tweet was posted on October 15, 2022. At the time of writing, it was still online at

  10. We wanted a representative sample of all messages on that day and therefore included retweets.

  11. The percentages of antisemitic tweets from 2019 to 2022 are results of an ongoing research project of the Social Media & Hate Research Lab at Indiana University’s Institute for the Study of Contemporary Antisemitism. They have been published only in part so far, see [9, 19].

  12. The screenshot shows the date of the original tweet, sent out on October 29. However, this tweet was among the tweets in our sample from October 30 because it was retweeted on that day.


  1. Finklea, K. (2022). Law enforcement and technology: using social media. Congressional Research Service. Accessed 6 Jul 2023.

  2. Macdonald, S., Correia, S. G., & Watkin, A.-L. (2019). Regulating terrorist content on social media: Automation and the rule of law. International Journal of Law in Context, 15, 183–197.

    Article  Google Scholar 

  3. Krishnan, N., Gu, J., Tromble, R., & Abroms, L. C. (2021). Research note: Examining how various social media platforms have responded to COVID-19 misinformation. Harvard Kennedy School Misinformation Review.

    Article  Google Scholar 

  4. Koltay, A. (2019). New media and freedom of expression: Rethinking the constitutional foundations of the public sphere. Hart.

    Book  Google Scholar 

  5. Klonick, K., & Kadri, T. (2019). Facebook v. Sullivan: Building constitutional law for online speech. SSRN Electronic Journal.

    Article  Google Scholar 

  6. Yin, W., & Zubiaga, A. (2021). Towards generalisable hate speech detection: A review on obstacles and solutions. PeerJ Computer Science, 7, e598.

    Article  Google Scholar 

  7. MacAvaney, S., Yao, H.-R., Yang, E., Russell, K., Goharian, N., & Frieder, O. (2019). Hate speech detection: Challenges and solutions. PLoS ONE, 14, e0221152.

    Article  Google Scholar 

  8. Becker, M. J., Troschke, H., & Allington, D. (2021). Decoding antisemitism. An AI-driven study on hate speech and imagery online. First discourse report. Center for Research on Antisemitism (Technical University Berlin) and King’s College London.

    Google Scholar 

  9. Jikeli, G., Axelrod, D., Fischer, R. K., Forouzesh, E., Jeong, W., Miehling, D., & Soemer, K. (2022). Differences between antisemitic and non-antisemitic English language tweets. Computational and Mathematical Organization Theory.

    Article  Google Scholar 

  10. Hübscher, M., & Von Mering, S. (2022). Antisemitism on social media. Routledge Taylor and Francis Group.

    Book  Google Scholar 

  11. DataReportal. (2023). Digital 2023: Global Overview Report. In: DataReportal – Glob. Digit. Insights. Accessed 8 May 2023

  12. Meta Stories Ads. (2023). In: Meta Bus. Accessed 8 May 2023

  13. Twitter Support [@TwitterSupport]. (2015). 500 million Tweets are sent every day! Learn how to search through all that content:

  14. Zamith, R., & Lewis, S. C. (2015). Content analysis and the algorithmic coder: What computational social science means for traditional modes of media analysis. The Annals of the American Academy of Political and Social Science, 659, 307–318.

    Article  Google Scholar 

  15. Elia-Shalev, A. (2022). “Now that Elon is taking over”: Online antisemites see opening as Twitter is handed over to Musk. In: Jew. Telegr. Agency. Accessed 17 Nov 2022

  16. Thompson, S. A. (2022). Antisemitic campaign tries to capitalize on Elon Musk’s Twitter takeover. N. Y. Times.

    Google Scholar 

  17. Finkelstein, J., Donohue, J., Goldenberg, A., Jussim, L., Ross, A. R., Vasko, C., Ramos, C., Glover, T., Jagdeep, A., Boyle, A., & Modi, J. (2023). #Twittertakeover: How the Musk acquisition became a launchpad for Gen-Z Neo-Nazis, Ye, and widespread antisemitism. Network Contagion Research Institute.

    Google Scholar 

  18. Miller, C., Weir, D., Ring, S., Marsh, O., Inskip, C., & Chavana, N. P. (2023). Antisemitism on twitter before and after Elon Musk’s acquisition. Institute for Strategic Dialogue.

    Google Scholar 

  19. Jikeli, G., Karali, S., Miehling, D., & Soemer, K. (2023). Antisemitic messages? A guide to high-quality annotation and a labeled dataset of tweets. Arxivpreprint.

    Article  Google Scholar 

  20. Jikeli, G., Cavar, D., & Miehling, D. (2019). Annotating antisemitic online content. Towards an applicable definition of antisemitism. ArXiv191001214 CsCY.

    Article  Google Scholar 

  21. Jikeli, G., Cavar, D., Joeng, W., Miehling, D., Wagh, P., & Pak, D. (2022). Towards an AI definition of antisemitism? In M. Hübscher & S. Von Mering (Eds.), Antisemitism on social media (pp. 193–212). Routledge Taylor & Francis Group.

    Chapter  Google Scholar 

  22. McClain, C., Widjaya, R., Rivero, G., & Smith, A. (2021). The behaviors and attitudes of U.S. adults on twitter. In: Pew Res. Cent. Internet Sci. Tech. Accessed 3 Apr 2023

  23. Cramer, P. (2022). “Kanye is right about the Jews”, antisemitic group says on Los Angeles highway banner. Jewish Telegraphic Agency.

    Google Scholar 

  24. Anglesey, A. (2022). Antisemitic Kanye message projected on to stadium at Florida football game. In: Newsweek. Accessed 20 Nov 2022

  25. Gerken, T., & Sardarizadeh, S. (2023). Elon Musk: I will say what I want even if it costs me. BBC News.

    Google Scholar 

  26. Rosenberg, Y. (2023). Elon Musk among the anti-semites. In: The Atlantic. Accessed 22 Jun 2023

Download references


This work used Jetstream2 at Indiana University through allocation HUM200003 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. We are grateful for the support of Indiana University’s Observatory on Social Media (OSoMe), Sameer Karali for his help with data collection, and other members of the Social Media & Hate Research Lab at Indiana University’s Institute for the Study of Contemporary Antisemitism, Elisha S. Breton, Kathryn Rose Cooper, Robin Forstenhäusler, Sophie von Máriássy, Daniel Miehling, Mabel Pointdexter, Jenna Solomon, Clara Schilling, and Victor Tschiskale.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Katharina Soemer.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jikeli, G., Soemer, K. The value of manual annotation in assessing trends of hate speech on social media: was antisemitism on the rise during the tumultuous weeks of Elon Musk’s Twitter takeover?. J Comput Soc Sc 6, 943–971 (2023).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: