Skip to main content
SpringerLink
Log in

Hate Speech Detection Using Transformer Ensembles on the HASOC Dataset

  • Conference paper
  • First Online:
Speech and Computer (SPECOM 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12335))

Included in the following conference series:

Abstract

With the ubiquity and anonymity of the Internet, the spread of hate speech has been a growing concern for many years now. The language used for the purpose of dehumanizing, defaming or threatening individuals and marginalized groups not only threatens the mental health of its targets, as well as their democratic access to the Internet, but also the fabric of our society. Because of this, much effort has been devoted to manual moderation. The amount of data generated each day, particularly on social media platforms such as Facebook and twitter, however makes this a Sisyphean task. This has led to an increased demand for automatic methods of hate speech detection.

Here, to contribute towards solving the task of hate speech detection, we worked with a simple ensemble of transformer models on a twitter-based hate speech benchmark. Using this method, we attained a weighted \(F_1\)-score of 0.8426, which we managed to further improve by leveraging more training data, achieving a weighted \(F_1\)-score of 0.8504. Thus markedly outperforming the best performing system in the literature.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Badjatiya, P., Gupta, S., Gupta, M., Varma, V.: Deep learning for hate speech detection in tweets. In: Proceedings of the 26th International Conference on World Wide Web Companion, WWW ’17 Companion, pp. 759–760 (2017)

    Google Scholar 

  2. Barendt, E.: What is the harm of hate speech? Ethic theory, moral prac., vol. 22 (2019). https://doi.org/10.1007/s10677-019-10002-0

  3. Basile, V., et al.: SemEval-2019 task 5: multilingual detection of hate speech against immigrants and women in Twitter. In: Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 54–63 (2019). https://doi.org/10.18653/v1/S19-2007

  4. Brown, A.: What is so special about online (as compared to offline) hate speech? Ethnicities 18(3), 297–326 (2018). https://doi.org/10.1177/1468796817709846

    Article  Google Scholar 

  5. Burnap, P., Williams, M.L.: Cyber hate speech on twitter: an application of machine classification and statistical modeling for policy and decision making. Policy Internet 7(2), 223–242 (2015). https://doi.org/10.1002/poi3.85

    Article  Google Scholar 

  6. Davidson, T., Warmsley, D., Macy, M., Weber, I.: Automated hate speech detection and the problem of offensive language. In: Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM 2017, pp. 512–515 (2017)

    Google Scholar 

  7. Del Vigna, F., Cimino, A., Dell’Orletta, F., Petrocchi, M., Tesconi, M.: Hate me, hate me not: Hate speech detection on Facebook. In: ITASEC, January 2017

    Google Scholar 

  8. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota, June 2019. https://doi.org/10.18653/v1/N19-1423

  9. Djuric, N., Zhou, J., Morris, R., Grbovic, M., Radosavljevic, V., Bhamidipati, N.: Hate speech detection with comment embeddings. In: Proceedings of the 24th International Conference on World Wide Web, WWW 2015 Companion, pp. 29–30. Association for Computing Machinery, New York (2015). https://doi.org/10.1145/2740908.2742760

  10. Do, H.T.T., Huynh, H.D., Nguyen, K.V., Nguyen, N.L.T., Nguyen, A.G.T.: Hate speech detection on vietnamese social media text using the bidirectional-LSTM model (2019), arXiv:1911.03648

  11. Dworkin, R.: A new map of censorship. Index Censorship 35(1), 130–133 (2006). https://doi.org/10.1080/03064220500532412

    Article  Google Scholar 

  12. Gambäck, B., Sikdar, U.K.: Using convolutional neural networks to classify hate-speech. In: Proceedings of the First Workshop on Abusive Language Online, pp. 85–90. Association for Computational Linguistics, Vancouver, BC, Canada, August 2017. https://doi.org/10.18653/v1/W17-3013. https://www.aclweb.org/anthology/W17-3013

  13. Greevy, E., Smeaton, A.F.: Classifying racist texts using a support vector machine. In: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2004, pp. 468–469. Association for Computing Machinery, New York (2004). https://doi.org/10.1145/1008992.1009074

  14. Gröndahl, T., Pajola, L., Juuti, M., Conti, M., Asokan, N.: All you need is “love”: evading hate speech detection. In: Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security, AISec 2018, pp. 2–12. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3270101.3270103

  15. Hern, A.: Revealed: catastrophic effects of working as a Facebook moderator. The Guardian (2019). https://www.theguardian.com/technology/2019/sep/17/revealed-catastrophic-effects-working-facebook-moderator. Accessed 26 Apr 2020

  16. Heyman, S.: Hate speech, public discourse, and the first amendment. In: Hare, I., Weinstein, J. (eds.) Extreme Speech and Democracy. Oxford Scholarship Online (2009). https://doi.org/10.1093/acprof:oso/9780199548781.003.0010

  17. Huynh, T.V., Nguyen, V.D., Nguyen, K.V., Nguyen, N.L.T., Nguyen, A.G.T.: Hate speech detection on Vietnamese social media text using the bi-gru-lstm-cnn model. arXiv:1911.03644 (2019)

  18. Immpermium: detecting insults in social commentary. https://kaggle.com/c/detecting-insults-in-social-commentary. Accessed 27 April 2020

  19. Kwok, I., Wang, Y.: Locate the hate: detecting tweets against blacks. In: Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2013, pp. 1621–1622. AAAI Press (2013)

    Google Scholar 

  20. Liu, Y., et al.: Roberta: A robustly optimized bert pretraining approach (2019)

    Google Scholar 

  21. MacAvaney, S., Yao, H.R., Yang, E., Russell, K., Goharian, N., Frieder, O.: Hate speech detection: Challenges and solutions. PLOS ONE 14(8), 1–16 (2019). https://doi.org/10.1371/journal.pone.0221152

  22. Mandl, T., Modha, S., Mandlia, C., Patel, D., Patel, A., Dave, M.: HASOC - Hate Speech and Offensive Content identification in indo-European languages. https://hasoc2019.github.io. Accessed 20 Sep 2019

  23. Mandl, T., Modha, S., Patel, D., Dave, M., Mandlia, C., Patel, A.: Overview of the HASOC track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo-European Languages). In: Proceedings of the 11th Annual Meeting of the Forum for Information Retrieval Evaluation, December 2019

    Google Scholar 

  24. Matsuda, M.J.: Public response to racist spech: considering the victim’s story. In: Matsuda, M.J., Lawrence III, C.R. (ed.) Words that Wound: Critical Race Theory, Assaultive Speech, and the First Amendment, pp. 17–52. Routledge, New York (1993)

    Google Scholar 

  25. Mehdad, Y., Tetreault, J.: Do characters abuse more than words? In: Proceedings of the SIGDIAL2016 Conference, pp. 299–303, January 2016. https://doi.org/10.18653/v1/W16-3638

  26. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Proceedings of the NIPS, pp. 3111–3119 (2013)

    Google Scholar 

  27. Mondal, M., Silva, L.A., Benevenuto, F.: A measurement study of hate speech in social media. In: Proceedings of the 28th ACM Conference on Hypertext and Social Media, HT 2017, pp. 85–94. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3078714.3078723

  28. Nina-Alcocer, V.: Vito at HASOC 2019: Detecting hate speech and offensive content through ensembles. In: Mehta, P., Rosso, P., Majumder, P., Mitra, M. (eds.) Working Notes of FIRE 2019 - Forum for Information Retrieval Evaluation, Kolkata, India, 12–15 December, 2019. CEUR Workshop Proceedings, vol. 2517, pp. 214–220. CEUR-WS.org (2019). http://ceur-ws.org/Vol-2517/T3-5.pdf

  29. Njagi, D., Zuping, Z., Hanyurwimfura, D., Long, J.: A lexicon-based approach for hate speech detection. Int. J. Multimed. Ubiquitous Eng. 10, 215–230 (2015). https://doi.org/10.14257/ijmue.2015.10.4.21

  30. Nourbakhsh, A., Vermeer, F., Wiltvank, G., van der Goot, R.: sthruggle at SemEval-2019 task 5: an ensemble approach to hate speech detection. In: Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 484–488. Association for Computational Linguistics, Minneapolis, Minnesota, June 2019. https://doi.org/10.18653/v1/S19-2086

  31. Otter, D.W., Medina, J.R., Kalita, J.K.: A survey of the usages of deep learning for natural language processing. IEEE Trans. Neural Networks Learn. Syst., 1–21 (2020)

    Google Scholar 

  32. Park, J., Fung, P.: One-step and two-step classification for abusive language detection on Twitter. In: ALW1: 1st Workshop on Abusive Language Online, June 2017

    Google Scholar 

  33. Alonso, P., Rajkumar Saini, G.K.: The North at HASOC 2019 hate speech detection in social media data. In: Proceedings of the 11th Anual Meeting of the Forum for Information Retrieval Evaluation, December 2019

    Google Scholar 

  34. Ross, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., Wojatzki, M.: Measuring the reliability of hate speech annotations: the case of the European refugee crisis. In: Beißwenger, M., Wojatzki, M., Zesch, T. (eds.) Proceedings of NLP4CMC III: 3rd Workshop on Natural Language Processing for Computer-Mediated Communication, pp. 6–9, September 2016. https://doi.org/10.17185/duepublico/42132

  35. Seganti, A., Sobol, H., Orlova, I., Kim, H., Staniszewski, J., Krumholc, T., Koziel, K.: Nlpr@srpol at semeval-2019 task 6 and task 5: linguistically enhanced deep learning offensive sentence classifier. In: SemEval@NAACL-HLT (2019)

    Google Scholar 

  36. Spertus, E.: Smokey: Automatic recognition of hostile messages. In: Proceedings of the 14th National Conference on Artificial Intelligence and 9th Innovative Applications of Artificial Intelligence Conference (AAAI-97/IAAI-97), pp. 1058–1065. AAAI Press, Menlo Park (1997. http://www.ai.mit.edu/people/ellens/smokey.ps

  37. Sun, C., Qiu, X., Xu, Y., Huang, X.: How to fine-tune BERT for text classification? In: Sun, M., Huang, X., Ji, H., Liu, Z., Liu, Y. (eds.) CCL 2019. LNCS (LNAI), vol. 11856, pp. 194–206. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32381-3_16

    Chapter  Google Scholar 

  38. Wang, B., Ding, Y., Liu, S., Zhou, X.: Ynu\(\_\)wb at HASOC 2019: Ordered neurons LSTM with attention for identifying hate speech and offensive language. In: Mehta, P., Rosso, P., Majumder, P., Mitra, M. (eds.) Working Notes of FIRE 2019 - Forum for Information Retrieval Evaluation, Kolkata, India, 12–15 December, 2019, pp. 191–198 (2019). http://ceur-ws.org/Vol-2517/T3-2.pdf

  39. Warner, W., Hirschberg, J.: Detecting hate speech on the world wide web. In: Proceedings of the Second Workshop on Language in Social Media, pp. 19–26. Association for Computational Linguistics, Montréal, Canada, June 2012. https://www.aclweb.org/anthology/W12-2103

  40. Waseem, Z., Hovy, D.: Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In: Proceedings of the NAACL Student Research Workshop, pp. 88–93. Association for Computational Linguistics, San Diego, California, June 2016. https://doi.org/10.18653/v1/N16-2013. https://www.aclweb.org/anthology/N16-2013

  41. Wei, X., Lin, H., Yang, L., Yu, Y.: A convolution-LSTM-based deep neural network for cross-domain MOOC forum post classification. Information 8, 92 (2017). https://doi.org/10.3390/info8030092

    Article  Google Scholar 

  42. Wiegand, M., Siegel, M., Ruppenhofer, J.: Overview of the germeval 2018 shared task on the identification of offensive language. In: Proceedings of the GermEval 2018 Workshop, pp. 1–11 (2018)

    Google Scholar 

  43. Wolf, T., et al.: Huggingface’s transformers: State-of-the-art natural language processing. arXiv:1910.03771 (2019)

  44. Young, T., Hazarika, D., Poria, S., Cambria, E.: Recent trends in deep learning based natural language processing (2017), arXiv:1708.02709 Comment: Added BERT, ELMo, Transformer

  45. Yuan, S., Wu, X., Xiang, Y.: A two phase deep learning model for identifying discrimination from tweets. In: Pitoura, E., et al. (eds.) Proceedings of the 19th International Conference on Extending Database Technology, EDBT 2016, Bordeaux, France, March 15–16, 2016, Bordeaux, France, 15–16 March, 2016, pp. 696–697. OpenProceedings.org (2016). https://doi.org/10.5441/002/edbt.2016.92

  46. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., Kumar, R.: Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). In: Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 75–86 (2019)

    Google Scholar 

  47. Zampieri, M., et al.: SemEval-2020 Task 12: multilingual offensive language identification in social media (OffensEval 2020). In: Proceedings of SemEval (2020)

    Google Scholar 

  48. Zhang, Z., Luo, L.: Hate speech detection: a solved problem? the challenging case of long tail on twitter. Semantic Web Accepted, October 2018. https://doi.org/10.3233/SW-180338

  49. Zhang, Z., Robinson, D., Tepper, J.: Detecting hate speech on Twitter using a convolution-GRU based deep neural network. In: Gangemi, A., Navigli, R., Vidal, M.-E., Hitzler, P., Troncy, R., Hollink, L., Tordai, A., Alam, M. (eds.) ESWC 2018. LNCS, vol. 10843, pp. 745–760. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93417-4_48

    Chapter  Google Scholar 

  50. Zimbardo, P.G.: The human choice: individuation, reason, and order versus deindividuation, impulse, and chaos. Nebr. Symp. Motiv. 17, 237–307 (1969)

    Google Scholar 

  51. Zimmerman, S., Kruschwitz, U., Fox, C.: Improving hate speech detection with deep learning ensembles. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan, May 2018. https://www.aclweb.org/anthology/L18-1404

Download references

Acknowledgements

Part of this work has been funded by the Vinnova project “Language models for Swedish authorities” (ref. number: 2019-02996).

Author information

Authors and Affiliations

  1. Embedded Internet Systems Lab, Luleå University of Technology, Luleå, Sweden

    Pedro Alonso, Rajkumar Saini & György Kovács

  2. MTA-SZTE Research Group on Artificial Intelligence, Szeged, Hungary

    György Kovács

Authors
  1. Pedro Alonso
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rajkumar Saini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. György Kovács
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to György Kovács .

Editor information

Editors and Affiliations

  1. St. Petersburg Institute for Informatics and Automation, Russian Academy of Sciences, St. Petersburg, Russia

    Alexey Karpov

  2. Institute for Applied and Mathematical Linguistics, Moscow State Linguistic University, Moscow, Russia

    Rodmonga Potapova

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Alonso, P., Saini, R., Kovács, G. (2020). Hate Speech Detection Using Transformer Ensembles on the HASOC Dataset. In: Karpov, A., Potapova, R. (eds) Speech and Computer. SPECOM 2020. Lecture Notes in Computer Science(), vol 12335. Springer, Cham. https://doi.org/10.1007/978-3-030-60276-5_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-60276-5_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-60275-8

  • Online ISBN: 978-3-030-60276-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation