Skip to main content
SpringerLink
Log in

BERT-based ensemble learning for multi-aspect hate speech detection

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

The social media world nowadays is overwhelmed with unfiltered content ranging from cyberbullying and cyberstalking to hate speech. Therefore, identifying and cleaning up such toxic language presents a big challenge and an active area of research. This study is dedicated to multi-aspect hate speech detection based on classifying text in multi-labels including ‘identity hate’, ‘threat’, ‘insult’, ‘obscene’, ‘toxic’ and ‘severe toxic’. The proposed approach is based on the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model combined with Deep Learning (DL) models to compose several ensemble learning architectures. The DL models used are built by stacking Bidirectional Long-Short Term Memory (Bi-LSTM) and/or Bidirectional Gated Recurrent Unit (Bi-GRU) on GloVe and FastText word embeddings. Whereby, these models and BERT are trained individually on multi-label hateful dataset and used in combination for hate speech detection tasks on social media. Thus, we demonstrate that encoding texts by using recent word embedding techniques as FastText and GloVe alongside Bi-LSTM and Bi-GRU can create models that, when combined with BERT, can enhance the ROC-AUC score to 98.63%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

The data that support the findings of this study are publicly available in kaggle Datasets (www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge/data).

Notes

  1. https://www.kaggle.com/swetaagrawal/formspring-data-for-cyberbullying-detection

  2. https://fasttext.cc/docs/en/crawl-vectors.htm

  3. https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge/data

  4. https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge

References

  1. Kaplan, A.M., Haenlein, M.: Users of the world, unite! The challenges and opportunities of Social Media. Bus. Horiz. 53, 59–68 (2010). https://doi.org/10.1016/j.bushor.2009.09.003

    Article  Google Scholar 

  2. Smith, P.K., Mahdavi, J., Carvalho, M., Fisher, S., Russell, S., Tippett, N.: Cyberbullying: its nature and impact in secondary school pupils. J. Child Psychol. Psychiatry. 49, 376–385 (2008). https://doi.org/10.1111/j.1469-7610.2007.01846.x

    Article  PubMed  Google Scholar 

  3. Grigg, D.W.: Cyber-aggression: definition and concept of cyberbullying. J. Psychol. Couns. Sch. 20, 143–156 (2010). https://doi.org/10.1375/ajgc.20.2.143

    Article  Google Scholar 

  4. Davidson, T., Warmsley, D., Macy, M., Weber, I.: Automated Hate Speech Detection and the Problem of Offensive Language. In: Proceedings of the 11th International Conference on Web and Social Media, ICWSM 2017. 512–515 (2017)

  5. Fortuna, P., Nunes, S.: A survey on automatic detection of hate speech in text. ACM Comput. Surv. 51, 1–30 (2018)

    Article  Google Scholar 

  6. Basile, V., Bosco, C., Fersini, E., Debora, N., Patti, V., Pardo, F.M.R., Rosso, P., Sanguinetti, M.: Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In: 13th International Workshop on Semantic Evaluation. pp. 54–63. Association for Computational Linguistics (2019)

  7. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998–6008 (2017)

  8. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv Prepr. http://arxiv.org/1810.04805. (2018)

  9. Badjatiya, P., Gupta, S., Gupta, M., Varma, V.: Deep Learning for Hate Speech Detection in Tweets. In: Proceedings of the 26th International Conference on World Wide Web Companion - WWW ’17 Companion. pp. 759–760. ACM Press, New York, New York, USA (2017). https://doi.org/10.1145/3041021.3054223

  10. Waseem, Z., Hovy, D.: Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. In: Proceedings of the NAACL Student Research Workshop. pp. 88–93. Association for Computational Linguistics, Stroudsburg, PA, USA, PA, USA (2016). https://doi.org/10.18653/v1/N16-2013

  11. Srivastava, S., Khurana, P., Tewari, V.: Identifying aggression and toxicity in comments using capsule network. In: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018). pp. 98–105 (2018)

  12. Aroyehun, S.T., Gelbukh, A.: Aggression detection in social media Using deep neural networks, data augmentation, and pseudo labeling. In: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018). pp. 90–97 (2018)

  13. Agrawal, S., Awekar, A.: Deep Learning for Detecting Cyberbullying Across Multiple Social Media Platforms. In: European Conference on Information Retrieval. pp. 303–315 (2018). https://doi.org/10.1007/978-3-319-76941-7

  14. Founta, A.M., Chatzakou, D., Kourtellis, N., Blackburn, J., Vakali, A., Leontiadis, I.: A Unified Deep Learning Architecture for Abuse Detection. In: Proceedings of the 10th ACM Conference on Web Science - WebSci ’19. pp. 105–114. ACM Press, New York, New York, USA (2019). https://doi.org/10.1145/3292522.3326028

  15. Mossie, Z., Wang, J.H.: Vulnerable community identification using hate speech detection on social media. Inf. Process. Manag. 57, 102087 (2020)

    Article  Google Scholar 

  16. Kapil, P., Ekbal, A.: A deep neural network based multi-task learning approach to hate speech detection. Knowledge-Based Syst. 210, 106458 (2020). https://doi.org/10.1016/j.knosys.2020.106458

    Article  Google Scholar 

  17. Mangaonkar, A., Pawar, R., Chowdhury, N.S., Raje, R.R.: Enhancing collaborative detection of cyberbullying behavior in Twitter data. Cluster Comput. 25, 1263–1277 (2022). https://doi.org/10.1007/s10586-021-03483-1

    Article  Google Scholar 

  18. Kammakomati, M., Tarun Kumar, P. V, Radhika, K.: Comparison of Machine Learning Algorithms for Hate and Offensive Speech Detection. In: Evolutionary Computing and Mobile Sustainable Networks. pp. 873–881. Springer (2022)

  19. Liu, P., Li, W., Zou, L.: NULI at SemEval-2019 Task 6: Transfer Learning for Offensive Language Detection using Bidirectional Transformers. In: Proceedings of the 13th International Workshop on Semantic Evaluation. pp. 87–91. Association for Computational Linguistics, Stroudsburg, PA, USA (2019). https://doi.org/10.18653/v1/S19-2011

  20. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., Kumar, R.: SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval). In: Proceedings of the 13th International Workshop on Semantic Evaluation. 652–656 (2019). https://doi.org/10.18653/v1/S19-2116

  21. Mozafari, M., Farahbakhsh, R., Crespi, N.: A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media. In: International Conference on Complex Networks and Their Applications. pp. 928–940 (2020). https://doi.org/10.1007/978-3-030-36687-2_77

  22. Waseem, Z.: Are you a racist or am i seeing things annotator influence on hate speech detection on twitter. In: Proceedings of the first workshop on NLP and computational social science. pp. 138–142 (2016)

  23. Modha, S., Majumder, P., Mandl, T., Mandalia, C.: Detecting and visualizing hate speech in social media: a cyber Watchdog for surveillance. Expert Syst. Appl. 161, 113725 (2020). https://doi.org/10.1016/j.eswa.2020.113725

    Article  Google Scholar 

  24. Pamungkas, E.W., Basile, V., Patti, V.: Misogyny detection in twitter: a multilingual and cross-domain study. Inf. Process. Manag. 57, 102360 (2020). https://doi.org/10.1016/j.ipm.2020.102360

    Article  Google Scholar 

  25. Wei, B., Li, J., Gupta, A., Umair, H., Vovor, A., Durzynski, N.: Offensive Language and Hate Speech Detection with Deep Learning and Transfer Learning. arXiv Prepr. http://arxiv.org/2108.03305. (2021)

  26. Plaza-del-Arco, F.M., Molina-González, M.D., Urena-López, L.A., Martín-Valdivia, M.T.: Comparing pre-trained language models for Spanish hate speech detection. Expert Syst. Appl. 166, 114120 (2021)

    Article  Google Scholar 

  27. Rosa, H., Matos, D., Ribeiro, R., Coheur, L., Carvalho, J.P.: A “Deeper” Look at Detecting Cyberbullying in Social Networks. In: 2018 International Joint Conference on Neural Networks (IJCNN). pp. 1–8. IEEE (2018). https://doi.org/10.1109/IJCNN.2018.8489211

  28. Kim, Y.: Convolutional Neural Networks for Sentence Classification. arXiv Prepr. http://arxiv.org/1408.5882. (2014)

  29. Zhou, C., Sun, C., Liu, Z., Lau, F.C.M.: A C-LSTM Neural Network for Text Classification. arXiv Prepr. http://arxiv.org/1511.08630. (2015)

  30. Ghosh, A., Veale, T.: Fracking sarcasm using neural network. In: Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis. pp. 161–169 (2016)

  31. Pitsilis, G.K., Ramampiaro, H., Langseth, H.: Effective hate-speech detection in Twitter data using recurrent neural networks. Appl. Intell. 48, 4730–4742 (2018). https://doi.org/10.1007/s10489-018-1242-y

    Article  Google Scholar 

  32. Mahata, D., Zhang, H., Uppal, K., Kumar, Y., Shah, R., Shahid, S., Mehnaz, L., Anand, S.: MIDAS at SemEval-2019 task 6 Identifying offensive posts and targeted offense from twitter. In: Proceedings of the 13th International Workshop on Semantic Evaluation. pp. 683–690 (2019)

  33. Sun, X., Zhang, C., Li, L.: Dynamic emotion modelling and anomaly detection in conversation based on emotional transition tensor. Inf. Fusion. 46, 11–22 (2019)

    Article  Google Scholar 

  34. Sadiq, S., Mehmood, A., Ullah, S., Ahmad, M., Choi, G.S., On, B.-W.: Aggression detection through deep neural model on Twitter. Futur. Gener. Comput. Syst. 114, 120–129 (2020)

    Article  Google Scholar 

  35. Nascimento, F.R.S., Cavalcanti, G.D.C., Da Costa-Abreu, M.: Unintended bias evaluation: an analysis of hate speech detection and gender bias mitigation on social media using ensemble learning. Expert Syst. Appl. 201, 117032 (2022)

    Article  Google Scholar 

  36. Lin, S.Y., Kung, Y.C., Leu, F.Y.: Predictive intelligence in harmful news identification by BERT-based ensemble learning model with text sentiment analysis. Inf. Process. Manag. 59, 102872 (2022)

    Article  Google Scholar 

  37. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)

    Article  Google Scholar 

  38. Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532–1543 (2014)

  39. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. 1st International Conference Learn. Represent. ICLR 2013 - Work. Track Proc. 1–12 (2013)

  40. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735

    Article  CAS  PubMed  Google Scholar 

  41. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv Prepr. http://arxiv.org/1406.1078. (2014)

  42. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986)

    Article  ADS  Google Scholar 

  43. Hochreiter, S., Bengio, Y., Frasconi, P., Schmidhuber, J.: Gradient flow in recurrent nets the difficulty of learning long-term dependencies. IEEE Press, Piscataway (2001)

    Google Scholar 

  44. Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45, 2673–2681 (1997)

    Article  ADS  Google Scholar 

  45. Lu, N., Wu, G., Zhang, Z., Zheng, Y., Ren, Y., Choo, K.K.R.: Cyberbullying detection in social media text based on character-level convolutional neural network with shortcuts. Concurr. Comput. Pract. Exp. (2020). https://doi.org/10.1002/cpe.5627

    Article  Google Scholar 

  46. Ratadiya, P., Mishra, D.: An Attention Ensemble Based Approach for Multilabel Profanity Detection. In: 2019 International Conference on Data Mining Workshops (ICDMW). pp. 544–550. IEEE (2019). https://doi.org/10.1109/ICDMW.2019.00083

  47. Saeed, H.H., Shahzad, K., Kamiran, F.: Overlapping Toxic Sentiment Classification Using Deep Neural Architectures. In: 2018 IEEE International Conference on Data Mining Workshops (ICDMW). pp. 1361–1366. IEEE (2018). https://doi.org/10.1109/ICDMW.2018.00193

Download references

Funding

No funding sources are applicable for this research.

Author information

Authors and Affiliations

  1. LSEA Laboratory, Mathematics and Computer Science Department, University of Médéa, Médéa, Algeria

    Ahmed Cherif Mazari & Nesrine Boudoukhani

  2. EEDIS Laboratory, Computer Science Department, University of Sidi Bel Abbès, Sidi Bel Abbès, Algeria

    Nesrine Boudoukhani

  3. LESIA Laboratory, Computer Science Department, University of Biskra, Biskra, Algeria

    Abdelhamid Djeffal

Authors
  1. Ahmed Cherif Mazari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nesrine Boudoukhani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abdelhamid Djeffal
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

ACM, NB: Conceptualization. NB: Testing. ACM, NB: Analysis and investigation, Writing original draft. ACM, NB & AD: Review and editing.

Corresponding author

Correspondence to Ahmed Cherif Mazari.

Ethics declarations

Conflict of interest

Not applicable.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mazari, A.C., Boudoukhani, N. & Djeffal, A. BERT-based ensemble learning for multi-aspect hate speech detection. Cluster Comput 27, 325–339 (2024). https://doi.org/10.1007/s10586-022-03956-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-022-03956-x

Keywords

Search

Navigation