Skip to main content
SpringerLink
Log in

Exploring Multi-Task Multi-Lingual Learning of Transformer Models for Hate Speech and Offensive Speech Identification in Social Media

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Hate Speech has become a major content moderation issue for online social media platforms. Given the volume and velocity of online content production, it is impossible to manually moderate hate speech related content on any platform. In this paper we utilize a multi-task and multi-lingual approach based on recently proposed Transformer Neural Networks to solve three sub-tasks for hate speech. These sub-tasks were part of the 2019 shared task on hate speech and offensive content (HASOC) identification in Indo-European languages. We expand on our submission to that competition by utilizing multi-task models which are trained using three approaches, (a) multi-task learning with separate task heads, (b) back-translation, and (c) multi-lingual training. Finally, we investigate the performance of various models and identify instances where the Transformer based models perform differently and better. We show that it is possible to to utilize different combined approaches to obtain models that can generalize easily on different languages and tasks, while trading off slight accuracy (in some cases) for a much reduced inference time compute cost. We open source an updated version of our HASOC 2019 code with the new improvements at https://github.com/socialmediaie/MTML_HateSpeech.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification.

  2. https://github.com/huggingface/transformers.

  3. https://colab.research.google.com/.

  4. https://cloud.google.com/translate/docs.

References

  1. Badjatiya P, Gupta S, Gupta M, Varma V. Deep learning for hate speech detection in tweets. In Proceedings of the 26th International Conference on World Wide Web Companion, WWW’17 Companion, page 759–760, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee; 2017.

  2. Bashar MA, Nayak R. Qutnocturnal@ hasoc’19: Cnn for hate speech and offensive content identification in hindi language; 2020. arXiv:2008.12448.

  3. Basile V, Bosco C, Fersini E, Nozza D, Patti V, Rangel Pardo FM, Rosso P, Sanguinetti M. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In: Proceedings of the 13th International Workshop on Semantic Evaluation, pp 54–63, Minneapolis, Minnesota, USA. Association for Computational Linguistics; 2019.

  4. Burnap P, Williams ML. Cyber hate speech on twitter: an application of machine classification and statistical modeling for policy and decision making. Policy Internet. 2015;7(2):223–42.

    Article  Google Scholar 

  5. Davidson T, Bhattacharya D, Weber I. Racial bias in hate speech and abusive language detection datasets. In: Proceedings of the third workshop on abusive language online, pp 25–35, Stroudsburg, PA, USA. Association for Computational Linguistics; 2019.

  6. Davidson T, Warmsley D, Macy MW, Weber I. Automated hate speech detection and the problem of offensive language. In: Proceedings of the International AAAI Conference on Web and Social Media 2017; 2017.

  7. Devlin J, Chang M-W, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. 2019.

  8. Duggan M, Smith A, Caiazza T. Online Harassment 2017. In: Pew Research Center: Technical report; 2017.

  9. Eisenstein J. What to do about bad language on the internet. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 359–369, Atlanta, Georgia. Association for Computational Linguistics; 2013.

  10. Florio K, Basile V, Polignano M, Basile P, Patti V. Time of your hate: the challenge of time in hate speech detection on social media. Appl Sci. 2020;10(12):4180.

    Article  Google Scholar 

  11. Gomez R, Gibert J, Gomez L, Karatzas D. Exploring hate speech detection in multimodal publications. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). 2020.

  12. Joulin A, Grave E, Bojanowski P, Mikolov T. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp 427–431, Valencia, Spain. Association for Computational Linguistics. 2017.

  13. Kiela D, Firooz H, Mohan A, Goswami V, Singh A, Ringshia P, Testuggine D. The hateful memes challenge: detecting hate speech in multimodal memes. 2020.

  14. Koehn P. Europarl: a parallel corpus for statistical machine translation. MT Summit. 2005.

  15. Kumar R, Ojha AK, Malmasi S, Zampieri M. Benchmarking aggression identification in social media. In: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pp 1–11, Santa Fe, New Mexico, USA. Association for Computational Linguistics. 2018.

  16. Kumar R, Ojha AK, Malmasi S, Zampieri M. Evaluating aggression identification in social media. In: Kumar R, Ojha AK, Lahiri B, Zampieri M, Malmasi S, Murdock V, Kadar D (eds) Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying (TRAC-2020), Paris, France. European Language Resources Association (ELRA). 2020.

  17. Le QV, Mikolov T. Distributed representations of sentences and documents. CoRR, abs/1405.4053. 2014.

  18. Liu P, Qiu X, Huang X. Deep multi-task learning with shared memory for text classification. In: Proceedings of the 2016 conference on empirical methods in natural language processing, pp 118–127, Stroudsburg, PA, USA. Association for Computational Linguistics. 2016.

  19. Mandl T, Modha S, Majumder P, Patel D, Dave M, Mandlia C, Patel A. Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in indo-european languages. In: Proceedings of the 11th Forum for Information Retrieval Evaluation, FIRE ’19, pp 14–17, New York, NY, USA. Association for Computing Machinery. 2019.

  20. Mishra S. Multi-dataset-multi-task Neural Sequence Tagging for Information Extraction from Tweets. In: Proceedings of the 30th ACM Conference on Hypertext and Social Media—HT ’19, pp 283–284, New York, New York, USA. ACM Press. 2019.

  21. Mishra S. Information extraction from digital social trace data with applications to social media and scholarly communication data. ACM SIGIR Forum. 2020;54:1.

    Article  Google Scholar 

  22. Mishra S. Information extraction from digital social trace data with applications to social media and scholarly communication data. In: PhD thesis, University of Illinois at Urbana-Champaign. 2020b.

  23. Mishra S. Non-neural Structured Prediction for Event Detection from News in Indian Languages. In: Mehta P, Mandl T, Majumder P, Mitra M (eds) Working Notes of FIRE 2020—Forum for Information Retrieval Evaluation, Hyderabad, India. CEUR Workshop Proceedings, CEUR-WS.org. 2020c.

  24. Mishra S, Agarwal S, Guo J, Phelps K, Picco J, Diesner J. Enthusiasm and support: alternative sentiment classification for social movements on social media. In: Proceedings of the 2014 ACM conference on Web science - WebSci ’14, pp 261–262, Bloomington, Indiana, USA. ACM Press. 2014.

  25. Mishra S, Diesner J. Semi-supervised Named Entity Recognition in noisy-text. In: Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT), pp 203–212, Osaka, Japan. The COLING 2016 Organizing Committee. 2016.

  26. Mishra S, Diesner J. Capturing Signals of Enthusiasm and Support Towards Social Issues from Twitter. In: Proceedings of the 5th International Workshop on Social Media World Sensors - SIdEWayS’19, pp 19–24, New York, New York, USA. ACM Press. 2019.

  27. Mishra S, Mishra S. 3Idiots at HASOC 2019: fine-tuning transformer neural networks for hate speech identification in Indo-European Languages. In: Proceedings of the 11th annual meeting of the Forum for Information Retrieval Evaluation, pp 208–213, Kolkata, India. 2019.

  28. Mishra S, Prasad S, Mishra S. Model and predictions for multi-task multi-lingual learning of transformer models for hate speech and offensive speech identification in social media. https://doi.org/10.13012/B2IDB-3565123_V1. 2020a.

  29. Mishra S, Prasad S, Mishra S. Multilingual Joint Fine-tuning of Transformer models for identifying Trolling,Aggression and Cyberbullying at TRAC 2020. In: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying (TRAC-2020). 2020b.

  30. Mondal M, Silva LA, Benevenuto F. A measurement study of hate speech in social media. In: Proceedings of the 28th ACM Conference on Hypertext and Social Media - HT ’17, pp 85–94, New York, New York, USA. ACM Press. 2017.

  31. Mozafari M, Farahbakhsh R, Crespi N. A bert-based transfer learning approach for hate speech detection in online social media. In: Cherifi H, Gaito S, Mendes JF, Moro E, Rocha LM (eds) Complex Networks and Their Applications VIII, pp 928–940, Cham. Springer International Publishing. 2020.

  32. Mujadia V, Mishra P, Sharma DM. Iiit-hyderabad at hasoc 2019: Hate speech detection. 2019.

  33. Perrin A. Social Media Usage:2005–2015. In: Pew Research Center: Technical report; 2015.

  34. Plank B. All-in-1 at IJCNLP-2017 task 4: short text classification with one model for all languages. In: Proceedings of the IJCNLP 2017, Shared Tasks, pp 143–148, Taipei, Taiwan. Asian Federation of Natural Language Processing. 2017.

  35. Ranasinghe T, Zampieri M. Multilingual offensive language identification with cross-lingual embeddings. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp 5838–5844. Association for Computational Linguistics. 2020.

  36. Razavi AH, Inkpen D, Uritsky S, Matwin S. Offensive language detection using multi-level classification. In: Proceedings of the 23rd Canadian Conference on Advances in Artificial Intelligence, AI’10, pp 16–27, Berlin, Heidelberg. Springer. 2010.

  37. Risch J, Krestel R. Aggression identification using deep learning and data augmentation. In: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (co-located with COLING); 2018. pp. 150–158.

  38. Risch J, Krestel R. Bagging bert models for robust aggression identification. In: Proceedings of the Workshop on Trolling, Aggression and Cyberbullying (TRAC@LREC). 2020.

  39. Ruiter D, Rahman MA, Klakow D. Lsv-uds at HASOC 2019: the problem of defining hate. In: Mehta P, Rosso P, Majumder P, Mitra M (eds) Working Notes of FIRE 2019—Forum for Information Retrieval Evaluation, Kolkata, India, December 12-15, 2019, volume 2517 of CEUR Workshop Proceedings, pp 263–270. CEUR-WS.org. 2019.

  40. Saha P, Mathew B, Goyal P, Mukherjee A. Hatemonitors: language agnostic abuse detection in social media. 2019.

  41. Salminen J, Almerekhi H, Milenković M, gyo Jung S, An J, Kwak H, Jansen B. Anatomy of online hate: developing a taxonomy and machine learning models for identifying and classifying hate in online news media. 2018.

  42. Schmidt A, Wiegand M. A survey on hate speech detection using natural language processing. In: Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pp 1–10, Stroudsburg, PA, USA. Association for Computational Linguistics. 2017.

  43. Sennrich R, Haddow B, Birch A. Improving neural machine translation models with monolingual data. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 86–96, Stroudsburg, PA, USA. Association for Computational Linguistics. 2016.

  44. Søgaard A, Goldberg Y. Deep multi-task learning with low level tasks supervised at lower layers. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp 231–235. Association for Computational Linguistics. 2016.

  45. Sticca F, Perren S. Is cyberbullying worse than traditional bullying? examining the differential roles of medium, publicity, and anonymity for the perceived severity of bullying. J Youth Adolesc. 2013;42(5):739–50.

    Article  Google Scholar 

  46. Struß J, Siegel M, Ruppenhofer J, Wiegand M, Klenner M. Overview of germeval task 2, 2019 shared task on the identification of offensive language. In: KONVENS. 2019.

  47. Van Hee C, Lefever E, Verhoeven B, Mennes J, Desmet B, De Pauw G, Daelemans W, Hoste V. Detection and fine-grained classification of cyberbullying events. In: Angelova G, Bontcheva K, Mitkov R (eds) Proceedings of Recent Advances in Natural Language Processing, Proceedings; 2015. pp 672–680.

  48. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. Adv Neural Inf Process Syst. 2017;2017:5998–6008.

    Google Scholar 

  49. Vidgen B, Harris A, Nguyen D, Tromble R, Hale S, Margetts H. Challenges and frontiers in abusive content detection. In: Proceedings of the Third Workshop on Abusive Language Online, pp 80–93, Florence, Italy. Association for Computational Linguistics. 2019.

  50. Wang B, Ding Y, Liu S, Zhou X. Ynu\_wb at hasoc 2019: ordered neurons lstm with attention for identifying hate speech and offensive language. 2019.

  51. Wang C. Interpreting neural network hate speech classifiers. In: Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pp 86–92, Brussels, Belgium. Association for Computational Linguistics. 2018.

  52. Waseem Z, Davidson T, Warmsley D, Weber I. Understanding abuse: A typology of abusive language detection subtasks. In: Proceedings of the First Workshop on Abusive Language Online, pp 78–84, Vancouver, BC, Canada. Association for Computational Linguistics. 2017.

  53. Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, Brew J. Huggingface’s transformers: state-of-the-art natural language processing. 2019.

  54. Yang F, Peng X, Ghosh G, Shilon R, Ma H, Moore E, Predovic G. Exploring deep multimodal fusion of text and photo for hate speech classification. In: Proceedings of the Third Workshop on Abusive Language Online, pp 11–18, Florence, Italy. Association for Computational Linguistics. 2019.

  55. Zampieri M, Malmasi S, Nakov P, Rosenthal S, Farra N, Kumar R. SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In: Proceedings of the 13th International Workshop on Semantic Evaluation, pp 75–86, Minneapolis, Minnesota, USA. Association for Computational Linguistics. 2019.

Download references

Author information

Authors and Affiliations

  1. Indian Institute of Technology Kanpur, Kanpur, India

    Sudhanshu Mishra

  2. University of Illinois at Urbana-Champaign, Champaign, USA

    Shivangi Prasad & Shubhanshu Mishra

Authors
  1. Sudhanshu Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shivangi Prasad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shubhanshu Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shubhanshu Mishra.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection “Social Media Analytics and its Evaluation” guest edited by Thomas Mandl, Sandip Modha and Prasenjit Majumder.

Appendix

Appendix

Label Distribution

See Fig. 4, 5 and 6.

Fig. 4
figure 4

English Data class wise distribution

Full size image
Fig. 5
figure 5

German Data class wise distribution

Full size image
Fig. 6
figure 6

Hindi Data class wise distribution

Full size image

Back Translation Top Changed Words

Here we list the top 5 words per label for each task obtained after removing the top 50 words which were either introduced or removed via back translation. We do not list the top words for Hindi because of the encoding issue in Latex.

figure a
figure b

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mishra, S., Prasad, S. & Mishra, S. Exploring Multi-Task Multi-Lingual Learning of Transformer Models for Hate Speech and Offensive Speech Identification in Social Media. SN COMPUT. SCI. 2, 72 (2021). https://doi.org/10.1007/s42979-021-00455-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-021-00455-5

Keywords

Search

Navigation