Abstract
Modern neural networks are quite demanding regarding the size and coverage of adequate training evidences, as far as complex inferences are involved. This is the case of offensive language detection that focuses on a phenomenon, the recognition of offensive uses of language, that is elusive and multifaceted. In this scenarios gathering training data can be prohibitively expensive and the dynamics and multidimensional nature of the abusive language phenomena are also demanding of timely and evolving evidence for training in a continuous fashion.
The MT-GAN-BERT approach proposed here aims to reduce the requirements of neural approaches both in terms of the amount of annotated data and the computational cost required at classification time. It focuses corresponds to a general BERT-based architecture for multi faceted text classification tasks. On the one side, MT-GAN-BERT enables semi-supervised learning for Transformers based on the Generative Adversarial Learning paradigm. It also implements a Multi-task Learning approach able to train over and solve multiple tasks, simultaneously. A single BERT-based model is used to encode the input examples, while multiple linear layers are used to implement the classification steps, with a significant reduction of the computational costs. In the experimental evaluations we studied six classification tasks related to the detection of abusive uses of language in Italian. Outcomes suggest that MT-GAN-BERT is sustainable and generally improves the raw adoption of multiple BERT-based models, with much lighter requirements in terms of annotated data and computational costs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
MT-GAN-BERT is publicly available at: https://github.com/crux82/mt-ganbert.
- 2.
The remaining \(h_{w_k}\) embeddings can be used for other tasks, such as sequence labeling tasks, not considered in this work.
- 3.
The original code repositories are available at https://github.com/namisan/mt-dnn and https://github.com/crux82/ganbert.
- 4.
- 5.
The number of parameters of \(\mathcal {D}\) are negligible if compared to the encoder.
References
Barbieri, F., Basile, V., Croce, D., Nissim, M., Novielli, N., Patti, V.: Overview of the Evalita 2016 sentiment polarity classification task. In: Proceedings of EVALITA 2016, Napoli, CEUR Workshop Proceedings, vol. 1749 (2016)
Chapelle, O., Schlkopf, B., Zien, A.: Semi-Supervised Learning. The MIT Press, 1st edn. (2010)
Croce, D., Castellucci, G., Basili, R.: GAN-BERT: generative adversarial learning for robust text classification with a bunch of labeled examples. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, 5–10 July 2020, pp. 2114–2119. Association for Computational Linguistics (2020)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1 (Long and Short Papers), pp. 4171–4186. Minneapolis, Minnesota (2019)
Fersini, E., Nozza, D., Rosso, P.: Overview of the Evalita 2018 task on automatic misogyny identification (AMI). In: Proceedings of EVALITA 2018, Turin, 12–13 December 2018. CEUR Workshop Proceedings, vol. 2263. CEUR-WS.org (2018). http://ceur-ws.org/Vol-2263/paper009.pdf
Founta, A., Chatzakou, D., Kourtellis, N., Blackburn, J., Vakali, A., Leontiadis, I.: A unified deep learning architecture for abuse detection. arXiv preprint arXiv:1802.00385 (2018)
Goldberg, Y.: A primer on neural network models for natural language processing. J. Artif. Int. Res. 57(1), 345–420 (2016)
Goodfellow, I.J.: NIPS 2016 tutorial: generative adversarial networks. arXiv preprint arXiv:1701.00160 (2017)
Hendrycks, D., Gimpel, K.: Bridging nonlinearities and stochastic regularizers with Gaussian error linear units. arXiv preprint arXiv:1606.08415 (2016)
Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, 25–29 October 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1746–1751 (2014)
Kim, Y., Jernite, Y., Sontag, D., Rush, A.M.: Character-aware neural language models. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 12–17 February 2016, Phoenix, Arizona, pp. 2741–2749 (2016)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Liu, X., He, P., Chen, W., Gao, J.: Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504 (2019)
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
Miliani, M., Giorgi, G., Rama, I., Anselmi, G., Lebani, G.E.: Dankmemes @ Evalita2020: The memeing of life: memes, multimodality and politics. In: Basile, V., Croce, D., Di Maro, M., Passaro, L.C. (eds.) Proceedings of Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020). CEUR.org, Online (2020)
Mishra, P., Yannakoudakis, H., Shutova, E.: Tackling online abuse: a survey of automated abuse detection methods. arXiv preprint arXiv:1908.06024 (2019)
Poletto, F., Stranisci, M., Sanguinetti, M., Patti, V., Bosco, C.: Hate speech annotation: analysis of an Italian Twitter corpus. In: CLiC-it (2017)
Rajamanickam, S., Mishra, P., Yannakoudakis, H., Shutova, E.: Joint modelling of emotion and abusive language detection. arXiv preprint arXiv:2005.14028 (2020)
Salimans, T., et al.: Improved techniques for training gans. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 2234–2242. Curran Associates, Inc. (2016)
Sanguinetti, M., Poletto, F., Bosco, C., Patti, V., Stranisci, M.: An Italian Twitter corpus of hate speech against immigrants. In: LREC (2018)
Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)
Sharir, O., Peleg, B., Shoham, Y.: The cost of training NLP models: a concise overview. arXiv preprint arXiv:2004.08900 (2020)
Shen, S., et al.: Q-BERT: hessian based ultra low precision quantization of BERT. arXiv preprint arXiv:1909.05840 (2019)
Vidgen, B., Derczynski, L.: Directions in abusive language training data, a systematic review: garbage in, garbage out. PLOS ONE 15(12), 1–32 (2021). https://doi.org/10.1371/journal.pone.0243300
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., Titov, I.: Analyzing multi-head self-attention: specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418 (2019)
Weston, J., Ratle, F., Collobert, R.: Deep learning via semi-supervised embedding. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1168–1175. ICML 2008, ACM, NY (2008)
Wolf, T., et al.: Huggingface’s transformers: state-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019)
Yang, Z., Cohen, W.W., Salakhutdinov, R.: Revisiting semi-supervised learning with graph embeddings. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning, vol. 48, pp. 40–48. ICML 2016, JMLR.org (2016). http://dl.acm.org/citation.cfm?id=3045390.3045396
Zhang, Y., Yang, Q.: A survey on multi-task learning. arXiv preprint arXiv:1707.08114 (2017)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Breazzano, C., Croce, D., Basili, R. (2022). Multi-task and Generative Adversarial Learning for Robust and Sustainable Text Classification. In: Bandini, S., Gasparini, F., Mascardi, V., Palmonari, M., Vizzari, G. (eds) AIxIA 2021 – Advances in Artificial Intelligence. AIxIA 2021. Lecture Notes in Computer Science(), vol 13196. Springer, Cham. https://doi.org/10.1007/978-3-031-08421-8_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-08421-8_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-08420-1
Online ISBN: 978-3-031-08421-8
eBook Packages: Computer ScienceComputer Science (R0)