Abstract
In this paper, we study the problem of imbalanced text classification based on the pre-trained language models. We propose the Adaptable Focal Loss (AFL) method to solve this problem. Firstly, we use the word embeddings from the pre-trained models to construct the sentence level prior by the sum of the word embeddings in the sentence. Then, we extend the Focal Loss, which is widely used in the field of object detection, by replacing the task-special parameters with the scaled-softmax of the distance between the fine-tuned embeddings and the prior embeddings from the pre-trained models. By removing the task-special parameters in Focal Loss, not only can the parameters of arbitrary imbalanced proportion distribution be adjusted automatically according to the task, but also the sentences that are difficult to classify can be given a higher weight. Experimental results show that our methods can easily combine with the common classifier models and significantly improve their performances.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Xu, W.D., Sun, H.Z., Deng, C., et al.: Variational autoencoder for semi-supervised text classification. In: 31st AAAI Conference on Artificial Intelligence, pp. 3358–3364. AAAI, CA (2017)
Zhang, T., Huang, M., Zhao, L.: Learning structured representation for text classification via reinforcement learning. In: 32nd AAAI Conference on Artificial Intelligence, pp. 6053–6060. AAAI, CA (2018)
You, R.H., Zhang, Z.H., Wang, Z.Y., et al.: AttentionXML: label tree-based attention-aware deep model for high-performance extreme multi-label text classification. Adv. Neural. Inf. Process. Syst. 32, 5820–5830 (2019)
Liu, Y.X, Jin, X.M., Lan, Y., et al.: Adaptive region embedding for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7314–7321. AAAI, CA (2019)
Chen, X., Qiu, Z.Z.: Research on Chinese text classification based on WAE and SVM. In: 2021 3rd International Conference on Natural Language Processing (ICNLP), pp. 14–19. IEEE, NJ (2021)
Chen, Q.X., Yao, L.X., Yang, J.: Short text classification based on LDA topic model. In: 2016 International Conference on Audio, Language and Image Processing (ICALIP), pp. 749–753. IEEE, NJ (2016)
Radford, A., Wu, J., Child, R., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
Mikolov, T., Sutskever, I., Chen, K., et al.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119. Curran Associates, NY (2013)
Li, Z.C., Wang, R., Chen, K.H., et al.: Data-dependent Gaussian prior objective for language generation. In: International Conference on Learning Representations (ICLR), pp. 1–18. ICLR (2020)
Fernández, A., del RÃo, S., Chawla, N.V., et al.: An insight into imbalanced big data classification: outcomes and challenges. Complex Intel. Syst. 3(2), 105–120 (2017)
Li, Y.X., Chai, Y., Hu, Y.Q., et al.: Review of imbalanced data classification methods. Control Decis. 34(4), 673–688 (2019)
Gu, X., Angelov, P.P., Soares, E.A.: A self-adaptive synthetic over-sampling technique for imbalanced classification. Int. J. Intell. Syst. 35(6), 923–943 (2020)
Ng, W.W.Y., Hu, J.J., Yeung, D.S., et al.: Diversified sensitivity-based undersampling for imbalance classification problems. IEEE Trans. Cybern. 45(11), 2402–2412 (2014)
Zhou, Z.H., Liu, X.Y.: Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Trans. Knowl. Data Eng. 18(1), 63–77 (2005)
Khan, S.H., Hayat, M., Bennamoun, M., et al.: Cost-sensitive learning of deep feature representations from imbalanced data. IEEE Trans. Neural Netw. Learn. Syst. 29(8), 3573–3587 (2017)
Castro, C.L., Braga, A.P.: Novel cost-sensitive approach to improve the multilayer perceptron performance on imbalanced data. IEEE Trans. Neural Netw. Learn. Syst. 24(6), 888–899 (2013)
Shrivastava, A., Gupta, A., Girshick, R.: Training region-based object detectors with online hard example mining. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 761–769. IEEE, NJ (2016)
Lin, T.Y., Goyal, P., Girshick, R., et al.: Focal loss for dense object detection. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 2980–2988. IEEE, NJ (2017)
Acknowledgments
This work is supported by the Key-Area Research and Development Program of Guangdong Province (No. 2020B010164003), National Key Research and Development Plan’s Key Special Program on High Performance Computing of China (No. 2017YFB0203201), The National Natural Science Foundation of China (Grant No. 6177010044); Basic and Applied Basic Research Fund of Guangdong Province (Grant No. 2019A1515010716); Key Projects of Basic and Applied Basic Research in General Universities of Guangdong Province (Grant No. 2018KZDXM073); Special Project in key Areas of Artificial Intelligence in Guangdong Universities (No. 2019KZDZX1017).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Cao, L., Liu, X., Shen, H. (2022). Adaptable Focal Loss for Imbalanced Text Classification. In: Shen, H., et al. Parallel and Distributed Computing, Applications and Technologies. PDCAT 2021. Lecture Notes in Computer Science(), vol 13148. Springer, Cham. https://doi.org/10.1007/978-3-030-96772-7_43
Download citation
DOI: https://doi.org/10.1007/978-3-030-96772-7_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-96771-0
Online ISBN: 978-3-030-96772-7
eBook Packages: Computer ScienceComputer Science (R0)