Skip to main content
SpringerLink
Log in

Improving Classification Robustness for Noisy Texts with Robust Word Vectors

  • Published:
Journal of Mathematical Sciences Aims and scope Submit manuscript

Text classification is a fundamental task in natural language processing, and with a huge and rapidly growing body of research devoted to it. However, there has been little work on investigating noise robustness for the developed approaches. In this work, we are bridging this gap, introducing results on noise robustness testing of modern text classification architectures for English and Russian languages. We benchmark the CharCNN and SentenceCNN models and introduce a new model, called RoVe (Robust Vectors), that we show to be the most robust to noise.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of Tricks for Efficient Text Classification,” arXiv:1607.01759 (2016).

  2. V. Malykh, “Robust Word Vectors: Embeddings for Noisy Texts,” arXiv:1607.01759 (2018).

  3. Y. Kim, Y. Jernite, D. Sontag, and A. M. Rush, “Character-Aware Neural Language Models,” in: Proceedings of the AAAI Conference on Artificial Intelligence (2016), pp. 2741– 2749.

  4. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and Ch. Potts, “Learning Word Vectors for Sentiment Analysis,” in: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (2011), pp. 142–150.

  5. S. Cucerzan and E. Brill, “Spelling correction as an iterative process that exploits the collective knowledge of web users,” in: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (2004).

  6. A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of tricks for efficient text classification,” arXiv:1607.01759 (2016).

  7. J. Howard and S. Ruder, “Fine-tuned Language Models for Text Classification,” arXiv:1801.06146 (2018).

  8. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and I. Polosukhin, “Attention is all you need,” in: Advances in Neural Information Processing Systems (2017), pp. 6000–6010.

  9. X. Zhang, J. J. Zhao, and Y. LeCun, “Character-level Convolutional Networks for Text Classification,” arXiv:1509.01626 (2017).

  10. Y. Kim, “Convolutional Neural Networks for Sentence Classification,” arXiv:1408.5882 (2014).

  11. K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio, “On the Properties of Neural Machine Translation: Encoder-Decoder Approaches,” arXiv:1409.1259 (2014).

  12. D. Bahdanau, K. Cho, and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” arXiv:1409.0473 (2014).

  13. Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE transactions on neural networks, 5, No. 2, 157–166 (1994).

    Article  Google Scholar 

  14. E. Tutubalina and S. Nikolenko, “Inferring sentiment-based priors in topic models,” in: Mexican International Conference on Artificial Intelligence (2015), pp. 92–104.

  15. J. Niu, Y. Yang, S. Zhang, Z. Sun, and W. Zhang, “Multi-task Character-Level Attentional Networks for Medical Concept Normalization,” Neural Processing Letters (2018), pp. 1–18.

  16. Y. Li, T. Cohn, and Y. Baldwin, “Learning robust representations of text,” in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016), pp. 1979–1985.

  17. N. V. Loukachevitch, et al., “SentiRuEval: Testing Object-oriented Sentiment Analysis Systems in Russian,” in: Proceedings of International Conference “Dialog” (2015).

  18. V. Malykh, “Generalizable Architecture for Robust Word Vectors Tested by Noisy Paraphrases,” in: Supplementary Proceedings of the Sixth International Conference on Analysis of Images, Social Networks and Texts (AIST 2017), Moscow, Russia (2017).

Download references

Author information

Authors and Affiliations

  1. St. Petersburg Department of Steklov Mathematical Institute RAS, St. Petersburg, Russia

    V. Malykh

  2. Moscow Institute of Physics and Technology, Dolgoprudny, Moscow Region, Russia

    V. Malykh & V. Lyalin

  3. Institute for Systems Analysis, Federal Research Center “Computer Science and Control” of Russian Academy of Sciences, Moscow, Russia

    V. Malykh

Authors
  1. V. Malykh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. V. Lyalin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to V. Malykh.

Additional information

Published in Zapiski Nauchnykh Seminarov POMI, Vol. 499, 2021, pp. 236–247.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Malykh, V., Lyalin, V. Improving Classification Robustness for Noisy Texts with Robust Word Vectors. J Math Sci 273, 605–613 (2023). https://doi.org/10.1007/s10958-023-06522-x

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10958-023-06522-x

Search

Navigation