Skip to main content

Improving Results on Russian Sentiment Datasets

Part of the Communications in Computer and Information Science book series (CCIS,volume 1292)

Abstract

In this study, we test standard neural network architectures (CNN, LSTM, BiLSTM) and recently appeared BERT architectures on previous Russian sentiment evaluation datasets. We compare two variants of Russian BERT and show that for all sentiment tasks in this study the conversational variant of Russian BERT performs better. The best results were achieved by BERT-NLI model, which treats sentiment classification tasks as a natural language inference task. On one of the datasets, this model practically achieves the human level .

Keywords

  • Targeted sentiment analysis
  • BERT
  • Natural language inference

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-59082-6_8
  • Chapter length: 13 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   79.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-59082-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   99.99
Price excludes VAT (USA)

Notes

  1. 1.

    http://docs.deeppavlov.ai/en/master/features/pretrained_vectors.html.

  2. 2.

    https://rusvectores.org/ru/models/.

  3. 3.

    http://www.cs.cmu.edu/~afm/projects/multilingual_embeddings.html.

  4. 4.

    https://scikit-learn.org/stable/.

  5. 5.

    http://docs.deeppavlov.ai/en/master/features/models/bert.html.

  6. 6.

    https://github.com/antongolubev5/Targeted-SA-for-Russian-Datasets.

  7. 7.

    https://github.com/LAIR-RCC/Russian-Sentiment-Analysis-Evaluation-Datasets.

References

  1. Amigó, E., et al.: Overview of RepLab 2013: evaluating online reputation monitoring systems. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 333–352. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40802-1_31

    CrossRef  Google Scholar 

  2. Arkhipenko, K., Kozlov, I., Trofimovich, J., Skorniakov, K., Gomzin, A., Turdakov, D.: Comparison of neural network architectures for sentiment analysis of Russian tweets. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference Dialogue, pp. 50–59 (2016)

    Google Scholar 

  3. Burtsev, M.: DeepPavlov: open-source library for dialogue systems. In: Proceedings of ACL 2018, System Demonstrations, pp. 122–127 (2018)

    Google Scholar 

  4. Chetviorkin, I., Loukachevitch, N.: Evaluating sentiment analysis systems in Russian. In: Proceedings of the 4th Biennial International Workshop on Balto-Slavic Natural Language Processing, pp. 12–17 (2013)

    Google Scholar 

  5. Cliche, M.: BB twtr at SemEval-2017 task 4: twitter sentiment analysis with CNNs and LSTMs. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 573–580 (2017)

    Google Scholar 

  6. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 (2018)

  7. Kuratov, Y., Arkhipov, M.: Adaptation of deep bidirectional multilingual transformers for Russian language (2019)

    Google Scholar 

  8. Kuznetsova, E., Loukachevitch, N., Chetviorkin, I.: Testing rules for a sentiment analysis system. In: Proceedings of International Conference Dialog, pp. 71–80 (2013)

    Google Scholar 

  9. Loukachevitch, N., Rubtsova, Y.: Entity-oriented sentiment analysis of tweets: results and problems. In: Král, P., Matoušek, V. (eds.) TSD 2015. LNCS (LNAI), vol. 9302, pp. 551–559. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24033-6_62

    CrossRef  Google Scholar 

  10. Loukachevitch, N., Rubtsova, Y.: SentiRuEval-2016: overcoming time gap and data sparsity in tweet sentiment analysis. In: Proceedings of International Conference Dialog-2016 (2016)

    Google Scholar 

  11. Loukachevitch, N., Rusnachenko, N.: Extracting sentiment attitudes from analytical texts. In: Proceedings of Computational Linguistics and Intellectual Technologies, Papers from the Annual Conference Dialog-2018, pp. 459–468 (2018)

    Google Scholar 

  12. Maas, A., Daly, R., Pham, P., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 142–150 (2011)

    Google Scholar 

  13. Nakov, P., Ritter, A., Rosenthal, S., Sebastiani, F., Stoyanov, V.: Semeval-2016 task 4: sentiment analysis in twitter. In: Proceedings of the 10th International Workshop on Semantic Evaluations, SemEval-2016, pp. 502–518 (2016)

    Google Scholar 

  14. Rogers, A., Romanov, A., Rumshisky, A., Volkova, S., Gronas, M., Gribov, A.: RuSentiment: an enriched sentiment analysis dataset for social media in Russian. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 755–763 (2018)

    Google Scholar 

  15. Rosenthal, S., Farra, N., Nakov, P.: Semeval-2017 task 4: sentiment analysis in twitter. In: Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017) (2017)

    Google Scholar 

  16. Rubtsova, Y.: Constructing a corpus for sentiment classification training. Softw. Syst. 109, 72–78 (2015)

    CrossRef  Google Scholar 

  17. Rubtsova, Y.: Reducing the deterioration of sentiment analysis results due to the time impact. Information 9(8), 184 (2018)

    CrossRef  Google Scholar 

  18. Rusnachenko, N., Loukachevitch, N., Tutubalina, E.: Distant supervision for sentiment attitude extraction. In: Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pp. 1022–1030 (2019)

    Google Scholar 

  19. Socher, R., et al.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642 (2013)

    Google Scholar 

  20. Sun, C., Huang, L., Qiu, X.: Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 380–385 (2019)

    Google Scholar 

  21. Svetlov, K., Platonov, K.: Sentiment analysis of posts and comments in the accounts of Russian politicians on the social network. In: 2019 25th Conference of Open Innovations Association (FRUCT), pp. 299–305. IEEE (2019)

    Google Scholar 

  22. Zhang, Y., Wallace, B.: A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification. arXiv preprint arXiv:1510.03820, pp. 573–580 (2015)

  23. Zvonarev, A., Bilyi, A.: A comparison of machine learning methods of sentiment analysis based on Russian language twitter data. In: The 11th Majorov International Conference on Software Engineering and Computer Systems (2019)

    Google Scholar 

Download references

Acknowledgments

The reported study was funded by RFBR according to the research project № 20-07-01059.

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Anton Golubev or Natalia Loukachevitch .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Golubev, A., Loukachevitch, N. (2020). Improving Results on Russian Sentiment Datasets. In: Filchenkov, A., Kauttonen, J., Pivovarova, L. (eds) Artificial Intelligence and Natural Language. AINL 2020. Communications in Computer and Information Science, vol 1292. Springer, Cham. https://doi.org/10.1007/978-3-030-59082-6_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-59082-6_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-59081-9

  • Online ISBN: 978-3-030-59082-6

  • eBook Packages: Computer ScienceComputer Science (R0)