Skip to main content
SpringerLink
Log in

Does BERT Look at Sentiment Lexicon?

  • Conference paper
  • First Online:
Recent Trends in Analysis of Images, Social Networks and Texts (AIST 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1573))

  • 280 Accesses

Abstract

The main approaches to sentiment analysis are rule-based methods and machine learning, in particular, deep neural network models with the Transformer architecture, including BERT. The performance of neural network models in the tasks of sentiment analysis is superior to the performance of rule-based methods. The reasons for this situation remain unclear due to the poor interpretability of deep neural network models. One of the main keys to understanding the fundamental differences between the two approaches is the analysis of how sentiment lexicon is taken into account in neural network models. To this end, we study the attention weights matrices of the Russian-language RuBERT model. We fine-tune RuBERT on sentiment text corpora and compare the distributions of attention weights for sentiment and neutral lexicons. It turns out that, on average, 3/4 of the heads of various model variants statistically pay more attention to the sentiment lexicon compared to the neutral one.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://paperswithcode.com/task/sentiment-analysis.

  2. 2.

    RuBERT model for Russian uses BPE (Byte Pair Encoding) tokenization [26].

  3. 3.

    The pymorphy2 [12] library was used for lemmatization.

References

  1. Barnes, J., Ovrelid, L., Velldal, E.: Sentiment analysis is not solved! Assessing and probing sentiment classification. In: Proceedings of the ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 12–23 (2019)

    Google Scholar 

  2. Belinkov, Y., Gehrmann, S., Pavlick, E.: Tutorial proposal: interpretability and analysis in neural NLP. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1–5 (2020)

    Google Scholar 

  3. Birjali, M., Kasri, M., Beni-Hssane, A.: A comprehensive survey on sentiment analysis: approaches, challenges and trends. Knowl.-Based Syst. 226, 1–26 (2021)

    Article  Google Scholar 

  4. Blinov, P.D., Klekovkina, M.V., Kotelnikov, E.V., Pestov, O.A.: Research of lexical approach and machine learning methods for sentiment analysis. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialogue”, vol. 12, no. 19, pp. 51–61 (2013)

    Google Scholar 

  5. Cao, N.D., Schlichtkrull, M.S., Aziz, W., Titov, I.: How do decisions emerge across layers in neural models? Interpretation with differentiable masking. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 3243–3255 (2020)

    Google Scholar 

  6. Chen, Y., Skiena, S.: Building sentiment lexicons for all major languages. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pp. 383–389 (2014)

    Google Scholar 

  7. Chetviorkin, I.I., Loukachevitch, N.V.: Sentiment analysis track at ROMIP 2012. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog”, vol. 2, pp. 40–50 (2013)

    Google Scholar 

  8. Clark, K., Khandelwal, U., Levy, O., Manning, C.D.: What does BERT look at? An analysis of BERT’s attention. In: Proceedings of the ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276–286 (2019)

    Google Scholar 

  9. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of 7th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), pp. 4171–4186 (2019)

    Google Scholar 

  10. Kim, S., Yi, J., Kim, E., Yoon, S.: Interpretation of NLP models through input marginalization. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 3154–3167 (2020)

    Google Scholar 

  11. Koltsova, O.Y., Alexeeva, S.V., Kolcov, S.N.: An opinion word lexicon and a training dataset for Russian sentiment analysis of social media. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog”, pp. 277–287 (2016)

    Google Scholar 

  12. Korobov, M.: Morphological analyzer and generator for Russian and Ukrainian languages. In: Khachay, M.Y., Konstantinova, N., Panchenko, A., Ignatov, D.I., Labunets, V.G. (eds.) AIST 2015. CCIS, vol. 542, pp. 320–332. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-26123-2_31

    Chapter  Google Scholar 

  13. Kotelnikov, E., Bushmeleva, N., Razova, E., Peskisheva, T., Pletneva, M.: Manually created sentiment lexicons: research and development. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog”, vol. 15(22), pp. 300–314 (2016)

    Google Scholar 

  14. Kotelnikov, E., Peskisheva, T., Kotelnikova, A., Razova, E.: A comparative study of publicly available Russian sentiment lexicons. In: Ustalov, D., Filchenkov, A., Pivovarova, L., Žižka, J. (eds.) AINL 2018. CCIS, vol. 930, pp. 139–151. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01204-5_14

    Chapter  Google Scholar 

  15. Kotelnikova, A., Kotelnikov, E.: SentiRusColl: Russian collocation lexicon for sentiment analysis. In: Ustalov, D., Filchenkov, A., Pivovarova, L. (eds.) AINL 2019. CCIS, vol. 1119, pp. 18–32. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-34518-1_2

    Chapter  Google Scholar 

  16. Kotelnikova, A.V., Pashchenko, D.E., Kotelnikov, E.V., Bochenina, K.O.: Lexicon-based methods vs. BERT for text sentiment analysis. In: Proceedings of the 10th International Conference on Analysis of Images, Social Networks and Texts (AIST) (2021)

    Google Scholar 

  17. Kulagin, D.: Russian word sentiment polarity dictionary: a publicly available dataset. In: Artificial Intelligence and Natural Language. AINL 2019 (2019)

    Google Scholar 

  18. Kuratov, Y., Arkhipov, M.: Adaptation of deep bidirectional multilingual transformers for Russian language. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog”, pp. 333–340 (2019)

    Google Scholar 

  19. Lalor, J.P., Wu, H., Munkhdalai, T., Yu, H.: Understanding deep learning performance through an examination of test set difficulty: a psychometric case study. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 4711–4716 (2018)

    Google Scholar 

  20. Loukachevitch, N., Levchik, A.: Creating a general Russian sentiment lexicon. In: Proceedings of Language Resources and Evaluation Conference (LREC), pp. 1171–1176 (2016)

    Google Scholar 

  21. Loukashevitch, N.V., Blinov, P.D., Kotelnikov, E.V., Rubtsova, Y.V., Ivanov, V.V., Tutubalina, E.V.: SentiRuEval: testing object-oriented sentiment analysis systems in Russian. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog”, vol. 2, pp. 2–13 (2015)

    Google Scholar 

  22. MacKay, D.: Information Theory, Inference, and Learning Algorithms. Cambridge University Press, Cambridge (2003)

    MATH  Google Scholar 

  23. Mohammad, S.M., Turney, P.D.: Crowdsourcing a word-emotion association lexicon. Comput. Intell. 29(3), 436–465 (2013)

    Article  MathSciNet  Google Scholar 

  24. Ong, D., Wu, Z., Tan, Z.-X., Reddan, M., Kahhale, I., et al.: Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset. IEEE Trans. Affect. Comput. 12, 570–594 (2021)

    Google Scholar 

  25. Rogers, A., Romanov, A., Rumshisky, A., Volkova, S., Gronas, M., Gribov, A.: RuSentiment: an enriched sentiment analysis dataset for social media in Russian. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 755–763 (2018)

    Google Scholar 

  26. Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1715–1725 (2016)

    Google Scholar 

  27. Socher, R., et al.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642 (2013)

    Google Scholar 

  28. Taboada, M., Brooke, J., Tofiloski, M., Voll, K., Stede, M.: Lexicon-based methods for sentiment analysis. Comput. Linguist. 37(2), 267–307 (2011)

    Article  Google Scholar 

  29. Taboada, M.: Sentiment analysis: an overview from linguistics. Ann. Rev. Linguist. 2, 325–347 (2016)

    Article  Google Scholar 

  30. Tutubalina, E.V.: Extraction and summarization methods for critical user reviews of a product. Ph.D. thesis, Kazan Federal University, Kazan, Russia (2016)

    Google Scholar 

  31. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., et al.: Attention is all you need. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS), vol. 30, pp. 6000–6010 (2017)

    Google Scholar 

  32. Voita, E., Talbot, D., Moiseev, F., Sennrich, R., Titov, I.: Analyzing multi-head self-attention: specialized heads do the heavy lifting, the rest can be pruned. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797–5808 (2019)

    Google Scholar 

  33. Warriner, A.B., Kuperman, V., Brysbaert, M.: Norms of valence, arousal, and dominance for 13,915 English lemmas. Behav. Res. Methods 45(4), 1191–1207 (2013). https://doi.org/10.3758/s13428-012-0314-x

    Article  Google Scholar 

  34. Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bull. 6(1), 80–83 (1945)

    Article  Google Scholar 

  35. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv:1609.08144 (2016)

  36. Wu, Z., Nguyen, T.-S., Ong, D.: Structured self-attention weights encode semantics in sentiment analysis. In: Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 255–264 (2020)

    Google Scholar 

  37. Wu, Z., Ong, D.C.: On explaining your explanations of BERT: an empirical study with sequence classification. arXiv:2101.00196 (2021)

Download references

Author information

Authors and Affiliations

  1. Vyatka State University, Kirov, Russia

    Elena Razova, Sergey Vychegzhanin & Evgeny Kotelnikov

  2. ITMO University, Saint-Petersburg, Russia

    Evgeny Kotelnikov

Authors
  1. Elena Razova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sergey Vychegzhanin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Evgeny Kotelnikov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Evgeny Kotelnikov .

Editor information

Editors and Affiliations

  1. Skolkovo Institute of Science and Technology, Moscow, Russia

    Evgeny Burnaev

  2. National Research University Higher School of Economics, Moscow, Russia

    Dmitry I. Ignatov

  3. Skolkovo Institute of Science and Technology, Moscow, Russia

    Sergei Ivanov

  4. Krasovskii Institute of Mathematics and Mechanics of Russian Academy of Sciences, Yekaterinburg, Russia

    Michael Khachay

  5. National Research University Higher School of Economics, St. Petersburg, Russia

    Olessia Koltsova

  6. University of Oslo, Oslo, Norway

    Andrei Kutuzov

  7. National Research University Higher School of Economics, Moscow, Russia

    Sergei O. Kuznetsov

  8. Research Computing Center, Lomonosov Moscow State University, Moscow, Russia

    Natalia Loukachevitch

  9. LORIA, Vandœuvre lès Nancy, France

    Amedeo Napoli

  10. Skolkovo Institute of Science and Technology, Moscow, Russia

    Alexander Panchenko

  11. Industrial and Systems Engineering, University of Florida, Gainesville, USA

    Panos M. Pardalos

  12. Aalto University, Espoo, Finland

    Jari Saramäki

  13. National Research University Higher School of Economics, Nizhny Novgorod, Russia

    Andrey V. Savchenko

  14. Yandex LLC, Moscow, Russia

    Evgenii Tsymbalov

  15. Kazan Federal University, Kazan, Russia

    Elena Tutubalina

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Razova, E., Vychegzhanin, S., Kotelnikov, E. (2022). Does BERT Look at Sentiment Lexicon?. In: Burnaev, E., et al. Recent Trends in Analysis of Images, Social Networks and Texts. AIST 2021. Communications in Computer and Information Science, vol 1573. Springer, Cham. https://doi.org/10.1007/978-3-031-15168-2_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-15168-2_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-15167-5

  • Online ISBN: 978-3-031-15168-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation