Skip to main content
Log in

Open evaluation of sentiment-analysis systems based on the material of the Russian language

  • Published:
Scientific and Technical Information Processing Aims and scope

Abstract

This paper describes the experience from carrying out open evaluation of the methods for the sentiment analysis in Russian based on the ROMIP seminar in 2011–2012. Several training collections, which are now in free access, have been created during the conduction of the track. This paper reviews the current state of the art in processing subjective texts in Russian and describes the major tasks and characteristics of collections, as well as quality-assessment measures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abdul-Mageed, M., Diab, M., Korayem, M., Subjectivity and sentiment analysis of modern standard arabic, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, 2011, pp. 587–591.

    Google Scholar 

  2. Amigo, E., Corujo, A., Gonzalo, J., Meij, E., Rijke, Md., Overview of replab 2012: Evaluating online reputation management systems, Proceedings of the CLEF 2012 Labs and Workshop Notebook Papers, 2012, pp. 1–24.

    Google Scholar 

  3. Awadallah, R., Ramanath, M., Weikum, G., PolariCQ: Polarity classification of political quotations, Proceedings of CIKM-2012, 2012., pp. 1945–1949.

    Google Scholar 

  4. Balasubramanyan, R., Cohen, W., Pierce, D., Redlawsk D., Modeling polarizing topics: When do different political communities respond differently to the same news? Proceedings of ICWSM, 2012.

    Google Scholar 

  5. Blinov, P., Klekovkina, M., Kotelnikov, E., Pestov, O., Research of lexical approach and machine learning methods for sentiment analysis, Proceedings of Dialog, vol. 2, 2013, pp. 51–61.

    Google Scholar 

  6. Chetviorkin, I., Braslavskiy, P., Loukachevich, N., Sentiment analysis track at ROMIP 2011, Proceedings of International Conference Dialog-2012, vol. 2, 2012, pp. 1–14.

    Google Scholar 

  7. Chetviorkin, I., Loukachevitch, N., Extraction of Russian sentiment lexicon for product meta-domain, Proceedings of COLING 2012, 2012, pp. 593–610.

    Google Scholar 

  8. Chetvirokin, I., Loukachevitch, N., Sentiment analysis track at ROMIP 2012, Proceedings of International Conference Dialog-2013, vol. 2, 2013, pp. 40–50.

    Google Scholar 

  9. Choi, Y., Cardie, C., Adapting a polarity lexicon using integer linear programming for domain-specific sentiment classification, Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2009, pp. 590–598.

    Google Scholar 

  10. Ermakov, A.E., Extraction of knowledge from a text and its processing: the state and prospects, Inform. Tekhnol., no. 7, 2009, pp. 50–55.

    Google Scholar 

  11. Kotel’nikov, E.V., Klekovkina, M.V., Automated Text Sentiment Analysis Based on Machine Learning Methods. Computer Linguistics and Intelligent Technologies: According to the Materials of the Yearly International Conference Dialog, 2012, vol. 2, pp. 27–36.

    Google Scholar 

  12. Kuznetsova, E.S., Loukachevitch, N.V., Chetviorkin, I.I., Testing rules for sentiment analysis system. Computational Linguistics and Intellectual Technologies, Proc. of International Conference Dialog-2013, vol. 2, 2013, pp.71–80.

    Google Scholar 

  13. Macdonald, C., Ounis, I., Soboroff, I., Overview of the TREC 2007 blog track, Proceedings of TREC-2007, Gaithersburg, USA, 2008.

    Google Scholar 

  14. Macdonald, C., Ounis, I., Soboroff, I., Overview of the TREC 2009 blog track, Proceedings of TREC-2009, Gaithersburg, USA, 2010.

    Google Scholar 

  15. Manning, C.D., Raghavan, P., Schiitze, H., Introduction to information retrieval, Cambridge: Cambridge University Press, 2008.

    Book  MATH  Google Scholar 

  16. Mihalcea, R., Banea, C., Wiebe, J., Learning multilingual subjective language via cross-lingual projections, Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, Prague, Czech Republic, 2007, pp. 976–983.

    Google Scholar 

  17. Morante, R., Blanco, E., SEM 2012 shared task: Resolving the scope and focus of negation, Proceedings of the First Joint Conference on Lexical and Computational Semantics, Montreal, 2012, pp. 265–274

    Google Scholar 

  18. Pak, A., Paroubek, P., Language independent approach to sentiment analysis (LIMSI Participation in ROMIP’ll) Computational Linguistics and Intellectual Technologies, Proc. of International Conference Dialog-2012, vol. 2, 2012, pp. 37–50.

    Google Scholar 

  19. Pan, S.J., Ni, X., Sun, J.-T., Yang, Q., Chen, Z., Cross-domain sentiment classification via spectral feature alignment, Proceedings of the World Wide Web Conference, 2010, pp. 751–760.

    Google Scholar 

  20. Pang, B., Lee, L., Opinion mining and sentiment analysis. Foundations and Trends-in Information Retrieval. Now Publishers, 2008.

    Google Scholar 

  21. Panicheva, P. The Rule-Based ATEX Sentiment Analysis System Based during Processing Texts of Different Themes. Computer Linguistics and Intelligent Technologies: According to the Materials of the Yearly International Conference Dialog, 2013, vol. 2, pp. 101–113.

    Google Scholar 

  22. Pazal’skaya, A.G., Solovyov, A.N., The Sentiment Recognition Method for Texts in Russian. Computer Linguistics and Intelligent Technologies: According to the Materials of the Yearly International Conference Dialog, 2011, pp. 510–522.

    Google Scholar 

  23. Perez-Rosas, V., Banea, C, Mihalcea, R., Learning sentiment lexicons in Spanish, Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), 2012.

    Google Scholar 

  24. Polyakov, P.Yu., Kalinina, M.V., Pleshko, V.V. Study of the Applicability of the Thematic Classification Methods in the Task of Classifying Reports about Books. Computer Linguistics and Intelligent Technologies: According to the Materials of the Yearly International Conference Dialog, 2012, vol. 2, pp. 51–59.

    Google Scholar 

  25. Pestian, J., Matykiewicz, P., Linn-Gust, M., Sentiment analysis of suicide notes: A shared task. Biomedical Informatics Insights. 2012; 5(Suppl. 1), 2012, pp. 3–16.

    Article  Google Scholar 

  26. Ounis, L., de Rijke, M., Macdonald, C, Mishne, G., Soboroff, I., Overview of TREC-2006 Blog track, Proceedings of TREC-2006, Gaithersburg, USA, 2007.

    Google Scholar 

  27. Ounis, I., Macdonald, C., Soboroff, I., Overview of the TREC 2008 blog track, Proceedings of TREC-2008, Gaithersburg, USA, 2009.

    Google Scholar 

  28. Ounis, I., Macdonald, C., Soboroff, I., Overview of the TREC 2010 blog track, Proceedings of TREC-2010, Gaithersburg, USA, 2011.

    Google Scholar 

  29. Seki, Y., Evans, D., Ku. L., Chen, H., Kando, N., Lin, C., Overview of opinion analysis pilot task at NTCIR-6, Proceedings of NTCIR-6 Workshop Meeting, 2007, pp. 265–278.

    Google Scholar 

  30. Steinberger, J., Lenkova, P., Ebrahim, M., Ehrmann, M., Hurriyetogly, A., Kabadjov, M., Steinberger, R., Tanev, H., Zavarella, V., Vazquez, S., Creating sentiment dictionaries via triangulation, Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis, ACL-HLT, 2011, pp. 28–36.

    Google Scholar 

  31. Taboada, M., Brooke, J., Tofiloski, M., Voll, K., Stede, M., Lexicon-based methods for Sentiment Analysis, Comp. Ling., 2011, vol., 37(2), pp. 267–307.

    Article  Google Scholar 

  32. Wu, Y., Jin, P., Semeval-2010 task 18: Disambiguating sentiment ambiguous adjectives, Proceedings of the 5th International Workshop on Semantic Evaluation, 2010, pp. 81–85.

    Google Scholar 

  33. Zagibalov, T., Belyatskaya, K., Carroll, J. Comparable English-Russian book review corpora for sentiment analysis, Proceedings of the 1st Workshop on Computational Approaches to Subjectivity and Sentiment Analysis WASSA, 2010, pp. 67–72.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to N. V. Loukachevitch.

Additional information

Original Russian Text © N.V. Loukachevitch, I.I. Chetviorkin, 2014, published in Iskusstvennyi Intellekt i Prinyatie Reshenii, 2014, No. 1, pp. 25–33.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Loukachevitch, N.V., Chetviorkin, I.I. Open evaluation of sentiment-analysis systems based on the material of the Russian language. Sci. Tech.Inf. Proc. 41, 370–376 (2014). https://doi.org/10.3103/S0147688214060057

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.3103/S0147688214060057

Keywords

Navigation