Skip to main content
SpringerLink
Log in

Size vs. Structure in Training Corpora for Word Embedding Models: Araneum Russicum Maximum and Russian National Corpus

  • Conference paper
  • First Online:
Analysis of Images, Social Networks and Texts (AIST 2017)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10716))

Abstract

In this paper, we present a distributional word embedding model trained on one of the largest available Russian corpora: Araneum Russicum Maximum (over 10 billion words crawled from the web). We compare this model to the model trained on the Russian National Corpus (RNC). The two corpora are much different in their size and compilation procedures. We test these differences by evaluating the trained models against the Russian part of the Multilingual SimLex999 semantic similarity dataset. We detect and describe numerous issues in this dataset and publish a new corrected version. Aside from the already known fact that the RNC is generally a better training corpus than web corpora, we enumerate and explain fine differences in how the models process semantic similarity task, what parts of the evaluation set are difficult for particular models and why. Additionally, the learning curves for both models are described, showing that the RNC is generally more robust as training material for this task.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://rusvectores.org/static/testsets/.

  2. 2.

    http://rusvectores.org/models/.

  3. 3.

    http://rusvectores.org/static/rus_araneum_maxicum.txt.gz.

References

  1. Turney, P., Pantel, P., et al.: From frequency to meaning: vector space models of semantics. J. Artif. Intell. Res. 37(1), 141–188 (2010)

    MathSciNet  MATH  Google Scholar 

  2. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. Adv. Neural Inf. Process. Syst. 26, 3111–3119 (2013)

    Google Scholar 

  3. Baroni, M., Dinu, G., Kruszewski, G.: Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, vol. 1, Baltimore, USA, pp. 238–247 (2014)

    Google Scholar 

  4. Panchenko, A., Ustalov, D., Arefyev, N., Paperno, D., Konstantinova, N., Loukachevitch, N., Biemann, C.: Human and machine judgements about Russian semantic relatedness. In: Proceedings of the 5th Conference on Analysis of Images, Social Networks and Texts (AIST 2016), Communications in Computer and Information Science (CCIS), pp. 174–183 (2016)

    Google Scholar 

  5. Kutuzov, A., Andreev, I.: Texts in, meaning out: neural language models in semantic similarity task for Russian. In: Proceedings of the Dialog Conference, vol. 2, Moscow, Russia, pp. 133–145 (2015)

    Google Scholar 

  6. Kutuzov, A., Kuzmenko, E.: Comparing neural lexical models of a classic national corpus and a web corpus: the case for russian. In: Gelbukh, A. (ed.) CICLing 2015. LNCS, vol. 9041, pp. 47–58. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-18111-0_4

    Google Scholar 

  7. Benko, V., Zakharov, V.: Very large Russian corpora: new opportunities and new challenges. Kompjuternaja Lingvistika I Intellektuanyje Technologii: Po Materialam Medunarodnoj konferencii “Dialog”, vol. 15(22), pp. 79–93 (2016)

    Google Scholar 

  8. Segalovich, I.: A fast morphological algorithm with unknown word guessing induced by a dictionary for a web search engine. In: MLMTA, pp. 273–280 (2003)

    Google Scholar 

  9. Petrov, S., Das, D., McDonald, R.: A universal part-of-speech tagset. In: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), European Language Resources Association (ELRA) (2012)

    Google Scholar 

  10. Goldberg, Y.: A primer on neural network models for natural language processing. J. Artif. Intell. Res. 57, 345–420 (2016)

    MathSciNet  MATH  Google Scholar 

  11. Hill, F., Reichart, R., Korhonen, A.: Simlex-999: evaluating semantic models with (genuine) similarity estimation. Comput. Linguist. 41(4), 665–695 (2015)

    Article  MathSciNet  Google Scholar 

  12. Leviant, I., Reichart, R.: Separated by an un-common language: towards judgment language informed vector space modeling. arXiv preprint arXiv:1508.00106 (2015)

  13. Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., Ruppin, E.: Placing search in context: the concept revisited. In: Proceedings of the 10th International Conference on World Wide Web, pp. 406–414. ACM (2001)

    Google Scholar 

  14. Avraham, O., Goldberg, Y.: Improving reliability of word similarity evaluation by redesigning annotation task and performance measure. In: Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP. Association for Computational Linguistics, pp. 106–110 (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Oslo, Oslo, Norway

    Andrey Kutuzov

  2. University of Tyumen, Tyumen, Russia

    Maria Kunilovskaya

Authors
  1. Andrey Kutuzov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Kunilovskaya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Kunilovskaya .

Editor information

Editors and Affiliations

  1. Eindhoven University of Technology, Eindhoven, The Netherlands

    Wil M.P. van der Aalst

  2. National Research University Higher School of Economics, Moscow, Russia

    Dmitry I. Ignatov

  3. Krasovsky Institute of Mathematics and Mechanics, Ekaterinburg, Russia

    Michael Khachay

  4. National Research University Higher School of Economics, Moscow, Russia

    Sergei O. Kuznetsov

  5. Skolkovo Institute of Science and Technology, Moscow, Russia

    Victor Lempitsky

  6. National Research University Higher School of Economics, Moscow, Russia

    Irina A. Lomazova

  7. Moscow State University, Moscow, Russia

    Natalia Loukachevitch

  8. LORIA, Campus Scientifique, Vandœuvre lès Nancy, France

    Amedeo Napoli

  9. University of Hamburg, Hamburg, Germany

    Alexander Panchenko

  10. University of Florida, Gainesville, Florida, USA

    Panos M. Pardalos

  11. National Research University Higher School of Economics, Nizhny Novgorod, Russia

    Andrey V. Savchenko

  12. Indiana University, Bloomington, Indiana, USA

    Stanley Wasserman

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kutuzov, A., Kunilovskaya, M. (2018). Size vs. Structure in Training Corpora for Word Embedding Models: Araneum Russicum Maximum and Russian National Corpus. In: van der Aalst, W., et al. Analysis of Images, Social Networks and Texts. AIST 2017. Lecture Notes in Computer Science(), vol 10716. Springer, Cham. https://doi.org/10.1007/978-3-319-73013-4_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-73013-4_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-73012-7

  • Online ISBN: 978-3-319-73013-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation