A Word Embeddings Training Method Based on Modified Skip-Gram and Align

  • Chang-shuai XingEmail author
  • Gang Zhou
  • Ji-Cang Lu
  • Feng-juan Zhang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11068)


To solve the problems that there is no sufficient annotated data in low-resource languages and it is hard to mine the deep semantic correspondence between languages via existing bilingual word embedding learning methods, this paper presents an effective text processing method based on transfer learning and bilingual word embedding model CWDR-BiGRU (Cross-context window of dynamic ratio bidirectional Gated Recurrent Unit) which contains an enhanced Skip-gram called cross-context window of dynamic ratio and encoder-decoder. The method can process low-resource language text effectively only using sentence-aligned corpus of bilingual resource languages and annotated data of high-resource language. The experimental results of semantic reasoning and word embedding visualization show that CWDR-BiGRU can effectively train bilingual word embeddings. In the task of Chinese-Tibetan cross-lingual document classification, the accuracy of transfer learning method based on CWDR-BiGRU is higher than the conventional method by 13.5%, and higher than the existing Bilingual Autoencoder, BilBOWA, BiCCV and BiSkip by 7.4%, 5.8%, 3.1% and 1.6% respectively, indicating CWDR-BiGRU which has reduced the difficulty of acquiring corpora for bilingual word embeddings can accurately excavate the deep alignment relationship and semantic properties.


Bilingual word embeddings Transfer learning Low-resource language Cross-context window of dynamic ratio 


  1. 1.
    Mikolov, T., Le, Q.V., Sutskever, I.: Exploiting similarities among languages for machine translation. Computer Science (2013)Google Scholar
  2. 2.
    Hermann, K.M., Blunsom, P.: Multilingual models for compositional distributed semantics. arXiv:1404.4641 (2014)
  3. 3.
    Zou, W.Y., Socher, R., Cer, D.M.: Bilingual word embedding for phrase-based machine translation. In: Empirical Methods in Natural Language Processing, pp. 1393–1398 (2013)Google Scholar
  4. 4.
    Xing, C., Wang, D., Liu, C.: Normalized word embedding and orthogonal transform for bilingual word translation. In: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1006–1011 (2015)Google Scholar
  5. 5.
    Vulić, I., Moens, M.F.: Bilingual word embeddings from non-parallel document-aligned data applied to bilingual lexicon induction. In: Meeting of the Association for Computational Linguistics and the, International Joint Conference on Natural Language Processing, pp. 719–725(2015)Google Scholar
  6. 6.
    Chandar, A.P.S., Lauly, S., Larochelle, H.: An autoencoder approach to learning bilingual word representations, pp. 1853–1861 (2014)Google Scholar
  7. 7.
    Luong, T., Pham, H., Manning, C.D.: Bilingual word representations with monolingual quality in mind. In: The Workshop on Vector Space Modeling for Natural Language Processing, pp. 151–159 (2015)Google Scholar
  8. 8.
    Gouws, S., Søgaard, A.: Simple task-specific bilingual word embeddings. In: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1386–1390 (2016)Google Scholar
  9. 9.
    Vyas, Y., Carpuat, M.: Sparse bilingual word representations for cross-lingual lexical entailment. In: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1187–1197 (2016)Google Scholar
  10. 10.
    Coulmance, J., Marty, J.M., Wenzek, G.: Trans-gram, fast cross-lingual word-embeddings (2016)Google Scholar
  11. 11.
    Vulić, I., Moens, M.F.: Bilingual distributed word representations from document-aligned comparable data. Computer Science, pp. 748–756 (2015)Google Scholar
  12. 12.
    Mikolov, T., Chen, K., Corrado, G.: Efficient estimation of word representations in vector space. Computer Science (2013)Google Scholar
  13. 13.
    Cho, K., Merrienboer, B.V., Gulcehre, C.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. Computer Science (2014)Google Scholar
  14. 14.
    Maaten, L.V.D., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(2605), 2579–2605 (2008)zbMATHGoogle Scholar
  15. 15.
    Bhattarai, B., Klementiev, A., Titov, I.: Inducing crosslingual distributed representations of words. J. Comput. Syst. Sci. 55(1), 36–43 (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Chang-shuai Xing
    • 1
    Email author
  • Gang Zhou
    • 1
  • Ji-Cang Lu
    • 1
  • Feng-juan Zhang
    • 1
  1. 1.State Key Laboratory of Mathematical Engineering and Advanced ComputerZhengzhouChina

Personalised recommendations