Advertisement

On the Reproducibility and Generalisation of the Linear Transformation of Word Embeddings

  • Xiao YangEmail author
  • Iadh Ounis
  • Richard McCreadie
  • Craig Macdonald
  • Anjie Fang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10772)

Abstract

Linear transformation is a way to learn a linear relationship between two word embeddings, such that words in the two different embedding spaces can be semantically related. In this paper, we examine the reproducibility and generalisation of the linear transformation of word embeddings. Linear transformation is particularly useful when translating word embedding models in different languages, since it can capture the semantic relationships between two models. We first reproduce two linear transformation approaches, a recent one using orthogonal transformation and the original one using simple matrix transformation. Previous findings on a machine translation task are re-examined, validating that linear transformation is indeed an effective way to transform word embedding models in different languages. In particular, we show that the orthogonal transformation can better relate the different embedding models. Following the verification of previous findings, we then study the generalisation of linear transformation in a multi-language Twitter election classification task. We observe that the orthogonal transformation outperforms the matrix transformation. In particular, it significantly outperforms the random classifier by at least 10% under the F1 metric across English and Spanish datasets. In addition, we also provide best practices when using linear transformation for multi-language Twitter election classification.

Keywords

Embedding Linear transformation Twitter classification 

Notes

Acknowledgements

This paper was supported by a grant from the Economic and Social Research Council, (ES/L016435/1). The authors would like to thank the assessors for their efforts in reviewing tweets.

References

  1. 1.
    Mikolov, T., Le, Q.V., Sutskever, I.: Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168 (2013)
  2. 2.
    Chandar, S., Lauly, S., Larochelle, H., Khapra, M., Ravindran, B., Raykar, V.C., Saha, A.: An autoencoder approach to learning bilingual word representations. In: Proceedings of NIPS (2014)Google Scholar
  3. 3.
    Eger, S., Hoenen, A.: Language classification from bilingual word embedding graphs. arXiv preprint arXiv:1607.05014 (2016)
  4. 4.
    Zhou, H., Chen, L., Shi, F., Huang, D.: Learning bilingual sentiment word embeddings for cross-language sentiment classification. In: Proceedings of ACL (2015)Google Scholar
  5. 5.
    Dinu, G., Lazaridou, A., Baroni, M.: Improving zero-shot learning by mitigating the hubness problem. arXiv preprint arXiv:1412.6568 (2014)
  6. 6.
    Ammar, W., Mulcaire, G., Tsvetkov, Y., Lample, G., Dyer, C., Smith, N.A.: Massively multilingual word embeddings. arXiv preprint arXiv:1602.01925 (2016)
  7. 7.
    Faruqui, M., Dyer, C.: Improving vector space word representations using multilingual correlation. In: Proceedings of EACL (2014)Google Scholar
  8. 8.
    Artetxe, M., Labaka, G., Agirre, E.: Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In: Proceedings of EMNLP (2016)Google Scholar
  9. 9.
    Smith, S.L., Turban, D.H.P., Hamblin, S., Hammerla, N.Y.: Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In: Proceedings of ICLR (2017)Google Scholar
  10. 10.
    Xing, C., Wang, D., Liu, C., Lin, Y.: Normalized word embedding and orthogonal transform for bilingual word translation. In: Proceedings of HLT-NAACL (2015)Google Scholar
  11. 11.
    Mitra, B., Nalisnick, E., Craswell, N., Caruana, R.: A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137 (2016)
  12. 12.
    Moran, S., McCreadie, R., Macdonald, C., Ounis, I.: Enhancing first story detection using word embeddings. In: Proceedings of ACM SIGIR (2016)Google Scholar
  13. 13.
    Fang, A., Macdonald, C., Ounis, I., Habel, P., Yang, X.: Exploring time-sensitive variational Bayesian inference LDA for social media data. In: Jose, J.M., Hauff, C., Altıngovde, I.S., Song, D., Albakour, D., Watt, S., Tait, J. (eds.) ECIR 2017. LNCS, vol. 10193, pp. 252–265. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-56608-5_20 CrossRefGoogle Scholar
  14. 14.
    Yang, X., Macdonald, C., Ounis, I.: Using word embeddings in Twitter election classification. In: Proceedings of Neu-IR Workshop at SIGIR (2016)Google Scholar
  15. 15.
    Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of EMNLP (2014)Google Scholar
  16. 16.
    Severyn, A., Nicosia, M., Barlacchi, G., Moschitti, A.: Distributional neural networks for automatic resolution of crossword puzzles. In: Proceedings of IJCNLP (2015)Google Scholar
  17. 17.
    Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606 (2016)
  18. 18.
    Voorhees, E.M., Harman, D.K.: TREC: Experiment and Evaluation in IR. MIT Press, Cambridge (2005)Google Scholar
  19. 19.
    Macdonald, C., McCreadie, R., Santos, R.L., Ounis, I.: From puppy to maturity: experiences in developing Terrier. In: Proceedings of OSIR Workshop at SIGIR (2012)Google Scholar
  20. 20.
    Amati, G., Amodeo, G., Bianchi, M., Marcone, G., Bordoni, F.U., Gaibisso, C., Gambosi, G., Celi, A., Di Nicola, C., Flammini, M.: FUB, IASI-CNR, UNIVAQ at TREC 2011 Microblog track. In: Proceedings of TREC (2011)Google Scholar
  21. 21.
    Severyn, A., Moschitti, A.: UNITN: Training deep convolutional neural network for Twitter sentiment classification. In: Proceedings of SemEval (2015)Google Scholar
  22. 22.
    Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T., Qin, B.: Learning sentiment-specific word embedding for Twitter sentiment classification. In: Proceedings of ACL (2014)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Xiao Yang
    • 1
    Email author
  • Iadh Ounis
    • 1
  • Richard McCreadie
    • 1
  • Craig Macdonald
    • 1
  • Anjie Fang
    • 1
  1. 1.University of GlasgowGlasgowUK

Personalised recommendations