Exploiting Synonymy and Hypernymy to Learn Efficient Meaning Representations

  • Thomas PerianinEmail author
  • Hajime Senuma
  • Akiko Aizawa
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10075)


Word representation learning methods such as word2vec usually associate one vector per word; however, in order to face polysemy problems, it’s important to produce distributed representations for each meaning, not for each surface form of a word. In this paper, we propose an extension for the existing AutoExtend model, an auto-encoder architecture that utilises synonymy relations to learn sense representations. We introduce a new layer in the architecture to exploit hypernymy relations predominantly present in existing ontologies. We evaluate the quality of the obtained vectors on word-sense disambiguation tasks and show that the use of the hypernymy relation leads to improvements of 1.2 % accuracy on Senseval-3 and 0.8 % on Semeval-2007 English lexical sample tasks, compared to the original model.


Sense embedding Semantic relation Auto-encoder Hypernymy Word-sense disambiguation 


  1. 1.
    Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R.M., Makhoul, J.: Fast and robust neural network joint models for statistical machine translation. In: ACL (1), pp. 1370–1380. Citeseer (2014)Google Scholar
  2. 2.
    Jauhar, S.K., Dyer, C., Hovy, E.: Ontologically grounded multi-sense representation learning for semantic vector space models. In: Proceedings NAACL, vol. 1 (2015)Google Scholar
  3. 3.
    Mihalcea, R., Chklovski, T.A., Kilgarriff, A.: The senseval-3 english lexical sample task. In: Association for Computational Linguistics (2004)Google Scholar
  4. 4.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp. 3111–3119 (2013)Google Scholar
  5. 5.
    Miller, G.A.: WORDNET: A lexical database for english. In: Human Language Technology, Proceedings of a Workshop held at Plainsboro, New Jerey, USA, March 8–11, 1994.
  6. 6.
    Neelakantan, A., Shankar, J., Passos, A., McCallum, A.: Efficient non-parametric estimation of multiple embeddings per word in vector space (2015). arXiv preprint: arXiv:1504.06654
  7. 7.
    Pradhan, S.S., Loper, E., Dligach, D., Palmer, M.: Semeval-2007 task 17: English lexical sample, srl and all words. In: Proceedings of the 4th International Workshop on Semantic Evaluations, pp. 87–92. ACL (2007)Google Scholar
  8. 8.
    Qiu, S., Cui, Q., Bian, J., Gao, B., Liu, T.Y.: Co-learning of word representations and morpheme representations. In: COLING, pp. 141–150 (2014)Google Scholar
  9. 9.
    Reisinger, J., Mooney, R.J.: Multi-prototype vector-space models of word meaning. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 109–117. Association for Computational Linguistics (2010)Google Scholar
  10. 10.
    Rothe, S., Schütze, H.: Autoextend: Extending word embeddings to embeddings for synsets and lexemes. In: Proceedings of the ACL (2015)Google Scholar
  11. 11.
    Turney, P.D., Littman, M.L.: Measuring praise and criticism: Inference of semantic orientation from association. ACM Trans. Inf. Syst. 21(4), 315–346 (2003). CrossRefGoogle Scholar
  12. 12.
    Xu, C., Bai, Y., Bian, J., Gao, B., Wang, G., Liu, X., Liu, T.Y.: Rc-net: A general framework for incorporating knowledge into word representations. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pp. 1219–1228. ACM (2014)Google Scholar
  13. 13.
    Zhong, Z., Ng, H.T.: It makes sense: A wide-coverage word sense disambiguation system for free text. In: Proceedings of the ACL 2010 System Demonstrations, pp. 78–83. Association for Computational Linguistics (2010)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Thomas Perianin
    • 1
    • 3
    Email author
  • Hajime Senuma
    • 2
  • Akiko Aizawa
    • 2
    • 3
  1. 1.Université Pierre et Marie CurieParisFrance
  2. 2.The University of TokyoBunkyoJapan
  3. 3.National Institute of InformaticsChiyodaJapan

Personalised recommendations