Exploration of a Threshold for Similarity Based on Uncertainty in Word Embedding

  • Navid RekabsazEmail author
  • Mihai Lupu
  • Allan Hanbury
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10193)


Word embedding promises a quantification of the similarity between terms. However, it is not clear to what extent this similarity value can be of practical use for subsequent information access tasks. In particular, which range of similarity values is indicative of the actual term relatedness? We first observe and quantify the uncertainty of word embedding models with respect to the similarity values they generate. Based on this, we introduce a general threshold which effectively filters related terms. We explore the effect of dimensionality on this general threshold by conducting the experiments in different vector dimensions. Our evaluation on four test collections with four relevance scoring models supports the effectiveness of our approach, as the results of the proposed threshold are significantly better than the baseline while being equal to, or statistically indistinguishable from, the optimal results.


Related Term Optimal Threshold Query Term Retrieval Task Test Collection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work is funded by: Self-Optimizer (FFG 852624) in the EUROSTARS programme, funded by EUREKA, the BMWFW and the European Union, and ADMIRE (P 25905-N23) by FWF. Thanks to Joni Sayeler and Linus Wretblad for their contributions in the SelfOptimizer project.


  1. 1.
    Baroni, M., Dinu, G., Kruszewski, G.: Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In: Proceedings of ACL Conference (2014)Google Scholar
  2. 2.
    Berger, A., Lafferty, J.: Information retrieval as statistical translation. In: Proceedings of SIGIR (1999)Google Scholar
  3. 3.
    Cuba Gyllensten, A., Sahlgren, M.: Navigating the semantic horizon using relative neighborhood graphs. In: Proceedings of EMNLP, Lisbon, Portugal (2015)Google Scholar
  4. 4.
    De Vine, L., Zuccon, G., Koopman, B., Sitbon, L., Bruza, P.: Medical semantic similarity with a neural language model. In: Proceedings of CIKMGoogle Scholar
  5. 5.
    Erk, K., Padó, S.: Exemplar-based models for word meaning in context. In: Proceedings of ACL (2010)Google Scholar
  6. 6.
    Ganguly, D., Roy, D., Mitra, M., Jones, G.J.: Word embedding based generalized language model for information retrieval. In: Proceedings of SIGIR (2015)Google Scholar
  7. 7.
    Grbovic, M., Djuric, N., Radosavljevic, V., Silvestri, F., Bhamidipati, N.: Context-and content-aware embeddings for query rewriting in sponsored search. In: Proceedings of SIGIR (2015)Google Scholar
  8. 8.
    Karlgren, J., Bohman, M., Ekgren, A., Isheden, G., Kullmann, E., Nilsson, D.: Semantic topology. In: Proceedings of CIKM Conference (2014)Google Scholar
  9. 9.
    Karlgren, J., Holst, A., Sahlgren, M.: Filaments of meaning in word space. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, pp. 531–538. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-78646-7_52 CrossRefGoogle Scholar
  10. 10.
    Kiela, D., Hill, F., Clark, S.: Specializing word embeddings for similarity or relatedness. In: Proceedings of EMNLP (2015)Google Scholar
  11. 11.
    Koopman, B., Zuccon, G., Bruza, P., Sitbon, L., Lawley, M.: An evaluation of corpus-driven measures of medical concept similarity for information retrieval. In: Proceedings of CIKM (2012)Google Scholar
  12. 12.
    Kruszewski, G., Baroni, M.: So similar and yet incompatible: toward automated identification of semantically compatible words. In: Proceedings of NAACL (2015)Google Scholar
  13. 13.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
  14. 14.
    Mitra, B.: Exploring session context using distributed representations of queries and reformulations. In: Proceedings of SIGIR (2015)Google Scholar
  15. 15.
    Ponte, J.M., Croft, W.B.: A language modeling approach to information retrieval. In: Proceedings of SIGIR (1998)Google Scholar
  16. 16.
    Rekabsaz, N., Bierig, R., Ionescu, B., Hanbury, A., Lupu, M.: On the use of statistical semantics for metadata-based social image retrieval. In: Proceedings of CBMI Conference (2015)Google Scholar
  17. 17.
    Rekabsaz, N., Lupu, M., Hanbury, A.: Generalizing translation models in the probabilistic relevance framework. In: Proceedings of CIKM (2016)Google Scholar
  18. 18.
    Sakai, T.: Alternatives to bpref. In: Proceedings of SIGIR (2007)Google Scholar
  19. 19.
    Schnabel, T., Labutov, I., Mimno, D., Joachims, T.: Evaluation methods for unsupervised word embeddings. In: Proceedings of EMNLP (2015)Google Scholar
  20. 20.
    Severyn, A., Moschitti, A.: Learning to rank short text pairs with convolutional deep neural networks. In: Proceedings of SIGIR (2015)Google Scholar
  21. 21.
    Tsvetkov, Y., Faruqui, M., Ling, W., Lample, G., Dyer, C.: Evaluation of word vector representations by subspace alignment. In: Proceedings of EMNLP (2015)Google Scholar
  22. 22.
    Vulić, I., Moens, M.-F.: Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings. In: Proceedings of SIGIR (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Institute of Software Technology and Interactive SystemsVienna University of TechnologyViennaAustria

Personalised recommendations