Advertisement

A Comparison Between Term-Based and Embedding-Based Methods for Initial Retrieval

  • Tonglei Guo
  • Jiafeng Guo
  • Yixing Fan
  • Yanyan Lan
  • Jun Xu
  • Xueqi Cheng
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11168)

Abstract

The initial retrieval stage of information retrieval aims to generate as many relevant candidate documents as possible in a simple yet efficient way. Traditional term based retrieval methods like BM25 deal with the problem based on Bag-of-Words (BoW) representation, thus they only focus on exact matching (i.e., syntactic) and lack the consideration for semantically related words. That causes the typical vocabulary mismatch problem and the reduction of performance in terms of recall. The advance of distributed representation (i.e., embedding) of words and documents provides an efficient way to measure the semantic relevance between words. Since embedding can alleviate the vocabulary mismatch problem, it is suitable for the initial retrieval task. We conduct several experiments to compare term based models with embedding based models in terms of recall. We compare above two branches of the initial retrieval models on three representative retrieval tasks (Web-QA, Ad-hoc retrieval and CQA respectively). The results show that embedding based method and term based method are complementary for each other and higher recall can be achieved by combining the above two types of models based on scores or ranking position. We find that combination of the two types of the models based on ranking position usually perform better than combination based on score. Furthermore, since queries and documents are in different forms for diverse application scenarios, it can be observed that the relative performance of the two types are almost same but the absolute performance are significant different regarding to distinct scenarios.

Keywords

Initial retrieval Embedding representation Recall 

References

  1. 1.
    Blooma, M.J., Kurian, J.C.: Research issues in community based question answering. In: PACIS, p. 29 (2011)Google Scholar
  2. 2.
    Burges, C., et al.: Learning to rank using gradient descent. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 89–96. ACM (2005)Google Scholar
  3. 3.
    Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., Li, H.: Learning to rank: from pairwise approach to listwise approach. In: Proceedings of the 24th International Conference on Machine Learning, pp. 129–136. ACM (2007)Google Scholar
  4. 4.
    Carmel, D., Mejer, A., Pinter, Y., Szpektor, I.: Improving term weighting for community question answering search using syntactic analysis. In: Proceedings of the 23rd ACM International Conference on Information and Knowledge Management, pp. 351–360. ACM (2014)Google Scholar
  5. 5.
    Dang, V., Bendersky, M., Croft, W.B.: Two-stage learning to rank for information retrieval. In: Serdyukov, P. (ed.) ECIR 2013. LNCS, vol. 7814, pp. 423–434. Springer, Heidelberg (2013).  https://doi.org/10.1007/978-3-642-36973-5_36CrossRefGoogle Scholar
  6. 6.
    Firth, J.R.: A synopsis of linguistic theory, 1930–1955. In: Studies in Linguistic Analysis (1957)Google Scholar
  7. 7.
    Galitsky, B.: Natural language question answering system: technique of semantic headers. Advanced Knowledge International (2003)Google Scholar
  8. 8.
    Guo, J., Fan, Y., Ai, Q., Croft, W.B.: A deep relevance matching model for ad-hoc retrieval. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 55–64. ACM (2016)Google Scholar
  9. 9.
    Guo, J., Fan, Y., Ai, Q., Croft, W.B.: Semantic matching by non-linear word transportation for information retrieval. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 701–710. ACM (2016)Google Scholar
  10. 10.
    Harris, Z.S.: Distributional structure. Word 10(2–3), 146–162 (1954)CrossRefGoogle Scholar
  11. 11.
    Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., Heck, L.: Learning deep structured semantic models for web search using clickthrough data. In: Proceedings of the 22nd ACM International Conference on Information and Knowledge Management, pp. 2333–2338. ACM (2013)Google Scholar
  12. 12.
    Huston, S., Croft, W.B.: Parameters learned in the comparison of retrieval models using term dependencies. Ir, University of Massachusetts (2014)Google Scholar
  13. 13.
    Krovetz, R.: Viewing morphology as an inference process. In: Proceedings of the 16th Annual International ACM SIGIR Conference on Research and development in Information Retrieval, pp. 191–202. ACM (1993)Google Scholar
  14. 14.
    Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: International Conference on Machine Learning, pp. 1188–1196 (2014)Google Scholar
  15. 15.
    Lin, J.: The web as a resource for question answering: perspectives and challenges. In: Proceedings of the Third International Conference on Language Resources and Evaluation (LREC 2002). Citeseer (2002)Google Scholar
  16. 16.
    Lu, Z., Li, H.: A deep architecture for matching short texts. In: Advances in Neural Information Processing Systems, pp. 1367–1375 (2013)Google Scholar
  17. 17.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)Google Scholar
  18. 18.
    Pang, L., Lan, Y., Guo, J., Xu, J., Xu, J., Cheng, X.: DeepRank: a new deep architecture for relevance ranking in information retrieval. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 257–266. ACM (2017)Google Scholar
  19. 19.
    Patra, B.: A survey of community question answering. arXiv preprint arXiv:1705.04009 (2017)
  20. 20.
    Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)Google Scholar
  21. 21.
    Perera, R.: IPedagogy: question answering system based on web information clustering. In: 2012 IEEE Fourth International Conference on Technology for Education (T4E), pp. 245–246. IEEE (2012)Google Scholar
  22. 22.
    Ponte, J.M., Croft, W.B.: A language modeling approach to information retrieval. In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 275–281. ACM (1998)Google Scholar
  23. 23.
    Robertson, S.E., Walker, S.: Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In: Croft, B.W., van Rijsbergen, C.J. (eds.) SIGIR ’94, pp. 232–241. Springer, New York (1994).  https://doi.org/10.1007/978-1-4471-2099-5_24CrossRefGoogle Scholar
  24. 24.
    Turtle, H., Croft, W.B.: Evaluation of an inference network-based retrieval model. ACM Trans. Inf. Syst. (TOIS) 9(3), 187–222 (1991)CrossRefGoogle Scholar
  25. 25.
    Vulić, I., Moens, M.-F.: Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 363–372. ACM (2015)Google Scholar
  26. 26.
    Zhai, C., Lafferty, J.: Two-stage language models for information retrieval. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 49–56. ACM (2002)Google Scholar
  27. 27.
    Zhai, C., Lafferty, J.: A study of smoothing methods for language models applied to ad hoc information retrieval. In: ACM SIGIR Forum, vol. 51, pp. 268–276. ACM (2017)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Tonglei Guo
    • 1
  • Jiafeng Guo
    • 1
  • Yixing Fan
    • 1
  • Yanyan Lan
    • 1
  • Jun Xu
    • 1
  • Xueqi Cheng
    • 1
  1. 1.CAS Key Lab of Network Data Science and Technology, Institute of Computing TechnologyChinese Academy of SciencesBeijingChina

Personalised recommendations