Towards Risk-Aware Resource Selection

  • Ilya Markov
  • Mark Carman
  • Fabio Crestani
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8870)

Abstract

When searching multiple sources of information it is crucial to select only relevant sources for a given query, thus filtering out non-relevant content. This task is known as resource selection and is used in many areas of information retrieval such as federated and aggregated search, blog distillation, etc. Resource selection often operates with limited and incomplete data and, therefore, is associated with a certain risk of selecting non-relevant sources due to the uncertainty in the produced source ranking. Despite the large volume of research on resource selection, the problem of risk within resource selection has been rarely addressed. In this work we propose a resource selection method based on document score distribution models that supports estimation of uncertainty of produced source scores and results in a novel risk-aware resource selection technique. We analyze two distributed retrieval scenarios and show that many queries are risk-sensitive and, because of that, the proposed risk-aware approach provides a basis for significant improvements in resource selection performance.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Callan, J.P., Lu, Z., Croft, W.B.: Searching distributed collections with inference networks. In: Proceedings of SIGIR, pp. 21–28 (1995)Google Scholar
  2. 2.
    Paltoglou, G., Salampasis, M., Satratzemi, M.: Integral based source selection for uncooperative distributed information retrieval environments. In: Proceeding of workshop on LSDS for IR, pp. 67–74 (2008)Google Scholar
  3. 3.
    Shokouhi, M.: Central-rank-based collection selection in uncooperative distributed information retrieval. In: Proceedings of ECIR, pp. 160–172 (2007)Google Scholar
  4. 4.
    Si, L., Callan, J.: Relevant document distribution estimation method for resource selection. In: Proceedings of SIGIR, pp. 298–305 (2003)Google Scholar
  5. 5.
    Thomas, P., Shokouhi, M.: Sushi: scoring scaled samples for server selection. In: Proceedings of SIGIR, pp. 419–426 (2009)Google Scholar
  6. 6.
    Callan, J.: Distributed Information Retrieval. In: Advances in Information Retrieval, pp. 127–150. Kluwer Academic Publishers (2000)Google Scholar
  7. 7.
    Crestani, F., Markov, I.: Distributed information retrieval and applications. In: Proceedings of ECIR, pp. 865–868 (2013)Google Scholar
  8. 8.
    Shokouhi, M., Si, L.: Federated search. Foundations and Trends in Information Retrieval 5, 1–102 (2011)CrossRefGoogle Scholar
  9. 9.
    Markov, I., Azzopardi, L., Crestani, F.: Reducing the uncertainty in resource selection. In: Proceedings of ECIR, pp. 507–519 (2013)Google Scholar
  10. 10.
    Zhu, J., Wang, J., Cox, I.J., Taylor, M.J.: Risky business: modeling and exploiting uncertainty in information retrieval. In: Proceedings of SIGIR, pp. 99–106 (2009)Google Scholar
  11. 11.
    Nguyen, D., Demeester, T., Trieschnigg, D., Hiemstra, D.: Federated search in the wild: the combined power of over a hundred search engines. In: Proceedings of CIKM, pp. 1874–1878 (2012)Google Scholar
  12. 12.
    Kulkarni, A., Tigelaar, A.S., Hiemstra, D., Callan, J.: Shard ranking and cutoff estimation for topically partitioned collections. In: Proceedings of CIKM, pp. 555–564 (2012)Google Scholar
  13. 13.
    Xu, J., Croft, W.B.: Cluster-based language models for distributed retrieval. In: Proceedings of SIGIR, pp. 254–261 (1999)Google Scholar
  14. 14.
    Markov, I., Crestani, F.: Theoretical, qualitative, and quantitative analyses of small-document approaches to resource selection. ACM Transactions on Information Systems 32(2), 9:1–9:37 (2014)Google Scholar
  15. 15.
    Aly, R., Hiemstra, D., Demeester, T.: Taily: shard selection using the tail of score distributions. In: Proceedings of SIGIR, pp. 673–682 (2013)Google Scholar
  16. 16.
    Baumgarten, C.: A probabilistic solution to the selection and fusion problem in distributed information retrieval. In: Proceedings of SIGIR, pp. 246–253 (1999)Google Scholar
  17. 17.
    Markov, I.: Modeling document scores for distributed information retrieval. In: Proceedings of SIGIR, pp. 1321–1322 (2011)Google Scholar
  18. 18.
    Arampatzis, A., Robertson, S.: Modeling score distributions in information retrieval. Information Retrieval 14(1), 26–46 (2011)CrossRefGoogle Scholar
  19. 19.
    Manmatha, R., Rath, T., Feng, F.: Modeling score distributions for combining the outputs of search engines. In: Proceedings of SIGIR, pp. 267–275 (2001)Google Scholar
  20. 20.
    Arguello, J., Callan, J., Diaz, F.: Classification-based resource selection. In: Proceedings of CIKM, pp. 1277–1286 (2009)Google Scholar
  21. 21.
    Callan, J., Connell, M.: Query-based sampling of text databases. ACM Transactions on Information Systems 19(2), 97–130 (2001)CrossRefGoogle Scholar
  22. 22.
    Shokouhi, M., Zobel, J., Scholer, F., Tahaghoghi, S.M.M.: Capturing collection size for distributed non-cooperative retrieval. In: Proceedings of SIGIR, pp. 316–323 (2006)Google Scholar
  23. 23.
    Markov, I., Arampatzis, A., Crestani, F.: On cori results merging. In: Proceedings of ECIR, pp. 752–755 (2013)Google Scholar
  24. 24.
    Zuccon, G., Azzopardi, L., van Rijsbergen, K.: Back to the roots: Mean-variance analysis of relevance estimations. In: Proceedings of ECIR, pp. 716–720 (2011)Google Scholar
  25. 25.
    Wang, J., Zhu, J.: Portfolio theory of information retrieval. In: Proceeding of SIGIR, pp. 115–122 (2009)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Ilya Markov
    • 1
  • Mark Carman
    • 2
  • Fabio Crestani
    • 1
  1. 1.University of Lugano (USI)LuganoSwitzerland
  2. 2.Monash UniversityAustralia

Personalised recommendations