A Belief Model of Query Difficulty That Uses Subjective Logic

  • Christina Lioma
  • Roi Blanco
  • Raquel Mochales Palau
  • Marie-Francine Moens
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5766)


The difficulty of a user query can affect the performance of Information Retrieval (IR) systems. This work presents a formal model for quantifying and reasoning about query difficulty as follows: Query difficulty is considered to be a subjective belief, which is formulated on the basis of various types of evidence. This allows us to define a belief model and a set of operators for combining evidence of query difficulty. The belief model uses subjective logic, a type of probabilistic logic for modeling uncertainties. An application of this model with semantic and pragmatic evidence about 150 TREC queries illustrates the potential flexibility of this framework in expressing and combining evidence. To our knowledge, this is the first application of subjective logic to IR.


Information Retrieval Belief Revision Query Term Mean Average Precision Subjective Logic 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    van Rijsbergen, C.J.: A non-classical logic for information retrieval. Comput. J. 29(6), 481–485 (1986)CrossRefzbMATHGoogle Scholar
  2. 2.
    van Rijsbergen, C.J.: The Geometry of Information Retrieval. CUP, Cambridge (2004)CrossRefzbMATHGoogle Scholar
  3. 3.
    van Rijsbergen, C.J., Crestani, F., Lalmas, M.: Information Retrieval: Uncertainty and Logics. Springer, Heidelberg (1998)Google Scholar
  4. 4.
    Carmel, D., Yom-Tov, E., Darlow, A., Pelleg, D.: What makes a query difficult? In: SIGIR, pp. 390–397 (2006)Google Scholar
  5. 5.
    Chiaramella, Y., Chevallet, J.-P.: About retrieval models and logic. Comput. J. 35(3), 233–242 (1992)CrossRefzbMATHGoogle Scholar
  6. 6.
    Cronen-Townsend, S., Zhou, Y., Croft, W.B.: Predicting query performance. In: SIGIR, pp. 299–306 (2002)Google Scholar
  7. 7.
    Dempster, A.P.: A generalization of Bayesian inference. Journal of the Royal Statistical Society B(30), 205–247 (1968)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Fishburn, P.C.: The axioms of subjective probability. Statistical Science 3(1), 335–345 (1986)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Hauff, C., Azzopardi, L., Hiemstra, D.: The combination and evaluation of query performance prediction methods. In: ECIR, pp. 301–312 (2009)Google Scholar
  10. 10.
    He, B., Ounis, I.: Inferring query performance using pre-retrieval predictors. In: Apostolico, A., Melucci, M. (eds.) SPIRE 2004. LNCS, vol. 3246, pp. 43–54. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  11. 11.
    Josang, A.: A logic for uncertain probabilities. Int. J. Uncertain. Fuzziness Knowl.-Based Syst. 9(3), 279–311 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Lalmas, M.: Information retrieval and Dempster-Shafer’s theory of evidence. In: Hunter, A., Parsons, S. (eds.) Applications of Uncertainty Formalisms. LNCS (LNAI), vol. 1455, pp. 157–176. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  13. 13.
    Lau, R.Y.K., Bruza, P.D., Song, D.: Towards a belief-revision-based adaptive and context-sensitive information retrieval system. ACM Trans. Inf. Syst. 26(2) (2008)Google Scholar
  14. 14.
    Logan, B., Reece, S., Sparck Jones, K.: Modelling information retrieval agents with belief revision. In: SIGIR, pp. 91–100 (1994)Google Scholar
  15. 15.
    Losada, D.E., Barreiro, A.: A logical model for information retrieval based on propositional logic and belief revision. Comput. J. 44(5), 410–424 (2001)CrossRefzbMATHGoogle Scholar
  16. 16.
    Mothe, J., Tanguy, L.: Linguistic features to predict query difficulty - a case study on previous TREC campaigns. In: SIGIR Workshop on Predicting Query Difficulty: Methods and Applications (2005)Google Scholar
  17. 17.
    Nie, J.-Y.: Towards a probabilistic modal logic for semantic-based information retrieval. In: SIGIR, pp. 140–151 (1992)Google Scholar
  18. 18.
    Plachouras, V., Ounis, I.: Dempster-Shafer theory for a query-biased combination of evidence on the web. Inf. Retr. 8(2), 197–218 (2005)CrossRefGoogle Scholar
  19. 19.
    Shafer, G.: A Mathematical Theory of Evidence. Princeton University Press, Princeton (1976)zbMATHGoogle Scholar
  20. 20.
    Shi, L., Nie, J.-Y., Cao, G.: Relating dependent indexes using Dempster-Shafer theory. In: CIKM, pp. 429–438 (2008)Google Scholar
  21. 21.
    Smets, P.: What is Dempster-Shafer’s model? Wiley, Chichester (1994)Google Scholar
  22. 22.
    Tomlinson, S.: Robust, web, and terabyte retrieval with Hummingbird SearchServer at TREC 2004. In: TREC (2004)Google Scholar
  23. 23.
    Tsikrika, T., Lalmas, M.: Combining evidence for web retrieval using the inference network model: an experimental study. Inf. Process. Manage. 40(5), 751–772 (2004)CrossRefGoogle Scholar
  24. 24.
    Voorhees, E.M., Harman, D.K.: TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge (2005)Google Scholar
  25. 25.
    Yom-Tov, E., Fine, S., Carmel, D., Darlow, A.: Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval. In: SIGIR, pp. 512–519 (2005)Google Scholar
  26. 26.
    Zhou, Y., Croft, W.B.: Ranking robustness: a novel framework to predict query performance. In: CIKM, pp. 567–574 (2006)Google Scholar
  27. 27.
    Zhou, Y., Croft, W.B.: Query performance prediction in web search environments. In: SIGIR, pp. 543–550 (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Christina Lioma
    • 1
  • Roi Blanco
    • 2
  • Raquel Mochales Palau
    • 1
  • Marie-Francine Moens
    • 1
  1. 1.Computer ScienceKatholieke Universiteit LeuvenBelgium
  2. 2.IRLab, Computer Science DepartmentA Coruña UniversitySpain

Personalised recommendations