Predicting Query Performance via Classification

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5993)


We investigate using topic prediction data, as a summary of document content, to compute measures of search result quality. Unlike existing quality measures such as query clarity that require the entire content of the top-ranked results, class-based statistics can be computed efficiently online, because class information is compact enough to precompute and store in the index. In an empirical study we compare the performance of class-based statistics to their language-model counterparts for two performance-related tasks: predicting query difficulty and expansion risk. Our findings suggest that using class predictions can offer comparable performance to full language models while reducing computation overhead.


Root Mean Square Error Average Precision Query Term Query Expansion Query Performance 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Amati, G., Carpineto, C., Romano, G.: Query difficulty, robustness, and selective application of query expansion. In: McDonald, S., Tait, J.I. (eds.) ECIR 2004. LNCS, vol. 2997, pp. 127–137. Springer, Heidelberg (2004)Google Scholar
  2. 2.
    Aslam, J.A., Pavlu, V.: Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions. In: Amati, G., Carpineto, C., Romano, G. (eds.) ECIR 2007. LNCS, vol. 4425, pp. 198–209. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  3. 3.
    Billerbeck, B.: Efficient Query Expansion. PhD thesis, RMIT University, Melbourne, Australia (2005)Google Scholar
  4. 4.
    Chickering, D., Heckerman, D., Meek, C.: A Bayesian approach to learning Bayesian networks with local structure. In: UAI 1997, pp. 80–89 (1997)Google Scholar
  5. 5.
    Chickering, D.M.: The WinMine toolkit. Technical Report MSR-TR-2002-103, Microsoft, Redmond, WA (2002)Google Scholar
  6. 6.
    Collins-Thompson, K., Callan, J.: Estimation and use of uncertainty in pseudo-relevance feedback. In: Proceedings of SIGIR 2007, pp. 303–310 (2007)Google Scholar
  7. 7.
    Cronen-Townsend, S., Croft, W.: Quantifying query ambiguity. In: Proceedings of HCL 2002, pp. 94–98 (2002)Google Scholar
  8. 8.
    Diaz, F.: Performance prediction using spatial autocorrelation. In: Proceedings of SIGIR 2007, pp. 583–590 (2007)Google Scholar
  9. 9.
    Hauff, C., Murdock, V., Baeza-Yates, R.: Improved query difficulty prediction for the web. In: Proceedings of CIKM 2008, pp. 439–448 (2008)Google Scholar
  10. 10.
    He, B., Ounis, I.: Query performance prediction. Information Systems 31, 585–594 (2006)CrossRefGoogle Scholar
  11. 11.
    Heckerman, D., Chickering, D., Meek, C., Rounthwaite, R., Kadie, C.: Dependency networks for inference, collaborative filtering, and data visualization. Journal of Machine Learning Research 1, 49–75 (2000)CrossRefGoogle Scholar
  12. 12.
    Lavrenko, V.: A Generative Theory of Relevance. PhD thesis, University of Massachusetts, Amherst (2004)Google Scholar
  13. 13.
    Lemur. Lemur toolkit for language modeling & retrieval (2002),
  14. 14.
    Metzler, D., Croft, W.B.: Latent concept expansion using Markov Random Fields. In: Proceedings of SIGIR 2007, pp. 311–318 (2007)Google Scholar
  15. 15.
    Netscape Communication Corp. Open directory project,
  16. 16.
    Qiu, G., Liu, K., Bu, J., Chen, C., Kang, Z.: Quantify query ambiguity using ODP metadata. In: Proceedings of SIGIR 2007, pp. 697–698 (2007)Google Scholar
  17. 17.
    Smeaton, A., van Rijsbergen, C.J.: The retrieval effects of query expansion on a feedback document retrieval system. The Computer Journal 26(3), 239–246 (1983)CrossRefGoogle Scholar
  18. 18.
    Song, R., Luo, Z., Wen, J.-R., Yu, Y., Hon, H.-W.: Identifying ambiguous queries in web search. In: Proceedings of WWW 2007, pp. 1169–1170 (2007)Google Scholar
  19. 19.
    Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: A language model-based search engine for complex queries. In: Proceedings of the International Conference on Intelligence Analysis (2004)Google Scholar
  20. 20.
    Vinay, V., Cox, I.J., Milic-Frayling, N., Wood, K.: On ranking the effectiveness of searches. In: Proceedings of SIGIR 2005, pp. 398–404 (2005)Google Scholar
  21. 21.
    Winaver, M., Kurland, O., Domshlak, C.: Towards robust query expansion: model selection in the language modeling framework. In: Proceedings of SIGIR 2007, pp. 729–730 (2007)Google Scholar
  22. 22.
    YomTov, E., Fine, S., Carmel, D., Darlow, A.: Learning to estimate query difficulty. In: Proceedings of SIGIR 2005, pp. 512–519 (2005)Google Scholar
  23. 23.
    Zhou, Y., Croft, W.B.: Ranking robustness: a novel framework to predict query performance. In: Proceedings of CIKM 2006, pp. 567–574 (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  1. 1.Microsoft ResearchRedmondUSA

Personalised recommendations