The Combination and Evaluation of Query Performance Prediction Methods

  • Claudia Hauff
  • Leif Azzopardi
  • Djoerd Hiemstra
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5478)


In this paper, we examine a number of newly applied methods for combining pre-retrieval query performance predictors in order to obtain a better prediction of the query’s performance. However, in order to adequately and appropriately compare such techniques, we critically examine the current evaluation methodology and show how using linear correlation coefficients (i) do not provide an intuitive measure indicative of a method’s quality, (ii) can provide a misleading indication of performance, and (iii) overstate the performance of combined methods. To address this, we extend the current evaluation methodology to include cross validation, report a more intuitive and descriptive statistic, and apply statistical testing to determine significant differences. During the course of a comprehensive empirical study over several TREC collections, we evaluate nineteen pre-retrieval predictors and three combination methods.


Root Mean Square Error Average Precision Retrieval Performance Query Term Query Performance 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Rank Correlation Methods. Hafner Publishing Co., New York (1955)Google Scholar
  2. 2.
    WordNet - An Electronic Lexical Database. MIT Press, Cambridge (1998)Google Scholar
  3. 3.
    Bach, F.R.: Bolasso: Model consistent lasso estimation through the bootstrap. In: ICML (2008)Google Scholar
  4. 4.
    Banerjee, S., Pedersen, T.: Extended gloss overlaps as a measure of semantic relatedness. In: IJCAI 2003, pp. 805–810 (2003)Google Scholar
  5. 5.
    Cronen-Townsend, S., Zhou, Y., Croft, W.B.: Predicting query performance. In: SIGIR 2002, pp. 299–306 (2002)Google Scholar
  6. 6.
    Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Ann. Statist. 32(2), 407–499 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    He, B., Ounis, I.: Inferring query performance using pre-retrieval predictors. In: Apostolico, A., Melucci, M. (eds.) SPIRE 2004. LNCS, vol. 3246, pp. 43–54. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  8. 8.
    He, J., Larson, M., de Rijke, M.: Using coherence-based measures to predict query difficulty. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, pp. 689–694. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  9. 9.
    Krovetz, R.: Viewing morphology as an inference process. In: SIGIR 1993, pp. 191–202 (1993)Google Scholar
  10. 10.
    Macdonald, C., He, B., Ounis, I.: Predicting query performance in intranet search. In: SIGIR 2005 Query Prediction Workshop (2005)Google Scholar
  11. 11.
    Meng, X., Rosenthal, R., Rubin, D.: Comparing correlated correlation coefficients. Psych. Bull. 111, 172–175 (1992)CrossRefGoogle Scholar
  12. 12.
    Mothe, J., Tanguy, L.: Linguistic features to predict query difficulty - a case study on previous trec campaigns. In: SIGIR 2005 Query Prediction Workshop (2005)Google Scholar
  13. 13.
    Scholer, F., Williams, H., Turpin, A.: Query association surrogates for web search. Journal of the American Society for Information Science and Technology 55(7), 637–650 (2004)CrossRefGoogle Scholar
  14. 14.
    Segal, M.R., Dahlquist, K.D., Conklin, B.R.: Regression approaches for microarray data analysis. J. Comput. Biol. 10(6), 961–980 (2003)CrossRefGoogle Scholar
  15. 15.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Statist. Soc. B 58(1), 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  16. 16.
    Vinay, V., Cox, I.J., Milic-Frayling, N., Wood, K.: On ranking the effectiveness of searches. In: SIGIR 2006, pp. 398–404 (2006)Google Scholar
  17. 17.
    Voorhees, E.: Overview of the trec 2003 robust retrieval track. In: Proceedings of the Twelfth Text REtrieval Conference (2003)Google Scholar
  18. 18.
    Zhai, C., Lafferty, J.: A study of smoothing methods for language models applied to ad hoc information retrieval. In: SIGIR 2001, pp. 334–342 (2001)Google Scholar
  19. 19.
    Zhao, Y., Scholer, F., Tsegay, Y.: Effective pre-retrieval query performance prediction using similarity and variability evidence. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, pp. 52–64. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  20. 20.
    Zhou, Y., Croft, W.B.: Query performance prediction in web search environments. In: SIGIR 2007, pp. 543–550 (2007)Google Scholar
  21. 21.
    Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Statist. Soc. B 67(2), 301–320 (2005)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Claudia Hauff
    • 1
  • Leif Azzopardi
    • 2
  • Djoerd Hiemstra
    • 1
  1. 1.University of TwenteThe Netherlands
  2. 2.University of GlasgowUnited Kingdom

Personalised recommendations