Query Performance Prediction: Evaluation Contrasted with Effectiveness

  • Claudia Hauff
  • Leif Azzopardi
  • Djoerd Hiemstra
  • Franciska de Jong
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5993)


Query performance predictors are commonly evaluated by reporting correlation coefficients to denote how well the methods perform at predicting the retrieval performance of a set of queries. Despite the amount of research dedicated to this area, one aspect remains neglected: how strong does the correlation need to be in order to realize an improvement in retrieval effectiveness in an operational setting? We address this issue in the context of two settings: Selective Query Expansion and Meta-Search. In an empirical study, we control the quality of a predictor in order to examine how the strength of the correlation achieved, affects the effectiveness of an adaptive retrieval system. The results of this study show that many existing predictors fail to achieve a correlation strong enough to reliably improve the retrieval effectiveness in the Selective Query Expansion as well as the Meta-Search setting.


Retrieval Performance Query Term Query Expansion Mean Average Precision Query Performance 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Amati, G., Carpineto, C., Romano, G.: Query difficulty, robustness and selective application of query expansion. In: McDonald, S., Tait, J.I. (eds.) ECIR 2004. LNCS, vol. 2997, pp. 127–137. Springer, Heidelberg (2004)Google Scholar
  2. 2.
    Billerbeck, B., Zobel, J.: When query expansion fails. In: SIGIR 2003, pp. 387–388 (2003)Google Scholar
  3. 3.
    Carmel, D., Farchi, E., Petruschka, Y., Soffer, A.: Automatic query refinement using lexical affinities with maximal information gain. In: SIGIR 2002, pp. 283–290 (2002)Google Scholar
  4. 4.
    Cronen-Townsend, S., Zhou, Y., Croft, W.B.: Predicting query performance. In: SIGIR 2002, pp. 299–306 (2002)Google Scholar
  5. 5.
    Cronen-Townsend, S., Zhou, Y., Croft, W.B.: A framework for selective query expansion. In: CIKM 2004, pp. 236–237 (2004)Google Scholar
  6. 6.
    Diaz, F.: Performance prediction using spatial autocorrelation. In: SIGIR 2007, pp. 583–590 (2007)Google Scholar
  7. 7.
    Hauff, C., Azzopardi, L.: When is query performance prediction effective? In: SIGIR 2009, pp. 829–830 (2009)Google Scholar
  8. 8.
    He, B., Ounis, I.: Inferring query performance using pre-retrieval predictors. In: Apostolico, A., Melucci, M. (eds.) SPIRE 2004. LNCS, vol. 3246, pp. 43–54. Springer, Heidelberg (2004)Google Scholar
  9. 9.
    He, B., Ounis, I.: Combining fields for query expansion and adaptive query expansion. Inf. Process. Manag. 43(5), 1294–1307 (2007)CrossRefGoogle Scholar
  10. 10.
    Kwok, K.: An attempt to identify weakest and strongest queries. In: SIGIR 2005 Query Prediction Workshop (2005)Google Scholar
  11. 11.
    Lavrenko, V., Croft, W.B.: Relevance based language models. In: SIGIR 2001, pp. 120–127 (2001)Google Scholar
  12. 12.
    Mitra, M., Singhal, A., Buckley, C.: Improving automatic query expansion. In: SIGIR 1998, pp. 206–214 (1998)Google Scholar
  13. 13.
    Vinay, V., Cox, I.J., Milic-Frayling, N., Wood, K.: On ranking the effectiveness of searches. In: SIGIR 2006, pp. 398–404 (2006)Google Scholar
  14. 14.
    Winaver, M., Kurland, O., Domshlak, C.: Towards robust query expansion: model selection in the language modeling framework. In: SIGIR 2007, pp. 729–730 (2007)Google Scholar
  15. 15.
    Wu, S., Crestani, F.: Data fusion with estimated weights. In: CIKM 2002, pp. 648–651 (2002)Google Scholar
  16. 16.
    Xu, J., Croft, W.B.: Query expansion using local and global document analysis. In: SIGIR 1996, pp. 4–11 (1996)Google Scholar
  17. 17.
    Xu, J., Croft, W.B.: Improving the effectiveness of information retrieval with local context analysis. ACM Trans. Inf. Syst. 18(1), 79–112 (2000)CrossRefGoogle Scholar
  18. 18.
    Yom-Tov, E., Fine, S., Carmel, D., Darlow, A.: Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval. In: SIGIR 2005, pp. 512–519 (2005)Google Scholar
  19. 19.
    Zhao, Y., Scholer, F., Tsegay, Y.: Effective pre-retrieval query performance prediction using similarity and variability evidence. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, pp. 52–64. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  20. 20.
    Zhou, Y., Croft, W.B.: Ranking robustness: a novel framework to predict query performance. In: CIKM 2006, pp. 567–574 (2006)Google Scholar
  21. 21.
    Zhou, Y., Croft, W.B.: Query performance prediction in web search environments. In: SIGIR 2007, pp. 543–550 (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Claudia Hauff
    • 1
  • Leif Azzopardi
    • 2
  • Djoerd Hiemstra
    • 1
  • Franciska de Jong
    • 1
  1. 1.University of TwenteEnschedeThe Netherlands
  2. 2.University of GlasgowGlasgowUK

Personalised recommendations