Information Retrieval

, Volume 10, Issue 3, pp 321–339 | Cite as

On rank-based effectiveness measures and optimization



Many current retrieval models and scoring functions contain free parameters which need to be set—ideally, optimized. The process of optimization normally involves some training corpus of the usual document-query-relevance judgement type, and some choice of measure that is to be optimized. The paper proposes a way to think about the process of exploring the space of parameter values, and how moving around in this space might be expected to affect different measures. One result, concerning local optima, is demonstrated for a range of rank-based evaluation measures.


Effectiveness metrics Ranking functions Optimization 



Thanks to Chris Burges, Michael Taylor and Nick Craswell for many discussions on optimization issues.


  1. Bartell, B. (1994). Optimizing ranking functions: A connectionist approach to adaptive information retrieval, Technical report, PhD thesis, University of California, San Diego.Google Scholar
  2. Belkin, N. J., Ingwersen, P., & Leong, M.-K., (Eds.) (2000). In SIGIR 2000: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM.Google Scholar
  3. Buckley, C., & Voorhees, E. (2000). Evaluating evaluation measure stability. In Belkin et al. (2000) (pp. 33–40).Google Scholar
  4. Burges, C. J. C. (2005). Ranking as learning structured outputs. In S. Agarwal et al. (Ed.), Proceedings of the NIPS 2005 Workshop on Learning to Rank.Google Scholar
  5. Burges, C. J. C., Shaked, T., Renshaw, E. et al. (2005). Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine Learning, Bonn.Google Scholar
  6. Cooper, W. S. (1968). Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. American Documentation, 19, 30–41.Google Scholar
  7. Cooper, W. S., Chen, A., & Gey, F. C. (1994). Full text retrieval based on probabilistic equations with coefficients fitted by logistic regression. In D. K. Harman (Ed.), The Second Text REtrieval Conference (TREC–2) (pp. 57–66). NIST Special Publication 500-215, Gaithersburg, MD: NIST.Google Scholar
  8. Freund, Y., Iyer, R., Schapire, R., & Singer, Y. (2003). An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4, 933–969.CrossRefMathSciNetGoogle Scholar
  9. Herbrich, R., Graepel, T., & Obermayer, K. (2000). Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers (pp. 115–132). MIT Press.Google Scholar
  10. Järvelin, K., & Kekäläinen, J. (2000). IR evaluation methods for retrieving highly relevant documents. In Belkin et al. (2000) (pp. 41–48).Google Scholar
  11. Kazai, G., Lalmas, M., & de Vries, A. P. (2004). The overlap problem in content-oriented XML retrieval evaluation. In K. Järvelin, J. Allan, P. Bruza & M. Sanderson (Eds.), SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 72–79). New York: ACM Press.Google Scholar
  12. Mitchell, M. (1996). An introduction to genetic algorithms. Cambridge, MA: MIT Press.Google Scholar
  13. Press, W. H., Teukolsky, S. A., Vettering, W. T., & Flannery, B. P. (2002). Numerical recipes in C++. The art of scientific computing, 2nd ed. Cambridge University Press.Google Scholar
  14. Robertson, S., & Soboroff, I. (2002). The TREC 2001 filtering track report. In E. M. Voorhees & D. K. Harman (Eds.), The Tenth Text REtrieval Conference, TREC 2001 (pp. 26–37). NIST Special Publication 500-250, Gaithersburg, MD: NIST.Google Scholar
  15. Robertson, S. E. (2002). Threshold setting and performance optimization in adaptive filtering. Information Retrieval, 5, 239–256.CrossRefGoogle Scholar
  16. Swets, J. A. (1963). Information retrieval systems. Science, 141(3577), 245– 250.CrossRefGoogle Scholar
  17. Voorhees, E. M. (2006). Overview of the TREC 2005 robust retrieval track. In E. M. Voorhees & L. P. Buckland (Eds.), The Fourteenth Text Retrieval Conference, TREC 2005. Gaithersburg, MD: NIST.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Microsoft ResearchCambridgeUK
  2. 2.Yahoo! ResearchBarcelonaSpain

Personalised recommendations