# On rank-based effectiveness measures and optimization

Article

First Online:

Received:

Accepted:

- 151 Downloads
- 11 Citations

## Abstract

Many current retrieval models and scoring functions contain free parameters which need to be set—ideally, optimized. The process of optimization normally involves some training corpus of the usual document-query-relevance judgement type, and some choice of measure that is to be optimized. The paper proposes a way to think about the process of exploring the space of parameter values, and how moving around in this space might be expected to affect different measures. One result, concerning local optima, is demonstrated for a range of rank-based evaluation measures.

### Keywords

Effectiveness metrics Ranking functions Optimization## Notes

### Acknowledgements

Thanks to Chris Burges, Michael Taylor and Nick Craswell for many discussions on optimization issues.

### References

- Bartell, B. (1994).
*Optimizing ranking functions: A connectionist approach to adaptive information retrieval*, Technical report, PhD thesis, University of California, San Diego.Google Scholar - Belkin, N. J., Ingwersen, P., & Leong, M.-K., (Eds.) (2000). In
*SIGIR 2000: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval*. New York: ACM.Google Scholar - Buckley, C., & Voorhees, E. (2000). Evaluating evaluation measure stability. In Belkin et al. (2000) (pp. 33–40).Google Scholar
- Burges, C. J. C. (2005). Ranking as learning structured outputs. In S. Agarwal et al. (Ed.),
*Proceedings of the NIPS 2005 Workshop on Learning to Rank*.Google Scholar - Burges, C. J. C., Shaked, T., Renshaw, E. et al. (2005). Learning to rank using gradient descent. In
*Proceedings of the 22nd International Conference on Machine Learning*, Bonn.Google Scholar - Cooper, W. S. (1968). Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems.
*American Documentation, 19*, 30–41.Google Scholar - Cooper, W. S., Chen, A., & Gey, F. C. (1994). Full text retrieval based on probabilistic equations with coefficients fitted by logistic regression. In D. K. Harman (Ed.),
*The Second Text REtrieval Conference (TREC–2)*(pp. 57–66). NIST Special Publication 500-215, Gaithersburg, MD: NIST.Google Scholar - Freund, Y., Iyer, R., Schapire, R., & Singer, Y. (2003). An efficient boosting algorithm for combining preferences.
*Journal of Machine Learning Research, 4*, 933–969.CrossRefMathSciNetGoogle Scholar - Herbrich, R., Graepel, T., & Obermayer, K. (2000). Large margin rank boundaries for ordinal regression. In
*Advances in Large Margin Classifiers*(pp. 115–132). MIT Press.Google Scholar - Järvelin, K., & Kekäläinen, J. (2000). IR evaluation methods for retrieving highly relevant documents. In Belkin et al. (2000) (pp. 41–48).Google Scholar
- Kazai, G., Lalmas, M., & de Vries, A. P. (2004). The overlap problem in content-oriented XML retrieval evaluation. In K. Järvelin, J. Allan, P. Bruza & M. Sanderson (Eds.),
*SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval*(pp. 72–79). New York: ACM Press.Google Scholar - Mitchell, M. (1996).
*An introduction to genetic algorithms*. Cambridge, MA: MIT Press.Google Scholar - Press, W. H., Teukolsky, S. A., Vettering, W. T., & Flannery, B. P. (2002).
*Numerical recipes in C++. The art of scientific computing*, 2nd ed. Cambridge University Press.Google Scholar - Robertson, S., & Soboroff, I. (2002). The TREC 2001 filtering track report. In E. M. Voorhees & D. K. Harman (Eds.),
*The Tenth Text REtrieval Conference, TREC 2001*(pp. 26–37). NIST Special Publication 500-250, Gaithersburg, MD: NIST.Google Scholar - Robertson, S. E. (2002). Threshold setting and performance optimization in adaptive filtering.
*Information Retrieval, 5*, 239–256.CrossRefGoogle Scholar - Swets, J. A. (1963). Information retrieval systems.
*Science, 141*(3577), 245– 250.CrossRefGoogle Scholar - Voorhees, E. M. (2006). Overview of the TREC 2005 robust retrieval track. In E. M. Voorhees & L. P. Buckland (Eds.),
*The Fourteenth Text Retrieval Conference, TREC 2005*. Gaithersburg, MD: NIST.Google Scholar

## Copyright information

© Springer Science+Business Media, LLC 2007