Information Retrieval

, Volume 13, Issue 4, pp 375–397 | Cite as

A general approximation framework for direct optimization of information retrieval measures

Article

Abstract

Recently direct optimization of information retrieval (IR) measures has become a new trend in learning to rank. In this paper, we propose a general framework for direct optimization of IR measures, which enjoys several theoretical advantages. The general framework, which can be used to optimize most IR measures, addresses the task by approximating the IR measures and optimizing the approximated surrogate functions. Theoretical analysis shows that a high approximation accuracy can be achieved by the framework. We take average precision (AP) and normalized discounted cumulated gains (NDCG) as examples to demonstrate how to realize the proposed framework. Experiments on benchmark datasets show that the algorithms deduced from our framework are very effective when compared to existing methods. The empirical results also agree well with the theoretical results obtained in the paper.

Keywords

Learning to rank Direct optimization of IR measures Position function approximation Truncation function approximation Accuracy analysis 

References

  1. Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., et al. (2005). Learning to rank using gradient descent. In ICML ’05: Proceedings of the 22nd international conference on Machine learning (pp. 89–96). New York, NY, USA: ACM Press.Google Scholar
  2. Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., & Li, H. (2007). Learning to rank: From pairwise approach to listwise approach. In ICML ’07: Proceedings of the 24th international conference on Machine learning (pp. 129–136). New York, NY, USA: ACM Press.Google Scholar
  3. Chakrabarti, S., Khanna, R., Sawant, U., & Bhattacharyya, C. (2008). Structured learning for non-smooth ranking losses. In KDD ’08: Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 88–96). New York, NY, USA: ACM.Google Scholar
  4. Chapelle, O., Le, Q., & Smola, A. (2007). Large margin optimization of ranking measures. In NIPS2007 workshop on machine learning for Web search.Google Scholar
  5. Duh, K., & Kirchhoff, K. (2008). Learning to rank with partially-labeled data. In SIGIR ’08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval (pp. 251–258). New York, NY, USA: ACM.Google Scholar
  6. Freund, Y., Iyer, R., Schapire, R. E., & Singer, Y. (2003). An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4, 933–969.Google Scholar
  7. Guiver, J., & Snelson, E. (2008). Learning to rank with softrank and gaussian processes. In SIGIR ’08: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval (pp. 259–266). New York, NY, USA: ACM.Google Scholar
  8. He, Y., & Liu, T.-Y. (2008). Are algorithms directly optimizing ir measures really direct? Technical Report MSR-TR-2008-154, Microsoft Corporation.Google Scholar
  9. Herbrich, R., Graepel, T., & Obermayer K. (1999). Support vector learning for ordinal regression. In ICANN1999 (pp. 97–102).Google Scholar
  10. Hu, X., Shonkwiller, R., & Spruill, M. (1994). Random restarts in global optimization. Technical Report, School of Mathematics, Georgia Institute of Technology, Atlanta.Google Scholar
  11. Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems, 20(4), 422–446.CrossRefGoogle Scholar
  12. Joachims T. (2002). Optimizing search engines using clickthrough data. In KDD ’02: Proceedings of the 8th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 133–142). New York, NY, USA: ACM Press.Google Scholar
  13. Kirkpatrick, S., Gelatt, C., & Vecchi, M. (1983). Optimization by simulated annealing. Science 220(4598), 671–680.CrossRefMathSciNetGoogle Scholar
  14. Liu, T.-Y., Xu, J., Qin, T., Xiong, W.-Y., & Li, H. (2007). Letor: benchmark dataset for research on learning to rank for information retrieval. In Proceedings of SIGIR 2007 workshop on learning to rank for information retrieval.Google Scholar
  15. Moffat, A., & Zobel, J. (2008). Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems, 27(1), 1–27.CrossRefGoogle Scholar
  16. Qin, T., Liu, T.-Y., & Li, H. (2008a). A general approximation framework for direct optimization of information retrieval measures. Technical Report MSR-TR-2008-164, Microsoft Corporation.Google Scholar
  17. Qin, T., Liu, T.-Y., Zhang, X.-D., Wang, D.-S., Xiong, W.-Y., & Li, H. (2008b). Learning to rank relational objects and its application to web search. In WWW ’08: Proceeding of the 17th international conference on World Wide Web (pp. 407–416). New York, NY, USA: ACM.Google Scholar
  18. Qin, T., Zhang, X.-D., Tsai, M.-F., Wang, D.-S., Liu, T.-Y., & Li, H. (2008c). Query-level loss functions for information retrieval. Information Processing & Management, 44(2), 838–855.CrossRefGoogle Scholar
  19. Robertson S. E., & Hull D. A. (2000). The TREC-9 filtering track final report. In TREC (pp. 25–40).Google Scholar
  20. Taylor, M., Guiver, J., Robertson, S., & Minka, T. (2008). Softrank: optimizing non-smooth rank metrics. In WSDM ’08: Proceedings of the international conference on Web search and web data mining (pp. 77–86). New York, NY, USA: ACM.Google Scholar
  21. Tsai, M.-F., Liu, T.-Y, Qin, T., Chen, H.-H., & Ma, W.-Y. (2007). Frank: a ranking method with fidelity loss. In SIGIR ’07 (pp. 383–390). New York, NY, USA: ACM Press.Google Scholar
  22. Vapnik, V. (1998). Statistical learning theory. New York: Wiley.Google Scholar
  23. Volkovs, M. N., & Zemel, R. S. (2009). Boltzrank: Learning to maximize expected ranking gain. In ICML ’09: Proceedings of the 26th annual international conference on machine learning (pp. 1089–1096). New York, NY, USA: ACM.Google Scholar
  24. Voorhees, E., & Harman, D. (2005). TREC: Experiment and evaluation in information retrieval. Cambridge: MIT Press.Google Scholar
  25. Xia, F., Liu, T.-Y., Wang, J., Zhang, W., & Li, H. (2008). Listwise approach to learning to rank: Theory and algorithm. In ICML ’08: Proceedings of the 25th international conference on machine learning (pp. 1192–1199). New York, NY, USA: ACM.Google Scholar
  26. Xu, J., & Li, H. (2007). Adarank: A boosting algorithm for information retrieval. In SIGIR ’07: Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval (pp. 391–398). New York, NY, USA: ACM Press.Google Scholar
  27. Xu, J., Liu, T.-Y., Lu, M., Li, H., & Ma, W.-Y. (2008). Directly optimizing evaluation measures in learning to rank. In SIGIR ’08: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval (pp. 107–114). New York, NY, USA: ACM.Google Scholar
  28. Yue, Y., Finley, T., Radlinski, F., & Joachims, T. (2007). A support vector method for optimizing average precision. In SIGIR ’07: Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval (pp. 271–278). New York, NY, USA: ACM Press.Google Scholar
  29. Zhai, C., & Lafferty, J. (2001). A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval (pp. 334–342). New York, NY, USA: ACM Press.Google Scholar
  30. Zheng, Z., Chen, K., Sun, G., & Zha, H. (2007). A regression framework for learning ranking functions using relative relevance judgments. In SIGIR ’07: Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval (pp. 287–294). New York, NY, USA: ACM.Google Scholar
  31. Zhou, K., Xue, G.-R., Zha, H., & Yu, Y. (2008). Learning to rank with ties. In SIGIR ’08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval (pp. 275–282). New York, NY, USA: ACM.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.Microsoft Research AsiaBeijingChina

Personalised recommendations