Journal of Signal Processing Systems

, Volume 74, Issue 3, pp 285–295 | Cite as

An Efficient Gradient-based Approach to Optimizing Average Precision Through Maximal Figure-of-Merit Learning

  • Ilseo Kim
  • Chin-Hui Lee


We propose an efficient algorithm that directly optimizes a ranking performance measure, with a focus on class average precision (AP). Instead of using pair-wise ranking approximation in defining a loss function by conventional approaches, we use an efficient gradient-based approach that approximates a discrete ranking performance measure. In particular, AP is considered as a staircase function with respect to each individual sample score after rank ordering is applied to all samples. Then, a combination of sigmoid functions is applied to approximate the staircase AP function as a continuous and differntiable function of the model parameters used to compute the sample scores. Compared to the use of pair-wise rankings, the proposed approach substantially reduces the computational complexity to a manageable level when estimating model parameters with a gradient descent algorithm. In terms of explicitly optimizing a target performance metric, the proposed algorithm can be considered as an extension of maximal figure-of-merit (MFoM) learning to optimization of a ranking performance measure. Our experiments on two challenging image-retrieval datasets showcased the usefulness of the proposed framework in both improving AP and achieving learning efficiency.


Average precision Automatic image annotation Maximal figure-of-merit MFoM Ranking optimization 


  1. 1.
    Toderici, G., Aradhye, H., Pasca, M., Sbaiz, L., Yagnik, J. (2010). Finding meaning on youtube: tag recommendation and category discovery. In CVPR.Google Scholar
  2. 2.
    Jiang, Y.-G., Ye, G., Chang, S.-F., Ellis, D., Loui, A.C. (2011). Consumer video understanding: a benchmark database and an evaluation of human and machine performance. In ACM ICMR.Google Scholar
  3. 3.
    Smeaton, A.F., Over, P., Kraaij, W. (2006). Evaluation campaigns and trecvid. In ACM MIR.Google Scholar
  4. 4.
    Wang, Z., Zhao, M., Song, Y., Kumar, S., Li, B. (2010). YouTubeCat: learning to categorize wild web videos. In CVPR.Google Scholar
  5. 5.
    Over, P., Awad, G., Fiscus, J., Michel, M., Smeaton, A.F., Kraaij, W. (2009). TRECVID 2009-goals, tasks, data, evaluation mechanisms and metrics. In Proceedings of TRECVID workshop.Google Scholar
  6. 6.
    Gao, S., Wu, W., Lee, C.-H., Chua, T.-S. (2004). A MFoM learning to robust multiclass multi-label text categorization. In Proceedings of ICML.Google Scholar
  7. 7.
    Katagiri, S., Juang, B.-H., Lee, C.-H. (1998). Pattern recognition using a family of design algorithm based upon the generalized probabilistic descent method. In Proceedings of the IEEE (pp. 2345–2373).Google Scholar
  8. 8.
    Joachims, T. (2002). Optimizing search engines using clickthrough data. In Proceedings of KDD.Google Scholar
  9. 9.
    Freund, Y., Iyer, R., Schapire, R.E., Singer, Y. (2003). An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4, 933–969.MathSciNetGoogle Scholar
  10. 10.
    Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., Hullender, G. (2005). Learning to rank using gradient descent. In Proceedings of ICML.Google Scholar
  11. 11.
    Gao, S., Lee, C.-H., Lim, J.H. (2006). An ensemble classifier learning approach to ROC optimization. In Proceedings of ICPR.Google Scholar
  12. 12.
    Gao, S., & Sun, Q. (2007). Improving semantic concept detection through optimizing ranking function. IEEE Transactions on Multimedia, 9, 1430–1442.CrossRefGoogle Scholar
  13. 13.
    Fawcett, T. (2006). Introduction to roc analysis. Pattern Recognition Letters, 27, 861–874.CrossRefGoogle Scholar
  14. 14.
    McFee, B. (2010). Metric learning to rank. In Proceedings of ICML.Google Scholar
  15. 15.
    Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., Li, H. (2007). Learning to rank: from pairwise approach to listwise approach. In Proceedings of ICML.Google Scholar
  16. 16.
    Xia, F., Liu, T.-Y., Wang, J., Zhang, W., Li, H. (2008). Listwise approach to learning to rank—theory and algorithm. In Proceedings of ICML.Google Scholar
  17. 17.
    Ma, C., & Lee, C.-H. (2008). An efficient gradient computation approach to discriminative fusion optimization in semantic concept detection. In Proceedings of ICPR.Google Scholar
  18. 18.
    Wang, L., Lin, J., Metzler, D. (2010). Learning to efficiently rank. In Proceedings of ACM SIGIR (pp. 138–145).Google Scholar
  19. 19.
    Cambazoglu, B.B., Zaragoza, H., Chapelle, O., Chen, J., Liao, C., Zheng, Z., Degenhardt, J. (2010). Early exit optimizations for additive machine learned ranking systems. In Proceedings of ACM WSDM.Google Scholar
  20. 20.
    Valizadegan, H., Jin, R., Zhang, R., Mao, J. (2009). Learning to rank by optimizing NDCG measure. In Proceedings of NIPS.Google Scholar
  21. 21.
    Gao, S., Wu, W., Lee, C.-H., Chua, T.-S. (2003). A maximal figure-of-merit approach to text categorization. In Proceedings of ACM SIGIR.Google Scholar
  22. 22.
    Kim, I., Oh, S., Byun, B., Perera, A.G.A., Lee, C.-H. (2012). Explicit performance metric optimization for fusion-based video retrieval. In Proceedings of ECCV workshops.Google Scholar
  23. 23.
    Yan, L., Dodier, R., Mozer, M.C., Wolniewicz, R. (2003). Optimizing classifier performance via an approximation to the Wilcoxon-Mann-Whitney statistic. In Proceedings of ICML.Google Scholar
  24. 24.
    Watanabe, H., Tokuno, J., Ohashi, T., Katagiri, S., Ohsaki, M. (2011). Minimum classification error training with automatic setting of loss smoothness. In MLSP.Google Scholar
  25. 25.
    Sun, W., & Yuan, Y.-X. (2006). Optimization theory and methods: nonlinear programming (pp. 102–117). New York: Springer.Google Scholar
  26. 26.
    Kennedy, L., & Hauptmann, A. (2006). LSCOM lexicon definition and annotation version 1.0. In Proceedings of DTO challenge workshop on large scale concept ontology for multimedia.Google Scholar
  27. 27.
    Over, P. (2006). Guidelines for the TRECVID 2006 evaluation. In Accessed 15 March 2012.
  28. 28.
    Gao, S., Wang, D.-H., Lee, C.-H. (2006). Automatic image annotation through multi-topic text categorization. In Proceedings of ICASSP.Google Scholar
  29. 29.
    Bellegarda, J.R. Exploiting latent semantic information in statistical language modeling (pp. 1279–1296).Google Scholar
  30. 30.
    Swain, M.J., & Ballard, D.H. (1991). Color indexing. International Journal of Computer Vision, 7, 11–32.CrossRefGoogle Scholar
  31. 31.
    Zhang, D., Chen, X., Lee, W.S. (2005). Text classification with kernels on the multinomial manifold. In Proceedings of Special Interest Group on Information Retrieval.Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Georgia Institute of TechnologyAtlantaUSA

Personalised recommendations