Multimedia Tools and Applications

, Volume 60, Issue 1, pp 161–180 | Cite as

A Learning to Rank framework applied to text-image retrieval

  • David Buffoni
  • Sabrina Tollari
  • Patrick Gallinari


We present a framework based on a Learning to Rank setting for a text-image retrieval task. In Information Retrieval, the goal is to compute the similarity between a document and an user query. In the context of text-image retrieval where several similarities exist, human intervention is often needed to decide on the way to combine them. On the other hand, with the Learning to Rank approach the combination of the similarities is done automatically. Learning to Rank is a paradigm where the learnt objective function is able to produce a ranked list of images when a user query is given. These score functions are generally a combination of similarities between a document and a query. In the past, Learning to Rank algorithms were successfully applied to text retrieval where they outperformed baselines such as BM25 or TFIDF. This inspired us to apply our state-of-the-art algorithm, called OWPC (Usunier et al. 2009), to the text-image retrieval task. At this time, no benchmarks are available, therefore we present a framework for building one. The empirical validation of this algorithm is done on the dataset constructed through comparison of typical text-image retrieval similarities. In both cases, visual only and text and visual, our algorithm performs better than a simple baseline.


Learning to Rank Text-image retrieval OWPC Visuo-textual fusion Pooling for Learning to Rank 



This work was partially supported by the French National Agency of Research (ANR-06-MDCA-002 AVEIR project).


  1. 1.
    Aslam JA, Kanoulas E, Pavlu V, Savev S, Yilmaz E (2009) Document selection methodologies for efficient and effective learning-to-rank. In: SIGIR ’09: proceedings of the 32nd international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, USA, pp 468–475CrossRefGoogle Scholar
  2. 2.
    Burges CJC, Ragno R, Le QV (2006) Learning to rank with nonsmooth cost functions. In: NIPS, pp 193–200Google Scholar
  3. 3.
    Burges CJC, Shaked T, Renshaw E, Lazier A, Deeds M, Hamilton N, Hullender GN (2005) Learning to rank using gradient descent. In: ICML, pp 89–96Google Scholar
  4. 4.
    Cao Y, Xu J, Liu T-Y, Li H, Huang Y, Hon H-W (2006) Adapting ranking svm to document retrieval. In: SIGIR, pp 186–193Google Scholar
  5. 5.
    Cao Z, Qin T, Liu T-Y, Tsai M-F, Li H (2007) Learning to rank: from pairwise approach to listwise approach. In: ICML, pp 129–136Google Scholar
  6. 6.
    La Cascia M, Sethi S, Sclaroff S (1998) Combining textual and visual cues for content-based image retrieval on the world wide web. In: In IEEE workshop on content-based access of image and video libraries, pp 24–28Google Scholar
  7. 7.
    Clough P, Grubinger M, Deselaers T, Hanbury A, Müller H (2006) Overview of the imageclef 2006 photographic retrieval and object annotation tasks. In: CLEF, pp 579–594Google Scholar
  8. 8.
    Cohen WW, Schapire RE, Singer Y (1997) Learning to order things. In: NIPSGoogle Scholar
  9. 9.
    Cossock D, Zhang T (2006) Subset ranking using regression. In: COLT, pp 605–619Google Scholar
  10. 10.
    Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2):1–60CrossRefGoogle Scholar
  11. 11.
    Faria FF, Veloso A, Almeida HM, Valle E, da Silva Torres R, Gonçalves MA, Meira Jr W (2010) Learning to rank for content-based image retrieval. In: MIR ’10: proceedings of the international conference on multimedia information retrieval. ACM, New York, NY, USA, pp 285–294CrossRefGoogle Scholar
  12. 12.
    Freund Y, Iyer R, Schapire RE, Singer Y (2003) An efficient boosting algorithm for combining preferences. JMLR 4:933–969MathSciNetGoogle Scholar
  13. 13.
    Grubinger M, Clough PD, Müller H, Deselaers T (2006) The iapr benchmark: a new evaluation resource for visual information systems. In: International conference on language resources and evaluationGoogle Scholar
  14. 14.
    Har-Peled S, Roth D, Zimak D (2002) Constraint classification for multiclass classification and ranking. In: NIPS, pp 785–792Google Scholar
  15. 15.
    Hu Y, Li MJ, Yu N (2008) Multiple-instance ranking: learning to rank images for image retrieval. In: CVPR08, pp 1–8Google Scholar
  16. 16.
    Järvelin K, Kekäläinen J (2000) Ir evaluation methods for retrieving highly relevant documents. In: SIGIR. ACM, New York, NY, USA, pp 41–48CrossRefGoogle Scholar
  17. 17.
    Joachims T (2002) Optimizing search engines using clickthrough data. In: KDD, pp 133–142Google Scholar
  18. 18.
    Li M, Zheng Y-T, Lin S-X, Zhang Y-D, Chua T-S (2008) Multimedia evidence fusion for video concept detection via owa operator. In: MMM ’09: proceedings of the 15th international multimedia modeling conference on advances in multimedia modeling. Springer, Berlin, Heidelberg, pp 208–216Google Scholar
  19. 19.
    Porter MF (1980) An algorithm for suffix stripping. Program 14(3):130–137CrossRefGoogle Scholar
  20. 20.
    Robertson SE, Walker S, Hancock-Beaulieu M, Gull A, Lau M (1992) Okapi at trec. In: TREC, pp 21–30Google Scholar
  21. 21.
    Rui Y, Huang T (2000) Optimizing learning in image retrieval. In: CVPR, pp 236–243Google Scholar
  22. 22.
    Taylor M, Guiver J, Robertson S, Minka T (2008) Softrank: optimizing non-smooth rank metrics. In: WSDM ’08. ACM, pp 77–86Google Scholar
  23. 23.
    Tollari S, Detyniecki M, Fakeri-Tabrizi A, Marsala C, Amini M-R, Gallinari P (2008) Using visual concepts and fast visual diversity to improve image retrieval. In: Peters C, Deselaers T, Ferro N, Gonzalo J, Jones GJF, Kurimo M, Mandl T, Peñas A, Petras V (eds) CLEF. Lecture notes in computer science, vol 5706. Springer, pp 577–584Google Scholar
  24. 24.
    Tollari S, Glotin H (2007) Web image retrieval on imageval: evidences on visualness and textualness concept dependency in fusion model. In: ACM international conference on image and video retrieval (ACM CIVR)Google Scholar
  25. 25.
    Tollari S, Glotin H (2008) Learning optimal visual features from web sampling in online image retrieval. In: IEEE international conference on acoustics, speech and signal processing (ICASSP)Google Scholar
  26. 26.
    Tong S, Chang E (2001) Support vector machine active learning for image retrieval. In: MULTIMEDIA ’01: proceedings of the ninth ACM international conference on multimedia. ACM, New York, NY, USA, pp 107–118CrossRefGoogle Scholar
  27. 27.
    Tsochantaridis I, Joachims T, Hofmann T, Altun Y (2005) Large margin methods for structured and interdependent output variables. J Mach Learn Res 6:1453–1484MathSciNetzbMATHGoogle Scholar
  28. 28.
    Usunier N, Buffoni D, Gallinari P (2009) Ranking with ordered weighted pairwise classification. In: Danyluk AP, Bottou L, Littman ML (eds) ICML. ACM international conference proceeding series, vol 382. ACM, p 133Google Scholar
  29. 29.
    Xu J, Li H (2007) Adarank: a boosting algorithm for information retrieval. In: SIGIR, pp 391–398Google Scholar
  30. 30.
    Xu J, Liu T-Y, Lu M, Li H, Ma W-Y (2008) Directly optimizing evaluation measures in learning to rank. In: SIGIR, pp 107–114Google Scholar
  31. 31.
    Yager RR (1988) On ordered weighted averaging aggregation operators in multi-criteria decision making. IEEE Trans Syst Man Cybern 18:183–190MathSciNetzbMATHCrossRefGoogle Scholar
  32. 32.
    Yates RB, Ribeiro-Neto B (1999) Modern information retrieval. Addison WesleyGoogle Scholar
  33. 33.
    Yue Y, Finley T, Radlinski F, Joachims T (2007) A support vector method for optimizing average precision. In: SIGIR, pp 271–278Google Scholar
  34. 34.
    Zhai C, Lafferty J (2004) A study of smoothing methods for language models applied to information retrieval. ACM Trans Inf Syst 22(2):179–214CrossRefGoogle Scholar
  35. 35.
    Zhou XS, Huang TS (2002) Unifying keywords and visual contents in image retrieval. IEEE Multimed 9(2):23–33MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • David Buffoni
    • 1
  • Sabrina Tollari
    • 1
  • Patrick Gallinari
    • 1
  1. 1.Université Pierre et Marie CURIE - Paris 6 / LIP6ParisFrance

Personalised recommendations