Image Retrieval and Ranking via Consistently Reconstructing Multi-attribute Queries

  • Xiaochun Cao
  • Hua Zhang
  • Xiaojie Guo
  • Si Liu
  • Xiaowu Chen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8689)


Image retrieval and ranking based on the multi-attribute queries is beneficial to various real world applications. Traditional methods on this problem often utilize intermediate representations generated by attribute classifiers to describe the images, and then the images in the database are sorted according to their similarities to the query. However, such a scheme has two main challenges: 1) how to exploit the correlation between query attributes and non-query attributes, and 2) how to handle noisy representations since the pre-defined attribute classifiers are probably unreliable. To overcome these challenges, we discover the correlation among attributes via expanding the query representation, and imposing the group sparsity on representations to reduce the disturbance of noisy data. Specifically, given a multi-attribute query matrix with each row corresponding to a query attribute and each column the pre-defined attribute, we firstly expand the query based on the correlation of the attributes learned from the training data. Then, the expanded query matrix is reconstructed by the images in the dataset with the ℓ2,1 regularization. Furthermore, we introduce the ranking SVM into the objective function to guarantee the ranking consistency. Finally, we adopt a graph regularization to preserve the local visual similarity among images. Extensive experiments on LFW, CUB-200-2011, and Shoes datasets are conducted to demonstrate the effectiveness of our proposed method.


Multi-Attribute Image Image Retrieval & Ranking Group Sparsity 


  1. 1.
    Berg, T.L., Berg, A.C., Shih, J.: Automatic attribute discovery and characterization from noisy web data. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part I. LNCS, vol. 6311, pp. 663–676. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  2. 2.
    Branson, S., Wah, C., Schroff, F., Babenko, B., Welinder, P., Perona, P., Belongie, S.: Visual recognition with humans in the loop. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part IV. LNCS, vol. 6314, pp. 438–451. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  3. 3.
    Chapelle, O., Le, Q., Smola, A.: Large margin optimization of ranking measures. In: NIPS Workshop on Learning to Rank (2007)Google Scholar
  4. 4.
    Chen, M., Zheng, A., Weinberger, K.: Fast image tagging. In: ICML (2013)Google Scholar
  5. 5.
    Duan, K., Parikh, D., Crandall, D., Grauman, K.: Discovering localized attributes for fine-grained recognition. In: CVPR (2012)Google Scholar
  6. 6.
    Jiang, Z., Lin, Z., Davis, L.: Label consistent k-svd: Learning a discriminative dictionary for recognition. TPAMI 35(11), 2651–2664 (2013)CrossRefGoogle Scholar
  7. 7.
    Joachims, T.: Optimizing search engines using clickthrough data. In: KDD (2002)Google Scholar
  8. 8.
    Kovashka, A., Parikh, D., Grauman, K.: Whittlesearch: Image search with relative attribute feedback. In: CVPR (2012)Google Scholar
  9. 9.
    Kumar, N., Belhumeur, P., Nayar, S.: FaceTracer: A search engine for large collections of images with faces. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part IV. LNCS, vol. 5305, pp. 340–353. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  10. 10.
    Kumar, N., Berg, A.C., Belhumeur, P.N., Nayar, S.K.: Attribute and simile classifiers for face verification. In: ICCV (2009)Google Scholar
  11. 11.
    Lin, Z., Ding, G., Hu, M., Wang, J., Ye, X.: Image tag completion via image-specific and tag-specific linear sparse reconstructions. In: CVPR (2013)Google Scholar
  12. 12.
    Liu, J., Ji, S., Ye, J.: Multi-task feature learning via efficient l2,1-norm minimization. In: UAI (2009)Google Scholar
  13. 13.
    Liu, S., Song, Z., Liu, G., Xu, C., Lu, H., Yan, S.: Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set. In: CVPR (2012)Google Scholar
  14. 14.
    Petterson, J., Caetano, T.: Reverse multi-label learning. In: NIPS (2010)Google Scholar
  15. 15.
    Rastegari, M., Diba, A., Parikh, D.: Multi-attribute queries: To merge or not to merge? In: CVPR (2013)Google Scholar
  16. 16.
    Scheirer, W., Kumar, N., Belhumeur, P.N., Boult, T.E.: Multi-attribute spaces: Calibration for attribute fusion and similarity search. In: CVPR (2012)Google Scholar
  17. 17.
    Siddiquie, B., Feris, R.S., Davis, L.S.: Image ranking and retrieval based on multi-attribute queries. In: CVPR (2011)Google Scholar
  18. 18.
    Tsochantaridis, I., Joachims, T., Hofmann, T., Altun, Y.: Large margin methods for structured and interdependent output variables. JMLR 6, 1453–1484 (2005)zbMATHMathSciNetGoogle Scholar
  19. 19.
    Wang, C., Yan, S., Zhang, L., Zhang, H.J.: Multi-label sparse coding for automatic image annotation. In: CVPR (2009)Google Scholar
  20. 20.
    Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: The caltech-ucsd birds-200-2011 dataset. California Institute of Technology, CNS-TR-2011-001 (2007)Google Scholar
  21. 21.
    Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Ma, Y.: Robust face recognition via sparse representation. TPAMI 31(2), 210–227 (2009)CrossRefGoogle Scholar
  22. 22.
    Wu, L., Jin, R., Jain, A.K.: Tag completion for image retrieval. TPAMI 35(3), 716–727 (2013)CrossRefGoogle Scholar
  23. 23.
    Yu, F., Ji, R., Tsai, M.H., Ye, G., Chang, S.F.: Weak attributes for large-scale image retrieval. In: CVPR (2012)Google Scholar
  24. 24.
    Zhang, M., Zhou, Z.: Ml-knn: A lazy learning approach to multi-label learning. PR 40(7), 2038–2048 (2007)zbMATHGoogle Scholar
  25. 25.
    Zhang, S., Huang, J., Li, H., Metaxas, D.N.: Automatic image annotation and retrieval using group sparsity. TSMC, Part B, 838–849 (2012)Google Scholar
  26. 26.
    Zheng, M., Bu, J., Chen, C., Wang, C., Zhang, L., Qiu, G., Cai, D.: Graph regularized sparse coding for image representation. TIP 20(5), 1327–1336 (2011)MathSciNetGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Xiaochun Cao
    • 1
    • 2
  • Hua Zhang
    • 1
  • Xiaojie Guo
    • 2
  • Si Liu
    • 3
  • Xiaowu Chen
    • 4
  1. 1.School of Computer Science and TechnologyTianjin UniversityTianjinChina
  2. 2.State Key Laboratory of Information SecurityIIE, Chinese Academy of SciencesChina
  3. 3.Department of Electrical & Computer EngineeringNational University of SingaporeSingapore
  4. 4.State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and EngineeringBeihang UniversityBeijingChina

Personalised recommendations