Query Specific Fusion for Image Retrieval

  • Shaoting Zhang
  • Ming Yang
  • Timothee Cour
  • Kai Yu
  • Dimitris N. Metaxas
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7573)


Recent image retrieval algorithms based on local features indexed by a vocabulary tree and holistic features indexed by compact hashing codes both demonstrate excellent scalability. However, their retrieval precision may vary dramatically among queries. This motivates us to investigate how to fuse the ordered retrieval sets given by multiple retrieval methods, to further enhance the retrieval precision. Thus, we propose a graph-based query specific fusion approach where multiple retrieval sets are merged and reranked by conducting a link analysis on a fused graph. The retrieval quality of an individual method is measured by the consistency of the top candidates’ nearest neighborhoods. Hence, the proposed method is capable of adaptively integrating the strengths of the retrieval methods using local or holistic features for different queries without any supervision. Extensive experiments demonstrate competitive performance on 4 public datasets, i.e., the UKbench, Corel-5K, Holidays and San Francisco Landmarks datasets.


Image Retrieval Query Image Retrieval Method Retrieval Result Rank Aggregation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Lowe, D.G.: Distinctive image features from scale invariant keypoints. Int’l Journal of Computer Vision 60, 91–110 (2004)CrossRefGoogle Scholar
  2. 2.
    Sivic, J., Zisserman, A.: Video google: A text retrieval approach to object matching in videos. In: ICCV (2003)Google Scholar
  3. 3.
    Nistér, D., Stewénius, H.: Scalable recognition with a vocabulary tree. In: CVPR (2006)Google Scholar
  4. 4.
    Oliva, A., Torralba, A.: Modeling the shape of the scene: A holistic representation of the spatial envelope. Int’l Journal of Computer Vision 42, 145–175 (2001)zbMATHCrossRefGoogle Scholar
  5. 5.
    Cai, D., He, X., Han, J.: Spectral regression: a unified subspace learning framework for content-based image retrieval. In: ACM Multimedia (2007)Google Scholar
  6. 6.
    Torralba, A., Fergus, R., Weiss, Y.: Small codes and large image databases for recognition. In: CVPR (2008)Google Scholar
  7. 7.
    Weiss, Y., Torralba, A., Fergus, R.: Spectral hashing. In: NIPS (2008)Google Scholar
  8. 8.
    Gehler, P., Nowozin, S.: On feature combination for multiclass object classification. In: ICCV (2009)Google Scholar
  9. 9.
    Zhang, S., Huang, J., Huang, Y., Yu, Y., Li, H., Metaxas, D.N.: Automatic image annotation using group sparsity. In: CVPR (2010)Google Scholar
  10. 10.
    Fagin, R., Kumar, R., Sivakumar, D.: Efficient similarity search and classification via rank aggregation. In: ACM SIGMOD (2003)Google Scholar
  11. 11.
    Jégou, H., Schmid, C., Harzallah, H., Verbeek, J.: Accurate image search using the contextual dissimilarity measure. IEEE Trans. Pattern Anal. Machine Intell. 32, 2–11 (2010)CrossRefGoogle Scholar
  12. 12.
    Qin, D., Gammeter, S., Bossard, L., Quack, T., van Cool, L.: Hello neighbor: accurate object retrieval with k-reciprocal nearest neighbors. In: CVPR (2011)Google Scholar
  13. 13.
    Jaccard, P.: The distribution of the flora in the alpine zone. New Phytologist 11, 37–50 (1912)CrossRefGoogle Scholar
  14. 14.
    Page, L., Brin, S., Motwani, R., Winograd, T.: The PageRank citation ranking: Bringing order to the web (1999)Google Scholar
  15. 15.
    Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: CVPR (2007)Google Scholar
  16. 16.
    Chum, O., Philbin, J., Sivic, J., Isard, M., Zisserman, A.: Total recall: Automatic query expansion with a generative feature model for object retrieval. In: ICCV (2007)Google Scholar
  17. 17.
    Jégou, H., Douze, M., Schmid, C.: Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 304–317. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  18. 18.
    Wu, Z., Ke, Q., Isard, M., Sun, J.: Bundling feature for large scale partial-duplicated web image search. In: CVPR (2009)Google Scholar
  19. 19.
    Zhou, W., Lu, Y., Li, H., Song, Y., Tian, Q.: Spatial coding for large scale partial-duplicate web image search. In: ACM Multimedia (2010)Google Scholar
  20. 20.
    Zhang, Y., Jia, Z., Chen, T.: Image retrieval with geometry-preserving visual phrases. In: CVPR (2011)Google Scholar
  21. 21.
    Andoni, A., Indyk, P.: Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. In: Symposium on Foundations of Computer Science, FOCS (2006)Google Scholar
  22. 22.
    Wang, J., Kumar, S., Chang, S.F.: Semi-supervised hashing for scalable image retrieval. In: CVPR (2010)Google Scholar
  23. 23.
    Gong, Y., Lazebnik, S.: Iterative quantization: A procrustean approach to learning binary codes. In: CVPR (2011)Google Scholar
  24. 24.
    Jing, Y., Balujia, S.: VisualRank: Applying PageRank to large-scale image search. In: IEEE Trans. Pattern Anal. Machine Intell., vol. 30, pp. 1877–1890 (2008)Google Scholar
  25. 25.
    Richardson, M., Domingos, P.: The intelligent surfer: Probabilistic combination of link and content information in PageRank. In: NIPS, vol. 14, pp. 1441–1448 (2002)Google Scholar
  26. 26.
    Duygulu, P., Barnard, K., de Freitas, J.F.G., Forsyth, D.: Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part IV. LNCS, vol. 2353, pp. 97–112. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  27. 27.
    Chen, D.M., Baatz, G., Köser, K., Tsai, S.S., Vedantham, R., Pylvänäinen, T., Roimela, K., Chen, X., Bach, J., Pollefeys, M., Girod, B., Grzeszczuk, R.: City-scale landmark indentification on mobile devices. In: CVPR (2011)Google Scholar
  28. 28.
    Wang, X., Yang, M., Cour, T., Zhu, S., Yu, K., Han, T.X.: Contextual weighting for vocabulary tree based image retrieval. In: ICCV (2011)Google Scholar
  29. 29.
    Vedaldi, A., Fulkerson, B.: VLFeat: An open and portable library of computer vision algorithms. In: ICME (2010)Google Scholar
  30. 30.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A large-scale hierarchical image database. In: CVPR (2009)Google Scholar
  31. 31.
    Jégou, H., Douze, M., Schmid, C.: On the burstiness of visual elements. In: CVPR (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Shaoting Zhang
    • 2
  • Ming Yang
    • 1
  • Timothee Cour
    • 1
  • Kai Yu
    • 1
  • Dimitris N. Metaxas
    • 2
  1. 1.NEC Laboratories America, Inc.CupertinoUSA
  2. 2.CS Dept.Rutgers UniversityPiscatawayUSA

Personalised recommendations