Retrieving Images by Multiple Samples via Fusing Deep Features

  • Kecai WuEmail author
  • Xueliang Liu
  • Jie Shao
  • Richang Hong
  • Tao Yang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9916)


Most existing image retrieval systems search similar images on a given single input, while querying based on multiple images is not a trivial. In this paper, we describe a novel image retrieval paradigm that users could input two images as query to search the images that include the content of the two input images-synchronously. In our solution, the deep CNN feature is extracted from each single query image and then fused as the query feature. Due to the role of the two query images is different and changeable, we propose the FWC (Feature weighting by Clustering), a novel algorithm to weight the two query features. All the CNN features in the whole dataset are clustered and the weight of each query is obtained by the distance to the mutual nearest cluster. The effectiveness of our algorithm is evaluated in PASCAL VOC2007 and Microsoft COCO datasets.


Image retrieval Feature fusion Convolutional neural network 



This work was partially supported by National High Technology Research and Development Program of China (Grant No. 2014AA015104), the Natural Science Foundation of China (NSFC) under Grant 61502139 and 61472116, The Natural Science Foundation of Anhui Province under Grant 1608085MF128, and the program from the Key Lab of Information Network Security, Ministry of Public Security under Grant C14605.


  1. 1.
    Chiang, J.L.: Knowledge-based principal component analysis for image fusion. Appl. Math. 8(1L), 223–230 (2014)Google Scholar
  2. 2.
    Elkan, C.: Using the triangle inequality to accelerate k-means. In: ICML, vol. 3, pp. 147–153 (2003)Google Scholar
  3. 3.
    Fernando, B., Tuytelaars, T.: Mining multiple queries for image retrieval: on-the-fly learning of an object-specific mid-level representation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2544–2551 (2013)Google Scholar
  4. 4.
    Fu, Y., Cao, L., Guo, G., et al.: Multiple feature fusion by subspace learning. In: Proceedings of the 2008 International Conference on Content-Based Image and Video Retrieval, pp. 127–134. ACM (2008)Google Scholar
  5. 5.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  6. 6.
    Ghodrati, A., Jia, X., Pedersoli, M., et al.: Swap retrieval: retrieving images of cats when the query shows a dog. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, pp. 395–402. ACM (2015)Google Scholar
  7. 7.
    Gawande, U., Zaveri, M., Kapur, A.: A novel algorithm for feature level fusion using SVM classifier for multibiometrics-based person identification. Appl. Comput. Intell. Soft Comput. 2013, 9 (2013)CrossRefGoogle Scholar
  8. 8.
    Girshick, R., Donahue, J., Darrell, T., et al.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587 (2014)Google Scholar
  9. 9.
    Hariharan, B., Arbeláez, P., Girshick, R., et al.: Hypercolumns for object segmentation and fine-grained localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 447–456 (2015)Google Scholar
  10. 10.
    Jia, Y., Shelhamer, E., Donahue, J., et al.: Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675–678. ACM (2014)Google Scholar
  11. 11.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  12. 12.
    Makadia, A.: Feature tracking for wide-baseline image retrieval. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 310–323. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15555-0_23 CrossRefGoogle Scholar
  13. 13.
    Nister, D., Stewenius, H.: Scalable recognition with a vocabulary tree. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), vol. 2, pp. 2161–2168. IEEE (2006)Google Scholar
  14. 14.
    Razavian, A.S., Azizpour, H., Sullivan, J., et al.: CNN features off-the-shelf: an astounding baseline for recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 806–813 (2014)Google Scholar
  15. 15.
    Vaca-Castano, G., Shah, M.: Semantic image search from multiple query images. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 887–890. ACM (2015)Google Scholar
  16. 16.
    Zhang, S., Yang, M., Cour, T., Yu, K., Metaxas, D.N.: Query specific fusion for image retrieval. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part II. LNCS, vol. 7573, pp. 660–673. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33709-3_47 CrossRefGoogle Scholar
  17. 17.
    Yang, X., Qian, X., Xue, Y.: Scalable mobile image retrieval by exploring contextual saliency. IEEE Trans. Image Proc. 24(6), 1709–1721 (2015)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Liu, X., Wang, M., Yin, B.-C., Huet, B., Li, X.: Event-based media enrichment using an adaptive probabilistic hypergraph model. IEEE Trans. Cybern. 45(11), 2461–2471 (2015)CrossRefGoogle Scholar
  19. 19.
    Wang, M., Li, W., Liu, D., Ni, B., Shen, J., Yan, S.: Facilitating image search with a scalable and compact semantic mapping. IEEE Trans. Cybern. 45(8), 1561–1574 (2015)CrossRefGoogle Scholar
  20. 20.
    Wang, M., Li, G., Lu, Z., Gao, Y., Chua, T.-S.: When Amazon meets Google: product visualization by exploring multiple information sources. ACM Trans. Internet Technol. 12(4), 1–17 (2013). Article 12CrossRefGoogle Scholar
  21. 21.
    Wang, M., Gao, Y., Ke, L., Rui, Y.: View-based discriminative probabilistic modeling for 3D object retrieval and recognition. IEEE Trans. Image Proces. 22(4), 1395–1407 (2013)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Kecai Wu
    • 1
    Email author
  • Xueliang Liu
    • 1
  • Jie Shao
    • 2
  • Richang Hong
    • 1
  • Tao Yang
    • 3
  1. 1.Hefei University of TechnologyHefeiChina
  2. 2.University of Electronic Science and Technology of ChinaChengduChina
  3. 3.The Third Research Institute of Ministry of Public SecurityBeijingChina

Personalised recommendations