Smooth Neighborhood Structure Mining on Multiple Affinity Graphs with Applications to Context-Sensitive Similarity

  • Song BaiEmail author
  • Shaoyan Sun
  • Xiang Bai
  • Zhaoxiang Zhang
  • Qi Tian
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9906)


Due to the ability of capturing geometry structures of the data manifold, diffusion process has demonstrated impressive performances in retrieval task by spreading the similarities on the affinity graph. In view of robustness to noise edges, diffusion process is usually localizedi.e., only propagating similarities via neighbors. However, selecting neighbors smoothly on graph-based manifolds is more or less ignored by previous works. In this paper, we propose a new algorithm called Smooth Neighborhood (SN) that mines the neighborhood structure to satisfy the manifold assumption. By doing so, nearby points on the underlying manifold are guaranteed to yield similar neighbors as much as possible. Moreover, SN is adjusted to tackle multiple affinity graphs by imposing a weight learning paradigm, and this is the primary difference compared with related works which are only applicable with one affinity graph. Exhausted experimental results and comparisons against other algorithms manifest the effectiveness of the proposed algorithm.


Diffusion process Image/shape retrieval Affinity graph 



The authors would to thank Pedronette DCG. for providing the codes on MPEG-7 dataset. This work was supported in part by NSFC 61573160, NSFC 61429201 and China Scholarship Council. This work was supported in part to Dr. Qi Tian by ARO grants W911NF-15-1-0290 and Faculty Research Gift Awards by NEC Laboratories of America and Blippar.


  1. 1.
    Jegou, H., Schmid, C., Harzallah, H., Verbeek, J.J.: Accurate image search using the contextual dissimilarity measure. TPAMI 32(1), 2–11 (2010)CrossRefGoogle Scholar
  2. 2.
    Bai, X., Yang, X., Latecki, L.J., Liu, W., Tu, Z.: Learning context-sensitive shape similarity by graph transduction. TPAMI 32(5), 861–874 (2010)CrossRefGoogle Scholar
  3. 3.
    Wang, J., Li, Y., Bai, X., Zhang, Y., Wang, C., Tang, N.: Learning context-sensitive similarity by shortest path propagation. Pattern Recogn. 44(10), 2367–2374 (2011)CrossRefGoogle Scholar
  4. 4.
    Bai, X., Wang, B., Wang, X., Liu, W., Tu, Z.: Co-transduction for shape retrieval. In: Maragos, P., Paragios, N., Daniilidis, K. (eds.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 328–341. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  5. 5.
    Yang, X., Latecki, L.J.: Affinity learning on a tensor product graph with applications to shape and image retrieval. In: CVPR, pp. 2369–2376 (2011)Google Scholar
  6. 6.
    Wang, B., Tu, Z.: Affinity learning via self-diffusion for image segmentation and clustering. In: CVPR, pp. 2312–2319 (2012)Google Scholar
  7. 7.
    Pedronette, D., Torres, R.: Image re-ranking and rank aggregation based on similarity of ranked lists. Pattern Recogn. 46(8), 2350–2360 (2013)CrossRefGoogle Scholar
  8. 8.
    Pedronette, D., Almeida, J., Torres, R.: A scalable re-ranking method for content-based image retrieval. Inf. Sci. 265, 91–104 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Bai, S., Bai, X.: Sparse contextual activation for efficient visual re-ranking. TIP 25(3), 1056–1069 (2016)MathSciNetGoogle Scholar
  10. 10.
    Chen, Y., Li, X., Dick, A., Hill, R.: Ranking consistency for image matching and object retrieval. Pattern Recogn. 47(3), 1349–1360 (2014)CrossRefGoogle Scholar
  11. 11.
    Zhang, S., Yang, M., Cour, T., Yu, K., Metaxas, D.N.: Query specific fusion for image retrieval. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part II. LNCS, vol. 7573, pp. 660–673. Springer, Heidelberg (2012)Google Scholar
  12. 12.
    Zhang, S., Yang, M., Cour, T., Yu, K., Metaxas, D.N.: Query specific rank fusion for image retrieval. TPAMI 37(4), 803–815 (2015)CrossRefGoogle Scholar
  13. 13.
    Donoser, M., Bischof, H.: Diffusion processes for retrieval revisited. In: CVPR, pp. 1320–1327 (2013)Google Scholar
  14. 14.
    Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500), 2323–2326 (2000)CrossRefGoogle Scholar
  15. 15.
    Kontschieder, P., Donoser, M., Bischof, H.: Beyond pairwise shape similarity analysis. In: Zha, H., Taniguchi, R., Maybank, S. (eds.) ACCV 2009, Part III. LNCS, vol. 5996, pp. 655–666. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  16. 16.
    Qin, D., Gammeter, S., Bossard, L., Quack, T., van Gool, L.: Hello neighbor: accurate object retrieval with k-reciprocal nearest neighbors. In: CVPR, pp. 777–784 (2011)Google Scholar
  17. 17.
    Pedronette, D., Penatti, O., Torres, R.: Unsupervised manifold learning using reciprocal kNN graphs in image re-ranking and rank aggregation tasks. Image Vis. Comput. 32(2), 120–130 (2014)CrossRefGoogle Scholar
  18. 18.
    Premachandran, V., Kakarala, R.: Consensus of k-NNs for robust neighborhood selection on graph-based manifolds. In: CVPR, pp. 1594–1601 (2013)Google Scholar
  19. 19.
    Yang, X., Koknar-Tezel, S., Latecki, L.J.: Locally constrained diffusion process on locally densified distance spaces with applications to shape retrieval. In: CVPR, pp. 357–364 (2009)Google Scholar
  20. 20.
    Luo, L., Shen, C., Zhang, C., van den Hengel, A.: Shape similarity analysis by self-tuning locally constrained mixed-diffusion. TMM 15(5), 1174–1183 (2013)Google Scholar
  21. 21.
    Pavan, M., Pelillo, M.: Dominant sets and pairwise clustering. TPAMI 29(1), 167–172 (2007)CrossRefGoogle Scholar
  22. 22.
    Pelillo, M.: Matching free trees with replicator equations. NIPS 2, 865–872 (2002)Google Scholar
  23. 23.
    Gao, S., Tsang, I.W.H., Chia, L.T., Zhao, P.: Local features are not lonely-laplacian sparse coding for image classification. In: CVPR, pp. 3555–3561 (2010)Google Scholar
  24. 24.
    Wang, J., Chang, S.F., Zhou, X., Wong, S.T.: Active microscopic cellular image annotation by superposable graph transduction with imbalanced labels. In: CVPR, pp. 1–8 (2008)Google Scholar
  25. 25.
    Zhu, X., Ghahramani, Z., Lafferty, J., et al.: Semi-supervised learning using Gaussian fields and harmonic functions. In: ICML, pp. 912–919 (2003)Google Scholar
  26. 26.
    Kuang, Z., Li, Z., Fan, J.: Discovering the latent similarities of the KNN graph by metric transformation. In: ICMR, pp. 471–474 (2015)Google Scholar
  27. 27.
    Latecki, L.J., Lakämper, R., Eckhardt, U.: Shape descriptors for non-rigid shapes with a single closed contour. In: CVPR, pp. 424–429 (2000)Google Scholar
  28. 28.
    Ling, H., Jacobs, D.W.: Shape classification using the inner-distance. TPAMI 29(2), 286–299 (2007)CrossRefGoogle Scholar
  29. 29.
    Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. TPAMI 24(4), 509–522 (2002)CrossRefGoogle Scholar
  30. 30.
    Ling, H., Yang, X., Latecki, L.J.: Balancing deformability and discriminability for shape matching. In: Maragos, P., Paragios, N., Daniilidis, K. (eds.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 411–424. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  31. 31.
    Gopalan, R., Turaga, P., Chellappa, R.: Articulation-invariant representation of non-planar shapes. In: Maragos, P., Paragios, N., Daniilidis, K. (eds.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 286–299. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  32. 32.
    Nistér, D., Stewénius, H.: Scalable recognition with a vocabulary tree. In: CVPR, pp. 2161–2168 (2006)Google Scholar
  33. 33.
    Arandjelović, R., Zisserman, A.: Three things everyone should know to improve object retrieval. In: CVPR, pp. 2911–2918 (2012)Google Scholar
  34. 34.
    Jegou, H., Douze, M., Schmid, C.: Hamming embedding and weak geometric consistency for large scale image search. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 304–317. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  35. 35.
    Shen, X., Lin, Z., Brandt, J., Avidan, S., Wu, Y.: Object retrieval and localization with spatially-constrained similarity measure and k-NN re-ranking. In: CVPR, pp. 3013–3020 (2012)Google Scholar
  36. 36.
    Zheng, L., Wang, S., Tian, L., He, F., Liu, Z., Tian, Q.: Query-adaptive late fusion for image search and person re-identification. In: CVPR, pp. 1741–1750 (2015)Google Scholar
  37. 37.
    Xie, L., Hong, R., Zhang, B., Tian, Q.: Image classification and retrieval are one. In: ICMR, pp. 3–10 (2015)Google Scholar
  38. 38.
    Tolias, G., Jégou, H.: Visual query expansion with or without geometry: refining local descriptors by feature aggregation. Pattern Recogn. 47(10), 3466–3476 (2014)CrossRefGoogle Scholar
  39. 39.
    Paulin, M., Douze, M., Harchaoui, Z., Mairal, J., Perronin, F., Schmid, C.: Local convolutional features with unsupervised training for image retrieval. In: ICCV, pp. 91–99 (2015)Google Scholar
  40. 40.
    Babenko, A., Lempitsky, V.: Aggregating deep convolutional features for image retrieval. In: ICCV, pp. 1269–1277 (2015)Google Scholar
  41. 41.
    Bai, S., Bai, X., Liu, W.: Multiple stage residual model for image classification and vector compression. TMM 18(7), 1351–1362 (2016)Google Scholar
  42. 42.
    Bai, S., Bai, X., Zhou, Z., Zhang, Z., Latecki, L.J.: Gift: a real-time and scalable 3D shape search engine. In: CVPR (2016)Google Scholar
  43. 43.
    Searle, S.R.: Matrix Algebra Useful for Statistics. Wiley-Interscience, Hoboken (1982)zbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Song Bai
    • 1
    Email author
  • Shaoyan Sun
    • 2
  • Xiang Bai
    • 1
  • Zhaoxiang Zhang
    • 3
  • Qi Tian
    • 4
  1. 1.Huazhong University of Science and TechnologyWuhanChina
  2. 2.University of Science and Technology of ChinaHefeiChina
  3. 3.CAS Center for Excellence in Brain Science and Intelligence Technology, CASIABeijingChina
  4. 4.University of Texas at San AntonioSan AntonioUSA

Personalised recommendations