International Journal of Computer Vision

, Volume 107, Issue 1, pp 1–19 | Cite as

Hough Pyramid Matching: Speeded-Up Geometry Re-ranking for Large Scale Image Retrieval

  • Yannis Avrithis
  • Giorgos ToliasEmail author


Exploiting local feature shape has made geometry indexing possible, but at a high cost of index space, while a sequential spatial verification and re-ranking stage is still indispensable for large scale image retrieval. In this work we investigate an accelerated approach for the latter problem. We develop a simple spatial matching model inspired by Hough voting in the transformation space, where votes arise from single feature correspondences. Using a histogram pyramid, we effectively compute pair-wise affinities of correspondences without ever enumerating all pairs. Our Hough pyramid matching algorithm is linear in the number of correspondences and allows for multiple matching surfaces or non-rigid objects under one-to-one mapping. We achieve re-ranking one order of magnitude more images at the same query time with superior performance compared to state of the art methods, while requiring the same index space. We show that soft assignment is compatible with this matching scheme, preserving one-to-one mapping and further increasing performance.


Image retrieval Spatial verification Relaxed spatial matching  Hough pyramid matching Geometric re-ranking 


  1. Avrithis, Y., Kalantidis, Y., Tolias, G., & Spyrou, E. (2010). Retrieving landmark and non-landmark images from community photo collections. Firenze, Italy: ACM Multimedia.Google Scholar
  2. Avrithis, Y., Tolias, G., & Kalantidis, Y. (2010). Feature map hashing: Sub-linear indexing of appearance and global geometry. Firenze, Italy: ACM Multimedia.Google Scholar
  3. Ballard, D. (1981). Generalizing the hough transform to detect arbitrary shapes. Pattern Recognition, 13(2), 111–122.CrossRefzbMATHGoogle Scholar
  4. Bay, H., Tuytelaars, T., & Van Gool, L. (2006). SURF: Speeded up robust features. In ECCV.Google Scholar
  5. Belongie, S., Malik, J., & Puzicha, J. (2000). Shape context: A new descriptor for shape matching and object recognition. NIPS, 12, 827–831.Google Scholar
  6. Berg, A., Berg, T., & Malik, J. (2005). Shape matching and object recognition using low distortion correspondences. In CVPR.Google Scholar
  7. Cao, Y., Wang, C., Li, Z., Zhang, L., & Zhang, L. (2010). Spatial-bag-of-features. In CVPR (pp. 3352–3359).Google Scholar
  8. Carneiro, G., & Jepson, A. (2007). Flexible spatial configuration of local image features. PAMI, 29(12), 2089–2104.CrossRefGoogle Scholar
  9. Cheng, Y. (1995). Mean shift, mode seeking, and clustering. PAMI, 17(8), 790–799.CrossRefGoogle Scholar
  10. Chum, O., Philbin, J., Sivic, J., Isard, M., & Zisserman, A. (2007). Total recall: Automatic query expansion with a generative feature model for object retrieval. In ICCV.Google Scholar
  11. Enqvist, O., Josephson, K., & Kahl, F. (2009). Optimal correspondences from pairwise constraints. In ICCV.Google Scholar
  12. Fischler, M., & Bolles, R. (1981). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381–395.CrossRefMathSciNetGoogle Scholar
  13. Grauman, K., & Darrell, T. (2007). The pyramid match kernel: Efficient learning with sets of features. Journal of Machine Learning Research, 8, 725–760.zbMATHGoogle Scholar
  14. Indyk, P., & Thaper, N. (2003). Fast image retrieval via embeddings. In Workshop on Statistical and Computational Theories of Vision.Google Scholar
  15. Jégou, H., Douze, M., & Schmid, C. (2008). Hamming embedding and weak geometric consistency for large scale image search. In ECCV.Google Scholar
  16. Jégou, H., Douze, M., & Schmid, C. (2009). On the burstiness of visual elements. In CVPR.Google Scholar
  17. Jégou, H., Douze, M., & Schmid, C. (2010). Improving bag-of-features for large scale image search. IJCV, 87(3), 316–336.Google Scholar
  18. Jiang, H., & Yu, S. X. (2009). Linear solution to scale and rotation invariant object matching. In CVPR.Google Scholar
  19. Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In CVPR (Vol. 2, p. 1).Google Scholar
  20. Leibe, B., Leonardis, A., & Schiele, B. (2008). Robust object detection with interleaved categorization and segmentation. IJCV, 77(1), 259–289.Google Scholar
  21. Leordeanu, M., & Hebert, M. (2005). A spectral technique for correspondence problems using pairwise constraints. In: ICCV, (Vol. 2, pp. 1482–1489). Google Scholar
  22. Lin, Z., & Brandt, J. (2010). A local bag-of-features model for large-scale object retrieval. In ECCV (pp. 294–308).Google Scholar
  23. Lowe, D. (2004). Distinctive image features from scale-invariant keypoints. IJCV, 60(2), 91–110.Google Scholar
  24. Mikulik, A., Perdoch, M., Chum, O., & Matas, J. (2010). Learning a fine vocabulary. In ECCV.Google Scholar
  25. Olsson, C., Eriksson, A., & Kahl, F. (2007). Solving large scale binary quadratic problems: Spectral methods vs. semidefinite programming. In CVPR.Google Scholar
  26. Perdoch, M., Chum, O., & Matas, J. (2009). Efficient representation of local geometry for large scale object retrieval. In CVPR.Google Scholar
  27. Philbin, J., Chum, O., Isard, M., Sivic, J., & Zisserman, A. (2007). Object retrieval with large vocabularies and fast spatial matching. In CVPR.Google Scholar
  28. Philbin, J., Chum, O., Sivic, J., Isard, M., & Zisserman, A. (2008). Lost in quantization: Improving particular object retrieval in large scale image databases. In CVPR.Google Scholar
  29. Raguram, R., & Frahm, J. M. (2011). Recon: Scale-adaptive robust estimation via residual consensus. In ICCV.Google Scholar
  30. Sahbi, H., Audibert, J. Y., Rabarisoa, J., & Keriven, R. (2008). Context-dependent kernel design for object matching and recognition. In CVPR.Google Scholar
  31. Scott, G., & Longuet-Higgins, H. (1991). An algorithm for associating the features of two images. Proceedings of the Royal Society of London, 244(1309), 21.CrossRefGoogle Scholar
  32. Shen, X., Lin, Z., Brandt, J., Avidan, S., & Wu, Y. (2012). Object retrieval and localization with spatially-constrained similarity measure and k-nn re-ranking. In CVPR. IEEE.Google Scholar
  33. Sivic, J., & Zisserman, A. (2003) Video Google: A text retrieval approach to object matching in videos. In: ICCV (pp. 1470–1477).Google Scholar
  34. Tolias, G., & Avrithis, Y. (2011). Speeded-up, relaxed spatial matching. In ICCV.Google Scholar
  35. Vedaldi, A., & Soatto, S. (2008). Quick shift and kernel methods for mode seeking. In ECCV.Google Scholar
  36. Vedaldi, A., & Soatto, S. (2008). Relaxed matching kernels for robust image comparison. In CVPR.Google Scholar
  37. Wu, Z., Ke, Q., Isard, M., & Sun, J. (2009). Bundling features for large scale partial-duplicate web image search. In CVPR.Google Scholar
  38. Zhang, Y., Jia, Z., & Chen, T. (2011). Image retrieval with geometry-preserving visual phrases. In CVPR. IEEE (pp. 809–816).Google Scholar
  39. Zhou, W., Lu, Y., Li, H., Song, Y., & Tian, Q. (2010). Spatial coding for large scale partial-duplicate web image search. Firenze, Italy: ACM Multimedia.Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.National Technical University of AthensZografouGreece

Personalised recommendations