What Is a Good Nearest Neighbors Algorithm for Finding Similar Patches in Images?

  • Neeraj Kumar
  • Li Zhang
  • Shree Nayar
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5303)


Many computer vision algorithms require searching a set of images for similar patches, which is a very expensive operation. In this work, we compare and evaluate a number of nearest neighbors algorithms for speeding up this task. Since image patches follow very different distributions from the uniform and Gaussian distributions that are typically used to evaluate nearest neighbors methods, we determine the method with the best performance via extensive experimentation on real images. Furthermore, we take advantage of the inherent structure and properties of images to achieve highly efficient implementations of these algorithms. Our results indicate that vantage point trees, which are not well known in the vision community, generally offer the best performance.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bentley, J.L.: Multidimensional binary search trees used for associative searching. Comm. ACM 18(9), 509–517 (1975)CrossRefzbMATHGoogle Scholar
  2. 2.
    Sproull, R.F.: Refinements to nearest-neighbor searching in k-dimensional trees. Algorithmica 6(4), 579–589 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Omohundro, S.M.: Five balltree construction algorithms. Technical Report 89-063 (1989)Google Scholar
  4. 4.
    Linde, Y., Buzo, A., Gray, R.M.: An algorithm for vector quantizer design. IEEE Transactions on Communications 28(1), 84–94 (1980)CrossRefGoogle Scholar
  5. 5.
    Yianilos, P.N.: Data structures and algorithms for nearest neighbour search in general metric spaces. In: Symposium on Discrete algorithms, pp. 311–321 (1993)Google Scholar
  6. 6.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. IJCV (2003)Google Scholar
  7. 7.
    Sivic, J., Zisserman, A.: Video google: A text retrieval approach to object matching in videos. In: ICCV, Washington, DC, p. 1470. IEEE Computer Society, Los Alamitos (2003)Google Scholar
  8. 8.
    Nister, D., Stewenius, H.: Scalable recognition with a vocabulary tree. CVPR (2006)Google Scholar
  9. 9.
    Shechtman, E., Irani, M.: Matching local self-similarities across images and videos. CVPR (2007)Google Scholar
  10. 10.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. CVPR (2005)Google Scholar
  11. 11.
    Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. SIGGRAPH (2001)Google Scholar
  12. 12.
    Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22(2), 56–65 (2002)CrossRefGoogle Scholar
  13. 13.
    Chávez, E., Navarro, G., Baeza-Yates, R., Marroquín, J.L.: Searching in metric spaces. ACM Computing Surveys 33(3), 273–321 (2001)CrossRefGoogle Scholar
  14. 14.
    Shakhnarovich, G., Darrell, T., Indyk, P.: Nearest-Neighbor Methods in Learning and Vision. MIT Press, Cambridge (2006)Google Scholar
  15. 15.
    Bentley, J.L.: K-d trees for semidynamic point sets. In: Symposium on Computational geometry, pp. 187–197 (1990)Google Scholar
  16. 16.
    Beis, J.S., Lowe, D.G.: Shape indexing using approximate nearest-neighbour search in high-dimensional spaces. CVPR (1997)Google Scholar
  17. 17.
    Hays, J., Efros, A.A.: Scene completion using millions of photographs. ACM Transactions on Graphics (SIGGRAPH) 26(3) (2007)Google Scholar
  18. 18.
    Hotelling, H.: Analysis of a complex of statistical variables with principal components. Journal of Educational Psychology 24, 417–441 (1933)CrossRefzbMATHGoogle Scholar
  19. 19.
    MacQueen, J.: Some methods for classification and analysis of multivariate observations. In: Symposium on Mathematical Statistics and Probability, pp. 281–297 (1967)Google Scholar
  20. 20.
    Edelsbrunner, H.: Algorithms in combinatorial geometry. Springer, New York (1987)CrossRefzbMATHGoogle Scholar
  21. 21.
    Nene, S.A., Nayar, S.K.: A Simple Algorithm for Nearest Neighbour Search in High Dimensions. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(9), 989–1003 (1997)CrossRefGoogle Scholar
  22. 22.
    Arya, S., Mount, D.M., Netanyahu, N.S., Silverman, R., Wu, A.Y.: An optimal algorithm for approximate nearest neighbor searching in fixed dimensions. J. ACM 45(6), 891–923 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Indyk, P., Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: Symposium on Theory of Computing, pp. 604–613 (1998)Google Scholar
  24. 24.
    Mikolajczyk, K., Matas, J.: Improving descriptors for fast tree matching by optimal linear projection. In: ICCV (2007)Google Scholar
  25. 25.
    Ruderman, D.L.: The statistics of natural images. Network: Computation in Neural Systems 5, 517–548 (1996)CrossRefzbMATHGoogle Scholar
  26. 26.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. CVPR (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Neeraj Kumar
    • 1
  • Li Zhang
    • 2
  • Shree Nayar
    • 1
  1. 1.Columbia UniversityUSA
  2. 2.University of Wisconsin-MadisonUSA

Personalised recommendations