Advertisement

The Generalized PatchMatch Correspondence Algorithm

  • Connelly Barnes
  • Eli Shechtman
  • Dan B. Goldman
  • Adam Finkelstein
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6313)

Abstract

PatchMatch is a fast algorithm for computing dense approximate nearest neighbor correspondences between patches of two image regions [1]. This paper generalizes PatchMatch in three ways: (1) to find k nearest neighbors, as opposed to just one, (2) to search across scales and rotations, in addition to just translations, and (3) to match using arbitrary descriptors and distances, not just sum-of-squared-differences on patch colors. In addition, we offer new search and parallelization strategies that further accelerate the method, and we show performance improvements over standard kd-tree techniques across a variety of inputs. In contrast to many previous matching algorithms, which for efficiency reasons have restricted matching to sparse interest points, or spatially proximate matches, our algorithm can efficiently find global, dense matches, even while matching across all scales and rotations. This is especially useful for computer vision applications, where our algorithm can be used as an efficient general-purpose component. We explore a variety of vision applications: denoising, finding forgeries by detecting cloned regions, symmetry detection, and object detection.

Keywords

Object Detection Image Denoising Sift Descriptor Average PSNR Clone Detection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Supplementary material

978-3-642-15558-1_3_MOESM1_ESM.pdf (913 kb)
Electronic Supplementary Material (914 KB)

References

  1. 1.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (Proc. SIGGRAPH) 28, 24 (2009)Google Scholar
  2. 2.
    Buades, A., Coll, B., Morel, J.: A non-local algorithm for image denoising. In: Proc. CVPR II, p. 60 (2005)Google Scholar
  3. 3.
    Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R.: A database and evaluation methodology for optical flow. In: Proc. ICCV, vol. 5 (2007)Google Scholar
  4. 4.
    Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1615–1630 (2005)Google Scholar
  5. 5.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 91–110 (2004)CrossRefGoogle Scholar
  6. 6.
    Brox, T., Malik, J.: Large displacement optical flow. In: CVPR (2009)Google Scholar
  7. 7.
    Simon, I., Seitz, S.: A probabilistic model for object recognition, segmentation, and non-rigid correspondence. In: Proc. CVPR, pp. 1–7 (2007)Google Scholar
  8. 8.
    Hays, J., Leordeanu, M., Efros, A., Liu, Y.: Discovering texture regularity as a higher-order correspondence problem. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, p. 522. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  9. 9.
    Boiman, O., Irani, M.: Detecting irregularities in images and in video. International Journal of Computer Vision 74, 17–31 (2007)CrossRefGoogle Scholar
  10. 10.
    Bagon, S., Boiman, O., Irani, M.: What is a good image segment? A unified approach to segment extraction. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part IV. LNCS, vol. 5305, p. 44. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  11. 11.
    Popescu, A., Farid, H.: Exposing digital forgeries by detecting duplicated image regions. Department of Computer Science, Dartmouth College (2004)Google Scholar
  12. 12.
    Freeman, W., Jones, T., Pasztor, E.: Example-based super-resolution. IEEE Computer Graphics and Applications, 56–65 (2002)Google Scholar
  13. 13.
    Efros, A., Leung, T.: Texture synthesis by non-parametric sampling. In: Proc. ICCV, vol. 2, pp. 1033–1038 (1999)Google Scholar
  14. 14.
    Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. on Image Processing 13 (2004)Google Scholar
  15. 15.
    Simakov, D., Caspi, Y., Shechtman, E., Irani, M.: Summarizing visual data using bidirectional similarity. In: Proc. CVPR (2008)Google Scholar
  16. 16.
    Hertzmann, A., Jacobs, C., Oliver, N., Curless, B., Salesin, D.: Image analogies. ACM Transactions on Graphics (Proc. SIGGRAPH), 327–340 (2001)Google Scholar
  17. 17.
    Mount, D.M., Arya, S.: ANN: A library for approx. nearest neighbor search (1997)Google Scholar
  18. 18.
    Boiman, O., Shechtman, E., Irani, M.: In defense of nearest-neighbor based image classification. In: Proc. CVPR., vol. 2, p. 6 (2008)Google Scholar
  19. 19.
    Niyogi, S., Freeman, W.: Example-based head tracking. In: Proc. of Conf. on Automatic Face and Gesture Recognition (FG 1996), p. 374 (1996)Google Scholar
  20. 20.
    Kumar, N., Zhang, L., Nayar, S.K.: What is a good nearest neighbors algorithm for finding similar patches in images? In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 364–378. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  21. 21.
    Nister, D., Stewenius, H.: Scalable recognition with a vocabulary tree. In: Proc. CVPR, vol. 5 (2006)Google Scholar
  22. 22.
    Muja, M., Lowe, D.: Fast approximate nearest neighbors with automatic algorithm configuration. In: VISAPP (2009)Google Scholar
  23. 23.
    Datar, M., Immorlica, N., Indyk, P., Mirrokni, V.: Locality-sensitive hashing scheme based on p-stable distributions. In: Symp. on Comp. Geom., pp. 253–262 (2004)Google Scholar
  24. 24.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Processing 16 (2007)Google Scholar
  25. 25.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G., Zisserman, A.: Non-local sparse models for image restoration. In: Proc. of ICCV (2009)Google Scholar
  26. 26.
    Brox, T., Kleinschmidt, O., Cremers, D.: Efficient nonlocal means for denoising of textural patterns. IEEE Transactions on Image Processing 17, 1083–1092 (2008)CrossRefMathSciNetGoogle Scholar
  27. 27.
    Bayram, S., Sencar, H., Memon, N.: A Survey of Copy-Move Forgery Detection Techniques. In: IEEE Western New York Image Processing Workshop (2008)Google Scholar
  28. 28.
    Park, M., Leey, S., Cheny, P., Kashyap, S., Butty, A., Liuy, Y.: Performance evaluation of state-of-the-art discrete symmetry detection. In: CVPR (2008)Google Scholar
  29. 29.
    Fischler, M., Bolles, R.: Random sample consensus: A paradigm for model fitting with apps. to image analysis and automated cartography. ACM Comm. 24 (1981)Google Scholar
  30. 30.
    Jain, A., Zhong, Y., Dubuisson-Jolly, M.: Deformable template models: A review. Signal Processing 71, 109–129 (1998)zbMATHCrossRefGoogle Scholar
  31. 31.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proc. CVPR (2001)Google Scholar
  32. 32.
    Guo, G., Dyer, C.: Patch-based image correlation with rapid filtering. In: The 2nd Beyond Patches Workshop, in Conj. with IEEE CVPR 2007 (2007)Google Scholar
  33. 33.
    Liu, C., Yuen, J., Torralba, A., Sivic, J., Freeman, W.T.: SIFT flow: Dense correspondence across different scenes. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, p. 28. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  34. 34.
    Glasner, D., Bagon, S., Irani, M.: Super-Resolution from a Single Image. In: Proc. of ICCV (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Connelly Barnes
    • 1
  • Eli Shechtman
    • 2
  • Dan B. Goldman
    • 2
  • Adam Finkelstein
    • 1
  1. 1.Princeton University 
  2. 2.Adobe Systems 

Personalised recommendations