Skip to main content
SpringerLink
Log in

Exemplar-Based Image Inpainting Using an Affine Invariant Similarity Measure

  • Conference paper
  • First Online:
Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2016)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 693))

  • 1103 Accesses

Abstract

Patch-based approaches are used in state-of-the-art methods for image inpainting. This paper presents a new method for exemplar-based image inpainting using transformed patches. The transformation is determined for each patch in a fully automatic way from a surrounding texture content. We build upon a recent affine invariant patch similarity measure that performs an appropriate patch comparison by automatically adapting the size and shape of the patches. As a consequence, it intrinsically extends the set of available source patches to copy information from. We incorporate this measure into a variational formulation for inpainting and present a numerical algorithm for optimizing it. We show that our method can be applied to complete a perspectively distorted texture as well as to automatically inpaint one view of a scene using other view of the same scene as a source. We present experimental results both for gray and color images, and a comparison with some exemplar-based image inpainting methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Fedorov, V., Arias, P., Sadek, R., Facciolo, G., Ballester, C.: Linear multiscale analysis of similarities between images on riemannian manifolds: practical formula and affine covariant metrics. SIAM J. Imaging Sci. 8, 2021–2069 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  2. Masnou, S., Morel, J.M.: Level lines based disocclusion. In: Proceedings of IEEE ICIP, vol. 3, pp. 259–263 (1998)

    Google Scholar 

  3. Ballester, C., Bertalmío, M., Caselles, V., Sapiro, G., Verdera, J.: Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans. Image Process. 10, 1200–1211 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  4. Masnou, S.: Disocclusion: a variational approach using level lines. IEEE Trans. Image Process. 11, 68–76 (2002)

    Article  MathSciNet  Google Scholar 

  5. Cao, F., Gousseau, Y., Masnou, S.: Geometrically guided exemplar-based inpainting. SIAM J. Imaging Sci. 4, 1143–1179 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  6. Bertalmío, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of SIGGRAPH, pp. 417–424 (2000)

    Google Scholar 

  7. Chan, T., Shen, J.H.: Mathematical models for local nontexture inpaintings. SIAM J. App. Math. 62, 1019–43 (2001)

    MathSciNet  MATH  Google Scholar 

  8. Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of the IEEE ICCV, pp. 1033–1038 (1999)

    Google Scholar 

  9. Foi, A., Boracchi, G.: Foveated self-similarity in nonlocal image filtering. Hum. Vis. Electron. Imaging XVII(8291), 829110 (2012)

    Google Scholar 

  10. Buades, A., Coll, B., Morel, J.M.: A non local algorithm for image denoising. In: Proceedings of the IEEE Conference on CVPR, vol. 2, pp. 60–65 (2005)

    Google Scholar 

  11. Gilboa, G., Osher, S.J.: Nonlocal operators with applications to image processing. Multiscale Model. Simul. 7, 1005–1028 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  12. Peyré, G.: Manifold models for signals and images. Comput. Vis. Image Underst. 113, 249–260 (2009)

    Article  Google Scholar 

  13. Pizarro, L., Mrázek, P., Didas, S., Grewenig, S., Weickert, J.: Generalised nonlocal image smoothing. Int. J. Comput. Vis. 90, 62–87 (2010)

    Article  MathSciNet  Google Scholar 

  14. Hays, J., Efros, A.: Scene completion using millions of photographs. In: SIGGRAPH. ACM, New York (2007)

    Google Scholar 

  15. Demanet, L., Song, B., Chan, T.: Image inpainting by correspondence maps: a deterministic approach. Appl. Comput. Math. 1100, 217–50 (2003)

    Google Scholar 

  16. Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based inpainting. IEEE Trans. IP 13, 1200–1212 (2004)

    Google Scholar 

  17. Wexler, Y., Shechtman, E., Irani, M.: Space-time completion of video. IEEE Trans. PAMI 29, 463–476 (2007)

    Article  Google Scholar 

  18. Kawai, N., Sato, T., Yokoya, N.: Image inpainting considering brightness change and spatial locality of textures and its evaluation. In: Wada, T., Huang, F., Lin, S. (eds.) PSIVT 2009. LNCS, vol. 5414, pp. 271–282. Springer, Heidelberg (2009). doi:10.1007/978-3-540-92957-4_24

    Chapter  Google Scholar 

  19. Aujol, J.F., Ladjal, S., Masnou, S.: Exemplar-based inpainting from a variational point of view. SIAM J. Math. Anal. 42, 1246–1285 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  20. Arias, P., Facciolo, G., Caselles, V., Sapiro, G.: A variational framework for exemplar-based image inpainting. Int. J. Comput. Vis. 93, 319–347 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  21. Drori, I., Cohen-Or, D., Yeshurun, H.: Fragment-based image completion. In: ACM SIGGRAPH 2003 Papers, vol. 22, pp. 303–312 (2003)

    Google Scholar 

  22. Mansfield, A., Prasad, M., Rother, C., Sharp, T., Kohli, P., van Gool, L.: Transforming image completion. In: Proceedings of BMVC, vol. 121, pp. 121.1–121.11 (2011)

    Google Scholar 

  23. Barnes, C., Shechtman, E., Goldman, D.B., Finkelstein, A.: The generalized PatchMatch correspondence algorithm. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6313, pp. 29–43. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15558-1_3

    Chapter  Google Scholar 

  24. Pavić, D., Schonefeld, V., Kobbelt, L.: Interactive image completion with perspective correction. Vis. Comput. 22, 671–681 (2006)

    Article  Google Scholar 

  25. Huang, J.B., Kopf, J., Ahuja, N., Kang, S.B.: Transformation guided image completion. In: International Conference on Computational Photography, pp. 1–9 (2013)

    Google Scholar 

  26. Huang, J.B., Kang, S.B., Ahuja, N., Kopf, J.: Image completion using planar structure guidance. ACM Trans. Graph. (Proc. SIGGRAPH 2014) 33, 129:1–129:10 (2014)

    Google Scholar 

  27. Gårding, J.: Shape from texture for smooth curved surfaces in perspective projection. J. Math. Imaging Vis. 2, 327–350 (1992)

    Article  Google Scholar 

  28. Gårding, J., Lindeberg, T.: Direct computation of shape cues using scale-adapted spatial derivative operators. Int. J. Comput. Vis. 17, 163–191 (1996)

    Article  Google Scholar 

  29. Ballester, C., Gonzalez, M.: Affine invariant texture segmentation and shape from texture by variational methods. J. Math. Imaging Vis. 9, 141–171 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  30. Wang, Z.: Image affine inpainting. In: Campilho, A., Kamel, M. (eds.) ICIAR 2008. LNCS, vol. 5112, pp. 1061–1070. Springer, Heidelberg (2008). doi:10.1007/978-3-540-69812-8_106

    Chapter  Google Scholar 

  31. Mikolajczyk, K., Schmid, C.: Scale and affine invariant interest point detectors. Int. J. Comput. Vis. 60, 63–86 (2004)

    Article  Google Scholar 

  32. Matas, J., Chum, O., Urban, M., Pajdla, T.: Robust wide-baseline stereo from maximally stable extremal regions. Image Vis. Comput. 22, 761–767 (2004)

    Article  Google Scholar 

  33. Gårding, J., Lindeberg, T.: Direct estimation of local surface shape in a fixating binocular vision system. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 800, pp. 365–376. Springer, Heidelberg (1994). doi:10.1007/3-540-57956-7_40

    Chapter  Google Scholar 

  34. Ballester, C., Calderero, F., Caselles, V., Facciolo, G.: Multiscale analysis of similarities between images on riemannian manifolds. SIAM J. Multiscale Model. Simul. 12, 616–649 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  35. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)

    Article  Google Scholar 

  36. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch: a randomized correspondence algorithm for structural image editing. In: ACM SIGGRAPH 2009 Papers, pp. 1–11. ACM, New York (2009)

    Google Scholar 

  37. Nadaraya, E.A.: On estimating regression. Theory Probab. Appl. 9, 141–142 (1964)

    Article  MATH  Google Scholar 

  38. Watson, G.S.: Smooth regression analysis. Sankhya: Indian J. Stat. Ser. A (1961–2002) 26, 359–372 (1964)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The first, second and fourth authors acknowledge partial support by the MINECO/FEDER project with reference TIN2015-70410-C2-1-R, the MICINN project with reference MTM2012-30772, and by GRC reference 2014 SGR 1301, Generalitat de Catalunya. The second and third authors were partly founded by the Centre National dEtudes Spatiales (CNES, MISS Project), BPIFrance and Région Ile de France, in the framework of the FUI 18 Plein Phare project, the European Research Council (advanced grant Twelve Labours n246961), the Office of Naval research (ONR grant N00014-14-1-0023), and ANR-DGA project ANR-12-ASTR-0035.

Author information

Authors and Affiliations

  1. University Pompeu Fabra, Barcelona, Spain

    Vadim Fedorov & Coloma Ballester

  2. CMLA, ENS Cachan, Cachan, France

    Pablo Arias

  3. LIGM, UMR 8049, École des Ponts, UPE, Champs-sur-Marne, France

    Gabriele Facciolo

Authors
  1. Vadim Fedorov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pablo Arias
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gabriele Facciolo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Coloma Ballester
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vadim Fedorov .

Editor information

Editors and Affiliations

  1. Escola Superior de Tecnologia do IPS, Setúbal, Portugal

    José Braz

  2. MiraLab, University of Geneva, Carouge, Switzerland

    Nadia Magnenat-Thalmann

  3. LISA - ISTIA, University of Angers, Angers, France

    Paul Richard

  4. Department of Computer Science and Electrical Engineering, Jacobs University, Bremen, Germany

    Lars Linsen

  5. University of Groningen, Groningen, The Netherlands

    Alexandru Telea

  6. Università di Catania, Catania, Italy

    Sebastiano Battiato

  7. Research Innovation Center, Canon U.S.A. Inc., San Jose, California, USA

    Francisco Imai

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Fedorov, V., Arias, P., Facciolo, G., Ballester, C. (2017). Exemplar-Based Image Inpainting Using an Affine Invariant Similarity Measure. In: Braz, J., et al. Computer Vision, Imaging and Computer Graphics Theory and Applications. VISIGRAPP 2016. Communications in Computer and Information Science, vol 693. Springer, Cham. https://doi.org/10.1007/978-3-319-64870-5_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-64870-5_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-64869-9

  • Online ISBN: 978-3-319-64870-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation