A Variational Framework for Non-local Image Inpainting

  • Pablo Arias
  • Vicent Caselles
  • Guillermo Sapiro
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5681)


Non-local methods for image denoising and inpainting have gained considerable attention in recent years. This is in part due to their superior performance in textured images, a known weakness of purely local methods. Local methods on the other hand have demonstrated to be very appropriate for the recovering of geometric structure such as image edges. The synthesis of both types of methods is a trend in current research. Variational analysis in particular is an appropriate tool for a unified treatment of local and non-local methods. In this work we propose a general variational framework for the problem of non-local image inpainting, from which several previous inpainting schemes can be derived, in addition to leading to novel ones. We explicitly study some of these, relating them to previous work and showing results on synthetic and real images.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aujol, J.-F., Ladjal, S., Masnou, S.: Exemplar-based inpainting from a variational point of view (submitted) (2008)Google Scholar
  2. 2.
    Awate, S.P., Whitaker, R.T.: Unsupervised, information-theoretic, adaptive image filtering for image restoration. IEEE Trans. on PAMI 28(3), 364–376 (2006)CrossRefGoogle Scholar
  3. 3.
    Ballester, C., Bertalmío, M., Caselles, V., Sapiro, G., Verdera, J.: Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans. on IP 10(8), 1200–1211 (2001)MathSciNetMATHGoogle Scholar
  4. 4.
    Bertalmío, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proc. of SIGGRAPH (2000)Google Scholar
  5. 5.
    Bertalmío, M., Vese, L., Sapiro, G., Osher, S.: Simultaneous structure and texture inpainting. IEEE Trans. on Image Processing 12(8), 882–889 (2003)CrossRefGoogle Scholar
  6. 6.
    Bornard, R., Lecan, E., Laborelli, L., Chenot, J.-H.: Missing data correction in still images and image sequences. In: Proc. ACM Int. Conf. on Multimedia (2002)Google Scholar
  7. 7.
    Bornemann, F., März, T.: Fast image inpainting based on coherence transport. J. of Math. Imag. and Vis. 28(3), 259–278 (2007)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Brox, T., Kleinschmidt, O., Cremers, D.: Efficient nonlocal means for denoising of textural patterns. IEEE Trans. on IP 17(7), 1057–1092 (2008)MathSciNetGoogle Scholar
  9. 9.
    Buades, A., Coll, B., Morel, J.M.: A non local algorithm for image denoising. In: Proc. of the IEEE Conf. on CVPR, vol. 2, pp. 60–65 (2005)Google Scholar
  10. 10.
    Cao, F., Gousseau, Y., Masnou, S., Pérez, P.: Geometrically-guided exemplar-based inpainting (submitted) (2008)Google Scholar
  11. 11.
    Chan, T., Kang, S.H., Shen, J.H.: Euler’s elastica and curvature based inpaintings. SIAM J. App. Math. 63(2), 564–592 (2002)MathSciNetMATHGoogle Scholar
  12. 12.
    Chan, T., Shen, J.H.: Mathematical models for local nontexture inpaintings. SIAM J. App. Math. 62(3), 1019–1043 (2001)MathSciNetMATHGoogle Scholar
  13. 13.
    Cheng, Y.: Mean shift, mode seeking and clustering. IEEE Trans. on PAMI 17(8), 790–799 (1995)CrossRefGoogle Scholar
  14. 14.
    Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based inpainting. IEEE Trans. on IP 13(9), 1200–1212 (2004)Google Scholar
  15. 15.
    Csiszár, I.: Axiomatic characterizations of information measures. Entropy 10(3), 261–273 (2008)CrossRefMATHGoogle Scholar
  16. 16.
    Demanet, L., Song, B., Chan, T.: Image inpainting by correspondence maps: a deterministic approach. Technical report, UCLA (2003)Google Scholar
  17. 17.
    Drori, I., Cohen-Or, D., Yeshurun, H.: Fragment-based image completion. ACM Trans. on Graphics. Special issue: Proc. of ACM SIGGRAPH 22(3), 303–312 (2003)CrossRefGoogle Scholar
  18. 18.
    Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proc. of the IEEE Conf. on CVPR, September 1999, pp. 1033–1038 (1999)Google Scholar
  19. 19.
    Esedoglu, S., Shen, J.H.: Digital image inpainting by the Mumford-Shah-Euler image model. European J. App. Math. 13, 353–370 (2002)MATHGoogle Scholar
  20. 20.
    Gilboa, G., Osher, S.J.: Nonlocal linear image regularization and supervised segmentation. SIAM Mult. Mod. and Sim. 6(2), 595–630 (2007)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Igehy, H., Pereira, L.: Image replacement through texture synthesis. In: Proc. of the IEEE Conf. on CVPR (October 1997)Google Scholar
  22. 22.
    Jaynes, E.T.: Information theory and statistical mechanics. Physical Review 106(4), 620–630 (1957)MathSciNetCrossRefMATHGoogle Scholar
  23. 23.
    Jia, J., Tang, C.-K.: Inference of segmented color and texture description by tensor voting. IEEE Trans. on PAMI 26(6), 771–786 (2004)CrossRefGoogle Scholar
  24. 24.
    Komodakis, N., Tziritas, G.: Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Trans. on IP 16(11), 2649–2661 (2007)MathSciNetGoogle Scholar
  25. 25.
    Levin, A., Zomet, A., Weiss, Y.: Learning how to inpaint from global image statistics. In: Proc. of ICCV (2003)Google Scholar
  26. 26.
    Levina, E., Bickel, P.: Texture synthesis and non-parametric resampling of random fields. Annals of Statistics 34(4) (2006)Google Scholar
  27. 27.
    Mahmoudi, M., Sapiro, G.: Fast image and video denoising via nonlocal means of similar neighborhoods. Signal Processing Letters 12(12), 839–842 (2005)CrossRefGoogle Scholar
  28. 28.
    Masnou, S.: Disocclusion: a variational approach using level lines. IEEE Trans. on IP 11(2), 68–76 (2002)MathSciNetGoogle Scholar
  29. 29.
    Masnou, S., Morel, J.-M.: Level lines based disocclusion. In: Proc. of IEEE ICIP (1998)Google Scholar
  30. 30.
    Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. In: Proc. of SIGGRAPH (2003)Google Scholar
  31. 31.
    Pérez, P., Gangnet, M., Blake, A.: PatchWorks: Example-based region tiling for image editing. Technical report, Microsoft Research (2004)Google Scholar
  32. 32.
    Peyré, G.: Manifold models for signals and images. Comp. Vis. and Im. Unders. 113(2), 249–260 (2009)CrossRefGoogle Scholar
  33. 33.
    Peyré, G., Bougleux, S., Cohen, L.: Non-local regularization of inverse problems. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, pp. 57–68. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  34. 34.
    Protter, M., Elad, M., Takeda, H., Milanfar, P.: Generalizing the non-local-means to super-resolution reconstruction. IEEE Trans. on IP 18(1), 36–51 (2009)Google Scholar
  35. 35.
    Singer, A., Shkolnisky, Y., Nadler, B.: Diffusion interpretation of non-local neighborhood filters for signal denoising. SIAM J. on Imag. Sci. 2(1), 118–139 (2009)CrossRefMATHGoogle Scholar
  36. 36.
    Tschumperlé, D., Deriche, R.: Vector-valued image regularization with PDE’s: a common framework for different applications. IEEE Trans. on PAMI 27(4) (2005)Google Scholar
  37. 37.
    Wei, L.-Y., Levoy, M.: Fast texture synthesis using tree-structured vector quantization. In: Proc. of the SIGGRAPH (2000)Google Scholar
  38. 38.
    Wexler, Y., Shechtman, E., Irani, M.: Space-time completion of video. IEEE Trans. on PAMI 29(3), 463–476 (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Pablo Arias
    • 1
  • Vicent Caselles
    • 1
  • Guillermo Sapiro
    • 2
  1. 1.Universitat Pompeu Fabra, DTICBarcelonaSpain
  2. 2.University of Minnesota, ECEMinneapolisUSA

Personalised recommendations