International Journal of Computer Vision

, Volume 93, Issue 3, pp 319–347 | Cite as

A Variational Framework for Exemplar-Based Image Inpainting

  • Pablo Arias
  • Gabriele Facciolo
  • Vicent Caselles
  • Guillermo Sapiro


Non-local methods for image denoising and inpainting have gained considerable attention in recent years. This is in part due to their superior performance in textured images, a known weakness of purely local methods. Local methods on the other hand have demonstrated to be very appropriate for the recovering of geometric structures such as image edges. The synthesis of both types of methods is a trend in current research. Variational analysis in particular is an appropriate tool for a unified treatment of local and non-local methods. In this work we propose a general variational framework for non-local image inpainting, from which important and representative previous inpainting schemes can be derived, in addition to leading to novel ones. We explicitly study some of these, relating them to previous work and showing results on synthetic and real images.


Inpainting Variational methods Self-similarity Non-local methods Exemplar-based methods 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Aharon, M., Elad, M., & Bruckstein, A. M. (2006). The K-SVD: an algorithm for designing of overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing, 54(11), 4311–4322. CrossRefGoogle Scholar
  2. Almansa, A., Caselles, V., Haro, G., & Rougé, B. (2006). Restoration and zoom of irregularly sampled, blurred, and noisy images by accurate total variation minimization with local constraints. Multiscale Modeling & Simulation, 5(1), 235–272. CrossRefMATHMathSciNetGoogle Scholar
  3. Arias, P., Caselles, V., & Sapiro, G. (2009). A variational framework for non-local image inpainting. In Lecture notes in computer science. EMMCVPR (pp. 345–358). Berlin: Springer. Google Scholar
  4. Aujol, J.-F., Ladjal, S., & Masnou, S. (2010). Exemplar-based inpainting from a variational point of view. SIAM Journal on Mathematical Analysis, 42(3), 1246–1285. CrossRefMathSciNetGoogle Scholar
  5. Awate, S. P., & Whitaker, R. T. (2005). Higher-order image statistics for unsupervised, information-theoretic, adaptive, image filtering. In Proc. of CVPR (pp. 44–51). Google Scholar
  6. Ballester, C., Bertalmío, M., Caselles, V., Sapiro, G., & Verdera, J. (2001). Filling-in by joint interpolation of vector fields and gray levels. IEEE Transactions on IP, 10(8), 1200–1211. MATHGoogle Scholar
  7. Barnes, C., Shechtman, E., Finkelstein, A., & Goldman, D. B. (2009). PatchMatch: a randomized correspondence algorithm for structural image editing. In Proc. of SIGGRAPH (pp. 1–11). New York: ACM. Google Scholar
  8. Bertalmío, M., Sapiro, G., Caselles, V., & Ballester, C. (2000). Image inpainting. In Proc. of SIGGRAPH (pp. 417–424). New York: ACM. CrossRefGoogle Scholar
  9. Bertalmío, M., Vese, L., Sapiro, G., & Osher, S. J. (2003). Simultaneous structure and texture inpainting. IEEE Transactions on IP, 12(8), 882–889. Google Scholar
  10. Bornard, R., Lecan, E., Laborelli, L., & Chenot, J.-H. (2002). Missing data correction in still images and image sequences. In Proc. ACM int. conf. on multimedia. Google Scholar
  11. Bornemann, F., & März, T. (2007). Fast image inpainting based on coherence transport. Journal of Mathematical Imaging and Vision, 28(3), 259–278. CrossRefMathSciNetGoogle Scholar
  12. Brox, T., Kleinschmidt, O., & Cremers, D. (2008). Efficient nonlocal means for denoising of textural patterns. IEEE Transaction on IP, 17(7), 1057–1092. MathSciNetGoogle Scholar
  13. Buades, A., Coll, B., & Morel, J.-M. (2005). A non local algorithm for image denoising. In Proc. of the IEEE conf. on CVPR (Vol. 2, pp. 60–65). Google Scholar
  14. Candes, E. J., & Wakin, M. B. (2008). An introduction to compressive sampling. IEEE Signal Processing Magazine, 25(2), 21–30. CrossRefGoogle Scholar
  15. Cao, F., Gousseau, Y., Masnou, S., & Pérez, P. (2009). Geometrically guided exemplar-based inpainting. Google Scholar
  16. Chambolle, A. (2004). An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision, 20(1–2), 89–97. MathSciNetGoogle Scholar
  17. Chan, T., & Shen, J. H. (2001). Mathematical models for local nontexture inpaintings. SIAM Journal on Applied Mathematics, 62(3), 1019–1043. MathSciNetGoogle Scholar
  18. Chan, T., Kang, S. H., & Shen, J. H. (2002). Euler’s elastica and curvature based inpaintings. SIAM Journal on Applied Mathematics, 63(2), 564–592. MATHMathSciNetGoogle Scholar
  19. Cheng, Y. (1995). Mean shift, mode seeking and clustering. IEEE Transactions on PAMI, 17(8), 790–799. Google Scholar
  20. Criminisi, A., Pérez, P., & Toyama, K. (2004). Region filling and object removal by exemplar-based inpainting. IEEE Transactions on IP, 13(9), 1200–1212. Google Scholar
  21. Csiszár, I. (2008). Axiomatic characterizations of information measures. Entropy, 10(3), 261–273. CrossRefMATHGoogle Scholar
  22. Demanet, L., Song, B., & Chan, T. (2003). Image inpainting by correspondence maps: a deterministic approach (Technical report). UCLA. Google Scholar
  23. Drori, I., Cohen-Or, D., & Yeshurun, H. (2003). Fragment-based image completion. In Proc. of ACM SIGGRAPH (pp. 303–312). New York: ACM. Google Scholar
  24. Efros, A. A., & Leung, T. K. (1999). Texture synthesis by non-parametric sampling. In Proc. of the IEEE ICCV, September 1999 (pp. 1033–1038). Google Scholar
  25. Elad, M., Starck, J. L., Querre, P., & Donoho, D. L. (2005). Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA). Applied and Computational Harmonic Analysis, 19(3), 340–358. CrossRefMATHMathSciNetGoogle Scholar
  26. Esedoglu, S., & Shen, J. H. (2002). Digital image inpainting by the Mumford-Shah-Euler image model. European Journal of Applied Mathematics, 13, 353–370. CrossRefMATHMathSciNetGoogle Scholar
  27. Facciolo, G., Arias, P., Caselles, V., & Sapiro, G. (2009). Exemplar-based interpolation of sparsely sampled images. In Lecture notes in computer science. EMMCVPR (pp. 331–344). Berlin: Springer. Google Scholar
  28. Fang, C.-W., & Lien, J.-J. J. (2009). Rapid image completion system using multiresolution patch-based directional and nondirectional approaches. IEEE Transactions on IP, 18(12), 2769–2779. Google Scholar
  29. Gilboa, G., & Osher, S. J. (2007). Nonlocal linear image regularization and supervised segmentation. SIAM Multiscale Modeling and Simulation, 6(2), 595–630. CrossRefMATHMathSciNetGoogle Scholar
  30. Gilboa, G., & Osher, S. (2008). Nonlocal operators with applications to image processing. Multiscale Modeling & Simulation, 7(3), 1005–1028. CrossRefMATHMathSciNetGoogle Scholar
  31. Han, J., Zhou, K., Wei, L.-Y., Gong, M., Bao, H., Zhang, X., & Guo, B. (2006). Fast example-based surface texture synthesis via discrete optimization. The Visual Computer, 22(9), 918–925. CrossRefGoogle Scholar
  32. Harrison, P. (2005). Texture tools. PhD thesis, Monash University. Google Scholar
  33. Holtzman-Gazit, M., & Yavneh, I. (2008). A scale-consistent approach to image completion. International Journal of Multiscale Computer Engineering, 6(6), 617–628. CrossRefGoogle Scholar
  34. Igehy, H., & Pereira, L. (1997). Image replacement through texture synthesis. In Proc. of the IEEE ICIP. Google Scholar
  35. Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical Review, 106(4), 620–630. CrossRefMathSciNetGoogle Scholar
  36. Jia, J., & Tang, C.-K. (2004). Inference of segmented color and texture description by tensor voting. IEEE Transactions on PAMI, 26(6), 771–786. Google Scholar
  37. Kawai, N., Sato, T., & Yokoya, N. (2009). Image inpainting considering brightness change and spatial locality of textures and its evaluation. In Ad. in image and video tech. (pp. 271–282). Berlin: Springer. CrossRefGoogle Scholar
  38. Kimball, S., Mattis, P., & the GIMP Dev. Team (2009). GIMP: GNU Image Manipulation Program. Version 2.6.8 released on December 2009.
  39. Komodakis, N., & Tziritas, G. (2007). Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Transactions on IP, 16(11), 2649–2661. MathSciNetGoogle Scholar
  40. Kwatra, V., Essa, I., Bobick, A., & Kwatra, N. (2005). Texture optimization for example-based synthesis. ACM Transactions on Graphics, 24(3), 795–802. CrossRefGoogle Scholar
  41. Levin, A., Zomet, A., & Weiss, Y. (2003). Learning how to inpaint from global image statistics. In Proc. of IEEE ICCV. Google Scholar
  42. Levina, E., & Bickel, P. (2006). Texture synthesis and non-parametric resampling of random fields. Annals of Statistics, 34(4). Google Scholar
  43. Lezoray, O., Elmoataz, A., & Bougleux, S. (2007). Graph regularization for color image processing. Computer Vision Image Understanding, 107(1–2), 38–55. CrossRefGoogle Scholar
  44. Mairal, J., Sapiro, G., & Elad, M. (2008). Learning multiscale sparse representations for image and video restoration. SIAM Multiscale Modeling and Simulation, 7(1), 214–241. CrossRefMATHMathSciNetGoogle Scholar
  45. Masnou, S. (2002). Disocclusion: a variational approach using level lines. IEEE Transactions on IP, 11(2), 68–76. MathSciNetGoogle Scholar
  46. Masnou, S., & Morel, J.-M. (1998). Level lines based disocclusion. In Proc. of IEEE ICIP. Google Scholar
  47. Morel, J.-M., & Yu, G. (2008). On the consistency of the SIFT method. Preprint, CMLA, 26. Google Scholar
  48. Pérez, P., Gangnet, M., & Blake, A. (2003). Poisson image editing. In Proc. of SIGGRAPH (pp. 313–318). New York: ACM. Google Scholar
  49. Pérez, P., Gangnet, M., & Blake, A. (2004). PatchWorks: example-based region tiling for image editing (Technical report). Microsoft Research. Google Scholar
  50. Peyré, G. (2009). Manifold models for signals and images. Computer Vision and Image Understanding, 113(2), 249–260. CrossRefGoogle Scholar
  51. Peyré, G., Bougleux, S., & Cohen, L. (2008). Non-local regularization of inverse problems. In ECCV ’08 (pp. 57–68). Berlin: Springer. Google Scholar
  52. Peyré, G., Bougleux, S., & Cohen, L. D. (2009). Non-local regularization of inverse problems. Preprint Hal-00419791. Google Scholar
  53. Pizarro, L., Mrázek, P., Didas, S., Grewenig, S., & Weickert, J. (2010). Generalised nonlocal image smoothing. International Journal of Computer Vision, 90, 62–87. CrossRefGoogle Scholar
  54. Protter, M., Elad, M., Takeda, H., & Milanfar, P. (2009). Generalizing the non-local-means to super-resolution reconstruction. IEEE Transactions on IP, 18(1), 36–51. MathSciNetGoogle Scholar
  55. Shen, J., Jin, X., & Zhou, C. (2005). Gradient based image completion by solving Poisson equation. In Ad. in multimedia information processing (pp. 257–68). CrossRefGoogle Scholar
  56. Sun, J., Yuan, L., Jia, J., & Shum, H. Y. (2005). Image completion with structure propagation. In Proc. of SIGGRAPH (pp. 861–868). New York: ACM. Google Scholar
  57. Tong, X., Zhang, J., Liu, L., Wang, X., Guo, B., & Shum, H.-Y. (2002). Synthesis of bidirectional texture functions on arbitrary surfaces. ACM Transactions on Graphics, 21(3), 665–672. CrossRefGoogle Scholar
  58. Tschumperlé, D., & Deriche, R. (2005). Vector-valued image regularization with PDE’s: a common framework for different applications. IEEE Transactions on PAMI, 27(4). Google Scholar
  59. Wei, L.-Y., & Levoy, M. (2000). Fast texture synthesis using tree-structured vector quantization. In Proc. of the SIGGRAPH (pp. 479–488). New York: ACM. CrossRefGoogle Scholar
  60. Wexler, Y., Shechtman, E., & Irani, M. (2007). Space-time completion of video. IEEE Transactions on PAMI, 29(3), 463–476. Google Scholar
  61. Zhou, D., & Schölkopf, B. (2005). Regularization on discrete spaces. In Proceedings of the 27th DAGM symposium (pp. 361–368). Berlin: Springer. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Pablo Arias
    • 1
  • Gabriele Facciolo
    • 1
  • Vicent Caselles
    • 1
  • Guillermo Sapiro
    • 2
  1. 1.Dept. of Information and Communication TechnologiesUniversitat Pompeu FabraBarcelonaSpain
  2. 2.Dept. of Electrical and Computer EngineeringUniversity of MinnesotaMinneapolisUSA

Personalised recommendations