Learning Reaction-Diffusion Models for Image Inpainting

  • Wei YuEmail author
  • Stefan Heber
  • Thomas Pock
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9358)


In this paper we present a trained diffusion model for image inpainting based on the structural similarity measure. The proposed diffusion model uses several parametrized linear filters and influence functions. Those parameters are learned in a loss based approach, where we first perform a greedy training before conducting a joint training to further improve the inpainting performance. We provide a detailed comparison to state-of-the-art inpainting algorithms based on the TUM-image inpainting database. The experimental results show that the proposed diffusion model is efficient and achieves superior performance. Moreover, we also demonstrate that the proposed method has a texture preserving property, that makes it stand out from previous PDE based methods.


  1. 1.
    Bertalmio, M., Vese, L., Sapiro, G., Osher, S.: Simultaneous structure and texture image inpainting. In: proceedings of CVPR, vol. 2, pp. II-707-12 (2003)Google Scholar
  2. 2.
    Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Conference on Computer graphics and interactive techniques, pp. 417–424. ACM Press/Addison-Wesley Publishing Co. (2000)Google Scholar
  3. 3.
    Bugeau, A., Bertalmio, M., Caselles, V., Sapiro, G.: A comprehensive framework for image inpainting. Image Process. 19(10), 2634–2645 (2010)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Cao, F., Gousseau, Y., Masnou, S., Pérez, P.: Geometrically guided exemplar-based inpainting. SIAM J. Img. Sci. 4(4), 1143–1179 (2011)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Caselles, V.: Exemplar-based image inpainting and applications. SIAM News 44(10), 1–3 (2011)Google Scholar
  6. 6.
    Chan, T.F., Shen, J.: Local inpainting models and TV inpainting. SIAM J. Appl. Math. 62(3), 1019–1043 (2001)MathSciNetGoogle Scholar
  7. 7.
    Chan, T.F., Shen, J.: Nontexture inpainting by curvature-driven diffusions. J. Vis. Commun. Image Represent. 12(4), 436–449 (2001)CrossRefGoogle Scholar
  8. 8.
    Chen, Y., Ranftl, R., Pock, T.: A bi-level view of inpainting-based image compression (2014). arXiv preprint arXiv:1401.4112
  9. 9.
    Chen, Y., Ranftl, R., Pock, T.: Insights into analysis operator learning: from patch-based sparse models to higher order MRFs. Image Process. 23(3), 1060–1072 (2014)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Chen, Y., Yu, W., Pock, T.: On learning optimized reaction diffusion processes for effective image restoration. In: Proceeding of CVPR (2015)Google Scholar
  11. 11.
    Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. Trans. Img. Proc. 13(9), 1200–1212 (2004)CrossRefGoogle Scholar
  12. 12.
    Efros, A., Leung, T.: Texture synthesis by non-parametric sampling. In: proceeding of ICCV, vol. 2, pp. 1033–1038 (1999)Google Scholar
  13. 13.
    Esedoglu, S., Shen, J.: Digital inpainting based on the Mumford-Shah-Euler image model. Eur. J. Appl. Math. 13(04), 353–370 (2002)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Facciolo, G., Arias, P., Caselles, V., Sapiro, G.: Exemplar-based interpolation of sparsely sampled images. In: Cremers, D., Boykov, Y., Blake, A., Schmidt, F.R. (eds.) EMMCVPR 2009. LNCS, vol. 5681, pp. 331–344. Springer, Heidelberg (2009) CrossRefGoogle Scholar
  15. 15.
    Galić, I., Weickert, J., Welk, M., Bruhn, A., Belyaev, A., Seidel, H.P.: Image compression with anisotropic diffusion. JMIV 31(2–3), 255–269 (2008)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Getreuer, P.: Total variation inpainting using split bregman. Image Process. On Line 2, 147–157 (2012)CrossRefGoogle Scholar
  17. 17.
    Grossauer, H.: A combined PDE and texture synthesis approach to inpainting. In: Pajdla, T., Matas, J.G. (eds.) ECCV 2004. LNCS, vol. 3022, pp. 214–224. Springer, Heidelberg (2004) CrossRefGoogle Scholar
  18. 18.
    Hel-Or, Y., Shaked, D.: A discriminative approach for wavelet denoising. IEEE Trans. Image Process. 17(4), 443–457 (2008)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Herling, J., Broll, W.: Pixmix: A real-time approach to high-quality diminished reality. In: International Symposium on Mixed and Augmented Reality (ISMAR), pp. 141–150. IEEE (2012)Google Scholar
  20. 20.
    Jain, V., Seung, S.: Natural image denoising with convolutional networks. In: Advances in Neural Information Processing Systems, pp. 769–776 (2009)Google Scholar
  21. 21.
    Kokaram, A.C., Morris, R.D., Fitzgerald, W.J., Rayner, P.J.: Interpolation of missing data in image sequences. Image Process. 4(11), 1509–1519 (1995)CrossRefGoogle Scholar
  22. 22.
    Komodakis, N., Tziritas, G.: Image completion using efficient belief propagation via priority scheduling and dynamic pruning. Trans. Img. Proc. 16(11), 2649–2661 (2007)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Kong, X., Li, K., Yang, Q., Wenyin, L., Yang, M.H.: A new image quality metric for image auto-denoising. In: Proceeding of ICCV, pp. 2888–2895. IEEE (2013)Google Scholar
  24. 24.
    Levin, A., Zomet, A., Weiss, Y.: Learning how to inpaint from global image statistics. In: poceeding of ICCV, pp. 305–312 (2003)Google Scholar
  25. 25.
    Liu, A., Lin, W., Narwaria, M.: Image quality assessment based on gradient similarity. Image Process. 21(4), 1500–1512 (2012)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Liu, D., Sun, X., Wu, F., Li, S., Zhang, Y.Q.: Image compression with edge-based inpainting. Circuits Syst. Video Technol. 17(10), 1273–1287 (2007)CrossRefGoogle Scholar
  27. 27.
    Liu, D.C., Nocedal, J.: On the limited memory BFGS method for large scale optimization. Math. Program. 45(1–3), 503–528 (1989)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Masnou, S., Morel, J.M.: Level lines based disocclusion. In: Proceeding of ICIP, vol. 3, pp. 259–263 (1998)Google Scholar
  29. 29.
    Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and associated variational problems. Comm. Pure Appl. Math. 42(5), 577–685 (1989)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Nikolaidis, N., Pitas, I.: Digital image processing in painting restoration and archiving. In: proceeding of ICCV, vol. 1, pp. 586–589. IEEE (2001)Google Scholar
  31. 31.
    Peter, P., Weickert, J.: Compressing images with diffusion- and exemplar-based inpainting. In: Aujol, J.-F., Nikolova, M., Papadakis, N. (eds.) SSVM 2015. LNCS, vol. 9087, pp. 154–165. Springer, Heidelberg (2015) Google Scholar
  32. 32.
    Rehman, A., Wang, Z.: Ssim-based non-local means image denoising. In: proceeding of ICIP, pp. 217–220. IEEE (2011)Google Scholar
  33. 33.
    Roth, S., Black, M.: Fields of experts: a framework for learning image priors. In: proceeding of CVPR, vol. 2, pp. 860–867 (2005)Google Scholar
  34. 34.
    Roth, S., Black, M.: Steerable random fields. In: proceeding of ICCV, pp. 1–8 (2007)Google Scholar
  35. 35.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60(1–4), 259–268 (1992)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Schmidt, U., Gao, Q., Roth, S.: A generative perspective on MRFs in low-level vision. In: proceeding of CVPR, pp. 1751–1758. IEEE (2010)Google Scholar
  37. 37.
    Tiefenbacher, P., Bogischef, V., Merget, D., Rigoll, G.: Subjective and objective evaluation of image inpainting quality. In: proceeding of ICIP. IEEE (2015)Google Scholar
  38. 38.
    TUM-image inpainting database.
  39. 39.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  40. 40.
    Xu, Z., Sun, J.: Image inpainting by patch propagation using patch sparsity. Image Process. 19(5), 1153–1165 (2010)MathSciNetCrossRefGoogle Scholar
  41. 41.
    Zhu, S.C., Mumford, D.: Prior learning and gibbs reaction-diffusion. Pattern Anal. Mach. Intell. 19(11), 1236–1250 (1997)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (, which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Authors and Affiliations

  1. 1.Institute for Computer Graphics and VisionGraz University of TechnologyGrazAustria
  2. 2.Safety & Security DepartmentAIT Austrian Institute of TechnologyGrazAustria

Personalised recommendations