Skip to main content
SpringerLink
Log in

Total Variation Wavelet Inpainting

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

We consider the problem of filling in missing or damaged wavelet coefficients due to lossy image transmission or communication. The task is closely related to classical inpainting problems, but also remarkably differs in that the inpainting regions are in the wavelet domain. New challenges include that the resulting inpainting regions in the pixel domain are usually not geometrically well defined, as well as that degradation is often spatially inhomogeneous. We propose two related variational models to meet such challenges, which combine the total variation (TV) minimization technique with wavelet representations. The associated Euler-Lagrange equations lead to nonlinear partial differential equations (PDE’s) in the wavelet domain, and proper numerical algorithms and schemes are designed to handle their computation. The proposed models can have effective and automatic control over geometric features of the inpainted images including sharp edges, even in the presence of substantial loss of wavelet coefficients, including in the low frequencies. Existence and uniqueness of the optimal inpaintings are also carefully investigated.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. R. Acar and C.R. Vogel, “Analysis of total variation penalty methods for ill-posed problems,” Inverse Prob., Vol. 10, pp. 1217–1229, 1994.

    Article  MATH  MathSciNet  Google Scholar 

  2. C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera, “Filling-in by joint interpolation of vector fields and grey levels,” IEEE Trans. Image Processing, Vol. 10, No. 8, pp. 1200–1211, 2001.

  3. M. Bertalmio, A.L. Bertozzi, and G. Sapiro, “Navier-Stokes, fluid dynamics, and image and video inpainting,” 2001 IEEE Conference on Computer Vision and Pattern Recognition, December. Kauai, Hawaii. Available at http://www.ece.umn.edu/users/marcelo/final-cvpr.pdf.

  4. M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, Image Inpainting, Tech. Report, ECE-University of Minnesota, 1999.

  5. M. Bertalmio, L. Vese, G. Sapiro, and S. Osher, Simultaneous Structure and Texture Image Inpainting, UCLA CAM Report 02-47, July 2002,

  6. E. Candès and D. Donoho, “Curvelets—A surprisingly effective nonadaptive representation for objects with edges,” in Curves and Surfaces, L.L. Schumaker et al. (Eds.), Vanderbilt University Press: Nashville, TN, 1999.

  7. E.J. Candès and F. Guo, “Edge-preserving image reconstruction from noisy radon data” (Invited Special Issue of the Journal of Signal Processing on Image and Video Coding Beyond Standards.), 2001.

  8. E. Candès, J. Romberg, and T. Tao, Robust Uncertainty Principles: Exact Signal Reconstruction from Highly Incomplete Frequency Information, Preprint: arXiv:math.GM/0409186, Sept. 2004.

  9. E. Candès and T. Tao, “Near optimal signal recovery from random projections and universal encoding strategies,” Preprint, submitted to IEEE Information Theory, Oct. 2004.

  10. V. Caselles, R. Kimmel, and G. Sapiro, “On geodesic active contours,” Int. Journal of Computer Vision, Vol. 22, No. 1, pp. 61–79, 1997.

    Article  MATH  Google Scholar 

  11. A. Chambolle, R.A. DeVore, N.-Y. Lee, and B.J. Lucier, “Nonlinear wavelet image processing: Variational problems, compression and noise removal through wavelet shrinkage,” IEEE Trans. Image Processing, Vol. 7, No. 3, pp. 319–335, 1998.

  12. A. Chambolle and P.L. Lions, “Image recovery via Total Variational minimization and related problems,” Numer. Math., Vol. 76, pp. 167–188, 1997.

    Article  MATH  MathSciNet  Google Scholar 

  13. T.F. Chan, S.H. Kang, and J. Shen, “Euler’s elastica and curvature based inpainting,” SIAM J. Appl. Math., Vol. 63, No. 2, pp. 564–592, 2002.

    Article  MATH  MathSciNet  Google Scholar 

  14. T.F. Chan, S. Osher, and J. Shen, “The digital tv filter and nonlinear denoising,” IEEE Trans. Image Process., Vol. 10, No. 2, pp. 231–241, 2001.

  15. T.F. Chan and J. Shen, “Mathematical models for local non-texture inpainting,” SIAM J. Appl. Math., Vol. 62, No. 3, pp. 1019–1043, 2002.

    Article  MATH  MathSciNet  Google Scholar 

  16. T.F. Chan and J. Shen, Morphologically Invariant PDE Inpaintings, UCLA CAM Report 01-15, 2001.

  17. T.F. Chan, J. Shen, and L. Vese, “Variational PDE models in image processing,” Notices of AMS, Vol. 50, No. 1, 2003.

  18. T.F. Chan and L. Vese, “Active contour without edges,” Submit to IEEE Tran. on Image Proc., 1998.

  19. T.F. Chan and C.K. Wong, “Total variation blind deconvolution,” IEEE Trans. Image Processing, Vol. 7, pp. 370– 375, 1998.

  20. T.F. Chan and H.M. Zhou, “ENO-wavelet transforms for piecewise smooth functions,” SIAM J. Numer. Anal., Vol. 40, No. 4, pp. 1369–1404, 2002.

    Article  MATH  MathSciNet  Google Scholar 

  21. T.F. Chan and H.M. Zhou, ”Total variation minimizing wavelet coefficients for image compression and denoising,” submitted to SIAM Journal on Scientific Computing.

  22. T.F. Chan and H.M. Zhou, Optimal Constructions of Wavelet Coefficients Using Total Variation Regularization in Image Compression, CAM Report, No. 00-27, Dept. of Math., UCLA, July 2000.

  23. M.Y. Cheng, H.Y. Huang, and A.W.Y. Su, A NURBS-based error concealment techniques for corrupted images from packet loss, in Proc. ICIP, bf 2, 2002, pp. 705–708.

  24. A. Cohen, R. DeVore, P. Petrushev, and H. Xu. “Nonlinear approximation and the space BV(R 2),” Amer. J. Math., Vol. 121, pp. 587–628, 1999.

    MATH  MathSciNet  Google Scholar 

  25. I. Daubechies, Ten Lectures on Wavelets, SIAM: Philadelphia, 1992.

  26. R.A. DeVore, B. Jawerth, and B.J. Lucier, “Image compression through wavelet transform coding,” IEEE Trans. Information Theory, Vol. 38, No. 2, pp. 719–746, 1992.

    Article  MATH  MathSciNet  Google Scholar 

  27. L. Demanet, B. Song, and T. Chan, Image Inpainting by Correspondence Maps: a Deterministic Approach, UCLA CAM Report 03-40, August 2003.

  28. D. Dobson and C.R. Vogel, “Convergence of an iterative method for total variation denoising,” SIAM Journal on Numerical Analysis, Vol. 34, pp. 1779–1971, (1997).

    Article  MATH  MathSciNet  Google Scholar 

  29. D.L. Donoho, “De-noising by soft-thresholding,” IEEE Trans. Information Theory, Vol. 41, No. 3, pp. 613–627, 1995.

    Article  MATH  MathSciNet  Google Scholar 

  30. D.L. Donoho and I.M. Johnstone, “Ideal spacial adaption by wavelet shrinkage,” Biometrika, Vol. 81, pp. 425–455, 1994.

    Article  MATH  MathSciNet  Google Scholar 

  31. D. Dugatkin, H.M. Zhou, T.F. Chan, and M. Effros, “Lagrangian optimization of a group testing for ENO wavelets algorithm,” Proceedings to the 2002 Conference on Information Sciences and Systems, Princeton University, New Jersey, March 20–22, 2002.

  32. S. Durand and J. Froment, “Artifact free signal denoising with wavelets,” in Proceedings of ICASSP’01, Vol. 6, pp. 3685–3688, 2001.

  33. M. Elad, J.-L. Starck, P. Querre, and D. Donoho, “Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA),” to appear in the J. on Applied and Computational Harmonic Analysis.

  34. S. Esedoglu and J. Shen, “Digital inpainting based on the Mumford-Shah-Euler image model,” European J. Appl. Math., Vol. 13, pp. 353–370, 2002.

    Article  MATH  MathSciNet  Google Scholar 

  35. W.H. Fleming and R. Rishel, “An integral formula for total gradient variation,” Arch. Math., Vol. 11, pp. 218–222, 1960.

    Article  MATH  MathSciNet  Google Scholar 

  36. E. De Giorgi, “Complementi alla teoria della misura (n−1)-dimensionale in uno spazio n-dimensionale,” Sem. Mat. Scuola Norm. Sup. Pisa, pp. 1960–1961.

  37. E. Giusti, Minimal Surfaces and Functions of Bounded Variation. Birkhäuser: Boston, 1984.

  38. “Special issue on partial differential equations and geometry-driven diffusion in image processing and analysis,” IEEE Tran. on Image Proc., Vol. 7, No. 3, 1998.

  39. P.C. Hansen, “The L-curve and Its Use in the Numerical Treatment of Inverse Problems,” Tech. Report, IMM-REP 99-15, Dept. of Math. Model., Tech. Univ. of Denmark, 1999.

  40. S.S. Hemami and R.M. Gray, “Subband-coded image reconstruction for lossy packet networks,” IEEE Trans. Image Proc., Vol. 6, No. 4, pp. 523–539, 1997.

  41. H. Igehy and L. Pereira, “Image replacement through texture synthesis,” Proceedings to IEEE ICIP, 1997.

  42. K.-H. Jung, J.-H. Chang, and C.W. Lee, “Error concealment technique using data for block-based image coding,” SPIE, Vol. 2308, pp. 1466–1477, 1994.

  43. S.H. Kang, T.F. Chan, and S. Soatto, Inpainting from multiple view, UCLA CAM Report 02–31, May 2002.

  44. E.H. Lieb and M. Loss, Analysis, second edition, Amer. Math. Soc., 2001.

  45. F. Malgouyres, Increase in the Resolution of Digital Images: Variational Theory and Applications, Ph.D. thesis, Ecole Normale Supérieure de Cachan, 2000, Cachan, France.

  46. F. Malgouyres, “Mathematical analysis of a model which combines total variation and wavelet for image restoration,” Journal of Information Processes, Vol. 2, No. 1, pp. 1–10, 2002.

  47. F. Malgouyres and F. Guichard, “Edge direction preserving image zooming: a mathematical and numerical analysis,” SIAM, J. Num. Anal., Vol. 39, No. 1, pp. 1–37, 2001.

    Article  MATH  MathSciNet  Google Scholar 

  48. L. Moisan, Extrapolation de Spectre et Variation Totale Ponderee, actes du GRETSI, 2001.

  49. M. Masnou and J. Morel, “Level-lines based disocclusion,” IEEE ICIP, Oct. 1998, pp. 259–263.

  50. D. Marr, Vision, Freeman: San Francisco, 1980.

    Google Scholar 

  51. Y. Meyer, Oscillating Patterns in Image Processing and Nonlinear Evolution Equations, volume 22 of University Lecture Series, AMS, Providence, 2001.

  52. D. Mumford and J. Shah, “Optimal approximations by piecewise smooth functions and associated variational problems,” Comm. Pure Applied. Math., Vol. 42, pp. 577–685, 1989.

    MATH  MathSciNet  Google Scholar 

  53. M. Nitzberg, D. Mumford, and T. Shiota, “Filtering, segmentation, and depth, Lecture Notes in Comp. Sci., Vol. 662, Springer-Verlag: Berlin, 1993.

  54. Y. Niu and T. Poston, Harmonic Postprocessing to Conceal for Transmission Errors in DWT Coded Images, preprint, Institute of Eng. Sci., National Univ. of Singapore, 2003.

  55. J.W. Park and S.U. Lee, “Recovery of corrupted image data based on the NURBS interpolation,” IEEE Trans. Circuits Syst. Video Tech., Vol. 9, No. 7, pp. 1003–1008, 1999.

    Article  MathSciNet  Google Scholar 

  56. I. Peterson, Filling in blanks, Science News, Vol. 161/19, May 11, 2002.

  57. S. Rane, J. Remus, and G. Sapiro, “Wavelet-domain reconstruction of lost blocks in wireless image transmission and packet-switched networks,” IEEE ICIP, 2002.

  58. S. Rane, G. Sapiro, and M. Bertalmio, “Structure and texture filling-in of missing image blocks in wireless transmission and compression applications,” IEEE Trans. Image Processing, Vol. 12, pp. 296–303, 2003.

  59. L. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D, Vol. 60, pp. 259–268, 1992.

    Article  MATH  Google Scholar 

  60. G. Sapiro, Geometric Partial Differential Equations and Image Processing, Cambridge University Press, 2001.

  61. G. Sapiro and A. Tannenbaum, “Affine invariant scale-space, Internet. J. Comput. Vision, Vol. 11, pp. 25–44, 1993.

    Article  Google Scholar 

  62. J.L. Starck, M. Elad, and D. Donoho, “Image decomposition via the combination of sparse representations and a variational approach,” to appear in the IEEE Trans. Image Processing.

  63. G. Strang and T. Nguyen, Wavelets and Filter Banks, Wellesley-Cambridge Press: Wellesley, MA, 1996.

    Google Scholar 

  64. Y. Wang and Q.F. Zhu, “Error control and concealment for video communication: A review,” Proc. IEEE, Vol. 86, No. 5, pp. 974–997.

  65. A. Yezzi, A. Tsai, and A. Willsky, “A fully global approach to image segmentation via coupled curve evolution equations, Submitted to Journal of Visual Communication and Image Representation.

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, University of California, Los Angeles, CA, 90095-1555

    Tony F. Chan

  2. Department of Mathematics, University of Minnesota, 206 Church St. SE Minneapolis, MN, 55455

    Jianhong Shen

  3. School of Mathematics, Georgia Institute of Technology, Atlanta, GA, 30332

    Hao-Min Zhou

Authors
  1. Tony F. Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianhong Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hao-Min Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tony F. Chan.

Additional information

Research supported in part by grants ONR-N00014-03-1-0888, NSF DMS-9973341, DMS-0202565 and DMS-0410062, and NIH contract P 20 MH65166.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chan, T.F., Shen, J. & Zhou, HM. Total Variation Wavelet Inpainting. J Math Imaging Vis 25, 107–125 (2006). https://doi.org/10.1007/s10851-006-5257-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-006-5257-3

Keywords

Search

Navigation