Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Orientation-Matching Minimization for Image Denoising and Inpainting

Abstract

In this paper, we propose an orientation-matching functional minimization for image denoising and image inpainting. Following the two-step TV-Stokes algorithm (Rahman et al. in Scale space and variational methods in computer vision, pp. 473–482, Springer, Heidelberg, 2007; Tai et al. in Image processing based on partial differential equations, pp. 3–22, Springer, Heidelberg, 2006; Bertalmio et al. in Proc. conf. comp. vision pattern rec., pp. 355–362, 2001), a regularized tangential vector field with zero divergence condition is first obtained. Then a novel approach to reconstruct the image is proposed. Instead of finding an image that fits the regularized normal direction from the first step, we propose to minimize an orientation matching cost measuring the alignment between the image gradient and the regularized normal direction. This functional yields a new nonlinear partial differential equation (PDE) for reconstructing denoised and inpainted images. The equation has an adaptive diffusivity depending on the orientation of the regularized normal vector field, providing reconstructed images which have sharp edges and smooth regions. The additive operator splitting (AOS) scheme is used for discretizing Euler-Lagrange equations. We present the results of various numerical experiments that illustrate the improvements obtained with the new functional.

This is a preview of subscription content, log in to check access.

References

  1. Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., & Verdera, J. (2001). Filling-in by joint interpolation of vector fields and gray levels. IEEE Transactions on Image Processing, 10(8), 1200–1211.

  2. Bertalmio, M. (2006). Strong-continuation contrast-invariant inpainting with a third-order optimal PDE. IEEE Transactions on Image Processing, 15(7), 1934–1938.

  3. Bertalmio, M., Sapiro, G., Caselles, V., & Ballester, C. (2000). Image inpainting. In Computer graphics SIGGRAPH (pp. 417–424).

  4. Bertalmio, M., Bertozzi, A. L., & Sapiro, G. (2001). Navier-Stokes, fluid dynamics, and image and video inpainting. In Proc. conf. comp. vision pattern rec. (pp. 355–362).

  5. Bresson, X., & Chan, T. F. (2008). Fast dual minimization of the vectorial total variation norm and applications to color image processing. Inverse Problems and Imaging, 2(4), 455–484.

  6. Brox, T., Weickert, J., Burgeth, B., & Mrázek, P. (2006). Nonlinear structure tensors. Image Vision Computer, 24, 41–55.

  7. Catté, F., Lions, P. L., Morel, J.-M., & Coll, T. (1992). Image selective smoothing and edge detection by nonlinear diffusion. SIAM Journal on Numerical Analysis, 29(1), 182–193.

  8. Chan, T. F., & Shen, J. (2001). Nontexture inpainting by curvature driven diffusion (CDD). Journal of Visual Communication and Image Representation, 12, 436–449.

  9. Chan, T. F., & Shen, J. (2002). Mathematical models for local nontexture inpaintings. SIAM Journal on Applied Mathematics, 62(3), 1019–1043.

  10. Chan, T. F., & Shen, J. (2005). Variational image inpainting. Communications on Pure Applied Mathematics, 58, 579–619.

  11. Chan, T., & Vese, L. (2001). Active contours without edges. IEEE Transactions on Image Processing, 10(2), 266–277.

  12. Chan, T. F., Kang, S.-H., & Shen, J. (2002). Euler’s elastica and curvature based inpaintings. SIAM Journal on Applied Mathematics, 63(2), 564–594.

  13. Chan, R. H., Setzer, S., & Steidl, G. (2008). Inpainting by flexible Haar-wavelet shrinkage. SIAM Journal on Imaging. Science, 1, 273–293.

  14. Chessel, A., Cao, F., & Fablet, R. (2006). Interpolating orientation fields: An axiomatic approach. In ECCV06 (pp. IV, pp. 241–254).

  15. Dong, F., Liu, Z., Kong, D., & Liu, K. (2009). An improved LOT model for image restoration. Journal of Mathematical Imaging and Vision, 34, 89–97.

  16. Hahn, J., & Lee, C.-O. (2009). A nonlinear structure tensor with the diffusivity matrix composed of the image gradient. Journal of Mathematical Imaging and Vision, 34, 137–151.

  17. Kimmel, R., & Bruckstein, A. M. (2001). Regularized Laplacian zero crossings as optimal edge integrators. International Journal of Computer Vision, 53, 225–243.

  18. Kornprobst, P., & Aubert, G. (2006). Explicit reconstruction for image inpainting. In Research report N. 5905 INRIA.

  19. Lindenbaum, M., Fischer, M., & Bruckstein, A. M. (1994). On Gabor’s contribution to image enhancement. Pattern Recognition, 27(1), 1–8.

  20. Litvinov, W. G., Rahman, T., & Tai, X.-C. (2009). A modified TV-Stokes model for image processing. Technical report, Preprints—Herausgeber: Institut für Mathematik der Universität Augsburg 2009-25.

  21. Lu, T., Neittaanmaki, P., & Tai, X.-C. (1992). A parallel splitting up method for partial differential equations and its application to Navier-Stokes equations. RAIRO Mathematical Modeling and Numerecal Analysis, 26(6), 673–708.

  22. Lysaker, M., & Tai, X.-C. (2006). Iterative image restoration combining total variation minimization and a second-order functional. International Journal of Computer Vision, 66, 5–18.

  23. Lysaker, M., Lundervold, A., & Tai, X.-C. (2003). Noise removal using fourth-order partial differential equation with applications to medical magnetic resonance images in space and time. IEEE Transaction on Image Processing, 12(12), 1579–1590.

  24. Lysaker, M., Osher, S., & Tai, X.-C. (2004). Noise removal using smoothed normals and surface fitting. IEEE Transactions on Image Processing, 13(10), 1345–1357.

  25. Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th int’l conf. computer vision (Vol. 2, pp. 416–423), July 2001.

  26. Masnou, S., & Morel, J.-M. (1998). Level lines based disocclusion. In Proc. IEEE int. conf. on image processing, Chicago, IL (pp. 259–263).

  27. Perona, P. (1998). Orientation diffusion. IEEE Transactions on Image Processing, 7(3), 457–467.

  28. Perona, P. & Malik, J. (1990). Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(7), 629–639.

  29. Rahman, T., Tai, X.-C., & Osher, S. (2007). A TV-Stokes denoising algorithm. In Scale space and variational methods in computer vision (pp. 473–482). Heidelberg: Springer.

  30. Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D, 60, 259–268.

  31. Sochen, N. A., Sagiv, C., & Kimmel, R. (2004). Stereographic combing a porcupine or studies on direction diffusion in image processing. SIAM Journal on Applied Mathematics, 64(5), 1477–1508.

  32. Spira, A., Kimmel, R., & Sochen, N. (2007). A short-time Beltrami kernel for smoothing images and manifolds. IEEE Transactions on Image Processing, 16, 1628–1636.

  33. Steidl, G., & Teuber, T. (2009). Anisotropic smoothing using double orientations. In SSVM ’09: proceedings of the second international conference on scale space and variational methods in computer vision (pp. 477–489). Berlin: Springer.

  34. Tai, X.-C., Osher, S., & Holm, R. (2006). Image inpainting using TV-Stokes equation. In Image processing based on partial differential equations (pp. 3–22). Heidelberg: Springer.

  35. Tang, B., Sapiro, G., & Caselles, V. (2000). Diffusion of general data on non-flat manifolds via harmonic maps theory: The direction diffusion case. International Journal of Computer Vision, 36, 149–161.

  36. Terzopoulos, D. (1988). The computation of visible-surface representations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(4), 417–438.

  37. Tschumperlé, D., & Deriche, R. (2002). Orthonormal vector sets regularization with PDE’s and applications. International Journal of Computer Vision, 50(3), 237–252.

  38. Vese, L. A., & Osher, S. (2002). Numerical methods for p-harmonic flows and applications to image processing. SIAM Journal on Numerical Analysis, 40(6), 2085–2104.

  39. Weickert, J. (1999). Coherence-enhancing diffusion filtering. International Journal of Computer Vision, 31, 111–127.

  40. Weickert, J., & Welk, M. (2006). Tensor field interpolation with PDEs. In Weickert, J., & Hagen, H. (Eds.) Visualization and processing of tensor fields (pp. 315–325). Berlin: Springer.

  41. Weickert, J., ter Harr Romeny, B. M., & Viergever, M. A. (2001). Efficient and reliable schemes for nonlinear diffusion filtering. IEEE Transactions on Image Processing, 7, 398–410.

Download references

Author information

Correspondence to Jooyoung Hahn.

Additional information

The research is supported by MOE (Ministry of Education) Tier II project T207N2202 and IDM project NRF2007IDMIDM002-010. In addition, the support from SUG 20/07 is also gratefully acknowledged. Professor A.M. Bruckstein’s work was supported in part by an NTU Joint visiting professorship at the School of Physical and Mathematical Sciences and the Institute for Media Innovations.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Hahn, J., Tai, X., Borok, S. et al. Orientation-Matching Minimization for Image Denoising and Inpainting. Int J Comput Vis 92, 308–324 (2011). https://doi.org/10.1007/s11263-010-0371-5

Download citation

Keywords

  • Orientation-matching minimization
  • TV-Stokes equation
  • Image denoising
  • Image inpainting