Journal of Mathematical Imaging and Vision

, Volume 55, Issue 3, pp 401–427 | Cite as

A Second-Order TV-Type Approach for Inpainting and Denoising Higher Dimensional Combined Cyclic and Vector Space Data

  • Ronny Bergmann
  • Andreas Weinmann


In this paper, we consider denoising and inpainting problems for higher dimensional combined cyclic and linear space-valued data. These kinds of data appear when dealing with nonlinear color spaces such as HSV, and they can be obtained by changing the space domain of, e.g., an optical flow field to polar coordinates. For such nonlinear data spaces, we develop algorithms for the solution of the corresponding second-order total variation-type problems for denoising, inpainting as well as the combination of both. We provide a convergence analysis and apply the algorithms to concrete problems.


Higher order total variation minimization Vector-valued TV Cyclic data Combined denoising and inpainting Cyclic proximal point algorithm 

Mathematics Subject Classification

65K05 65K10 68U10 94A08 



This research was started when RB visited the Helmholtz-Zentrum München in summer 2014. We thank Gabriele Steidl for valuable discussions and both reviewers for their valuable comments and suggestions. The authors acknowledge funding by the DFG project BE 5888/2-1 & WE 5886/3-1. AW is supported by the Helmholtz Association within the young investigator group VH-NG-526. AW also acknowledges the support by the DFG scientific network “Mathematical Methods in Magnetic Particle Imaging.”


  1. 1.
    Almeida, M., Figueiredo, M.: Deconvolving images with unknown boundaries using the alternating direction method of multipliers. IEEE Trans. Image Process. 22(8), 3074–3086 (2013)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Bačák, M.: The proximal point algorithm in metric spaces. Isr. J. Math. 194(2), 689–701 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Bačák, M.: Computing medians and means in Hadamard spaces. SIAM J. Optim. 24(3), 1542–1566 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., Verdera, J.: Filling in by joint interpolation of vector fields and gray levels. IEEE Trans. Image Process. 10(8), 1200–1211 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Basser, P., Mattiello, J., LeBihan, D.: MR diffusion tensor spectroscopy and imaging. Biophys. J. 66, 259–267 (1994)CrossRefGoogle Scholar
  6. 6.
    Bergmann, R., Laus, F., Steidl, G., Weinmann, A.: Second order differences of cyclic data and applications in variational denoising. SIAM J. Imaging Sci. 7(4), 2916–2953 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Bergmann, R., Weinmann, A.: Inpainting of cyclic data using first and second order differences. In: EMMCVPR 2015, pp. 155–168. Springer (2015)Google Scholar
  8. 8.
    Bertsekas, D.P.: Incremental gradient, subgradient, and proximal methods for convex optimization: a survey. Technical report LIDS-P-2848, laboratory for information and decision systems. MIT, Cambridge (2010)Google Scholar
  9. 9.
    Bertsekas, D.P.: Incremental proximal methods for large scale convex optimization. Math. Prog. Ser. B 129(2), 163–195 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Bhattacharya, R., Patrangenaru, V.: Large sample theory of intrinsic and extrinsic sample means on manifolds. Ann. Stat. 31(1), 1–29 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Bhattacharya, R., Patrangenaru, V.: Large sample theory of intrinsic and extrinsic sample means on manifolds. Ann. Stat. 33(3), 1225–1259 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Blomgren, P., Chan, T.: Color TV: total variation methods for restoration of vector-valued images. IEEE Trans. Image Process. 7(3), 304–309 (1998)CrossRefGoogle Scholar
  13. 13.
    Bornemann, F., März, T.: Fast image inpainting based on coherence transport. J. Math. Imaging Vis. 28(3), 259–278 (2007)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3(3), 492–526 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Bresson, X., Chan, T.F.: Fast dual minimization of the vectorial total variation norm and applications to color image processing. Inverse Prob. Imaging 2(4), 455–484 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Bugeau, A., Bertalmío, M., Caselles, V., Sapiro, G.: A comprehensive framework for image inpainting. IEEE Trans. Signal Process. 19(10), 2634–2645 (2010)MathSciNetGoogle Scholar
  17. 17.
    Cai, J.-F., Chan, R.H., Shen, Z.: Simultaneous cartoon and texture inpainting. Inverse Prob. Imaging 19(3), 379–395 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Cai, J.-F., Dong, B., Osher, S., Shen, Z.: Image restoration: total variation, wavelet frames, and beyond. J. Am. Math. Soc. 25(4), 1033–1089 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Caselles, V., Morel, J.-M., Sbert, C.: An axiomatic approach to image interpolation. IEEE Trans. Image Process. 7(3), 376–386 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Chambolle, A., Lions, P.-L.: Image recovery via total variation minimization and related problems. Numer. Math. 76(2), 167–188 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Chan, T.F., Esedoglu, S., Park, F.E.: Image decomposition combining staircase reduction and texture extraction. J. Vis. Commun. Image R 18(6), 464–486 (2007)CrossRefGoogle Scholar
  22. 22.
    Chan, T.F., Kang, S., Shen, J.: Total variation denoising and enhancement of color images based on the CB and HSV color models. J. Vis. Commun. Image R 12, 422–435 (2001)CrossRefGoogle Scholar
  23. 23.
    Chan, T.F., Kang, S.H.: Error analysis for image inpainting. J. Math. Imaging Vis. 26(1), 85–103 (2006)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Chan, T.F., Kang, S.H., Shen, J.: Euler’s elastica and curvature-based inpainting. SIAM J. Appl. Math. 63(2), 564–592 (2002)MathSciNetzbMATHGoogle Scholar
  25. 25.
    Chan, T.F., Marquina, A., Mulet, P.: High-order total variation-based image restoration. SIAM J. Sci. Comput. 22(2), 503–516 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Chan, T.F., Shen, J.: Local inpainting models and TV inpainting. SIAM J. Appl. Math. 62(3), 1019–1043 (2001)MathSciNetGoogle Scholar
  27. 27.
    Chan, T.F., Shen, J.J.: Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods. SIAM, Philadelphia (2005)CrossRefzbMATHGoogle Scholar
  28. 28.
    Chan, T.F., Yip, A.M., Park, F.E.: Simultaneous total variation image inpainting and blind deconvolution. Int. J. Imag. Syst. Tech. 15(1), 92–102 (2005)CrossRefGoogle Scholar
  29. 29.
    Chefd’Hotel, C., Tschumperlé, D., Deriche, R., Faugeras, O.: Regularizing flows for constrained matrix-valued images. J. Math. Imaging Vis. 20, 147–162 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Didas, S., Steidl, G., Setzer, S.: Combined \(\ell _2\) data and gradient fitting in conjunction with \(\ell _1\) regularization. Adv. Comput. Math. 30(1), 79–99 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Didas, S., Weickert, J., Burgeth, B.: Properties of higher order nonlinear diffusion filtering. J. Math. Imaging Vis. 35, 208–226 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Dong, B., Ji, H., Li, J., Shen, Z., Xu, Y.: Wavelet frame based blind image inpainting. Appl. Comput. Harmon. Anal. 32(2), 268–279 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Elad, M., Starck, J.-L., Querre, P., Donoho, D.L.: Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA). Appl. Comput. Harmon. Anal. 19(3), 340–358 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Esedoglu, S., Shen, J.: Digital inpainting based on the Mumford-Shah-Euler image model. Eur. J. Appl. Math. 13(4), 353–370 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Ferreira, O.P., Oliveira, P.R.: Proximal point algorithm on Riemannian manifolds. Optimization 51(2), 257–270 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Fisher, N.I.: Statistical Analysis of Circular Data. Cambridge University Press, Cambridge (1996)Google Scholar
  37. 37.
    Fletcher, P.: Geodesic regression and the theory of least squares on Riemannian manifolds. Int. J. Comput. Vis. 105(2), 171–185 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Giaquinta, M., Modica, G., Souček, J.: Variational problems for maps of bounded variation with values in \(S^1\). Calc. Var. 1(1), 87–121 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Giaquinta, M., Mucci, D.: The BV-energy of maps into a manifold: relaxation and density results. Ann. Sc. Norm. Super. Pisa 5(4), 483–548 (2006)MathSciNetzbMATHGoogle Scholar
  40. 40.
    Giaquinta, M., Mucci, D.: Maps of bounded variation with values into a manifold: total variation and relaxed energy. Pure Appl. Math. Q. 3(2), 513–538 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Goldluecke, B., Strekalovskiy, E., Cremers, D.: The natural vectorial total variation which arises from geometric measure theory. SIAM J. Imaging Sci. 5(2), 537–563 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Goldluecke, B., Strekalovskiy, E., Cremers, D.: Tight convex relaxations for vector-valued labeling. SIAM J. Imaging Sci. 6(3), 1626–1664 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Grohs, P., Hardering, H., Sander, O.: Optimal a priori discretization error bounds for geodesic finite elements. Found. Comput. Math. (2015)Google Scholar
  44. 44.
    Grohs, P., Wallner, J.: Interpolatory wavelets for manifold-valued data. Appl. Comput. Harmon. Anal. 27(3), 325–333 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Guillemot, C., Le Meur, O.: Image inpainting: overview and recent advances. IEEE Sig. Process. Mag. 31(1), 127–144 (2014)CrossRefGoogle Scholar
  46. 46.
    Harizanov, S., Oswald, P., Shingel, T.: Normal multi-scale transforms for curves. Found. Comput. Math. 11(6), 617–656 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Hatcher, A.: Algebraic Topology. Cambridge University Press, Cambridge (2002)zbMATHGoogle Scholar
  48. 48.
    Hinterberger, W., Scherzer, O.: Variational methods on the space of functions of bounded Hessian for convexification and denoising. Computing 76(1), 109–133 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  49. 49.
    Jammalamadaka, S.R., SenGupta, A.: Topics in Circular Statistics. World Scientific Publishing Company, River Edge (2001)Google Scholar
  50. 50.
    Kimmel, R., Sochen, N.: Orientation diffusion or how to comb a porcupine. J. Vis. Commun. Image R 13(1–2), 238–248 (2002)CrossRefGoogle Scholar
  51. 51.
    King, E.J., Kutyniok, G., Zhuang, X.: Analysis of inpainting via clustered sparsity and microlocal analysis. J. Math. Imaging Vis. 48(2), 205–234 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  52. 52.
    Lai, R., Osher, S.: A splitting method for orthogonality constrained problems. J. Sci. Comput. 58(2), 431–449 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  53. 53.
    Lefkimmiatis, S., Bourquard, A., Unser, M.: Hessian-based norm regularization for image restoration with biomedical applications. IEEE Trans. Image Process. 21(3), 983–995 (2012)MathSciNetCrossRefGoogle Scholar
  54. 54.
    Lellmann, J., Strekalovskiy, E., Koetter, S., Cremers, D.: Total variation regularization for functions with values in a manifold. IEEE ICCV 2013, 2944–2951 (2013)Google Scholar
  55. 55.
    Lysaker, M., Lundervold, A., Tai, X.-C.: Noise removal using fourth-order partial differential equations with applications to medical magnetic resonance images in space and time. IEEE Trans. Image Process. 12(12), 1579–1590 (2003)CrossRefzbMATHGoogle Scholar
  56. 56.
    Lysaker, M., Tai, X.-C.: Iterative image restoration combining total variation minimization and a second-order functional. Int. J. Comput. Vis. 66(1), 5–18 (2006)CrossRefzbMATHGoogle Scholar
  57. 57.
    März, T.: Image inpainting based on coherence transport with adapted distance functions. SIAM J. Imaging Sci. 4(4), 981–1000 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  58. 58.
    März, T.: A well-posedness framework for inpainting based on coherence transport. Found. Comput. Math. 15, 973–1033 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  59. 59.
    Masnou, S.: Disocclusion: a variational approach using level lines. IEEE Trans. Image Process. 11(2), 68–76 (2002)MathSciNetCrossRefGoogle Scholar
  60. 60.
    Masnou, S., Morel, J.-M.: Level lines based disocclusion. IEEE ICIP 1998, 259–263 (1998)MathSciNetGoogle Scholar
  61. 61.
    Möllenhoff, T., Strekalovskiy, E., Möller, M., Cremers, D.: Low rank priors for color image regularization. In: EMMCVPR 2015, pp. 126–140. Springer (2015)Google Scholar
  62. 62.
    Oller, J., Corcuera, J.: Intrinsic analysis of statistical estimation. Ann. Stat. 23(5), 1562–1581 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  63. 63.
    Papafitsoros, K., Schönlieb, C.: A combined first and second order variational approach for image reconstruction. J. Math. Imaging Vis. 48(2), 308–338 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  64. 64.
    Papafitsoros, K., Schoenlieb, C.B., Sengul, B.: Combined first and second order total variation inpainting using split bregman. Image Process. Line 3, 112–136 (2013)CrossRefGoogle Scholar
  65. 65.
    Pennec, X.: Intrinsic statistics on Riemannian manifolds: basic tools for geometric measurements. J. Math. Imaging Vis. 25(1), 127–154 (2006)MathSciNetCrossRefGoogle Scholar
  66. 66.
    Pennec, X., Fillard, P., Ayache, N.: A Riemannian framework for tensor computing. Int. J. Comput. Vis. 66(1), 41–66 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  67. 67.
    Rakêt, L.L., Roholm, L., Nielsen, M., Lauze, F.: TV-\(L^1\) optical flow for vector valued images. In: EMMCVPR 2011, pp. 329–343. Springer (2011)Google Scholar
  68. 68.
    Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14(5), 877–898 (1976)MathSciNetCrossRefzbMATHGoogle Scholar
  69. 69.
    Rosman, G., Bronstein, M., Bronstein, A., Wolf, A., Kimmel, R.: Group-valued regularization framework for motion segmentation of dynamic non-rigid shapes. In: SSVM 2011, LNCS 6667, pp. 725–736. Springer (2012)Google Scholar
  70. 70.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D. 60(1), 259–268 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  71. 71.
    Scherzer, O.: Denoising with higher order derivatives of bounded variation and an application to parameter estimation. Computing 60, 1–27 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  72. 72.
    Setzer, S., Steidl, G.: Variational methods with higher order derivatives in image processing. In: Neamtu, M., Schumaker, L., (ed.), Approximation XII: San Antonio 2007, pp. 360–385 (2008)Google Scholar
  73. 73.
    Setzer, S., Steidl, G., Teuber, T.: Infimal convolution regularizations with discrete l1-type functionals. Commun. Math. Sci. 9(3), 797–872 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  74. 74.
    Shen, J., Chan, T.F.: Mathematical models for local nontexture inpaintings. SIAM J. Appl. Math. 62(3), 1019–1043 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  75. 75.
    Strekalovskiy, E., Cremers, D.: Total variation for cyclic structures: convex relaxation and efficient minimization. IEEE CVPR 2011, 1905–1911 (2011)Google Scholar
  76. 76.
    Strekalovskiy, E., Cremers, D.: Total cyclic variation and generalizations. J. Math. Imaging Vis. 47(3), 258–277 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  77. 77.
    Tschumperlé, D.: Fast anisotropic smoothing of multi-valued images using curvature-preserving PDE’s. Int. J. Comput. Vis. 68(1), 65–82 (2006)CrossRefGoogle Scholar
  78. 78.
    Tschumperlé, D., Deriche, R.: Diffusion tensor regularization with constraints preservation. In: IEEE CVPR 2001, 1, pp. 948–953 (2001)Google Scholar
  79. 79.
    Ur Rahman, I., Drori, I., Stodden, V., Donoho, D., Schröder, P.: Multiscale representations for manifold-valued data. Multiscale Model. Sim. 4, 1201–1232 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  80. 80.
    Valkonen, T., Bredies, K., Knoll, F.: Total generalized variation in diffusion tensor imaging. SIAM J. Imaging Sci. 6(1), 487–525 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  81. 81.
    Vese, L., Osher, S.: Numerical methods for p-harmonic flows and applications to image processing. SIAM J. Numer. Anal. 40(6), 2085–2104 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  82. 82.
    Wallner, J., Dyn, N.: Convergence and \(C^1\) analysis of subdivision schemes on manifolds by proximity. Comput. Aided Geom. D. 22, 593–622 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  83. 83.
    Weinmann, A.: Nonlinear subdivision schemes on irregular meshes. Constr. Approx. 31(3), 395–415 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  84. 84.
    Weinmann, A.: Interpolatory multiscale representation for functions between manifolds. SIAM J. Math. Anal. 44, 162–191 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  85. 85.
    Weinmann, A., Demaret, L., Storath, M.: Total variation regularization for manifold-valued data. SIAM J. Imaging Sci. 7(4), 2226–2257 (2014)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Departement of MathematicsTechnische Universität KaiserslauternKaiserslauternGermany
  2. 2.Department of MathematicsTU MünchenMunichGermany
  3. 3.Department of Mathematics and Natural Sciences, Darmstadt University of Applied SciencesDarmstadtGermany
  4. 4.Fast Algorithms for Biomedical Imaging Group, Helmholtz Center MunichNeuherbergGermany

Personalised recommendations