Skip to main content
SpringerLink
Log in

The application of joint sparsity and total variation minimization algorithms to a real-life art restoration problem

  • Published:
Advances in Computational Mathematics Aims and scope Submit manuscript

Abstract

On March 11, 1944, the famous Eremitani Church in Padua (Italy) was destroyed in an Allied bombing along with the inestimable frescoes by Andrea Mantegna et al. contained in the Ovetari Chapel. In the last 60 years, several attempts have been made to restore the fresco fragments by traditional methods, but without much success. One of the authors contributed to the development of an efficient pattern recognition algorithm to map the original position and orientation of the fragments, based on comparisons with an old gray level image of the fresco prior to the damage. This innovative technique allowed for the partial reconstruction of the frescoes. Unfortunately, the surface covered by the colored fragments is only 77 m2, while the original area was of several hundreds. This means that we can reconstruct only a fraction (less than 8%) of this inestimable artwork. In particular the original color of the blanks is not known. This begs the question of whether it is possible to estimate mathematically the original colors of the frescoes by making use of the potential information given by the available fragments and the gray level of the pictures taken before the damage. Moreover, is it possible to estimate how faithful such a restoration is? In this paper we retrace the development of the recovery of the frescoes as an inspiring and challenging real-life problem for the development of new mathematical methods. Then we shortly review two models recently studied independently by the authors for the recovery of vector valued functions from incomplete data, with applications to the recolorization problem. The models are based on the minimization of a functional which is formed by the discrepancy with respect to the data and additional regularization constraints. The latter refer to joint sparsity measures with respect to frame expansions, in particular wavelet or curvelet expansions, for the first functional and functional total variation for the second. We establish relations between these two models. As a major contribution of this work we perform specific numerical test on the real-life problem of the A. Mantegna’s frescoes and we compare the results due to the two methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aubert, G., Vese, L.: A variational method in image recovery. SIAM J. Numer. Anal. 34(5), 1948–1979 (1997)

    Article  MATH  MathSciNet  Google Scholar 

  2. Ballester, C., Beltramio, M., Caselles, V., Sapiro, G., Verdera J.: Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans. Image Process. 10(8), 1200–1211 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  3. Beltramio, M., Sapiro, G., Caselles, V., Ballester, B.: Image inpainting. In: SIGGRAPH 2000, July (2001)

  4. Candès, E.J., Donoho, D.L.: New tight frames of curvelets and optimal representations of objects with piecewise C 2 singularities. Comm. Pure Appl. Math. 57(2), 219–266 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  5. Candès, E.J., Romberg, J., Tao, T.: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory 52(2), 489–509 (2006)

    Article  MathSciNet  Google Scholar 

  6. Candès, E.J., Tao, T.: Near optimal signal recovery from random projections: universal encoding strategies. IEEE Trans. Inform. Theory 52, 5406–5425 (2005)

    Article  Google Scholar 

  7. Caselles, V., Coll, V., Morel, V.J.-M.: Geometry and color in natural images. J. Math. Imaging Vision 16(2), 89–105 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  8. Cazzato, R.: Un Metodo per la Ricolorazione di Immagini e Altri Strumenti per il Restauro. Il Progetto Mantegna e gli Affreschi nella Chiesa degli Eremitani. Master thesis (Italian), University of Padova (2007)

  9. Cazzato, R., Costa, G., Dal Farra, A., Fornasier, M., Toniolo, D., Tosato, D., Zanuso, C.: Il Progetto Mantegna: storia e risultati. In: Spiazzi, D.T.A.M., De Nicolò Salmazo, A. (eds.) Andrea Mantegna e i Maestri della Cappella Ovetari: La Ricomposizione Virtuale e il Restauro, pp. 151–169. Skira (2006)

  10. Chambolle, A., Lions, P.-L.: Image recovery via total variation minimization and related problems. Numer. Math. 76(2), 167–188 (1997)

    Article  MATH  MathSciNet  Google Scholar 

  11. Chan, T.F., Kang, S.H., Shen, J.: Euler’s elastica and curvature-based inpainting. SIAM J. Appl. Math. 63(2), 564–592 (2002)

    MATH  MathSciNet  Google Scholar 

  12. Chan, T.F., Shen, J.: Inpainting based on nonlinear transport and diffusion. Contemp. Math. 313, 53–65 (2002)

    MathSciNet  Google Scholar 

  13. Chan, T.F., Shen, J.: Mathematical models for local nontexture inpaintings. SIAM J. Appl. Math. 62(3), 1019–1043 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  14. Chan, T.F., Shen, J.: Variational image inpainting. Comm. Pure Appl. Math. 58(5), 579–619 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  15. Chaux, C., Combettes, P.L., Pesquet, J.-C., Wajs, V.R.: A variational formulation for frame-based inverse problems. Inverse Problems, 23(4), 1495–1518 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  16. Cohen, A., Dahmen, W., Daubechies, I., DeVore R.: Harmonic analysis of the space BV. Rev. Mat. Iberoamericana 19(1), 235–263 (2003)

    MATH  MathSciNet  Google Scholar 

  17. Cohen, A., DeVore, R., Petrushev, P., Xu, H.: Nonlinear approximation and the space \(BV({\Bbb R}^2)\). Amer. J. Math. 121, 587–628 (1999)

    Article  MATH  MathSciNet  Google Scholar 

  18. Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  19. Daubechies, I., Defrise, M., DeMol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Comm. Pure Appl. Math 57, 1413–1541 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  20. Daubechies, I., Teschke, G.: Variational image restoration by means of wavelets: simultaneous decomposition, deblurring and denoising. Appl. Comput. Harmon. Anal. 19(1), 1–16 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  21. Daubechies, I., Teschke, G., Vese, L.: Iteratively solving linear inverse problems under general convex constraints. Inverse Problems Imaging 1(1), 29–46 (2007)

    MATH  MathSciNet  Google Scholar 

  22. Daubechies, I., Teschke, G., Vese, L.: On some iterative concepts for image restoration. In: Hawkes, P.W. (ed.) Advances in Imaging and Electron Physics, no. 150, pp. 2–52. Elsevier (2008)

  23. Donoho, D.L.: Compressed sensing. IEEE Trans. Inform. Theory 52(4), 1289–1306 (2006)

    Article  MathSciNet  Google Scholar 

  24. Elad, M., Starck, J.-L., Querre, P., Donoho, D.L.: Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA). Appl. Comput. Harmon. Anal. 19, 340–358 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  25. Elad, M., Milanfar, P., Rubinstein, R.: Analysis versus synthesis in signal priors. Inverse Problems 23(3), 947–968 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  26. Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Mathematics and its Applications (Dordrecht), p. 375. Kluwer Academic, Dordrecht (1996)

    Google Scholar 

  27. Fadili, M.J., Starck, J.-L., Murtagh, F.: Inpainting and zooming using sparse representations. Comput. J. (2007). doi:10.1093/comjnl/bxm055

  28. Fornasier, M.: Nonlinear projection recovery in digital inpainting for color image restoration. J. Math. Imaging Vision 24(3), 359–373 (2006)

    Article  MathSciNet  Google Scholar 

  29. Fornasier, M., March, R.: Restoration of color images by vector valued BV functions and variational calculus. SIAM J. Appl. Math. 68(2), 437–460 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  30. Fornasier, M., Rauhut, H.: Iterative thresholding algorithms. Appl. Comput. Harmon. Anal. (2008). doi:10.1016/j.acha.2007.10.005

  31. Fornasier, M., Rauhut, H.: Recovery algorithms for vector valued data with joint sparsity constraints. SIAM J. Numer. Anal. 46(2), 577–613 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  32. Fornasier, M., Toniolo, D.: Computer-based recomposition of the frescoes in the Ovetari Chapel in the Church of the Eremitani in Padua. Methodology and initial results, (English/Italian). In: Mantegna nella chiesa degli Eremitani a Padova. Il recupero possibile, Ed. Skira (2003)

  33. Fornasier, M., Toniolo, D.: Fast, robust, and efficient 2D pattern recognition for re-assembing fragmented images. Pattern Recogn. 38(11), 2074–2087 (2005)

    Article  Google Scholar 

  34. Kang, S.H., March, R.: Variational models for image colorization via chromaticity and brightness decomposition. IEEE Trans. Image Process. 16(9), 2251–2261 (2007)

    Article  MathSciNet  Google Scholar 

  35. Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. In: ACM SIGGRAPH 2004, pp. 689–694 (2004)

  36. Mallat, S.: A Wavelet Tour of Signal Processing. Academic, New York (1998)

    MATH  Google Scholar 

  37. Mairal, J., Elad, M., Sapiro, G.: Sparse representation for color image restoration. IEEE Trans. Image Process. 17(1), 53–69 (2008)

    Article  MathSciNet  Google Scholar 

  38. Masnou, S.: Disocclusion: a variational approach using level lines. IEEE Trans. Image Process. 11(2), 68–76 (1998)

    Article  MathSciNet  Google Scholar 

  39. Masnou, S., Morel, J.-M.: Level lines based disocclusion. In: Proceedings of 5th IEEE Intl Conf. on Image Process., Chicago, pp. 259–263 (1998)

  40. Mumford, D.: Elastica and computer vision. In: Bajaj, C. (ed.) Algebraic Geometry and Applications, pp. 491–506. Springer, Heidelberg (1994)

    Google Scholar 

  41. Ramlau, R., Teschke, G.: Tikhonov replacement functionals for iteratively solving nonlinear operator equations. Inverse Problems 21, 1571–1592 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  42. Ramlau, R., Teschke, G.: A projection iteration for nonlinear operator equations with sparsity constraints. Numer. Math. 104, 177–203 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  43. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica, D 60(1–4), 259–268 (1992)

    Article  MATH  Google Scholar 

  44. Sapiro, G.: Inpainting the colors. In: ICIP 2005. IEEE International Conference on Image Processing, pp. 698–701 (2005)

  45. Teschke, G., Ramlau, R.: An iterative algorithm for nonlinear inverse problems with joint sparsity constraints in vector valued regimes and an application to color image inpainting. Inverse Problems 23, 1851–1870 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  46. Tropp, J.: Algorithms for simultaneous sparse approximation. Part ii: Convex relaxation. Signal Process., special issue Sparse approximations in signal and image processing 86, 589–602 (2006)

    Google Scholar 

  47. Tropp, J., Gilbert, A.C., Strauss, M.J.: Algorithms for simultaneous sparse approximation. Part i: Greedy pursuit. Signal Process., special issue Sparse approximations in signal and image processing 86, 572–588 (2006)

    Google Scholar 

  48. Vese, L.: A study in the BV space of a denoising-deblurring variational problem. Appl. Math. Optim. 44, 131–161 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  49. Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. 15(5), 1120–1129 (2006)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Johann Radon Institute for Computational and Applied Mathematics (RICAM), Austrian Academy of Sciences, Altenbergerstr. 69, 4040, Linz, Austria

    Massimo Fornasier & Ronny Ramlau

  2. Institute for Computational Mathematics in Science and Technology, University of Applied Sciences Neubrandenburg, Brodaer Str. 2, 17033, Neubrandenburg, Germany

    Gerd Teschke

  3. Zuse-Institute Berlin, Berlin, Germany

    Gerd Teschke

Authors
  1. Massimo Fornasier
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ronny Ramlau
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gerd Teschke
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gerd Teschke.

Additional information

Communicated by Lixin Shen and Yuesheng Xu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fornasier, M., Ramlau, R. & Teschke, G. The application of joint sparsity and total variation minimization algorithms to a real-life art restoration problem. Adv Comput Math 31, 157–184 (2009). https://doi.org/10.1007/s10444-008-9103-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10444-008-9103-6

Keywords

Mathematics Subject Classifications (2000)

Search

Navigation