Journal of Fourier Analysis and Applications

, Volume 14, Issue 5–6, pp 764–792 | Cite as

Accelerated Projected Gradient Method for Linear Inverse Problems with Sparsity Constraints

  • Ingrid Daubechies
  • Massimo Fornasier
  • Ignace Loris


Regularization of ill-posed linear inverse problems via 1 penalization has been proposed for cases where the solution is known to be (almost) sparse. One way to obtain the minimizer of such an 1 penalized functional is via an iterative soft-thresholding algorithm. We propose an alternative implementation to 1-constraints, using a gradient method, with projection on 1-balls. The corresponding algorithm uses again iterative soft-thresholding, now with a variable thresholding parameter. We also propose accelerated versions of this iterative method, using ingredients of the (linear) steepest descent method. We prove convergence in norm for one of these projected gradient methods, without and with acceleration.


Linear inverse problems Sparse recovery Projected gradient method 

Mathematics Subject Classification (2000)

15A29 49M30 65F22 65K10 90C25 52A41 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alber, Y.I., Iusem, A.N., Solodov, M.V.: On the projected subgradient method for nonsmooth convex optimization in a Hilbert space. Math. Program. Ser. A 81(1), 23–35 (1998) CrossRefMathSciNetGoogle Scholar
  2. 2.
    Anthoine, S.: Different Wavelet-based Approaches for the Separation of Noisy and Blurred Mixtures of Components. Application to Astrophysical Data. Ph.D. Thesis, Princeton University, 2005 Google Scholar
  3. 3.
    Brègman, L.M.: Finding the common point of convex sets by the method of successive projection. Dokl. Akad. Nauk SSSR 162, 487–490 (1965) MathSciNetGoogle Scholar
  4. 4.
    Candès, E.J., Donoho, D.L.: New tight frames of curvelets and optimal representations of objects with piecewise C 2 singularities. Commun. Pure Appl. Math. 57(2), 219–266 (2004) MATHCrossRefGoogle Scholar
  5. 5.
    Candès, E.J., Romberg, J.: Practical signal recovery from random projections. In: Wavelet Applications in Signal and Image Processing XI. Proc. SPIE Conf., vol. 5914 (2004) Google Scholar
  6. 6.
    Candès, E.J., Tao, T.: Near-optimal signal recovery from random projections: universal encoding strategies? IEEE Trans. Inf. Theory 52(12), 5406–5425 (2006) CrossRefGoogle Scholar
  7. 7.
    Candès, E., Romberg, J., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8), 1207–1223 (2006) MATHCrossRefGoogle Scholar
  8. 8.
    Candès, E.J., Romberg, J., Tao, T.: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006) CrossRefGoogle Scholar
  9. 9.
    Canuto, C., Urban, K.: Adaptive optimization of convex functionals in Banach spaces. SIAM J. Numer. Anal. 42(5), 2043–2075 (2004) CrossRefMathSciNetGoogle Scholar
  10. 10.
    Chambolle, A., DeVore, R.A., Lee, N.-Y., Lucier, B.J.: Nonlinear wavelet image processing: variational problems, compression, and noise removal through wavelet shrinkage. IEEE Trans. Image Process. 7(3), 319–335 (1998) MATHCrossRefMathSciNetGoogle Scholar
  11. 11.
    Christensen, O.: An Introduction to Frames and Riesz Bases. Birkhäuser, Boston (2003) MATHGoogle Scholar
  12. 12.
    Cohen, A.: Numerical Analysis of Wavelet Methods. Studies in Mathematics and Its Applications, vol. 32. North-Holland, Amsterdam (2003) MATHGoogle Scholar
  13. 13.
    Cohen, A., Dahmen, W., DeVore, R.: Adaptive wavelet methods for elliptic operator equations—Convergence rates. Math. Comput. 70, 27–75 (2001) MATHMathSciNetGoogle Scholar
  14. 14.
    Cohen, A., Dahmen, W., DeVore, R.: Adaptive wavelet methods II: Beyond the elliptic case. Found. Comput. Math. 2(3), 203–245 (2002) MATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    Cohen, A., Hoffmann, M., Reiss, M.: Adaptive wavelet Galerkin methods for linear inverse problems. SIAM J. Numer. Anal. 42(4), 1479–1501 (2004) MATHCrossRefMathSciNetGoogle Scholar
  16. 16.
    Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005) MATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    Dahlke, S., Maass, P.: An outline of adaptive wavelet Galerkin methods for Tikhonov regularization of inverse parabolic problems. In: Hon, J.-C., et al. (eds.) Recent Development in Theories and Numerics. Proceedings of the International Conference on Inverse Problems, Hong Kong, China, January 9–12 2002, pp. 56–66. World Scientific, Singapore (2003) CrossRefGoogle Scholar
  18. 18.
    Dahlke, S., Fornasier, M., Raasch, T.: Adaptive frame methods for elliptic operator equations. Adv. Comput. Math. 27(1), 27–63 (2007) MATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Dahlke, S., Fornasier, M., Raasch, T., Stevenson, R., Werner, M.: Adaptive frame methods for elliptic operator equations: The steepest descent approach. IMA J. Numer. Anal. 27, 717–740 (2007) MATHCrossRefMathSciNetGoogle Scholar
  20. 20.
    Daubechies, I.: Ten Lectures on Wavelets. SIAM, Philadelphia (1992) MATHGoogle Scholar
  21. 21.
    Daubechies, I., Teschke, G.: Variational image restoration by means of wavelets: Simultaneous decomposition, deblurring, and denoising. Appl. Comput. Harmon. Anal. 19(1), 1–16 (2005) MATHCrossRefMathSciNetGoogle Scholar
  22. 22.
    Daubechies, I., Defrise, M., DeMol, C.: An iterative thresholding algorithm for linear inverse problems. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004) MATHCrossRefMathSciNetGoogle Scholar
  23. 23.
    Donoho, D.L.: Superresolution via sparsity constraints. SIAM J. Math. Anal. 23(5), 1309–1331 (1992) MATHCrossRefMathSciNetGoogle Scholar
  24. 24.
    Donoho, D.L.: De-noising by soft-thresholding. IEEE Trans. Inf. Theory 41(3), 613–627 (1995) MATHCrossRefMathSciNetGoogle Scholar
  25. 25.
    Donoho, D.L.: Nonlinear solution of linear inverse problems by wavelet-vaguelette decomposition. Appl. Comput. Harmon. Anal. 2(2), 101–126 (1995) MATHCrossRefMathSciNetGoogle Scholar
  26. 26.
    Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006) CrossRefMathSciNetGoogle Scholar
  27. 27.
    Donoho, D.L., Logan, B.F.: Signal recovery and the large sieve. SIAM J. Appl. Math. 52, 577–591 (1992) MATHCrossRefMathSciNetGoogle Scholar
  28. 28.
    Donoho, D.L., Starck, P.: Uncertainty principles and signal recovery. SIAM J. Appl. Math. 49, 906–931 (1989) MATHCrossRefMathSciNetGoogle Scholar
  29. 29.
    Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Ann. Stat. 32(2), 407–499 (2004) MATHCrossRefMathSciNetGoogle Scholar
  30. 30.
    Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Mathematics and Its Applications, vol. 375. Kluwer Academic, Dordrecht (1996) MATHGoogle Scholar
  31. 31.
    Figueiredo, M.A.T., Nowak, R.D.: An EM algorithm for wavelet-based image restoration. IEEE Trans. Image Proc. 12(8), 906–916 (2003) CrossRefMathSciNetGoogle Scholar
  32. 32.
    Figueiredo, M.A.T., Nowak, R.D., Wright, S.J.: Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1(4), 586–597 (2007) CrossRefGoogle Scholar
  33. 33.
    Fornasier, M.: Domain decomposition methods for linear inverse problems with sparsity constraints. Inverse Probl. 23, 2505–2526 (2007) MATHCrossRefMathSciNetGoogle Scholar
  34. 34.
    Fornasier, M., Pitolli, F.: Adaptive iterative thresholding algorithms for magnetoenceophalography (MEG). J. Comput. Appl. Math. (2008, to appear). doi: 10.1016/ MathSciNetGoogle Scholar
  35. 35.
    Fornasier, M., Rauhut, H.: Iterative thrsholding algorithms. Appl. Comput. Harmon. Anal. 25(2), 187–208 (2008) MATHCrossRefMathSciNetGoogle Scholar
  36. 36.
    Fornasier, M., Rauhut, H.: Recovery algorithms for vector valued data with joint sparsity constraints. SIAM J. Numer. Anal. 46(2), 577–613 (2008) CrossRefMathSciNetGoogle Scholar
  37. 37.
    Logan, B.F.: Properties of High-Pass Signals. Ph.D. Thesis, Columbia University, 1965 Google Scholar
  38. 38.
    Loris, I., Nolet, G., Daubechies, I., Dahlen, F.A.: Tomographic inversion using 1-norm regularization of wavelet coefficients. Geophys. J. Int. 170(1), 359–370 (2007) CrossRefGoogle Scholar
  39. 39.
    Mallat, S.: A Wavelet Tour of Signal Processing, 2nd edn. Academic Press, San Diego (1999) MATHGoogle Scholar
  40. 40.
    Ramlau, R., Teschke, G.: Tikhonov replacement functionals for iteratively solving nonlinear operator equations. Inverse Probl. 21(5), 1571–1592 (2005) MATHCrossRefMathSciNetGoogle Scholar
  41. 41.
    Rauhut, H.: Random sampling of sparse trigonometric polynomials. Appl. Comput. Harmon. Anal. 22(1), 16–42 (2007) MATHCrossRefMathSciNetGoogle Scholar
  42. 42.
    Starck, J.-L., Candès, E.J., Donoho, D.L.: Astronomical image representation by curvelet transform. Astron. Astrophys. 298, 785–800 (2003) CrossRefGoogle Scholar
  43. 43.
    Starck, J.-L., Nguyen, M.K., Murtagh, F.: Wavelets and curvelets for image deconvolution: a combined approach. Signal Process. 83, 2279–2283 (2003) MATHCrossRefGoogle Scholar
  44. 44.
    Teschke, G.: Multi-frame representations in linear inverse problems with mixed multi-constraints. Appl. Comput. Harmon. Anal. 22(1), 43–60 (2007) MATHCrossRefMathSciNetGoogle Scholar
  45. 45.
    van den Berg, E., Friedlander, M.P.: In pursuit of a root. Preprint (2007) Google Scholar
  46. 46.
    Wolfram, S.: The Mathematica Book, 5th edn. Wolfram Media/Cambridge University Press, Cambridge (2003) Google Scholar

Copyright information

© Birkhäuser Boston 2008

Authors and Affiliations

  • Ingrid Daubechies
    • 1
  • Massimo Fornasier
    • 1
  • Ignace Loris
    • 2
  1. 1.Program in Computational and Applied MathematicsPrinceton UniversityPrincetonUSA
  2. 2.Dienst Theoretische NatuurkundeVrije Universiteit BrusselBrusselBelgium

Personalised recommendations