Journal of Fourier Analysis and Applications

, Volume 14, Issue 5–6, pp 813–837 | Cite as

Linear Convergence of Iterative Soft-Thresholding

  • Kristian Bredies
  • Dirk A. LorenzEmail author


In this article a unified approach to iterative soft-thresholding algorithms for the solution of linear operator equations in infinite dimensional Hilbert spaces is presented. We formulate the algorithm in the framework of generalized gradient methods and present a new convergence analysis. As main result we show that the algorithm converges with linear rate as soon as the underlying operator satisfies the so-called finite basis injectivity property or the minimizer possesses a so-called strict sparsity pattern. Moreover it is shown that the constants can be calculated explicitly in special cases (i.e. for compact operators). Furthermore, the techniques also can be used to establish linear convergence for related methods such as the iterative thresholding algorithm for joint sparsity and the accelerated gradient projection method.


Iterative soft-thresholding Inverse problems Sparsity constraints Convergence analysis Generalized gradient projection method 

Mathematics Subject Classification (2000)

65J22 46N10 49M05 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bauschke, H.H., Borwein, J.M., Combettes, P.L.: Bregman monotone optimization algorithms. SIAM J. Control Optim. 42(2), 596–636 (2003) zbMATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Bredies, K., Lorenz, D.A.: Iterated hard shrinkage for minimization problems with sparsity constraints. SIAM J. Sci. Comput. 30(2), 657–683 (2008) zbMATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Bredies, K., Lorenz, D.A., Maass, P.: A generalized conditional gradient method and its connection to an iterative shrinkage method. Comput. Optim. Appl. (2008). doi: 10.1007/s10589-007-9083-3
  4. 4.
    Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005) zbMATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Dahlke, S., Maass, P.: An outline of adaptive wavelet Galerkin methods for Tikhonov regularization of inverse parabolic problems. In: Hon, J.C.Y.-C., Yamamoto, M., Lee, J.-Y. (eds.) Recent Development in Theories & Numerics, pp. 56–66. World Scientific, Singapore (2003) CrossRefGoogle Scholar
  6. 6.
    Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure App. Math. 57(11), 1413–1457 (2004) zbMATHCrossRefGoogle Scholar
  7. 7.
    Daubechies, I., Fornasier, M., Loris, I.: Accelerated projected gradient method for linear inverse problems with sparsity constraints. J. Fourier Anal. Appl. (2008), this issue. doi: 10.1007/s00041-008-9039-8
  8. 8.
    Demyanov, V.F., Rubinov, A.M.: Approximate Methods in Optimization Problems. Modern Analytic and Computational Methods in Science and Mathematics, vol. 32. American Elsevier, New York (1970) Google Scholar
  9. 9.
    Donoho, D.L., Elad, M., Temlyakov, V.: Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Trans. Inf. Theory 52, 6–18 (2006) CrossRefMathSciNetGoogle Scholar
  10. 10.
    Duarte, M.F., Sarvotham, S., Baron, D., Wakin, M.B., Baraniuk, R.: Distributed compressed sensing of jointly sparse signals. In: Conference Record of the Thirty-Ninth Asilomar Conference on Signals, Systems And Computers, pp. 1537–1541 (2005) Google Scholar
  11. 11.
    Dunn, J.C.: Global and asymptotic convergence rate estimates for a class of projected gradient processes. SIAM J. Control Optim. 19(3), 368–400 (1981) zbMATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Ann. Stat. 32(2), 407–451 (2004) zbMATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Ekeland, I., Temam, R.: Convex Analysis and Variational Problems. North-Holland, Amsterdam (1976) zbMATHGoogle Scholar
  14. 14.
    Elad, M., Matalon, B., Zibulevsky, M.: Coordinate and subspace optimization methods for linear least squares with non-quadratic regularization. Appl. Comput. Harmon. Anal. 23, 346–367 (2007) zbMATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Mathematics and its Applications, vol. 375. Kluwer Academic, Dordrecht (2000) Google Scholar
  16. 16.
    Figueiredo, M.A.T., Nowak, R.D., Wright, S.J.: Gradient projection for sparse reconstruction: Applications to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 4, 586–597 (2007) CrossRefGoogle Scholar
  17. 17.
    Fornasier, M., Rauhut, H.: Recovery algorithms for vector valued data with joint sparsity constraints. SIAM J. Numer. Anal. 46(2), 577–613 (2008) CrossRefMathSciNetGoogle Scholar
  18. 18.
    Fuchs, J.-J.: Recovery of exact sparse representations in the presence of bounded noise. IEEE Trans. Inf. Theory 51(10), 3601–3608 (2005) CrossRefGoogle Scholar
  19. 19.
    Goldstein, A.A.: Convex programming in Hilbert space. Bull. Am. Math. Soc. 70, 709–710 (1964) zbMATHCrossRefGoogle Scholar
  20. 20.
    Gribonval, R., Nielsen, M.: Highly sparse representations from dictionaries are unique and independent of the sparseness measure. Appl. Comput. Harmon. Anal. 22(3), 335–355 (2007) zbMATHCrossRefMathSciNetGoogle Scholar
  21. 21.
    Griesse, R., Lorenz, D.A.: A semismooth Newton method for Tikhonov functionals with sparsity constraints. Inverse Probl. 24, 035007 (2008) CrossRefMathSciNetGoogle Scholar
  22. 22.
    Kim, S.-J., Koh, K., Lustig, M., Boyd, S., Gorinevsky, D.: A method for large-scale 1-regularized least squares problems with applications in signal processing and statistics. IEEE J. Sel. Top. Signal Process. 4, 606–617 (2007) CrossRefGoogle Scholar
  23. 23.
    Levitin, E.S., Polyak, B.T.: Constrained minimization problems. USSR Comput. Math. Math. Phys. 6, 1–50 (1966) CrossRefGoogle Scholar
  24. 24.
    Lorenz, D.A.: Convergence rates and source conditions for Tikhonov regularization with sparsity constraints. J. Inverse Ill-Posed Probl. (2008, to appear) Google Scholar
  25. 25.
    Maass, P.: The interior radon transform. SIAM J. Appl. Math. 52(3), 710–724 (1992) zbMATHCrossRefMathSciNetGoogle Scholar
  26. 26.
    Nachman, A.I.: Global uniqueness for a two-dimensional inverse boundary value problem. Ann. Math. 143, 71–96 (1996) zbMATHCrossRefMathSciNetGoogle Scholar
  27. 27.
    Osborne, M.R., Presnell, B., Turlach, B.A.: A new approach to variable selection in least squares problems. IMA J. Numer. Anal. 20, 389–404 (2000) zbMATHCrossRefMathSciNetGoogle Scholar
  28. 28.
    Resmerita, E.: Regularization of ill-posed problems in Banach spaces: convergence rates. Inverse Probl. 21(4), 1303–1314 (2005) zbMATHCrossRefMathSciNetGoogle Scholar
  29. 29.
    Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Springer, New York (1998) zbMATHGoogle Scholar
  30. 30.
    Schöpfer, F., Louis, A.K., Schuster, T.: Nonlinear iterative methods for linear ill-posed problems in Banach spaces. Inverse Probl. 22, 311–329 (2006) zbMATHCrossRefGoogle Scholar
  31. 31.
    Showalter, R.E.: Monotone Operators in Banach Space and Nonlinear Partial Differential Equations. Mathematical Surveys and Monographs, vol. 49. American Mathematical Society, Providence (1997) Google Scholar
  32. 32.
    Teschke, G., Ramlau, R.: An iterative algorithm for nonlinear inverse problems with joint sparsity constraints in vector-valued regimes and an application to color image inpainting. Inverse Probl. 23(5), 1851–1870 (2007) zbMATHCrossRefMathSciNetGoogle Scholar
  33. 33.
    Turlach, B.A.: On algorithms for solving least squares problems under an L 1 penalty or an L 1 constraint. In: 2004 Proceedings of the American Statistical Association, Statistical Computing Section, pp. 2572–2577. American Statistical Association, New York (2005) Google Scholar
  34. 34.
    Xu, Z.B., Roach, G.F.: Characteristic inequalities of uniformly convex and uniformly smooth Banach spaces. J. Math. Anal. Appl. 157(1), 189–210 (1991) zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Birkhäuser Boston 2008

Authors and Affiliations

  1. 1.Center for Industrial Mathematics / Fachbereich 3University of BremenBremenGermany

Personalised recommendations