Minimization of Non-smooth, Non-convex Functionals by Iterative Thresholding

  • Kristian Bredies
  • Dirk A. LorenzEmail author
  • Stefan Reiterer


Convergence analysis is carried out for a forward-backward splitting/generalized gradient projection method for the minimization of a special class of non-smooth and genuinely non-convex minimization problems in infinite-dimensional Hilbert spaces. The functionals under consideration are the sum of a smooth, possibly non-convex and non-smooth, necessarily non-convex functional. For separable constraints in the sequence space, we show that the generalized gradient projection method amounts to a discontinuous iterative thresholding procedure, which can easily be implemented. In this case we prove strong subsequential convergence and moreover show that the limit satisfies strengthened necessary conditions for a global minimizer, i.e., it avoids a certain set of non-global minimizers. Eventually, the method is applied to problems arising in the recovery of sparse data, where strong convergence of the whole sequence is shown, and numerical tests are presented.


Non-convex optimization Non-smooth optimization  Gradient projection method Iterative thresholding 

Mathematics Subject Classification

49M05 65K10 



Kristian Bredies acknowledges support by the SFB Research Center “Mathematical Optimization and Applications in Biomedical Sciences” at the University of Graz. Dirk Lorenz acknowledges support by the DFG under Grant LO 1436/2-1.


  1. 1.
    Nocedal, J., Wright, S.: Numerical Optimization. Springer Series in Operations Research and Financial Engineering, 2nd edn. Springer, New York (2006)zbMATHGoogle Scholar
  2. 2.
    Pytlak, R.: Conjugate Gradient Algorithms in Nonconvex Optimization, Nonconvex Optimization and Its Applications, vol. 89. Springer, Berlin (2009)Google Scholar
  3. 3.
    Nikolova, M.: Analysis of the recovery of edges in images and signals by minimizing nonconvex regularized least-squares. Multiscale Modell. Simul. 4(3), 960–991 (2005)CrossRefzbMATHMathSciNetGoogle Scholar
  4. 4.
    Nikolova, M.: Markovian reconstruction using a GNC approach. IEEE Trans. Image Process. 8(9), 1204–1220 (1999)CrossRefGoogle Scholar
  5. 5.
    Blake, A., Zisserman, A.: Visual Reconstruction. MIT Press, Cambridge (1987)Google Scholar
  6. 6.
    Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16(6), 964–979 (1979)CrossRefzbMATHMathSciNetGoogle Scholar
  7. 7.
    Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward–backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005)CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Bredies, K., Lorenz, D.A.: Linear convergence of iterative soft-thresholding. J. Fourier Anal. Appl. 14(5–6), 813–837 (2008). doi: 10.1007/s00041-008-9041-1 CrossRefzbMATHMathSciNetGoogle Scholar
  9. 9.
    Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Math. Program. Ser. B. (2011). doi: 10.1007/s10107-011-0484-9.
  10. 10.
    Ito, K., Kunisch, K.: A variational approach to sparsity optimization based on Lagrange multiplier theory. MOBIS SFB-Report 2011–2014 (2011).
  11. 11.
    Chouzenoux, E., Pesquet, J.C., Repetti, A.: Variable metric forward–backward algorithm for minimizing the sum of a differentiable function and a convex function. To appear J. Optim. Theory Appl. (2013).
  12. 12.
    Ochs, P., Chen, Y., Brox, T., Pock, T.: iPiano: Inertial proximal algorithm for non-convex optimization. (2013).
  13. 13.
    Kaplan, A., Tichatschke, R.: Proximal point methods and nonconvex optimization. J. Global Optim. 13, 389–406 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  14. 14.
    Lewis, A.S., Luke, D.R., Malick, J.: Local linear convergence for alternating and averaged nonconvex projections. Found. Comput. Math. 9, 485–513 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  15. 15.
    Rockafellar, R.T., Wets, R.J.B.: Variational Analysis, Grundlehren der Mathematischen Wissenschaften, vol. 317. Springer, Berlin (1998)Google Scholar
  16. 16.
    Penot, J.P.: Proximal mappings. J. Approx. Theory 94, 203–221 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  17. 17.
    Hare, W., Sagastizábal, C.: Computing proximal points of nonconvex functions. Math. Program. Ser. B 116(1–2), 221–258 (2009)CrossRefzbMATHGoogle Scholar
  18. 18.
    Demyanov, V.F., Rubinov, A.M.: Approximate Methods in Optimization Problems. Modern Analytic and Computational Methods in Science and Mathematics, vol. 32. American Elsevier, New York (1970)Google Scholar
  19. 19.
    Dunn, J.C.: Global and asymptotic convergence rate estimates for a class of projected gradient processes. SIAM J. Control Optim. 19(3), 368–400 (1981)CrossRefzbMATHMathSciNetGoogle Scholar
  20. 20.
    Clarke, F.H.: Optimization and Nonsmooth Analysis. CRM Université de Montréal, Montréal (1989)zbMATHGoogle Scholar
  21. 21.
    Bredies, K., Lorenz, D.A.: Regularization with non-convex separable constraints. Inverse Problems, 25(8), 085011 (2009). doi: 10.1088/0266-5611/25/8/085011
  22. 22.
    Blumensath, T., Yaghoobi, M., Davies, M.: Iterative hard thresholding and \(l^0\) regularisation. In: IEEE International Conference on Acoustics, Speech and Signal Processing (2007)Google Scholar
  23. 23.
    Moulin, P., Liu, J.: Analysis of multiresolution image denoising schemes using generalized Gaussian and complexity priors. IEEE Trans. Inform. Theory 45(3), 909–919 (1999)CrossRefzbMATHMathSciNetGoogle Scholar
  24. 24.
    Lorenz, D.A.: Non-convex variational denoising of images: interpolation between hard and soft wavelet shrinkage. Curr. Dev. Theory Appl. Wavelets 1(1), 31–56 (2007)zbMATHMathSciNetGoogle Scholar
  25. 25.
    Lorenz, D.A.: Convergence rates and source conditions for Tikhonov regularization with sparsity constraints. J. Inverse Ill-Posed Problems 16(5), 463–478 (2008). doi: 10.1515/JIIP.2008.025 CrossRefzbMATHMathSciNetGoogle Scholar
  26. 26.
    Grasmair, M.: Well-posedness classes for sparse regularisation. Reports of FSP S092-“Industrial Geometry”. University of Vienna, Austria (2010)Google Scholar
  27. 27.
    Lai, M.J., Wang, J.: An unconstrained \(\ell _q\) minimization with \(0 < q\le 1\) for sparse solution of underdetermined linear systems. SIAM J. Optim. 21(1), 82–101 (2011)Google Scholar
  28. 28.
    Sturmfels, B.: What is a Gröbner basis? Notices Am. Math. Soc. 52(10), 1199–1200 (2005)zbMATHMathSciNetGoogle Scholar
  29. 29.
    Buchberger, B.: Introduction to Gröbner bases. In: Buchberger, B., Winkler, F. (eds.) Gröbner Basis and Applications. London Mathematical Society Lecture Notes Series, vol. 251, pp. 3–31. Cambridge University Press, Cambridge (1998)Google Scholar
  30. 30.
    Kreuzer, M., Robbiano, L.: Computational Commutative Algebra, vol. 1. Springer, New York (2000)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Kristian Bredies
    • 1
  • Dirk A. Lorenz
    • 2
    Email author
  • Stefan Reiterer
    • 1
  1. 1.Institute of Mathematics and Scientific ComputingUniversity of GrazGrazAustria
  2. 2.Institute for Analysis and AlgebraTU BraunschweigBraunschweigGermany

Personalised recommendations