# Minimization of Non-smooth, Non-convex Functionals by Iterative Thresholding

- 965 Downloads
- 17 Citations

## Abstract

Convergence analysis is carried out for a forward-backward splitting/generalized gradient projection method for the minimization of a special class of non-smooth and genuinely non-convex minimization problems in infinite-dimensional Hilbert spaces. The functionals under consideration are the sum of a smooth, possibly non-convex and non-smooth, necessarily non-convex functional. For separable constraints in the sequence space, we show that the generalized gradient projection method amounts to a discontinuous iterative thresholding procedure, which can easily be implemented. In this case we prove strong subsequential convergence and moreover show that the limit satisfies strengthened necessary conditions for a global minimizer, i.e., it avoids a certain set of non-global minimizers. Eventually, the method is applied to problems arising in the recovery of sparse data, where strong convergence of the whole sequence is shown, and numerical tests are presented.

## Keywords

Non-convex optimization Non-smooth optimization Gradient projection method Iterative thresholding## Mathematics Subject Classification

49M05 65K10## Notes

### Acknowledgments

Kristian Bredies acknowledges support by the SFB Research Center “Mathematical Optimization and Applications in Biomedical Sciences” at the University of Graz. Dirk Lorenz acknowledges support by the DFG under Grant LO 1436/2-1.

## References

- 1.Nocedal, J., Wright, S.: Numerical Optimization. Springer Series in Operations Research and Financial Engineering, 2nd edn. Springer, New York (2006)zbMATHGoogle Scholar
- 2.Pytlak, R.: Conjugate Gradient Algorithms in Nonconvex Optimization, Nonconvex Optimization and Its Applications, vol. 89. Springer, Berlin (2009)Google Scholar
- 3.Nikolova, M.: Analysis of the recovery of edges in images and signals by minimizing nonconvex regularized least-squares. Multiscale Modell. Simul.
**4**(3), 960–991 (2005)CrossRefzbMATHMathSciNetGoogle Scholar - 4.Nikolova, M.: Markovian reconstruction using a GNC approach. IEEE Trans. Image Process.
**8**(9), 1204–1220 (1999)CrossRefGoogle Scholar - 5.Blake, A., Zisserman, A.: Visual Reconstruction. MIT Press, Cambridge (1987)Google Scholar
- 6.Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal.
**16**(6), 964–979 (1979)CrossRefzbMATHMathSciNetGoogle Scholar - 7.Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward–backward splitting. Multiscale Model. Simul.
**4**(4), 1168–1200 (2005)CrossRefzbMATHMathSciNetGoogle Scholar - 8.Bredies, K., Lorenz, D.A.: Linear convergence of iterative soft-thresholding. J. Fourier Anal. Appl.
**14**(5–6), 813–837 (2008). doi: 10.1007/s00041-008-9041-1 CrossRefzbMATHMathSciNetGoogle Scholar - 9.Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Math. Program. Ser. B. (2011). doi: 10.1007/s10107-011-0484-9.
- 10.Ito, K., Kunisch, K.: A variational approach to sparsity optimization based on Lagrange multiplier theory. MOBIS SFB-Report 2011–2014 (2011). http://math.uni-graz.at/mobis/publications/SFB-Report-2011-014.pdf.
- 11.Chouzenoux, E., Pesquet, J.C., Repetti, A.: Variable metric forward–backward algorithm for minimizing the sum of a differentiable function and a convex function. To appear J. Optim. Theory Appl. (2013). http://www.optimization-online.org/DB_HTML/2013/01/3749.html
- 12.Ochs, P., Chen, Y., Brox, T., Pock, T.: iPiano: Inertial proximal algorithm for non-convex optimization. (2013). http://lmb.informatik.uni-freiburg.de/Publications/2013/OB13
- 13.Kaplan, A., Tichatschke, R.: Proximal point methods and nonconvex optimization. J. Global Optim.
**13**, 389–406 (1998)CrossRefzbMATHMathSciNetGoogle Scholar - 14.Lewis, A.S., Luke, D.R., Malick, J.: Local linear convergence for alternating and averaged nonconvex projections. Found. Comput. Math.
**9**, 485–513 (2009)CrossRefzbMATHMathSciNetGoogle Scholar - 15.Rockafellar, R.T., Wets, R.J.B.: Variational Analysis, Grundlehren der Mathematischen Wissenschaften, vol. 317. Springer, Berlin (1998)Google Scholar
- 16.Penot, J.P.: Proximal mappings. J. Approx. Theory
**94**, 203–221 (1998)CrossRefzbMATHMathSciNetGoogle Scholar - 17.Hare, W., Sagastizábal, C.: Computing proximal points of nonconvex functions. Math. Program. Ser. B
**116**(1–2), 221–258 (2009)CrossRefzbMATHGoogle Scholar - 18.Demyanov, V.F., Rubinov, A.M.: Approximate Methods in Optimization Problems. Modern Analytic and Computational Methods in Science and Mathematics, vol. 32. American Elsevier, New York (1970)Google Scholar
- 19.Dunn, J.C.: Global and asymptotic convergence rate estimates for a class of projected gradient processes. SIAM J. Control Optim.
**19**(3), 368–400 (1981)CrossRefzbMATHMathSciNetGoogle Scholar - 20.Clarke, F.H.: Optimization and Nonsmooth Analysis. CRM Université de Montréal, Montréal (1989)zbMATHGoogle Scholar
- 21.Bredies, K., Lorenz, D.A.: Regularization with non-convex separable constraints. Inverse Problems,
**25**(8), 085011 (2009). doi: 10.1088/0266-5611/25/8/085011 - 22.Blumensath, T., Yaghoobi, M., Davies, M.: Iterative hard thresholding and \(l^0\) regularisation. In: IEEE International Conference on Acoustics, Speech and Signal Processing (2007)Google Scholar
- 23.Moulin, P., Liu, J.: Analysis of multiresolution image denoising schemes using generalized Gaussian and complexity priors. IEEE Trans. Inform. Theory
**45**(3), 909–919 (1999)CrossRefzbMATHMathSciNetGoogle Scholar - 24.Lorenz, D.A.: Non-convex variational denoising of images: interpolation between hard and soft wavelet shrinkage. Curr. Dev. Theory Appl. Wavelets
**1**(1), 31–56 (2007)zbMATHMathSciNetGoogle Scholar - 25.Lorenz, D.A.: Convergence rates and source conditions for Tikhonov regularization with sparsity constraints. J. Inverse Ill-Posed Problems
**16**(5), 463–478 (2008). doi: 10.1515/JIIP.2008.025 CrossRefzbMATHMathSciNetGoogle Scholar - 26.Grasmair, M.: Well-posedness classes for sparse regularisation. Reports of FSP S092-“Industrial Geometry”. University of Vienna, Austria (2010)Google Scholar
- 27.Lai, M.J., Wang, J.: An unconstrained \(\ell _q\) minimization with \(0 < q\le 1\) for sparse solution of underdetermined linear systems. SIAM J. Optim.
**21**(1), 82–101 (2011)Google Scholar - 28.Sturmfels, B.: What is a Gröbner basis? Notices Am. Math. Soc.
**52**(10), 1199–1200 (2005)zbMATHMathSciNetGoogle Scholar - 29.Buchberger, B.: Introduction to Gröbner bases. In: Buchberger, B., Winkler, F. (eds.) Gröbner Basis and Applications. London Mathematical Society Lecture Notes Series, vol. 251, pp. 3–31. Cambridge University Press, Cambridge (1998)Google Scholar
- 30.Kreuzer, M., Robbiano, L.: Computational Commutative Algebra, vol. 1. Springer, New York (2000)CrossRefzbMATHGoogle Scholar