Splitting Methods with Variable Metric for Kurdyka–Łojasiewicz Functions and General Convergence Rates

  • Pierre Frankel
  • Guillaume Garrigos
  • Juan Peypouquet
Article

Abstract

We study the convergence of general descent methods applied to a lower semi-continuous and nonconvex function, which satisfies the Kurdyka–Łojasiewicz inequality in a Hilbert space. We prove that any precompact sequence converges to a critical point of the function, and obtain new convergence rates both for the values and the iterates. The analysis covers alternating versions of the forward–backward method with variable metric and relative errors. As an example, a nonsmooth and nonconvex version of the Levenberg–Marquardt algorithm is detailed.

Keywords

Nonconvex and nonsmooth optimization Kurdyka–Łojasiewicz inequality Descent methods Convergence rates  Variable metric Gauss–Seidel method Newton-like method 

Mathematics Subject Classification

49M37 65K10 90C26 90C30 

References

  1. 1.
    Łojasiewicz, S.: Une propriété topologique des sous-ensembles analytiques réels. In: Les Équations aux Dérivées Partielles, pp. 87–89. Éditions du centre National de la Recherche Scientifique, Paris (1963)Google Scholar
  2. 2.
    Kurdyka, K.: On gradients of functions definable in o-minimal structures. Ann. Inst. Fourier 48, 769–783 (1998)CrossRefMATHMathSciNetGoogle Scholar
  3. 3.
    Łojasiewicz, S.: Sur la géométrie semi- et sous-analytique. Ann. Inst. Fourier 43, 1575–1595 (1993)CrossRefMATHGoogle Scholar
  4. 4.
    Kurdyka, K., Parusiński, A.: \(\mathbf{w}_f\)-stratification of subanalytic functions and the Łojasiewicz inequality. C. R. Acad. Paris 318, 129–133 (1994)MATHGoogle Scholar
  5. 5.
    Bolte, J., Daniilidis, A., Lewis, A.: The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim. 17, 1205–1223 (2006)CrossRefMathSciNetGoogle Scholar
  6. 6.
    Simon, L.: Asymptotics for a class of nonlinear evolution equations, with applications to geometric problems. Ann. Math. 118, 525–571 (1983)CrossRefMATHGoogle Scholar
  7. 7.
    Huang, S.-Z., Takáč, P.: Convergence in gradient-like systems which are asymptotically autonomous and analytic. Nonlinear Anal. Ser. A Theory Methods 46, 675–698 (2001)CrossRefMATHGoogle Scholar
  8. 8.
    Chill, R., Jendoubi, M.A.: Convergence to steady states in asymptotically autonomous semilinear evolution equations. Nonlinear Anal. 53, 1017–1039 (2003)CrossRefMATHMathSciNetGoogle Scholar
  9. 9.
    Haraux, A., Jendoubi, M.A.: Convergence of solutions of second-order gradient-like systems with analytic nonlinearities. J. Differ. Equ. 144(2), 313–320 (1998)CrossRefMATHMathSciNetGoogle Scholar
  10. 10.
    Haraux, A.: A hyperbolic variant of Simon’s convergence theorem. Evolution equations and their applications in physical and life sciences (Bad Herrenalb, 1998). Lecture Notes in Pure and Applied Mathematics. Dekker, New York (2001)Google Scholar
  11. 11.
    Baudoin, L., Salomon, J.: Constructive solution of a bilinear optimal control problem for a Schrödinger equation. Syst. Control Lett. 57(6), 453–464 (2008)CrossRefGoogle Scholar
  12. 12.
    Absil, P.-A., Mahony, R., Andrews, B.: Convergence of the iterates of descent methods for analytic cost functions. SIAM J. Optim. 16, 531–547 (2005)CrossRefMATHMathSciNetGoogle Scholar
  13. 13.
    Attouch, H., Bolte, J.: On the convergence of the proximal algorithm for nonsmooth functions involving analytic features. Math. Program. Ser. B 116, 5–16 (2009)CrossRefMATHMathSciNetGoogle Scholar
  14. 14.
    Bolte, J., Daniilidis, A., Ley, O., Mazet, L.: Characterizations of Łojasiewicz inequalities: subgradient flows, talweg, convexity. Trans. Am. Math. Soc. 362, 3319–3363 (2010)CrossRefMATHMathSciNetGoogle Scholar
  15. 15.
    Merlet, B., Pierre, M.: Convergence to equilibrium for the backward Euler scheme and applications. Commun. Pure Appl. Anal 9, 685–702 (2010)CrossRefMATHMathSciNetGoogle Scholar
  16. 16.
    Noll, D.: Convergence of non-smooth descent methods using the Kurdyka–Lojasiewicz inequality. J. Optim. Theory Appl. 160, 553–572 (2014)CrossRefMATHMathSciNetGoogle Scholar
  17. 17.
    Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss–Seidel methods. Math. Program. 137(1–2), 91–129 (2013)CrossRefMATHMathSciNetGoogle Scholar
  18. 18.
    Chouzenoux, E., Pesquet, J.C., Repetti, A.: Variable metric forward-backward algorithm for minimizing the sum of a differentiable function and a convex function. J. Optim. Theory Appl . (to appear) (2013)Google Scholar
  19. 19.
    Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems. An approach based on the Kurdyka–Łojasiewicz inequality. Math. Oper. Res. 35(2), 438–457 (2010)CrossRefMATHMathSciNetGoogle Scholar
  20. 20.
    Xu, Y., Yin, W.: A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM J. Imaging Sci. 6(3), 1758–1789 (2013)CrossRefMATHMathSciNetGoogle Scholar
  21. 21.
    Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Programm. 15, 1–36 (2013)Google Scholar
  22. 22.
    Chouzenoux, E., Pesquet, J.C., Repetti, A.: A block coordinate variable metric forward-backward algorithm (2014). http://www.optimization-online.org/DB_HTML/2013/12/4178.html
  23. 23.
    Li, D., Pang, L.P., Chen, S.: A proximal alternating linearization method for nonconvex optimization problems. Optim. Methods Softw. 29(4), 646–657 (2014)CrossRefMathSciNetGoogle Scholar
  24. 24.
    Bolte, J., Daniilidis, A., Lewis, A., Shiota, M.: Clarke subgradients of stratifiable functions. SIAM J. Optim. 18(2), 556–572 (2007)CrossRefMATHMathSciNetGoogle Scholar
  25. 25.
    van den Dries, L.: Tame topology and o-minimal structures. Bull. AMS 37(3), 351–357 (2000)CrossRefGoogle Scholar
  26. 26.
    van den Dries, L., Miller, C.: Geometric categories and o-minimal structures. Duke Math. J. 84, 497–540 (1996)CrossRefMATHMathSciNetGoogle Scholar
  27. 27.
    Haraux, A., Jendoubi, M.A.: The Łojasiewicz gradient inequality in the infinite dimensional Hilbert space framework. J. Funct. Anal. 260(9), 2826–2842 (2010)CrossRefMathSciNetGoogle Scholar
  28. 28.
    Chill, R.: The Łojasiewicz–Simon gradient inequality in Hilbert spaces. In: Jendoubi, M.A. (ed.) Proceedings of the 5th European-Maghrebian Workshop on Semigroup Theory, Evolution Equations, and Applications, pp. 25–36 (2006)Google Scholar
  29. 29.
    Attouch, H., Buttazo, G., Michaille, G.: Variational Analysis in Sobolev and \(BV\) Spaces. MPS-SIAM Series on Optimization. Springer, NewYork (2008)Google Scholar
  30. 30.
    Bertsekas, D.: Nonlinear Programming. Athena Scientific, Belmont (1999)MATHGoogle Scholar
  31. 31.
    Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14(5), 877–898 (1976)CrossRefMATHMathSciNetGoogle Scholar
  32. 32.
    Ferris, M.: Finite termination of the proximal point algorithm. Math. Program. 50, 359–366 (1991)CrossRefMATHMathSciNetGoogle Scholar
  33. 33.
    Peypouquet, J.: Asymptotic convergence to the optimal value of diagonal proximal iterations in convex minimization. J. Convex Anal. 16(1), 277–286 (2009)MATHMathSciNetGoogle Scholar
  34. 34.
    Chill, R., Fiorenza, A.: Convergence and decay rate to equilibrium of bounded solutions of quasilinear parabolic equations. J. Differ. Equ. 228, 611–632 (2006)CrossRefMATHMathSciNetGoogle Scholar
  35. 35.
    Cauchy, A.-L.: Méthode générale pour la résolution des systèmes d’équations simultanées. C. R. Acad. Sci. Paris 25, 536–538 (1847)Google Scholar
  36. 36.
    Martinet, B.: Régularisation d’inéquations variationnelles par approximations successives. Rev. Française Informat. Recherche Opérationnelle 4, Sér. R-3, 154–158 (1970)Google Scholar
  37. 37.
    Brézis, H., Lions, P.-L.: Produits infinis de résolvantes. Israel J. Math. 29(4), 329–345 (1978)CrossRefMATHMathSciNetGoogle Scholar
  38. 38.
    Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979)CrossRefMATHMathSciNetGoogle Scholar
  39. 39.
    Passty, G.: Ergodic convergence to a zero of the sum of monotone operators in Hilbert space. J. Math. Anal. Appl. 72(2), 383–390 (1979)CrossRefMATHMathSciNetGoogle Scholar
  40. 40.
    Alvarez, F., Bolte, J., Brahic, O.: Hessian Riemannian gradient flows in convex programming. SIAM J. Control Optim. 43(2), 477–501 (2004)CrossRefMATHMathSciNetGoogle Scholar
  41. 41.
    Alvarez, F., López, J., Ramírez, H.: Interior proximal algorithm with variable metric for second-order cone programming: applications to structural optimization and support vector machines. Optim. Methods Softw. 25(4–6), 859–881 (2010)CrossRefMATHMathSciNetGoogle Scholar
  42. 42.
    Attouch, H., Svaiter, B.F.: A continuous dynamical Newton-like approach to solving monotone inclusions. SIAM J. Optim. 49(2), 574–598 (2011)CrossRefMATHMathSciNetGoogle Scholar
  43. 43.
    Sra, S., Nowozin, S., Wright, S.J.: Optimization for Machine Learning. MIT Press, Cambridge (2011)Google Scholar
  44. 44.
    Gafni, E.M., Bertsekas, D.P.: Two-metric projection methods for constrained optimization. SIAM J. Control Optim. 22(6), 936–964 (1984)CrossRefMATHMathSciNetGoogle Scholar
  45. 45.
    Bertsekas, D.P.: Projected Newton methods for optimization problems with simple constraints. SIAM J. Control Optim. 20(2), 221–246 (1982)CrossRefMATHMathSciNetGoogle Scholar
  46. 46.
    Lewis, A.S., Luke, D.R., Malick, J.: Local linear convergence for alternating and averaged non convex projections. Found. Comput. Math. 9(4), 485–513 (2009)CrossRefMATHMathSciNetGoogle Scholar
  47. 47.
    Donoho, D.L.: For most Large underdetermined systems of linear equations the minimal \(\ell ^1\)-norm solution is also the sparsest solution. Commun. Pure Appl. Math. 6, 59, 797–829 (2006)CrossRefMathSciNetGoogle Scholar
  48. 48.
    Donoho, D.L., Tanner, J.: Counting the faces of randomly-projected hypercubes and orthants, with applications. Discret. Comput. Geom. 43, 522–541 (2010)CrossRefMATHMathSciNetGoogle Scholar
  49. 49.
    Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)CrossRefMATHGoogle Scholar
  50. 50.
    Chandrasekaran, V., Sanghavi, S., Parillo, P.A., Willsky, A.S.: Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim. 21(2), 572–596 (2011)CrossRefMATHMathSciNetGoogle Scholar
  51. 51.
    Ganesh, A., Lin, Z., Wright, J., Wu, L., Chen, M., Ma, Y.: Fast algorithms for recovering a corrupted low-rank matrix. In: Proceedings of the 3rd IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), pp. 213–216 (2009)Google Scholar
  52. 52.
    Yuan, X., Yang, J.: Sparse and low-rank matrix decomposition via alternating direction method. Pac. J. Optim. 9(1), 167–180 (2013)MATHMathSciNetGoogle Scholar
  53. 53.
    Recht, B., Fazel, B., Parillo, P.A.: Guaranteed minimum rank solutions to linear matrix equations via nuclear norm minimization. SIAM Rev. 52, 471–501 (2007)CrossRefGoogle Scholar
  54. 54.
    Nesterov, Y.E.: A method for solving the convex programming problem with convergence rate O(\(1/k^2\)). Dokl. Akad. Nauk SSSR 269, 543–547 (1983)MathSciNetGoogle Scholar
  55. 55.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2, 183–202 (2009)CrossRefMATHMathSciNetGoogle Scholar
  56. 56.
    Beck, A., Teboulle, M.: Gradient-based algorithms with applications in signal recovery problems. In: Palomar, D., Eldar, Y. (eds.) Convex Optimization in Signal Processing and Communications, pp. 33–88. Cambridge University Press, Cambridge (2010)Google Scholar
  57. 57.
    Ochs, P., Chen, Y., Brox, T.: Pock, T.: iPiano: Inertial Proximal Algorithm for Non-convex Optimization (2014). http://lmb.informatik.uni-freiburg.de/Publications/2014/OB14/
  58. 58.
    Casas, E., Herzog, R., Wachsmuth, G.: Approximation of sparse controls in semilinear equations by piecewise linear functions. Numer. Math. 122(4), 645–669 (2012)CrossRefMATHMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Pierre Frankel
    • 1
  • Guillaume Garrigos
    • 1
    • 2
  • Juan Peypouquet
    • 2
  1. 1.Institut de Mathématiques et Modélisation de Montpellier, UMR 5149 CNRSUniversité Montpellier 2Montpellier cedex 5France
  2. 2.Departamento de Matemática & AM2VUniversidad Técnica Federico Santa MaríaValparaisoChile

Personalised recommendations