Splitting Methods with Variable Metric for Kurdyka–Łojasiewicz Functions and General Convergence Rates

  • Pierre Frankel
  • Guillaume GarrigosEmail author
  • Juan Peypouquet


We study the convergence of general descent methods applied to a lower semi-continuous and nonconvex function, which satisfies the Kurdyka–Łojasiewicz inequality in a Hilbert space. We prove that any precompact sequence converges to a critical point of the function, and obtain new convergence rates both for the values and the iterates. The analysis covers alternating versions of the forward–backward method with variable metric and relative errors. As an example, a nonsmooth and nonconvex version of the Levenberg–Marquardt algorithm is detailed.


Nonconvex and nonsmooth optimization Kurdyka–Łojasiewicz inequality Descent methods Convergence rates  Variable metric Gauss–Seidel method Newton-like method 

Mathematics Subject Classification

49M37 65K10 90C26 90C30 



The authors thank H. Attouch for useful remarks. They would also like to thank the anonymous reviewer for his careful reading and constructive comments. The second and third authors are partly supported by Conicyt Anillo Project ACT-1106, ECOS-Conicyt Project C13E03 and Millenium Nucleus ICM/FIC P10-024F. The third author is also partly supported by FONDECYT Grant 1140829 and Basal Project CMM Universidad de Chile.


  1. 1.
    Łojasiewicz, S.: Une propriété topologique des sous-ensembles analytiques réels. In: Les Équations aux Dérivées Partielles, pp. 87–89. Éditions du centre National de la Recherche Scientifique, Paris (1963)Google Scholar
  2. 2.
    Kurdyka, K.: On gradients of functions definable in o-minimal structures. Ann. Inst. Fourier 48, 769–783 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  3. 3.
    Łojasiewicz, S.: Sur la géométrie semi- et sous-analytique. Ann. Inst. Fourier 43, 1575–1595 (1993)CrossRefzbMATHGoogle Scholar
  4. 4.
    Kurdyka, K., Parusiński, A.: \(\mathbf{w}_f\)-stratification of subanalytic functions and the Łojasiewicz inequality. C. R. Acad. Paris 318, 129–133 (1994)zbMATHGoogle Scholar
  5. 5.
    Bolte, J., Daniilidis, A., Lewis, A.: The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim. 17, 1205–1223 (2006)CrossRefMathSciNetGoogle Scholar
  6. 6.
    Simon, L.: Asymptotics for a class of nonlinear evolution equations, with applications to geometric problems. Ann. Math. 118, 525–571 (1983)CrossRefzbMATHGoogle Scholar
  7. 7.
    Huang, S.-Z., Takáč, P.: Convergence in gradient-like systems which are asymptotically autonomous and analytic. Nonlinear Anal. Ser. A Theory Methods 46, 675–698 (2001)CrossRefzbMATHGoogle Scholar
  8. 8.
    Chill, R., Jendoubi, M.A.: Convergence to steady states in asymptotically autonomous semilinear evolution equations. Nonlinear Anal. 53, 1017–1039 (2003)CrossRefzbMATHMathSciNetGoogle Scholar
  9. 9.
    Haraux, A., Jendoubi, M.A.: Convergence of solutions of second-order gradient-like systems with analytic nonlinearities. J. Differ. Equ. 144(2), 313–320 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  10. 10.
    Haraux, A.: A hyperbolic variant of Simon’s convergence theorem. Evolution equations and their applications in physical and life sciences (Bad Herrenalb, 1998). Lecture Notes in Pure and Applied Mathematics. Dekker, New York (2001)Google Scholar
  11. 11.
    Baudoin, L., Salomon, J.: Constructive solution of a bilinear optimal control problem for a Schrödinger equation. Syst. Control Lett. 57(6), 453–464 (2008)CrossRefGoogle Scholar
  12. 12.
    Absil, P.-A., Mahony, R., Andrews, B.: Convergence of the iterates of descent methods for analytic cost functions. SIAM J. Optim. 16, 531–547 (2005)CrossRefzbMATHMathSciNetGoogle Scholar
  13. 13.
    Attouch, H., Bolte, J.: On the convergence of the proximal algorithm for nonsmooth functions involving analytic features. Math. Program. Ser. B 116, 5–16 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  14. 14.
    Bolte, J., Daniilidis, A., Ley, O., Mazet, L.: Characterizations of Łojasiewicz inequalities: subgradient flows, talweg, convexity. Trans. Am. Math. Soc. 362, 3319–3363 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  15. 15.
    Merlet, B., Pierre, M.: Convergence to equilibrium for the backward Euler scheme and applications. Commun. Pure Appl. Anal 9, 685–702 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Noll, D.: Convergence of non-smooth descent methods using the Kurdyka–Lojasiewicz inequality. J. Optim. Theory Appl. 160, 553–572 (2014)CrossRefzbMATHMathSciNetGoogle Scholar
  17. 17.
    Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss–Seidel methods. Math. Program. 137(1–2), 91–129 (2013)CrossRefzbMATHMathSciNetGoogle Scholar
  18. 18.
    Chouzenoux, E., Pesquet, J.C., Repetti, A.: Variable metric forward-backward algorithm for minimizing the sum of a differentiable function and a convex function. J. Optim. Theory Appl . (to appear) (2013)Google Scholar
  19. 19.
    Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems. An approach based on the Kurdyka–Łojasiewicz inequality. Math. Oper. Res. 35(2), 438–457 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  20. 20.
    Xu, Y., Yin, W.: A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM J. Imaging Sci. 6(3), 1758–1789 (2013)CrossRefzbMATHMathSciNetGoogle Scholar
  21. 21.
    Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Programm. 15, 1–36 (2013)Google Scholar
  22. 22.
    Chouzenoux, E., Pesquet, J.C., Repetti, A.: A block coordinate variable metric forward-backward algorithm (2014).
  23. 23.
    Li, D., Pang, L.P., Chen, S.: A proximal alternating linearization method for nonconvex optimization problems. Optim. Methods Softw. 29(4), 646–657 (2014)CrossRefMathSciNetGoogle Scholar
  24. 24.
    Bolte, J., Daniilidis, A., Lewis, A., Shiota, M.: Clarke subgradients of stratifiable functions. SIAM J. Optim. 18(2), 556–572 (2007)CrossRefzbMATHMathSciNetGoogle Scholar
  25. 25.
    van den Dries, L.: Tame topology and o-minimal structures. Bull. AMS 37(3), 351–357 (2000)CrossRefGoogle Scholar
  26. 26.
    van den Dries, L., Miller, C.: Geometric categories and o-minimal structures. Duke Math. J. 84, 497–540 (1996)CrossRefzbMATHMathSciNetGoogle Scholar
  27. 27.
    Haraux, A., Jendoubi, M.A.: The Łojasiewicz gradient inequality in the infinite dimensional Hilbert space framework. J. Funct. Anal. 260(9), 2826–2842 (2010)CrossRefMathSciNetGoogle Scholar
  28. 28.
    Chill, R.: The Łojasiewicz–Simon gradient inequality in Hilbert spaces. In: Jendoubi, M.A. (ed.) Proceedings of the 5th European-Maghrebian Workshop on Semigroup Theory, Evolution Equations, and Applications, pp. 25–36 (2006)Google Scholar
  29. 29.
    Attouch, H., Buttazo, G., Michaille, G.: Variational Analysis in Sobolev and \(BV\) Spaces. MPS-SIAM Series on Optimization. Springer, NewYork (2008)Google Scholar
  30. 30.
    Bertsekas, D.: Nonlinear Programming. Athena Scientific, Belmont (1999)zbMATHGoogle Scholar
  31. 31.
    Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14(5), 877–898 (1976)CrossRefzbMATHMathSciNetGoogle Scholar
  32. 32.
    Ferris, M.: Finite termination of the proximal point algorithm. Math. Program. 50, 359–366 (1991)CrossRefzbMATHMathSciNetGoogle Scholar
  33. 33.
    Peypouquet, J.: Asymptotic convergence to the optimal value of diagonal proximal iterations in convex minimization. J. Convex Anal. 16(1), 277–286 (2009)zbMATHMathSciNetGoogle Scholar
  34. 34.
    Chill, R., Fiorenza, A.: Convergence and decay rate to equilibrium of bounded solutions of quasilinear parabolic equations. J. Differ. Equ. 228, 611–632 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  35. 35.
    Cauchy, A.-L.: Méthode générale pour la résolution des systèmes d’équations simultanées. C. R. Acad. Sci. Paris 25, 536–538 (1847)Google Scholar
  36. 36.
    Martinet, B.: Régularisation d’inéquations variationnelles par approximations successives. Rev. Française Informat. Recherche Opérationnelle 4, Sér. R-3, 154–158 (1970)Google Scholar
  37. 37.
    Brézis, H., Lions, P.-L.: Produits infinis de résolvantes. Israel J. Math. 29(4), 329–345 (1978)CrossRefzbMATHMathSciNetGoogle Scholar
  38. 38.
    Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979)CrossRefzbMATHMathSciNetGoogle Scholar
  39. 39.
    Passty, G.: Ergodic convergence to a zero of the sum of monotone operators in Hilbert space. J. Math. Anal. Appl. 72(2), 383–390 (1979)CrossRefzbMATHMathSciNetGoogle Scholar
  40. 40.
    Alvarez, F., Bolte, J., Brahic, O.: Hessian Riemannian gradient flows in convex programming. SIAM J. Control Optim. 43(2), 477–501 (2004)CrossRefzbMATHMathSciNetGoogle Scholar
  41. 41.
    Alvarez, F., López, J., Ramírez, H.: Interior proximal algorithm with variable metric for second-order cone programming: applications to structural optimization and support vector machines. Optim. Methods Softw. 25(4–6), 859–881 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  42. 42.
    Attouch, H., Svaiter, B.F.: A continuous dynamical Newton-like approach to solving monotone inclusions. SIAM J. Optim. 49(2), 574–598 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  43. 43.
    Sra, S., Nowozin, S., Wright, S.J.: Optimization for Machine Learning. MIT Press, Cambridge (2011)Google Scholar
  44. 44.
    Gafni, E.M., Bertsekas, D.P.: Two-metric projection methods for constrained optimization. SIAM J. Control Optim. 22(6), 936–964 (1984)CrossRefzbMATHMathSciNetGoogle Scholar
  45. 45.
    Bertsekas, D.P.: Projected Newton methods for optimization problems with simple constraints. SIAM J. Control Optim. 20(2), 221–246 (1982)CrossRefzbMATHMathSciNetGoogle Scholar
  46. 46.
    Lewis, A.S., Luke, D.R., Malick, J.: Local linear convergence for alternating and averaged non convex projections. Found. Comput. Math. 9(4), 485–513 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  47. 47.
    Donoho, D.L.: For most Large underdetermined systems of linear equations the minimal \(\ell ^1\)-norm solution is also the sparsest solution. Commun. Pure Appl. Math. 6, 59, 797–829 (2006)CrossRefMathSciNetGoogle Scholar
  48. 48.
    Donoho, D.L., Tanner, J.: Counting the faces of randomly-projected hypercubes and orthants, with applications. Discret. Comput. Geom. 43, 522–541 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  49. 49.
    Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)CrossRefzbMATHGoogle Scholar
  50. 50.
    Chandrasekaran, V., Sanghavi, S., Parillo, P.A., Willsky, A.S.: Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim. 21(2), 572–596 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  51. 51.
    Ganesh, A., Lin, Z., Wright, J., Wu, L., Chen, M., Ma, Y.: Fast algorithms for recovering a corrupted low-rank matrix. In: Proceedings of the 3rd IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), pp. 213–216 (2009)Google Scholar
  52. 52.
    Yuan, X., Yang, J.: Sparse and low-rank matrix decomposition via alternating direction method. Pac. J. Optim. 9(1), 167–180 (2013)zbMATHMathSciNetGoogle Scholar
  53. 53.
    Recht, B., Fazel, B., Parillo, P.A.: Guaranteed minimum rank solutions to linear matrix equations via nuclear norm minimization. SIAM Rev. 52, 471–501 (2007)CrossRefGoogle Scholar
  54. 54.
    Nesterov, Y.E.: A method for solving the convex programming problem with convergence rate O(\(1/k^2\)). Dokl. Akad. Nauk SSSR 269, 543–547 (1983)MathSciNetGoogle Scholar
  55. 55.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2, 183–202 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  56. 56.
    Beck, A., Teboulle, M.: Gradient-based algorithms with applications in signal recovery problems. In: Palomar, D., Eldar, Y. (eds.) Convex Optimization in Signal Processing and Communications, pp. 33–88. Cambridge University Press, Cambridge (2010)Google Scholar
  57. 57.
    Ochs, P., Chen, Y., Brox, T.: Pock, T.: iPiano: Inertial Proximal Algorithm for Non-convex Optimization (2014).
  58. 58.
    Casas, E., Herzog, R., Wachsmuth, G.: Approximation of sparse controls in semilinear equations by piecewise linear functions. Numer. Math. 122(4), 645–669 (2012)CrossRefzbMATHMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Pierre Frankel
    • 1
  • Guillaume Garrigos
    • 1
    • 2
    Email author
  • Juan Peypouquet
    • 2
  1. 1.Institut de Mathématiques et Modélisation de Montpellier, UMR 5149 CNRSUniversité Montpellier 2Montpellier cedex 5France
  2. 2.Departamento de Matemática & AM2VUniversidad Técnica Federico Santa MaríaValparaisoChile

Personalised recommendations