Mathematical Programming

, Volume 146, Issue 1–2, pp 459–494 | Cite as

Proximal alternating linearized minimization for nonconvex and nonsmooth problems

Full Length Paper Series A

Abstract

We introduce a proximal alternating linearized minimization (PALM) algorithm for solving a broad class of nonconvex and nonsmooth minimization problems. Building on the powerful Kurdyka–Łojasiewicz property, we derive a self-contained convergence analysis framework and establish that each bounded sequence generated by PALM globally converges to a critical point. Our approach allows to analyze various classes of nonconvex-nonsmooth problems and related nonconvex proximal forward–backward algorithms with semi-algebraic problem’s data, the later property being shared by many functions arising in a wide variety of fundamental applications. A by-product of our framework also shows that our results are new even in the convex setting. As an illustration of the results, we derive a new and simple globally convergent algorithm for solving the sparse nonnegative matrix factorization problem.

Keywords

Alternating minimization Block coordinate descent  Gauss-Seidel method Kurdyka–Łojasiewicz property  Nonconvex-nonsmooth minimization Proximal forward-backward   Sparse nonnegative matrix factorization 

Mathematics Subject Classification (2000)

90C26 90C30 49M37 65K10 47J25 49M27 

References

  1. 1.
    Attouch, H., Bolte, J.: On the convergence of the proximal algorithm for nonsmooth functions involving analytic features. Math. Program. 116, 5–16 (2009)CrossRefMATHMathSciNetGoogle Scholar
  2. 2.
    Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka–Łojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)CrossRefMATHMathSciNetGoogle Scholar
  3. 3.
    Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Math. Program. Ser. A 137, 91–129 (2013)CrossRefMATHMathSciNetGoogle Scholar
  4. 4.
    Auslender, A.: Méthodes numériques pour la décomposition et la minimisation de fonctions non différentiables. Numerische Mathematik 18, 213–223 (1971)CrossRefMATHMathSciNetGoogle Scholar
  5. 5.
    Auslender, A.: Optimisation—Méthodes numériques. Masson, Paris (1976)MATHGoogle Scholar
  6. 6.
    Auslender, A.: Asymptotic properties of the Fenchel dual functional and applications to decomposition problems. J. Optim. Theory Appl. 73, 427–449 (1992)CrossRefMATHMathSciNetGoogle Scholar
  7. 7.
    Auslender, A., Teboulle, M., Ben-Tiba, S.: Coupling the logarithmic-quadratic proximal method and the block nonlinear Gauss-Seidel algorithm for linearly constrained convex minimization. In: Thera, M., Tichastschke, R. (eds.) Lecture Notes in Economics and Mathematical Systems, vol. 477. pp. 35–47 (1998)Google Scholar
  8. 8.
    Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, New York (2011)CrossRefMATHGoogle Scholar
  9. 9.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci 2, 183–202 (2009)CrossRefMATHMathSciNetGoogle Scholar
  10. 10.
    Beck, A., Tetruashvili, L.: On the convergence of block coordinate descent type methods. Preprint (2011)Google Scholar
  11. 11.
    Berry, M., Browne, M., Langville, A., Pauca, P., Plemmons, R.J.: Algorithms and applications for approximation nonnegative matrix factorization. Comput. Stat. Data Anal. 52, 155–173 (2007)CrossRefMATHMathSciNetGoogle Scholar
  12. 12.
    Bertsekas, D.P., Tsitsiklis, J.N.: Parallel and Distributed Computation: Numerical Methods. Prentice-Hall, New Jersey (1989)MATHGoogle Scholar
  13. 13.
    Blum, M., Floyd, R.W., Pratt, V., Rivest, R., Tarjan, R.: Time bounds for selection. J. Comput. Syst. Sci. 7, 448–461 (1973)CrossRefMATHMathSciNetGoogle Scholar
  14. 14.
    Bolte, J., Combettes, P.L., Pesquet, J.-C.: Alternating proximal algorithm for blind image recovery. In: Proceedings of the 17-th IEEE International Conference on Image Processing,Hong-Kong, ICIP, pp. 1673–1676 (2010)Google Scholar
  15. 15.
    Bolte, J., Daniilidis, A., Ley, O., Mazet, L.: Characterizations of Łojasiewicz inequalities: subgradient flows, talweg, convexity. Trans. Am. Math. Soc. 362, 3319–3363 (2010)CrossRefMATHMathSciNetGoogle Scholar
  16. 16.
    Bolte, J., Daniilidis, A., Lewis, A.: The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim. 17, 1205–1223 (2006)CrossRefMathSciNetGoogle Scholar
  17. 17.
    Bolte, J., Daniilidis, A., Lewis, A., Shiota, M.: Clarke subgradients of stratifiable functions. SIAM J. Optim. 18, 556–572 (2007)CrossRefMATHMathSciNetGoogle Scholar
  18. 18.
    Cichocki, A., Zdunek, R., Phan, A.H., Amari, S.: Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-Way Data Analysis and Blind Source Separation. Wiley, New York (2009)CrossRefGoogle Scholar
  19. 19.
    Grippo, L., Sciandrone, M.: On the convergence of the block nonlinear Gauss-Seidel method under convex constraints. Oper. Res. Lett. 26, 127–136 (2000)CrossRefMATHMathSciNetGoogle Scholar
  20. 20.
    Heiler, M., Schnorr, C.: Learning sparse representations by non-negative matrix factorization and sequential cone programming. J. Mach. Learn. Res 7, 1385–1407 (2006)MATHMathSciNetGoogle Scholar
  21. 21.
    Hoyer, P.O.: Non-negative matrix factorization with sparseness constraints. J. Mach. Learn. Res. 5, 1457–1469 (2004)MATHMathSciNetGoogle Scholar
  22. 22.
    Kurdyka, K.: On gradients of functions definable in o-minimal structures. Annales de l’institut Fourier 48, 769–783 (1998)CrossRefMATHMathSciNetGoogle Scholar
  23. 23.
    Lee, D.D., Seung, H.S.: Learning the part of objects from nonnegative matrix factorization. Nature 401, 788–791 (1999)CrossRefGoogle Scholar
  24. 24.
    Lin, C.J.: Projected gradient methods for nonnegative matrix factorization. Neural Comput. 19, 2756–2779 (2007)CrossRefMATHMathSciNetGoogle Scholar
  25. 25.
    Łojasiewicz, S.: Une propriété topologique des sous-ensembles analytiques réels, Les Équations aux Dérivées Partielles. Éditions du centre National de la Recherche Scientifique, Paris, 8–89 (1963)Google Scholar
  26. 26.
    Luss, R., Teboulle, M.: Conditional gradient algorithms for rank-one matrix approximations with a sparsity constraint. SIAM Rev. 55, 65–98 (2013)CrossRefMATHMathSciNetGoogle Scholar
  27. 27.
    Mordukhovich, B.: Variational Analysis and Generalized Differentiation. I. Basic Theory, Grundlehren der Mathematischen Wissenschaften, vol. 330. Springer, Berlin (2006)CrossRefGoogle Scholar
  28. 28.
    Ortega, J.M., Rheinboldt, W.C.: Iterative Solution of Nonlinear Equations in Several Variables. Academic Press, New-York (1970)MATHGoogle Scholar
  29. 29.
    Palomar, D.P., Eldar, Y. (eds.): Convex Optimization in Signal Processing and Communications. Cambridge University Press, UK (2010)MATHGoogle Scholar
  30. 30.
    Powell, M.J.D.: On search directions for minimization algorithms. Math. Program. 4, 193–201 (1973)CrossRefMATHGoogle Scholar
  31. 31.
    Rockafellar, R.T., Wets, R.: Variational Analysis Grundlehren der Mathematischen Wissenschaften, vol. 317. Springer, Berlin (1998)Google Scholar
  32. 32.
    Sra, S., Nowozin, S., Wright, S.J. (eds.): Optimization for Machine Learning. The MIT Press, Cambridge (2011)Google Scholar
  33. 33.
    Tseng, P.: Convergence of a block coordinate descent method for nondifferentiable minimization. J. Optim. Theory Appl. 109, 475–494 (2001)Google Scholar
  34. 34.
    Zangwill, W.I.: Nonlinear Programming: A Unified Approach. Prentice Hall, Englewood Cliffs (1969)MATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg and Mathematical Optimization Society 2013

Authors and Affiliations

  1. 1.TSE (GREMAQ, Université Toulouse I), Manufacture des TabacsToulouseFrance
  2. 2.School of Mathematical SciencesTel-Aviv UniversityTel AvivIsrael

Personalised recommendations