Skip to main content
SpringerLink
Log in

A Block Inertial Bregman Proximal Algorithm for Nonsmooth Nonconvex Problems with Application to Symmetric Nonnegative Matrix Tri-Factorization

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

We propose BIBPA, a block inertial Bregman proximal algorithm for minimizing the sum of a block relatively smooth function (that is, relatively smooth concerning each block) and block separable nonsmooth nonconvex functions. We show that the cluster points of the sequence generated by BIBPA are critical points of the objective under standard assumptions, and this sequence converges globally when a regularization of the objective function satisfies the Kurdyka-Łojasiewicz (KL) property. We also provide the convergence rate when a regularization of the objective function satisfies the Łojasiewicz inequality. We apply BIBPA to the symmetric nonnegative matrix tri-factorization (SymTriNMF) problem, where we propose kernel functions for SymTriNMF and provide closed-form solutions for subproblems of BIBPA.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. A function \(\varphi :\mathbb {R}^n\rightarrow {\overline{\mathbb {R}}}\) said to be real analytic if it can be represented by a convergent power series.

References

  1. Ahookhosh, M.: Accelerated first-order methods for large-scale convex optimization: nearly optimal complexity under strong convexity. Math. Methods Oper. Res. 89(3), 319–353 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  2. Ahookhosh, M., Hien, L.T.K., Gillis, N., Patrinos, P.: Multi-block Bregman proximal alternating linearized minimization and its application to sparse orthogonal nonnegative matrix factorization. Comput. Optim. Appl. 1-35. https://doi.org/10.1007/s10589-021-00286-3 (2021)

  3. Ahookhosh, M., Themelis, A., Patrinos, P.: A Bregman forward-backward linesearch algorithm for nonconvex composite optimization: superlinear convergence to nonisolated local minima. SIAM J. Optim. 31(1), 653–685 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  4. Airoldi, E.M., Blei, D.M., Fienberg, S.E., Xing, E.P.: Mixed membership stochastic blockmodels. J. Mach. Learn. Res. 9, 1981–2014 (2008)

    MATH  Google Scholar 

  5. Attouch, H., Bolte, J.: On the convergence of the proximal algorithm for nonsmooth functions involving analytic features. Math. Program. 116(1–2), 5–16 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  6. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the Kurdyka-Łojasiewicz inequality. Math. Oper. Res. 35(2), 438–457 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  7. Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Math. Program. 137(1), 91–129 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  8. Attouch, H., Redont, P., Soubeyran, A.: A new class of alternating proximal minimization algorithms with costs-to-move. SIAM J. Optim. 18(3), 1061–1081 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  9. Auslender, A.: Optimisation méthodes numériques. 1976. Mason, Paris (1976)

    MATH  Google Scholar 

  10. Bauschke, H.H., Bolte, J., Chen, J., Teboulle, M., Wang, X.: On linear convergence of non-euclidean gradient methods without strong convexity and lipschitz gradient continuity. J. Optim. Theory Appl. 182(3), 1068–1087 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  11. Bauschke, H.H., Bolte, J., Teboulle, M.: A descent lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications. Math. Oper. Res. 42(2), 330–348 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  12. Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. CMS Books in Mathematics, Springer (2017)

    Book  MATH  Google Scholar 

  13. Beck, A.: First-Order Methods in Optimization. Society for Industrial and Applied Mathematics, Philadelphia, PA (2017)

    Book  MATH  Google Scholar 

  14. Beck, A., Pauwels, E., Sabach, S.: The cyclic block conditional gradient method for convex optimization problems. SIAM J. Optim. 25(4), 2024–2049 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  15. Beck, A., Tetruashvili, L.: On the convergence of block coordinate descent type methods. SIAM J. Optim. 23(4), 2037–2060 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  16. Bertsekas, D.P., Tsitsiklis, J.N.: Parallel and Distributed Computation: Numerical Methods. Prentice-Hall, Inc., New Jersey (1989)

    MATH  Google Scholar 

  17. Bolte, J., Daniilidis, A., Lewis, A.: The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim. 17(4), 1205–1223 (2007)

    Article  MATH  Google Scholar 

  18. Bolte, J., Daniilidis, A., Lewis, A., Shiota, M.: Clarke subgradients of stratifiable functions. SIAM J. Optim. 18(2), 556–572 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  19. Bolte, J., Daniilidis, A., Ley, O., Mazet, L.: Characterizations of Łojasiewicz inequalities: subgradient flows, talweg, convexity. Trans. Am. Math. Soc. 362(6), 3319–3363 (2010)

    Article  MATH  Google Scholar 

  20. Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146(1–2), 459–494 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  21. Bolte, J., Sabach, S., Teboulle, M., Vaisbourd, Y.: First order methods beyond convexity and Lipschitz gradient continuity with applications to quadratic inverse problems. SIAM J. Optim. 28(3), 2131–2151 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  22. Boţ, R.I., Csetnek, E.R., László, S.C.: An inertial forward-backward algorithm for the minimization of the sum of two nonconvex functions. EURO J. Comput. Optim. 4(1), 3–25 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  23. Boţ, R.I., Nguyen, D.K.: The proximal alternating direction method of multipliers in the nonconvex setting: convergence analysis and rates. Math. Oper. Res. 45(2), 682–712 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  24. Bregman, L.M.: The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phys. 7(3), 200–217 (1967)

    Article  MathSciNet  MATH  Google Scholar 

  25. Cichocki, A., Zdunek, R., Phan, A.H., Amari, S.i.: Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation. John Wiley & Sons (2009)

  26. Van den Dries, L.: Tame Topology and O-minimal Structures, vol. 248. Cambridge University Press, Cambridge (1998)

    Book  MATH  Google Scholar 

  27. Frankel, P., Garrigos, G., Peypouquet, J.: Splitting methods with variable metric for Kurdyka-Łojasiewicz functions and general convergence rates. J. Optim. Theory Appl. 165(3), 874–900 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  28. Gillis, N.: The why and how of nonnegative matrix factorization. Regul. Optim. Kernels Support Vector Mach. 12(257), 257–291 (2014)

    Google Scholar 

  29. Grippo, L., Sciandrone, M.: On the convergence of the block nonlinear Gauss-Seidel method under convex constraints. Oper. Res. Lett. 26(3), 127–136 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  30. Gutman, D.H., Peña, J.F.: Perturbed fenchel duality and first-order methods. (2018) arXiv:1812.10198

  31. Hanzely, F., Richtárik, P.: Fastest rates for stochastic mirror descent methods. Comput. Optim. Appl. 1-50. https://doi.org/10.1007/s10589-021-00284-5 (2018)

  32. Hanzely, F., Richtarik, P., Xiao, L.: Accelerated Bregman proximal gradient methods for relatively smooth convex optimization. arXiv:1808.03045 (2018)

  33. Kurdyka, K.: On gradients of functions definable in o-minimal structures. Ann. l’inst. Fourier 48(3), 769–783 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  34. László, S.C.: Forward-backward algorithms with different inertial terms for structured non-convex minimization problems. (2021) arXiv preprint arXiv:2002.07154

  35. Łojasiewicz, S.: Une propriété topologique des sous-ensembles analytiques réels. Les équations aux dérivées partielles pp. 87–89 (1963)

  36. Łojasiewicz, S.: Sur la géométrie semi- et sous- analytique. Ann. l’inst. Fourier 43(5), 1575–1595 (1993)

    Article  MATH  Google Scholar 

  37. Lu, H., Freund, R.M., Nesterov, Y.: Relatively smooth convex optimization by first-order methods, and applications. SIAM J. Optim. 28(1), 333–354 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  38. Nesterov, Y.: Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 22(2), 341–362 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  39. Nesterov, Y.: Universal gradient methods for convex optimization problems. Math. Program. 152(1–2), 381–404 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  40. Nesterov, Y.: Lectures on Convex Optimization, vol. 137. Springer, Berlin (2018)

  41. Ochs, P.: Unifying abstract inexact convergence theorems and block coordinate variable metric ipiano. SIAM J. Optim. 29(1), 541–570 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  42. Ochs, P., Chen, Y., Brox, T., Pock, T.: iPiano: inertial proximal algorithm for nonconvex optimization. SIAM J. Imaging Sci. 7(2), 1388–1419 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  43. Pock, T., Sabach, S.: Inertial proximal alternating linearized minimization (iPALM) for nonconvex and nonsmooth problems. SIAM J. Imaging Sci. 9(4), 1756–1787 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  44. Rockafellar, R.T., Wets, R.J.B.: Variational Analysis, vol. 317. Springer Science, Berlin (2011)

    MATH  Google Scholar 

  45. Shefi, R., Teboulle, M.: On the rate of convergence of the proximal alternating linearized minimization algorithm for convex problems. EURO J. Comput. Optimi. 4(1), 27–46 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  46. Tam, M.K.: Regularity properties of non-negative sparsity sets. J. Math. Anal. Appl. 447(2), 758–777 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  47. Teboulle, M.: A simplified view of first order methods for optimization. Math. Prog. pp. 1–30 (2018)

  48. Tseng, P.: Convergence of a block coordinate descent method for nondifferentiable minimization. J. Optim. Theory Appl. 109(3), 475–494 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  49. Tseng, P., Yun, S.: A coordinate gradient descent method for nonsmooth separable minimization. Math. Program. 117(1–2), 387–423 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  50. Van Nguyen, Q.: Forward-backward splitting with Bregman distances. Vietnam J. Math. 45(3), 519–539 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  51. Wang, H., Huang, H., Ding, C.: Simultaneous clustering of multi-type relational data via symmetric nonnegative matrix tri-factorization. In: Proceedings of the 20th ACM CIKM’11, pp. 279–284 (2011)

  52. Wang, X., Yuan, X., Zeng, S., Zhang, J., Zhou, J.: Block coordinate proximal gradient method for nonconvex optimization problems: convergence analysis. http://www.optimization-online.org/DB_HTML/2018/04/6573.html (2018)

  53. Zhang, X., Zhang, H., Peng, W.: Inertial Bregman proximal gradient algorithm for nonconvex problem with smooth adaptable property. (2019) arXiv preprint arXiv:1904.04436

  54. Zhang, Y., Yeung, D.Y.: Overlapping community detection via bounded nonnegative matrix tri-factorization. In: Proceedings of the 18th ACM SIGKDD, pp. 606–614 (2012)

Download references

Acknowledgements

The authors are grateful of the associate editor and anonymous referees for their helpful comments and suggestions that improved the quality of the paper.

Funding

Paragraph text.

Author information

Authors and Affiliations

  1. Department of Mathematics, University of Antwerp, Middelheimlaan 1, B-2020, Antwerp, Belgium

    Masoud Ahookhosh

  2. Department of Mathematics and Operational Research, Faculté polytechnique, Université de Mons, Rue de Houdain 9, 7000, Mons, Belgium

    Le Thi Khanh Hien & Nicolas Gillis

  3. Department of Electrical Engineering (ESAT-STADIUS) – KU Leuven, Kasteelpark Arenberg 10, 3001, Leuven, Belgium

    Panagiotis Patrinos

Authors
  1. Masoud Ahookhosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Le Thi Khanh Hien
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicolas Gillis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Panagiotis Patrinos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Masoud Ahookhosh.

Additional information

Communicated by Hedy Attouch.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahookhosh, M., Hien, L.T.K., Gillis, N. et al. A Block Inertial Bregman Proximal Algorithm for Nonsmooth Nonconvex Problems with Application to Symmetric Nonnegative Matrix Tri-Factorization. J Optim Theory Appl 190, 234–258 (2021). https://doi.org/10.1007/s10957-021-01880-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-021-01880-5

Keywords

Mathematics Subject Classification

Search

Navigation