Abstract
In this paper, we propose an algorithm combining Bregman alternating minimization algorithm with two-step inertial force for solving a minimization problem composed of two nonsmooth functions with a smooth one in the absence of convexity. For solving nonconvex and nonsmooth problems, we give an abstract convergence theorem for general descent methods satisfying a sufficient decrease assumption, and allowing a relative error tolerance. Our result holds under the assumption that the objective function satisfies the Kurdyka–Łojasiewicz inequality. The proposed algorithm is shown to satisfy the requirements of our abstract convergence theorem. The convergence is obtained provided an appropriate regularization of the objective function satisfies the Kurdyka–Łojasiewicz inequality. Finally, numerical results are reported to show the effectiveness of the proposed algorithm.
Similar content being viewed by others
References
Neumann, J.v.: Functional operators, Vol.22, Annals of Mathematics Studies, Princeton University Press, Princeton, NJ (1950)
Combettes, P.L., Wajs, V.: Signal recovery by proximal forward-backward splitting. Multiscale Model Sim 4(4), 1168–1200 (2005)
Donoho, D.L.: Compressed sensing. IEEE Trans. Inform. Theory 4, 1289–1306 (2006)
Lions, P.L.: On the Schwarz alternating method. III, A variant for nonoverlapping subdomains. In: Chan, T.F., Glowinski, R., Périaux, J., Widlund, O. (eds.) Third Internat, pp. 202–231. Sympos. Domain Decomposition Methods for Partial Differential Equations, SIAM, Philadelphia (1990)
Widrow, B., Wallach, E.: Adaptive Inverse Control. Prentice-Hall, Englewood Cliffs, NJ (1996)
Behmardi, B., Raich, R.: On provable exact low-rank recovery in topic models. In: proceedings of IEEE statistical signal processing workshop (SSP), Nice, (2011), 265-268
Xu, H., Caramanis, C., Mannor, S.: Outlier-robust PCA: the high-dimensional case. IEEE Trans. Inform. Theor. 59, 546–572 (2013)
Sra, S., Nowozin, S., Wright, S.J. (eds.): Optimization for Machine Learning. The MIT Press, Cambridge (2011)
Wang, F., Cao, W., Xu, Z.: Convergence of multi-block Bregman ADMM for nonconvex composite problems. Sci. China Inf. Sci. 61, 122101:1-122101:12 (2018)
Li, G., Pong, T.K.: Global convergence of splitting methods for nonconvex composite optimization. SIAM J. Optim. 25(4), 2434–2460 (2015)
Hong, M., Luo, Z.Q., Razaviyayn, M.: Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems. SIAM J. Optim. 26, 337–364 (2016)
Li, G., Pong, T.K.: Douglas-Rachford splitting for nonconvex optimization with application to nonconvex feasibility problems. Math. Program. Ser. A 159, 371–401 (2016)
Clason, C., Mazurenko, S., Valkonen, T.: Acceleration and global convergence of a first-order primal-dual method for nonconvex problems. SIAM J. Optim. 29(1), 933–963 (2019)
Wu, Z., Li, M., Wang, D.Z.W., Han, D.: A symmetric alternating direction method of multipliers for separable nonconvex minimization problems, Asia Pac. J. Oper. Res. 34(6), 1750030 (2017)
Wen, B., Chen, X., Pong, T.K.: A proximal difference-of-convex algorithm with extrapolation. Comput. Optim. Appl. 69, 297–324 (2018)
Liu, Z., Wu, C., Zhao, Y.: A new globally convergent algorithm for non-Lipschitz \(l_p-l_q\) minimization. Adv. Comput. Math. 45, 1369–1399 (2019)
Li, M., Wu, Z.: Convergence analysis of the generalized splitting methods for a class of nonconvex optimization problems. J. Optim. Theory Appl. 183(2), 535–565 (2019)
Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka-Łojasiewicz inequality. Math. Oper. Res. 35(2), 438–457 (2010)
Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. Ser. A 146, 459–494 (2014)
Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4, 1–17 (1964)
Ochs, P., Chen, Y., Brox, T., Pock, T.: iPiano: inertial proximal algorithm for nonconvex optimization. SIAM J. Imaging Sci. 7(2), 1388–1419 (2014)
Boţ, R.I., Csetnek, E.R.: An inertial Tseng’s type proximal algorithm for nonsmooth and nonconvex optimization problems. J. Optim. Theory Appl. 171(2), 600–616 (2016)
Zhang, Y., He, S.: Inertial proximal alternating minimization for nonconvex and nonsmooth problems. J. Inequal. Appl. 2017, 232 (2017)
Pock, T., Sabach, S.: Inertial proximal alternating linearized minimization (iPALM) for nonconvex and nonsmooth problems. SIAM J. Imaging Sci. 9, 1756–1787 (2017)
Gao, X., Cai, X., Han, D.: A Gauss-Seidel type inertial proximal alternating linearized minimization for a class of nonconvex optimization problems. J. Global Optim. 76, 863–887 (2020)
Polyak, B.T.: Introduction to Optimization, Optimization Software Inc., Publications Division: New York (1987)
Liang, J., Fadili, J.M., Peyrë, G.: A multi-step inertial forward-backward splitting method for non-convex optimization, Advances in Neural Information Processing Systems, Advances in Neural Information Processing Systems 29-Proceedings of the 2016 conference, (2016), 4042-4050. Adv. neural inf. proces. syst
Poon, C., Liang, J.: Geometry of first-order methods and adaptive acceleration. arXiv:2003.03910
Mordukhovich, B.: Variational Analysis and Generalized Differentiation, I: Basic Theory. Grundlehren der Mathematischen Wissenschaften, Vol. 330. Springer-Verlag, Berlin (2006)
Bolte, J., Daniilidis, A., Lewis, A.: The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim. 17(4), 1205–1223 (2006)
Rockafellar, R.T., Wets, R.: Variational analysis, Grundlehren der Mathematischen Wissenschaften, vol. 317. Springer, Berlin (1998)
Łojasiewicz, S., Une propriété topologique des sous-ensembles analytiques réels, in Les Équations aux Dérivées Partielles (Paris,: Éditions du centre National de la Recherche Scientifique. Paris 1963, 87–89 (1962)
Kurdyka, K.: On gradients of functions definable in o-minimal structures. Ann. Inst. Fourier 48, 769–783 (1998)
Xu, Y., Yin, W.: A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM J. Imaging Sci. 6, 1758–1789 (2013)
Hsieh, Y.-P., Kao, Y.-C., Mahabadi, R.K., Yurtsever, A., Kyrillidis, A., Cevher, V.: A non-Euclidean gradient descent framework for non-convex matrix factorization. IEEE Trans. Signal Process 66, 5917–5926 (2018)
Auslender, A., Teboulle, M.: Interior gradient and proximal methods for convex and conic optimization. SIAM J. Optim. 16, 697–725 (2006)
Auslender, A., Teboulle, M.: Projected subgradient methods with non-Euclidean distances for non-differentiable convex minimization and variational inequalities. Math. Program 120, 27–48 (2009)
Bauschke, H.H., Dao, M.N., Lindstrom, S.B.: Regularizing with Bregman-Moreau envelopes. SIAM J. Optim. 28, 3208–3228 (2018)
Bolte, J., Sabach, S., Teboulle, M., Vaisbourd, Y.: First-order methods beyond convexity and Lipschitz gradient continuity with applications to quadratic inverse problems. SIAM J. Optim. 28, 2131–2151 (2018)
Bauschke, H.H., Bolte, J., Chen, J., Teboulle, M., Wang, X.: On linear convergence of non-Euclidean gradient methods without strong convexity and Lipschitz gradient continuity. J. Optim. Theory Appl. 182, 1068–1087 (2019)
Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Math. Program. Ser. A 137, 91–129 (2013)
Sunthrayuth, P., Jolaoso, L.O., Cholamjiak, P.: New Bregman projection methods for solving pseudo-monotone variational inequality problem. J. Appl. Math. Comput. (2021). https://doi.org/10.1007/s12190-021-01581-2
Hieu, D.V., Cholamjiak, P.: Modified extragradient method with Bregman distance for variational inequalities. Appl. Anal. 101(2), 655–670 (2022)
Xu, Z.B., Chang, X.Y., Xu, F.M., et al.: \(L_{1/2}\) regularization: a thresholding representation theory and a fast solver. IEEE Trans. Neural Netw. Learning Syst. 23, 1013–1027 (2012)
Zeng, J., Xu, Z., Zhang, B., et al.: Accelerated \(L_{1/2}\) regularization based SAR imaging via BCR and reduced Newton skills. Signal Process. 93(7), 1831–1844 (2013)
Acknowledgements
We are grateful to Professor Chunlin Wu, Miss Yuan Li and Mr. Xuan Lin for valuable help in Numerical experiments. We express our thanks to two anonymous referees for their constructive suggestions, which significant improved the presentation of this paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supported by Scientific Research Project of Tianjin Municipal Education Commission (2020ZD02)
Rights and permissions
About this article
Cite this article
Zhao, J., Dong, QL., Rassias, M.T. et al. Two-step inertial Bregman alternating minimization algorithm for nonconvex and nonsmooth problems. J Glob Optim 84, 941–966 (2022). https://doi.org/10.1007/s10898-022-01176-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10898-022-01176-6
Keywords
- Nonconvex optimization
- Inertial algorithm
- Bregman distance
- Kurdyka–Łojasiewicz inequality
- Alternating minimization