Abstract
The nonlinear, or warped, resolvent recently explored by Giselsson and Bùi-Combettes has been used to model a large set of existing and new monotone inclusion algorithms. To establish convergent algorithms based on these resolvents, corrective projection steps are utilized in both works. We present a different way of ensuring convergence by means of a nonlinear momentum term, which in many cases leads to cheaper per-iteration cost. The expressiveness of our method is demonstrated by deriving a wide range of special cases. These cases cover and expand on the forward-reflected-backward method of Malitsky-Tam, the primal-dual methods of Vũ-Condat and Chambolle-Pock, and the forward-reflected-Douglas-Rachford method of Ryu-Vũ. A new primal-dual method that uses an extra resolvent step is also presented as well as a general approach for adding momentum to any special case of our nonlinear forward-backward method, in particular all the algorithms listed above.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Goldstein, A.A.: Convex programming in Hilbert space. Bull. Am. Math. Soc. 70(5), 709–711 (1964). https://doi.org/10.1090/S0002-9904-1964-11178-2
Levitin, E.S., Polyak, B.T.: Constrained minimization methods. USSR Comput. Math. Math. Phys. 6(5), 1–50 (1966)
Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16(6), 964–979 (1979). https://doi.org/10.1137/0716071
Davis, D., Yin, W.: A three-operator splitting scheme and its optimization applications. Set-Valued Var. Anal. 25(4), 829–858 (2017). https://doi.org/10.1007/s11228-017-0421-z
Combettes, P.L., Pesquet, J.-C.: Primal-dual splitting algorithm for solving inclusions with mixtures of composite, Lipschitzian, and parallel-sum type monotone operators. Set-Valued Var. Anal. 20(2), 307–330 (2012). https://doi.org/10.1007/s11228-011-0191-y
Boţ, R.I., Csetnek, E.R., Nagy, E.: Solving systems of monotone inclusions via primal-dual splitting techniques. Taiwan. J. Math. 17(6), 1983–2009 (2013). https://doi.org/10.11650/tjm.17.2013.3087
Raguet, H., Fadili, J., Peyré, G.: A generalized forward-backward splitting. SIAM J. Imaging Sci. 6(3), 1199–1226 (2013). https://doi.org/10.1137/120872802
Combettes, P.L., Eckstein, J.: Asynchronous block-iterative primal-dual decomposition methods for monotone inclusions. Math. Program. 168(1), 645–672 (2018). https://doi.org/10.1007/s10107-016-1044-0
Combettes, P.L., Glaudin, L.E.: Solving composite fixed point problems with block updates. Adv. Nonlinear Anal. 10(1), 1154–1177 (2021). https://doi.org/10.1515/anona-2020-0173
Bùi, M.N., Combettes, P.L.: Warped proximal iterations for monotone inclusions. J. Appl. Math. Anal. Appl. 491(1), 124315 (2020). https://doi.org/10.1016/j.jmaa.2020.124315
Giselsson, P.: Nonlinear forward-backward splitting with projection correction. SIAM J. Optim. 31, 2199–2226 (2021). https://doi.org/10.1137/20M1345062
Kassay, G.: The proximal points algorithm for reflexive Banach spaces. Stud. Univ. Babeş–Bolyai, Math. 30, 9–17 (1985)
Konnov, I.V.: Combined relaxation methods for generalized monotone variational inequalities. In: Generalized Convexity and Related Topics. Lecture Notes in Economics and Mathematical Systems, pp. 3–31. Springer, Berlin (2006). https://doi.org/10.1007/978-3-540-37007-9_1
Latafat, P., Patrinos, P.: Asymmetric forward–backward–adjoint splitting for solving monotone inclusions involving three operators. Comput. Optim. Appl. 68(1), 57–93 (2017). https://doi.org/10.1007/s10589-017-9909-6
Bregman, L.M.: The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phys. 7(3), 200–217 (1967). https://doi.org/10.1016/0041-5553(67)90040-7
Bauschke, H.H., Borwein, J.M., Combettes, P.L.: Bregman monotone optimization algorithms. SIAM J. Control Optim. 42(2), 596–636 (2003). https://doi.org/10.1137/S0363012902407120
Bùi, M.N., Combettes, P.L.: Bregman forward-backward operator splitting. Set-Valued Var. Anal. 29(3), 583–603 (2021). https://doi.org/10.1007/s11228-020-00563-z
Eckstein, J.: Nonlinear proximal point algorithms using Bregman functions, with applications to convex programming. Math. Oper. Res. 18(1), 202–226 (1993)
Bauschke, H.H., Combettes, P.L., Noll, D.: Joint minimization with alternating bregman proximity operators. Pac. J. Optim. (2006)
Burachik, R., Dutta, J.: Inexact proximal point methods for variational inequality problems. SIAM J. Optim. 20(5), 2653–2678 (2010). https://doi.org/10.1137/080733437
Bauschke, H.H., Dao, M.N., Lindstrom, S.B.: Regularizing with Bregman–Moreau envelopes. SIAM J. Optim. 28(4), 3208–3228 (2018). https://doi.org/10.1137/17M1130745
Malitsky, Y., Tam, M.K.: A forward-backward splitting method for monotone inclusions without cocoercivity. SIAM J. Optim. 30(2), 1451–1472 (2020). https://doi.org/10.1137/18M1207260
Tseng, P.: A modified forward-backward splitting method for maximal monotone mappings. SIAM J. Control Optim. 38(2), 431–446 (2000). https://doi.org/10.1137/S0363012998338806
Briceño-Arias, L.M., Davis, D.: Forward-backward-half forward algorithm for solving monotone inclusions. SIAM J. Optim. 28(4), 2839–2871 (2018). https://doi.org/10.1137/17M1120099
Vũ, B.C.: A splitting algorithm for dual monotone inclusions involving cocoercive operators. Adv. Comput. Math. 38(3), 667–681 (2013). https://doi.org/10.1007/s10444-011-9254-8
Condat, L.: A primal–dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms. J. Optim. Theory Appl. 158(2), 460–479 (2013). https://doi.org/10.1007/s10957-012-0245-9
Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011). https://doi.org/10.1007/s10851-010-0251-1
He, B., You, Y., Yuan, X.: On the convergence of primal-dual hybrid gradient algorithm. SIAM J. Imaging Sci. 7(4), 2526–2537 (2014). https://doi.org/10.1137/140963467
Ryu, E.K., Vũ, B.C.: Finding the forward-Douglas–Rachford-forward method. J. Optim. Theory Appl. 184(3), 858–876 (2020). https://doi.org/10.1007/s10957-019-01601-z
Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4(5), 1–17 (1964). https://doi.org/10.1016/0041-5553(64)90137-5
Alvarez, F., Attouch, H.: An inertial proximal method for maximal monotone operators via discretization of a nonlinear oscillator with damping. Set-Valued Anal. 9(1), 3–11 (2001). https://doi.org/10.1023/A:1011253113155
Moudafi, A., Oliny, M.: Convergence of a splitting inertial proximal method for monotone operators. J. Comput. Appl. Math. 155(2), 447–454 (2003). https://doi.org/10.1016/S0377-0427(02)00906-8
Boţ, R.I., Csetnek, E.R., Hendrich, C.: Inertial Douglas–Rachford splitting for monotone inclusion problems. Appl. Math. Comput. 256, 472–487 (2015). https://doi.org/10.1016/j.amc.2015.01.017
Lorenz, D.A., Pock, T.: An inertial forward-backward algorithm for monotone inclusions. J. Math. Imaging Vis. 51(2), 311–325 (2015). https://doi.org/10.1007/s10851-014-0523-2
Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009). https://doi.org/10.1137/080716542
Attouch, H., Cabot, A.: Convergence rates of inertial forward-backward algorithms. SIAM J. Optim. 28(1), 849–874 (2018). https://doi.org/10.1137/17M1114739
Boţ, R.I., Csetnek, E.R.: An inertial forward-backward-forward primal-dual splitting algorithm for solving monotone inclusion problems. Numer. Algorithms 71(3), 519–540 (2016). https://doi.org/10.1007/s11075-015-0007-5
Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces, 2nd edn. CMS Books in Mathematics. Springer, New York (2017)
Ghadimi, E., Feyzmahdavian, H.R., Johansson, M.: Global convergence of the heavy-ball method for convex optimization. In: 2015 European Control Conference (ECC), pp. 310–315 (2015). https://doi.org/10.1109/ECC.2015.7330562
Eckstein, J., Svaiter, B.F.: General projective splitting methods for sums of maximal monotone operators. SIAM J. Control Optim. 48(2), 787–811 (2009). https://doi.org/10.1137/070698816
Giselsson, P.: Nonlinear Forward-Backward Splitting with Projection Correction (2021). http://arxiv.org/abs/1908.07449v3
Bùi, M.N.: The warped resolvent of a set-valued operator: theory and applications. PhD thesis, North Carolia State University (2021) https://repository.lib.ncsu.edu/bitstream/handle/1840.20/39099/etd.pdf. Accessed 2021-10-10
O’Connor, D., Vandenberghe, L.: On the equivalence of the primal-dual hybrid gradient method and Douglas–Rachford splitting. Math. Program. 179(1), 85–108 (2020). https://doi.org/10.1007/s10107-018-1321-1
Briceño-Arias, L.M., Roldán, F.: Split-Douglas-Rachford for composite monotone inclusions and split-admm. SIAM J. Optim. 31, 2987–3013 (2021)
Briceño-Arias, L.M., Roldán, F.: Primal-dual splittings as fixed point iterations in the range of linear operators. J. Glob. Optim. 85, 847–866 (2023)
Funding
Open access funding provided by Lund University. All authors have been supported by ELLIIT: Excellence Center at Linköping-Lund in Information Technology as well as the Swedish Research Council. The Wallenberg AI, Autonomous Systems and Software Program (WASP) has supported the work of the second and last author.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Communicated by Teemu Pennanen
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Morin, M., Banert, S. & Giselsson, P. Nonlinear Forward-Backward Splitting with Momentum Correction. Set-Valued Var. Anal 31, 37 (2023). https://doi.org/10.1007/s11228-023-00700-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11228-023-00700-4
Keywords
- Monotone inclusions
- Nonlinear resolvent
- Momentum
- Forward-backward splitting
- Forward-reflected-backward splitting
- Primal-dual splitting
- Four-operator splitting