Abstract
Primal-dual hybrid gradient (PDHG) method is a canonical and popular prototype for solving saddle point problem (SPP). However, the nonlinear coupling term in SPP excludes the application of PDHG on far-reaching real-world problems. In this paper, following the seminal work by Valkonen (Inverse Problems 30, 2014), we devise a variant iterative scheme for solving SPP with nonlinear function by exerting an alternative extrapolation procedure. The novel iterative scheme falls exactly into the proximal point algorithmic framework without any residuals, which indicates that the associated inclusion problem is “nearer” to the KKT mapping induced by SPP. Under the metrically regular assumption on KKT mapping, we simplify the local convergence of the proposed method on contractive perspective. Numerical simulations on a PDE-constrained nonlinear inverse problem demonstrate the compelling performance of the proposed method.
Similar content being viewed by others
Data availability statement
All data generated or analyzed during this study are available from the corresponding author.
Notes
The critical point of SPP varies in different literature. We herein follow the definition in [13].
Codes of accelerated-NL-PDHGM are available at https://github.com/clason/nlpdegm.
References
Artacho, F.J.A., Dontchev, A.L., Gaydu, M., Geoffroy, M.H., Veliov, V.M.: Metric regularity of Newton’s iteration. SIAM J. Control Optim. 49, 339–362 (2011)
Başar, T., Olsder, G.J.: Dynamic Noncooperative Game Theory. SIAM (1998)
Ben-Tal, A., El Ghaoui, L., Nemirovski, A.: Robust Optimization. Princeton University Press, Princeton (2009)
Benzi, M., Golub, G.H., Liesen, J.: Numerical solution of saddle point problems. Acta Numer. 14, 1–137 (2005)
Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146, 459–494 (2014)
Cai, X., Han, D., Xu, L.: An improved first-order primal-dual algorithm with a new correction step. J. Global Optim. 57, 1419–1428 (2013)
Chambolle, A., Ehrhardt, M.J., Richtárik, P., Schönlieb, C.B.: Stochastic primal-dual hybrid gradient algorithm with arbitrary sampling and imaging applications. SIAM J. Optim. 28, 2783–2808 (2018)
Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vision 40, 120–145 (2011)
Chambolle, A., Pock, T.: On the ergodic convergence rates of a first-order primal-dual algorithm. Math. Program. 159, 253–287 (2016)
Chen, Y., Lan, G., Ouyang, Y.: Optimal primal-dual methods for a class of saddle point problems. SIAM J. Optim. 24, 1779–1814 (2014)
Chung, J., Nagy, J.G.: An efficient iterative approach for large-scale separable nonlinear inverse problems. SIAM J. Sci. Comput. 31, 4654–4674 (2010)
Clason, C., Jin, B.: A semismooth Newton method for nonlinear parameter identification problems with impulsive noise. SIAM J. Imaging Sci. 5, 505–536 (2012)
Clason, C., Mazurenko, S., Valkonen, T.: Acceleration and global convergence of a first-order primal-dual method for nonconvex problems. SIAM J. Optim. 29, 933–963 (2019)
Clason, C., Mazurenko, S., Valkonen, T.: Primal-dual proximal splitting and generalized conjugation in non-smooth non-convex optimization. Appl. Math. Optim. 84, 1239–1284 (2020)
Clason, C., Valkonen, T.: Primal-dual extragradient methods for nonlinear nonsmooth PDE-constrained optimization. SIAM J. Optim. 27, 1314–1339 (2017)
Clason, C., Valkonen, T.: Stability of saddle points via explicit coderivatives of pointwise subdifferentials. Set-Valued Var. Anal. 25, 69–112 (2017)
Condat, L.: A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms. J. Optim. Theory Appl. 158, 460–479 (2013)
Esser, E., Zhang, X., Chan, T.F.: A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM J. Imaging Sci. 3, 1015–1046 (2010)
Fercoq, O., Bianchi, P.: A coordinate-descent primal-dual algorithm with large step size and possibly nonseparable functions. SIAM J. Optim. 29, 100–134 (2019)
Fusek, P.: On metric regularity for weakly almost piecewise smooth functions and some applications in nonlinear semidefinite programming. SIAM J. Optim. 23, 1041–1061 (2013)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Proc. Adv. Neural Inf. Process. Syst. pp 2672–2680 (2014)
Gu, G., He, B., Yuan, X.: Customized proximal point algorithms for linearly constrained convex minimization and saddle-point problems: a unified approach. Comput. Optim. Appl. 59, 135–161 (2014)
Hart, P.E., Stork, D.G., Duda, R.O.: Pattern Classification. Wiley Hoboken, Hoboken (2000)
He, B., Ma, F., Yuan, X.: An algorithmic framework of generalized primal-dual hybrid gradient methods for saddle point problems. J. Math. Imaging Vision 58, 279–293 (2017)
He, B., You, Y., Yuan, X.: On the convergence of primal-dual hybrid gradient algorithm. SIAM J. Imaging Sci. 7, 2526–2537 (2014)
He, B., Yuan, X.: Convergence analysis of primal-dual algorithms for a saddle-point problem: from contraction perspective. SIAM J. Imaging Sci. 5, 119–149 (2012)
He, B., Yuan, X.: Balanced augmented Lagrangian method for convex programming. arXiv:2108.08554 (2021)
He, B., Yuan, X., Zhang, W.: A customized proximal point algorithm for convex minimization with linear constraints. Comput. Optim. Appl. 56, 559–572 (2013)
Hiriart-Urruty, J.B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms. Springer, Berlin (1993)
Jiang, F., Cai, X., Wu, Z., Han, D.: Approximate first-order primal-dual algorithms for saddle point problems. Math. Comput. 90, 1227–1262 (2021)
Keung, Y.L., Zou, J.: An efficient linear solver for nonlinear parameter identification problems. SIAM J. Sci. Comput. 22, 1511–1526 (2001)
Li, Z., Yan, M.: New convergence analysis of a primal-dual algorithm with large stepsizes. Adv. Comput. Math. 47, 1–20 (2021)
Liu, Y., Xu, Y., Yin, W.: Acceleration of primal-dual methods by preconditioning and simple subproblem procedures. J. Sci. Comput. 86, 1–34 (2021)
Malitsky, Y., Pock, T.: A first-order primal-dual algorithm with linesearch. SIAM J. Optim. 28, 411–432 (2018)
Mazurenko, S., Jauhiainen, J., Valkonen, T.: Primal-dual block-proximal splitting for a class of non-convex problems. Electron. Trans. Numer. Anal. 52, 509–552 (2020)
Rasch, J., Chambolle, A.: Inexact first-order primal-dual algorithms. Comput. Optim. Appl. 76, 381–430 (2020)
Robbins, H., Siegmund, D.: A convergence theorem for nonnegative almost supermartingales and some applications. Optimizing Methods in Statistics. In: Proc. Sympos., Ohio State Univ., Ohio, pp. 233–257 (1971)
Rockafellar, R.T., Wets, R.J.: Variational Analysis. Springer, Berlin (1998)
Uzawa, H.: Iterative methods for concave programming. Stud. Linear Nonlinear Program. (Arrow et al, eds) 6, 154–165 (1958)
Valkonen, T.: A primal-dual hybrid gradient method for nonlinear operators with applications to MRI. Inverse Problems 30, 45 (2014)
Valkonen, T.: Block-proximal methods with spatially adapted acceleration. Electron. Trans. Numer. Anal. 51, 15–49 (2019)
Valkonen, T.: First-order primal–dual methods for nonsmooth non-convex optimisation. In: Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging: Mathematical Imaging and Vision (Chen et al, eds) pp. 1–42 (2021)
Valkonen, T., Pock, T.: Acceleration of the PDHGM on partially strongly convex functions. J. Math. Imaging Vision 59, 394–414 (2017)
Vũ, B.C.: A splitting algorithm for dual monotone inclusions involving cocoercive operators. Adv. Comput. Math. 38, 667–681 (2013)
Acknowledgements
The authors would like to thank the anonymous referees for their valuable comments, which helped improving the manuscript substantially. This work is support by NSFC grant (11971003), and partially by Science and Technology Department of Sichuan Province (2021YFG0125) and ZYGX2019J090.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Gao, Y., Zhang, W. An alternative extrapolation scheme of PDHGM for saddle point problem with nonlinear function. Comput Optim Appl 85, 263–291 (2023). https://doi.org/10.1007/s10589-023-00453-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10589-023-00453-8