Abstract
Extrapolation, restart and stepsize are very powerful strategies for accelerating the convergence rates of first-order algorithms. In this paper, we propose a modified accelerated proximal gradient algorithm (modAPG), which incorporates the adaptive nonmonotone stepsize strategy, extrapolation and a modified rule of setting the extrapolation parameter to zero at some iterations, for solving a minimization problem composed of a smooth (possibly nonconvex) with Lipschitz continuous gradient and nonsmooth (possibly nonconvex) function. Moreover, the proposed algorithm has a simpler form for this problem with convex setting. We show that any cluster point of the sequence generated by modAPG is a critical point of the problem, and analyze the convergence rates of function values and iterates under the assumption that objective function has the Kurdyka-Łojasiewicz property. Finally, we conduct some preliminary numerical experiments for solving the convex problems, such as image deblurring problem and sparse logistic regression problem; and the nonconvex regularization problem, such as the \({L_{\frac{1}{2}}}\) penalty and the smoothly clipped absolute deviation (SCAD) penalty problem. Numerical experiments demonstrate the promising performance of the proposed method.
Similar content being viewed by others
Data availability
Data sharing is not applicable to this article as no new data were created or analyzed in this study.
References
Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka-Łojasiewicz inequality. Math. Oper. Res. 35(2), 438–457 (2010)
Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the Kurdyka-Łojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)
Attouch, H., Bolte, J., Svaier, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Math. Program. 137, 91–129 (2013)
Attouch, H., Peypouquet, J.: The rate of convergence of Nesterov’s accelerated forward_backward method is actually faster than \( {\frac{1}{{{k^2}}}} \). SIAM J. Optim. 26, 1824–1834 (2016)
Attouch, H., Cabot, A.: Convergence rates of inertial forward-backward algorithms. SIAM J. Optim. 28, 849–874 (2018)
Ahookhosh, M., Themelis, A., Patrinos, P.: A Bregman forward-backward linesearch algorithm for nonconvex composite optimization: superlinear convergence to nonisolated local minima (2019). arXiv preprint arXiv:1905.11904
Apidopoulos, V., Aujol, J., Dossal, C.: Convergence rate of inertial Forward-Backward algorithm beyond Nesterov’s rule. Math. Program. 180, 137–156 (2020)
Apidopoulos, V., Aujol, J., Dossal, C., et al.: Convergence rates of an inertial gradient descent algorithm under growth and flatness conditions. Math. Program. (2020). https://doi.org/10.1007/s10107-020-01476-3
Bertsekas, D.P., Tsitsiklis, J.N.: Parallel and Distributed Computation: Numerical Methods. PrenticeHall, New Jersey (1989)
Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)
Beck, A., Teboulle, M.: Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems.in IEEE Transactions on Image Processing. 18, 2419-2434 (2009)
Bolte, J., Sabach, S., Teboulle. M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146(1-2), 459-494 (2014)
Chambolle, A., Dossal, C.: On the convergence of the iterates of the fast iterative shrinkage-thresholding algorithm J. Optim. Theory Appl. 166, 968–982 (2015)
Chambolle, A., Pock, T.: An introduction to continuous optimization for imaging. Acta Numerica. 25, 161–319 (2015)
Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM. Trans. Intell. Syst. Technol. 2, 1–27 (2011)
Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)
Donghwan, K., Jeffrey, A.F.: Another look at the fast iterative shrinkage thresholding algorithm (FISTA). SIAM J. Optim. 28, 223–250 (2018)
Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96, 1348–1360 (2001)
Fercoq, O., Qu, Z.: Restarting accelerated gradient methods with a rough strong convexity estimate (2016). arXiv preprint arXiv:1609.07358
Fercoq, O., Qu, Z.: Adaptive restart of accelerated gradient methods under local quadratic growth condition. IMA J. Numer Anal. 39, 2069–2095 (2019)
Ghayem, F., Sadeghi, M., Babaie-Zadeh, M., Chatterjee, S., Skoglund, M., Jutten, C.: Sparse signal recovery using iterative proximal projection. IEEE Trans. Signal Process. 66, 879–894 (2018)
Hien, L.T.K., Gillis, N., Patrinos, P.: Inertial block mirror descent method for non-convex non-smooth optimization (2019). arXiv preprint arXiv:1903.01818
Johnstone, P.R., Moulin, P.: Local and global convergence of a general inertial proximal splitting scheme for minimizing composite functions. Comput. Optim. Appl. 67, 259–292 (2017)
Liu, H.W., Wang, T., Liu, Z.X.: Convergence rate of inertial forward-backward algorithms based on the local error bound condition. http://arxiv.org/pdf/2007.07432
Liu, H.W., Wang, T., Liu, Z.X.: Some modified fast iteration shrinkage thresholding algorithms with a new adaptive non-monotone stepsize strategy for nonsmooth and convex minimization problems. Optimization online. http://www.optimization-online.org/DB_HTML/2020/12/8169.html
Lin, Q., Xiao, L.: An adaptive accelerated proximal gradient method and its homotopy continuation for sparse optimization. International Conference on Machine Learning. PMLR, 73–81 (2014)
Liang, J, Schönlieb, C.B.: Improving FISTA: Faster, smarter and greedier (2018). arXiv preprint arXiv:1811.01430
Li, H., Lin, Z.: Accelerated proximal gradient methods for nonconvex programming. In: Proceedings of NeurIPS, 379-387 (2015)
Mukkamala, M.C., Ochs, P., Pock, T., et al.: Convex-concave backtracking for inertial Bregman proximal gradient algorithms in nonconvex optimization. SIAM J. Math. Data Sci. 2, 658–682 (2020)
Nesterov, Y.: A method for solving the convex programming problem with convergence rate \(O\left( {\frac{1}{{{k^2}}}} \right)\). Dokl. Akad. Nauk SSSR. 269, 543–547 (1983)
Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140, 125–161 (2013)
Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course. Academic Publishers, Boston (2004)
Ortega, J.M., Rheinboldt, W.C.: Iterative Solution of Nonlinear Equations in Several Variables. Academic Press, New-York (1970)
Ochs, P., Chen, Y., Brox, T., Pock, T., et al.: iPiano: Inertial proximal algorithm for nonconvex optimization. SIAM J. Imaging Sci. 7(2), 1388–1419 (2014)
O’Donoghue, B., Candès, E.: Adaptive restart for accelerated gradient schemes. Found Comput Math. 15, 715–732 (2015)
Pock, T., Sabach, S.: Inertial proximal alternating linearized minimization (iPALM) for nonconvex and nonsmooth problems. SIAM J. Imaging Sci. 9, 1756–1787 (2016)
Parikh, N., Boyd, S.: Proximal algorithms. Found. Trends Optim. 1, 127–239 (2014)
Roulet, V., d’Aspremont, A.: Sharpness, restart, and acceleration. SIAM J. Optim. 30, 262–289 (2020)
Su, W., Boyd, S., Candes, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights. J. Mach. Learn. Res. 17, 1–43 (2016)
Wen, B., Chen, X.J., Pong, T.K.: Linear convergence of proximal gradient algorithm with extrapolation for a class of nonconvex nonsmooth minimization problems. SIAM J. Optim. 27, 124–145 (2017)
Wu, Z.M., Li, C.S., Li, M., Lim, A.: Inertial proximal gradient methods with Bregman regularization for a class of nonconvex optimization problems. J Global Optim. https://doi.org/10.1007/s10898-020-00943-7
Wu, Z.M., Li, M.: General inertial proximal gradient method for a class of onconvex nonsmooth optimizaiton problems. Comput. Optim. Appl. 73, 129–158 (2019)
Xu, Z., Chang, X.Y., Xu, F.M., Zhang, H.: L1/2 Regularization: A Thresholding Representation Theory and a Fast Solver. IEEE Trans. Neural Netw. Learn. Syst. 23(7), 1013–1027 (2012)
Yang, L.: Proximal gradient method with extrapolation and line search for a class of nonconvex and nonsmooth problems (2017). arXiv preprint arXiv:1711.06831
Yang, L., Pong, T.K., Chen, X.: A non-monotone alternating updating method for a class of matrix factorization problems. SIAM J. Optim 28, 3402–3430 (2018)
Zhang, H., Hager, W.W.: A nonmonotone line search technique and its application to unconstrained optimization. SIAM J. Optim. 14, 1043–1056 (2004)
Zeng, L.M., Xie, J.: Group variable selection via SCAD-l2. Statistics. 48, 49–66 (2014)
Rockafellar, R.T., Wets, R.J-B.: Variational Analysis. Springer, 1998
Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Mathematical Programming. 137(1), 91–129 (2013)
Frankel, P., Garrigos, G., Peypouquet, J.: Splitting methods with variable metric for Kurdyka-Łojasiewicz functions and general convergence rates. Journal of Optimization Theory and Applications. 165(3), 874–900 (2015)
Funding
The work was supported by the National Science Foundation of China (No. 12261019).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, T., Liu, H. A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problems. Numer Algor 95, 207–241 (2024). https://doi.org/10.1007/s11075-023-01569-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11075-023-01569-y