Skip to main content
SpringerLink
Log in

A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problems

  • Original Paper
  • Published:
Numerical Algorithms Aims and scope Submit manuscript

Abstract

Extrapolation, restart and stepsize are very powerful strategies for accelerating the convergence rates of first-order algorithms. In this paper, we propose a modified accelerated proximal gradient algorithm (modAPG), which incorporates the adaptive nonmonotone stepsize strategy, extrapolation and a modified rule of setting the extrapolation parameter to zero at some iterations, for solving a minimization problem composed of a smooth (possibly nonconvex) with Lipschitz continuous gradient and nonsmooth (possibly nonconvex) function. Moreover, the proposed algorithm has a simpler form for this problem with convex setting. We show that any cluster point of the sequence generated by modAPG is a critical point of the problem, and analyze the convergence rates of function values and iterates under the assumption that objective function has the Kurdyka-Łojasiewicz property. Finally, we conduct some preliminary numerical experiments for solving the convex problems, such as image deblurring problem and sparse logistic regression problem; and the nonconvex regularization problem, such as the \({L_{\frac{1}{2}}}\) penalty and the smoothly clipped absolute deviation (SCAD) penalty problem. Numerical experiments demonstrate the promising performance of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

References

  1. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka-Łojasiewicz inequality. Math. Oper. Res. 35(2), 438–457 (2010)

    Article  MathSciNet  Google Scholar 

  2. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the Kurdyka-Łojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)

    Article  MathSciNet  Google Scholar 

  3. Attouch, H., Bolte, J., Svaier, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Math. Program. 137, 91–129 (2013)

    Article  MathSciNet  Google Scholar 

  4. Attouch, H., Peypouquet, J.: The rate of convergence of Nesterov’s accelerated forward_backward method is actually faster than \( {\frac{1}{{{k^2}}}} \). SIAM J. Optim. 26, 1824–1834 (2016)

    Article  MathSciNet  Google Scholar 

  5. Attouch, H., Cabot, A.: Convergence rates of inertial forward-backward algorithms. SIAM J. Optim. 28, 849–874 (2018)

    Article  MathSciNet  Google Scholar 

  6. Ahookhosh, M., Themelis, A., Patrinos, P.: A Bregman forward-backward linesearch algorithm for nonconvex composite optimization: superlinear convergence to nonisolated local minima (2019). arXiv preprint arXiv:1905.11904

  7. Apidopoulos, V., Aujol, J., Dossal, C.: Convergence rate of inertial Forward-Backward algorithm beyond Nesterov’s rule. Math. Program. 180, 137–156 (2020)

    Article  MathSciNet  Google Scholar 

  8. Apidopoulos, V., Aujol, J., Dossal, C., et al.: Convergence rates of an inertial gradient descent algorithm under growth and flatness conditions. Math. Program. (2020). https://doi.org/10.1007/s10107-020-01476-3

    Article  Google Scholar 

  9. Bertsekas, D.P., Tsitsiklis, J.N.: Parallel and Distributed Computation: Numerical Methods. PrenticeHall, New Jersey (1989)

    Google Scholar 

  10. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)

    Article  MathSciNet  Google Scholar 

  11. Beck, A., Teboulle, M.: Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems.in IEEE Transactions on Image Processing. 18, 2419-2434 (2009)

  12. Bolte, J., Sabach, S., Teboulle. M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146(1-2), 459-494 (2014)

  13. Chambolle, A., Dossal, C.: On the convergence of the iterates of the fast iterative shrinkage-thresholding algorithm J. Optim. Theory Appl. 166, 968–982 (2015)

  14. Chambolle, A., Pock, T.: An introduction to continuous optimization for imaging. Acta Numerica. 25, 161–319 (2015)

    Article  MathSciNet  Google Scholar 

  15. Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM. Trans. Intell. Syst. Technol. 2, 1–27 (2011)

    Article  Google Scholar 

  16. Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)

    Article  MathSciNet  Google Scholar 

  17. Donghwan, K., Jeffrey, A.F.: Another look at the fast iterative shrinkage thresholding algorithm (FISTA). SIAM J. Optim. 28, 223–250 (2018)

    Article  MathSciNet  Google Scholar 

  18. Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96, 1348–1360 (2001)

    Article  MathSciNet  Google Scholar 

  19. Fercoq, O., Qu, Z.: Restarting accelerated gradient methods with a rough strong convexity estimate (2016). arXiv preprint arXiv:1609.07358

  20. Fercoq, O., Qu, Z.: Adaptive restart of accelerated gradient methods under local quadratic growth condition. IMA J. Numer Anal. 39, 2069–2095 (2019)

    Article  MathSciNet  Google Scholar 

  21. Ghayem, F., Sadeghi, M., Babaie-Zadeh, M., Chatterjee, S., Skoglund, M., Jutten, C.: Sparse signal recovery using iterative proximal projection. IEEE Trans. Signal Process. 66, 879–894 (2018)

    Article  MathSciNet  Google Scholar 

  22. Hien, L.T.K., Gillis, N., Patrinos, P.: Inertial block mirror descent method for non-convex non-smooth optimization (2019). arXiv preprint arXiv:1903.01818

  23. Johnstone, P.R., Moulin, P.: Local and global convergence of a general inertial proximal splitting scheme for minimizing composite functions. Comput. Optim. Appl. 67, 259–292 (2017)

    Article  MathSciNet  Google Scholar 

  24. Liu, H.W., Wang, T., Liu, Z.X.: Convergence rate of inertial forward-backward algorithms based on the local error bound condition. http://arxiv.org/pdf/2007.07432

  25. Liu, H.W., Wang, T., Liu, Z.X.: Some modified fast iteration shrinkage thresholding algorithms with a new adaptive non-monotone stepsize strategy for nonsmooth and convex minimization problems. Optimization online. http://www.optimization-online.org/DB_HTML/2020/12/8169.html

  26. Lin, Q., Xiao, L.: An adaptive accelerated proximal gradient method and its homotopy continuation for sparse optimization. International Conference on Machine Learning. PMLR, 73–81 (2014)

  27. Liang, J, Schönlieb, C.B.: Improving FISTA: Faster, smarter and greedier (2018). arXiv preprint arXiv:1811.01430

  28. Li, H., Lin, Z.: Accelerated proximal gradient methods for nonconvex programming. In: Proceedings of NeurIPS, 379-387 (2015)

  29. Mukkamala, M.C., Ochs, P., Pock, T., et al.: Convex-concave backtracking for inertial Bregman proximal gradient algorithms in nonconvex optimization. SIAM J. Math. Data Sci. 2, 658–682 (2020)

    Article  MathSciNet  Google Scholar 

  30. Nesterov, Y.: A method for solving the convex programming problem with convergence rate \(O\left( {\frac{1}{{{k^2}}}} \right)\). Dokl. Akad. Nauk SSSR. 269, 543–547 (1983)

    MathSciNet  Google Scholar 

  31. Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140, 125–161 (2013)

    Article  MathSciNet  Google Scholar 

  32. Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course. Academic Publishers, Boston (2004)

    Book  Google Scholar 

  33. Ortega, J.M., Rheinboldt, W.C.: Iterative Solution of Nonlinear Equations in Several Variables. Academic Press, New-York (1970)

    Google Scholar 

  34. Ochs, P., Chen, Y., Brox, T., Pock, T., et al.: iPiano: Inertial proximal algorithm for nonconvex optimization. SIAM J. Imaging Sci. 7(2), 1388–1419 (2014)

    Article  MathSciNet  Google Scholar 

  35. O’Donoghue, B., Candès, E.: Adaptive restart for accelerated gradient schemes. Found Comput Math. 15, 715–732 (2015)

    Article  MathSciNet  Google Scholar 

  36. Pock, T., Sabach, S.: Inertial proximal alternating linearized minimization (iPALM) for nonconvex and nonsmooth problems. SIAM J. Imaging Sci. 9, 1756–1787 (2016)

    Article  MathSciNet  Google Scholar 

  37. Parikh, N., Boyd, S.: Proximal algorithms. Found. Trends Optim. 1, 127–239 (2014)

    Article  Google Scholar 

  38. Roulet, V., d’Aspremont, A.: Sharpness, restart, and acceleration. SIAM J. Optim. 30, 262–289 (2020)

    Article  MathSciNet  Google Scholar 

  39. Su, W., Boyd, S., Candes, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights. J. Mach. Learn. Res. 17, 1–43 (2016)

    MathSciNet  Google Scholar 

  40. Wen, B., Chen, X.J., Pong, T.K.: Linear convergence of proximal gradient algorithm with extrapolation for a class of nonconvex nonsmooth minimization problems. SIAM J. Optim. 27, 124–145 (2017)

    Article  MathSciNet  Google Scholar 

  41. Wu, Z.M., Li, C.S., Li, M., Lim, A.: Inertial proximal gradient methods with Bregman regularization for a class of nonconvex optimization problems. J Global Optim. https://doi.org/10.1007/s10898-020-00943-7

  42. Wu, Z.M., Li, M.: General inertial proximal gradient method for a class of onconvex nonsmooth optimizaiton problems. Comput. Optim. Appl. 73, 129–158 (2019)

    Article  MathSciNet  Google Scholar 

  43. Xu, Z., Chang, X.Y., Xu, F.M., Zhang, H.: L1/2 Regularization: A Thresholding Representation Theory and a Fast Solver. IEEE Trans. Neural Netw. Learn. Syst. 23(7), 1013–1027 (2012)

    Article  Google Scholar 

  44. Yang, L.: Proximal gradient method with extrapolation and line search for a class of nonconvex and nonsmooth problems (2017). arXiv preprint arXiv:1711.06831

  45. Yang, L., Pong, T.K., Chen, X.: A non-monotone alternating updating method for a class of matrix factorization problems. SIAM J. Optim 28, 3402–3430 (2018)

    Article  MathSciNet  Google Scholar 

  46. Zhang, H., Hager, W.W.: A nonmonotone line search technique and its application to unconstrained optimization. SIAM J. Optim. 14, 1043–1056 (2004)

    Article  MathSciNet  Google Scholar 

  47. Zeng, L.M., Xie, J.: Group variable selection via SCAD-l2. Statistics. 48, 49–66 (2014)

    Article  MathSciNet  Google Scholar 

  48. Rockafellar, R.T., Wets, R.J-B.: Variational Analysis. Springer, 1998

  49. Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Mathematical Programming. 137(1), 91–129 (2013)

    Article  MathSciNet  Google Scholar 

  50. Frankel, P., Garrigos, G., Peypouquet, J.: Splitting methods with variable metric for Kurdyka-Łojasiewicz functions and general convergence rates. Journal of Optimization Theory and Applications. 165(3), 874–900 (2015)

    Article  MathSciNet  Google Scholar 

Download references

Funding

The work was supported by the National Science Foundation of China (No. 12261019).

Author information

Authors and Affiliations

  1. School of Science, Xi’an University of Posts and Telecommunications, Xi’an, 710121, China

    Ting Wang

  2. School of Mathematics and Statistics, Xidian University, Xi’an, 710126, China

    Hongwei Liu

Authors
  1. Ting Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongwei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongwei Liu.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, T., Liu, H. A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problems. Numer Algor 95, 207–241 (2024). https://doi.org/10.1007/s11075-023-01569-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11075-023-01569-y

Keywords

Search

Navigation