Skip to main content
SpringerLink
Log in

An extrapolated proximal iteratively reweighted method for nonconvex composite optimization problems

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

We consider a class of problems where the objective function is the sum of a smooth function and a composition of nonconvex and nonsmooth functions. Such optimization problems arise frequently in machine learning and data processing. The proximal iteratively reweighted method has been widely used and popularized in solving these problems. In this paper, we develop an extrapolated proximal iteratively reweighted method that incorporates two different flexible inertial steps at each iteration. We first prove the subsequential convergence of the proposed method under parameter constraints. Moreover, if the objective function satisfies the Kurdyka-Łojasiewicz property, the global convergence of the new method is established. In addition, we analyze the local convergence rate by making assumptions on the Kurdyka-Łojasiewicz exponent of the objective function. Finally, numerical results on \(l_p\) minimization and feature selection problems are reported to show the effectiveness and superiority of the proposed algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availibility

The authors confirm that all data generated or analysed during this study are included in this paper.

References

  1. Ahmad, R., Schniter, P.: Iteratively reweighted \(l_1\) approaches to sparse composite regularization. IEEE Trans. Comput. Imag. 1, 220–235 (2015)

    Google Scholar 

  2. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka-Łojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)

    MathSciNet  MATH  Google Scholar 

  3. Bauschke, H.H., Combettes, P.L.: Convex analysis and monotone operator theory in Hilbert spaces. Springer, Berlin (2011)

    MATH  Google Scholar 

  4. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci. 2, 183–202 (2009)

    MathSciNet  MATH  Google Scholar 

  5. Bian, W., Chen, X.: Neural network for nonsmooth, nonconvex constrained minimization via smooth approximation. IEEE Trans. Neural Netw. Learn. Syst. 25, 545–556 (2014)

    Google Scholar 

  6. Bolte, J., Daniilidis, A., Lewis, A.S., Shiota, M.: Clark subgradients of stratifiable functions. SIAM J. Optim. 18, 556–572 (2007)

    MathSciNet  MATH  Google Scholar 

  7. Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146, 459–494 (2014)

    MathSciNet  MATH  Google Scholar 

  8. Boţ, R.I., Csetnek, E.R., László, S.C.: An inertial forward-backward algorithm for the minimization of the sum of two nonconvex functions. EURO J. Comput. Optim. 4, 3–25 (2016)

    MathSciNet  MATH  Google Scholar 

  9. Chartrand, R., Yin, W.: Iteratively reweighted algorithms for compressive sensing. In: IEEE International Conference on Acoustics, Speech and Signal Processing (IEEE-ICASSP) (2008)

  10. Chen, X.: Smoothing methods for nonsmooth, nonconvex minimization. Math. Program. 134, 71–99 (2012)

    MathSciNet  MATH  Google Scholar 

  11. Chen, X., Ng, M.K., Zhang, C.: Non-Lipschitz-regularization and box constrained model for image restoration. IEEE Trans. Image Process. 21, 4709–4721 (2012)

    MathSciNet  MATH  Google Scholar 

  12. Chen, X., Ge, D., Wang, Z., Ye, Y.: Complexity of unconstrained \(l_2\)-\(l_p\) minimization. Math. Program. 143, 371–383 (2014)

    MathSciNet  MATH  Google Scholar 

  13. Chen, X., Zhou, W.: Convergence of the reweighted \(l_1\) minimization algorithm for \(l_2\)-\(l_p\) minimization. Comput. Optim. Appl. 59, 47–61 (2014)

    MathSciNet  Google Scholar 

  14. Clarke, F.H.: Nonsmooth analysis and optimization. In: Proceedings of the International Congress of Mathematicians (ICM) (1983)

  15. Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. Multiscale Model. Sim. 4, 1168–1200 (2005)

    MathSciNet  MATH  Google Scholar 

  16. Foucart, S., Lai, M.: Sparsest solutions of underdetermined linear systems via \(l_q\)-minimization for \(0< q\le 1\). Appl. Comput. Harmon A. 26, 395–407 (2009)

    MATH  Google Scholar 

  17. Gasso, G., Rakotomamonjy, A., Canu, S.: Recovering sparse signals with a certain family of nonconvex penalties and DC programming. IEEE Trans. Signal Process. 57, 4686–4698 (2009)

    MathSciNet  MATH  Google Scholar 

  18. Ge, Z., Wu, Z., Zhang, X., Ni, Q.: A fast proximal iteratively reweighted nuclear norm algorithm for nonconvex low-rank matrix minimization problems. Appl. Numer. Math. 179, 66–86 (2022)

    MathSciNet  MATH  Google Scholar 

  19. Ghadimi, E., Feyzmahdavian, H.R., Johansson, M.: Global convergence of the heavy-ball method for convex optimization. arXiv:1412.7457 (2014)

  20. Gong, P., Ye, J., Zhang, C.: Multi-stage multitask feature learning. J. Mach. Learn. Res. 14, 2979–3010 (2013)

    MathSciNet  MATH  Google Scholar 

  21. Guo, K., Han, D.: A note on the Douglas-Rachford splitting method for optimization problems involving hypoconvex functions. J. Global Optim. 72, 431–441 (2018)

    MathSciNet  MATH  Google Scholar 

  22. Horst, R., Thoai, N.V.: DC programming: overview. J. Optim. Theory Appl. 103, 1–43 (1999)

    MathSciNet  MATH  Google Scholar 

  23. Jacob, L., Obozinski, G., Vert, J.P.: Group LASSO with overlap and graph LASSO. In: International Conference on Machine Learning, 2009. ICML 2009. 433–440 (2009)

  24. Jiang, B., Liu, Y., Wen, Z.: \(L_p\)-norm regularization algorithms for optimization over permutation matrices. SIAM J. Optim. 26, 2284–2313 (2016)

    MathSciNet  MATH  Google Scholar 

  25. Kang, M.: Approximate versions of proximal iteratively reweighted algorithms including an extended IP-ICMM for signal and image processing problems. J. Comput. Appl. Math. 376, 112837 (2020)

    MathSciNet  MATH  Google Scholar 

  26. Lai, M., Wang, J.: An unconstrained \(l_q\) minimization with \(0<q\le 1\) for sparse solution of underdetermined linear systems. SIAM J. Optim. 21, 82–101 (2011)

    MathSciNet  MATH  Google Scholar 

  27. Lai, M., Xu, Y., Yin, W.: Improved iteratively reweighted least squares for unconstrained smoothed \(l_q\) minimization. SIAM J. Numer. Anal. 51, 927–957 (2013)

    MathSciNet  MATH  Google Scholar 

  28. Liu, Y., Ma, S., Dai, Y., Zhang, S.: A smoothing SQP framework for a class of composite Lq minimization over polyhedron. Math. Program. 158, 467–500 (2016)

    MathSciNet  MATH  Google Scholar 

  29. Lu, C., Wei, Y., Lin, Z., Yan, S.: Proximal iteratively reweighted algorithm with multiple splitting for nonconvex sparsity optimization. In: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence (AAAI-CAI) (2014)

  30. Lu, C., Tang, J., Yan, S., Lin, Z.: Nonconvex nonsmooth low rank minimization via iteratively reweighted nuclear norm. IEEE Trans. Image Process. 25, 829–839 (2015)

    MathSciNet  MATH  Google Scholar 

  31. Mohan, K., Fazel, M.: Iterative reweighted algorithms for matrix rank minimization. J. Mach. Learn. Res. 13, 3441–3473 (2012)

    MathSciNet  MATH  Google Scholar 

  32. Nesterov, Y.: Introductory lectures on convex optimization: a basic course. Kluwer Academic Publishers, Boston (2004)

    MATH  Google Scholar 

  33. Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140, 125–161 (2007)

    MathSciNet  MATH  Google Scholar 

  34. Ochs, P., Chen, Y., Brox, T., Pock, T.: iPiano: Inertial proximal algorithm for nonconvex optimization. SIAM J. Imag. Sci. 7, 1388–1419 (2014)

    MathSciNet  MATH  Google Scholar 

  35. Ochs, P., Dosovitskiy, A., Brox, T., Pock, T.: On iteratively reweighted algorithms for nonsmooth nonconvex optimization in computer vision. SIAM J. Imag. Sci. 8, 331–372 (2015)

    MathSciNet  MATH  Google Scholar 

  36. Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4, 1–17 (1964)

    Google Scholar 

  37. Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (2015)

    Google Scholar 

  38. Sun, T., Jiang, H., Cheng, L.: Global convergence of proximal iteratively reweighted algorithm. J. Global Optim. 68, 815–826 (2017)

    MathSciNet  MATH  Google Scholar 

  39. Sun, T., Jiang, H., Cheng, L., Zhu, W.: Iteratively linearized reweighted alternating direction method of multipliers for a class of nonconvex problems. IEEE Trans. Signal Process. 66, 5380–5391 (2018)

    MathSciNet  MATH  Google Scholar 

  40. Tao, P.D., Le Thi Hoai, A.: Solving a class of linearly constrained indefinite quadratic problems by DC algorithms. J. Global Optim. 11, 253–285 (1997)

    MathSciNet  MATH  Google Scholar 

  41. Tao, P.D., Le Thi Hoai, A.: A DC optimization algorithm for solving the trust-region subproblem. SIAM J. Optim. 8, 476–505 (1998)

    MathSciNet  MATH  Google Scholar 

  42. Wen, B., Chen, X., Pong, T.K.: Linear convergence of proximal gradient algorithm with extrapolation for a class of nonconvex nonsmooth minimization problems. SIAM J. Optim. 27, 124–145 (2017)

    MathSciNet  MATH  Google Scholar 

  43. Wen, B., Xue, X.: On the convergence of the iterates of proximal gradient algorithm with extrapolation for convex nonsmooth minimization problems. J. Global Optim. 75, 767–787 (2019)

    MathSciNet  MATH  Google Scholar 

  44. Wen, Z., Yin, W., Goldfarb, D., Zhang, Y.: A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation. SIAM J. Sci. Comput. 32, 1832–1857 (2009)

    MathSciNet  MATH  Google Scholar 

  45. Weston, J., Elisseeff, A., Schölkopf, B., Kaelbling, P.: The use of zero-norm with linear models and kernel methods. J. Mach. Learn. Res. 3, 1439–1461 (2003)

    MathSciNet  MATH  Google Scholar 

  46. Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Ma, Y.: Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31, 210–227 (2009)

    Google Scholar 

  47. Wu, Z., Li, M.: General inertial proximal gradient method for a class of nonconvex nonsmooth optimization problems. Comput. Optim. Appl. 73, 129–158 (2019)

    MathSciNet  MATH  Google Scholar 

  48. Wu, Z., Li, C., Li, M., Lim, A.: Inertial proximal gradient methods with Bregman regularization for a class of nonconvex optimization problems. J. Global Optim. 79, 1–28 (2021)

    MathSciNet  MATH  Google Scholar 

  49. Xu, Y., Xu, Y., Yan, Y., Chen, J.: Distributed stochastic inertial-accelerated methods with delayed derivatives for nonconvex problems. arXiv:2107.11513 (2021)

  50. Yang, L.: Proximal gradient method with extrapolation and line search for a class of nonconvex and nonsmooth problems. arXiv:1711.06831 (2018)

  51. Zhang, T.: Analysis of multi-stage convex relaxation for sparse regularization. J. Mach. Learn. Res. 11, 1081–1107 (2010)

    MathSciNet  MATH  Google Scholar 

  52. Zhang, T.: Multi-stage convex relaxation for feature selection. Bernoulli 19, 2277–2293 (2013)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant Nos. 12001281, 12001286, and 11771210), the Project funded by China Postdoctoral Science Foundation (No. 2022M711672), Suqian Sci &Tech Program (Grant Nos. Z2020135 and K202112), the Startup Foundation for Introducing Talent of NUIST (No. 2020r003), and Qing Lan Project.

Author information

Authors and Affiliations

  1. School of Mathematics and Information Science, Nanjing Normal University of Special Education, Nanjing, 210038, China

    Zhili Ge

  2. School of Management Science and Engineering, Nanjing University of Information Science and Technology, Nanjing, 210044, China

    Zhongming Wu

  3. School of Sciences and Arts, Suqian University, Suqian, 223800, China

    Xin Zhang

  4. School of Mathematics, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Qin Ni

Authors
  1. Zhili Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhongming Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qin Ni
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ge, Z., Wu, Z., Zhang, X. et al. An extrapolated proximal iteratively reweighted method for nonconvex composite optimization problems. J Glob Optim 86, 821–844 (2023). https://doi.org/10.1007/s10898-023-01299-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-023-01299-4

Keywords

Search

Navigation