Skip to main content
SpringerLink
Log in

Convergence rate analysis of proximal iteratively reweighted \(\ell _1\) methods for \(\ell _p\) regularization problems

  • Original Paper
  • Published:
Optimization Letters Aims and scope Submit manuscript

Abstract

In this paper, we focus on the local convergence rate analysis of the proximal iteratively reweighted \(\ell _1\) algorithms for solving \(\ell _p\) regularization problems, which are widely applied for inducing sparse solutions. We show that if the Kurdyka–Łojasiewicz property is satisfied, the algorithm converges to a unique first-order stationary point; furthermore, the algorithm has local linear convergence or local sublinear convergence. The theoretical results we derived are much stronger than the existing results for iteratively reweighted \(\ell _1\) algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Attouch, H., Bolte, J.: On the convergence of the proximal algorithm for nonsmooth functions involving analytic features. Math. Program. 116(1), 5–16 (2009). https://doi.org/10.1007/s10107-007-0133-5

    Article  MathSciNet  MATH  Google Scholar 

  2. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the kurdyka-łojasiewicz inequality. Math. Oper. Res. 35(2), 438–457 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  3. Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized gauss-seidel methods. Math. Program. 137(1–2), 91–129 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146(1), 459–494 (2014). https://doi.org/10.1007/s10107-013-0701-9

    Article  MathSciNet  MATH  Google Scholar 

  5. Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146(1), 459–494 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  6. Bolte, J., Daniilidis, A., Lewis, A.: The łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim. 17(4), 1205–1223 (2007). https://doi.org/10.1137/050644641

    Article  MATH  Google Scholar 

  7. Candes, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(\ell _1\) minimization. J. Fourier Anal. Appl. 14(5–6), 877–905 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  8. Chen, X., Niu, L., Yuan, Y.X.: Optimality conditions and a smoothing trust region newton method for nonlipschitz optimization. SIAM J. Optim. 23, 1528–1552 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  9. Chen, X., Zhou, W.: Convergence of reweighted \(\ell _1\) minimization algorithms and unique solution of truncated lp minimization. The Hong Kong Polytechnic University, Department of Applied Mathematics (2010)

  10. Chun, H., Keleş, S.: Sparse partial least squares regression for simultaneous dimension reduction and variable selection. J. R. Stat. Soc. Ser. B (Statistical Methodology) 72(1), 3–25 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  11. Figueiredo, M.A., Bioucas-Dias, J.M., Nowak, R.D.: Majorization-minimization algorithms for wavelet-based image restoration. IEEE Trans. Image Process. 16(12), 2980–2991 (2007)

    Article  MathSciNet  Google Scholar 

  12. Ge, D., Jiang, X., Ye, Y.: A note on the complexity of \(\ell _p\) minimization. Math. Program. 129(2), 285–299 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  13. Hu, Y., Li, C., Meng, K., Yang, X.: Linear convergence of inexact descent method and inexact proximal gradient algorithms for lower-order regularization problems. J. Global Optim. 79(4), 853–883 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  14. Lai, M.J., Wang, J.: An unconstrained \( \ell _q \) minimization with \(0<q\le 1\) for sparse solution of underdetermined linear systems. SIAM J. Optim. 21(1), 82–101 (2011)

    Article  MathSciNet  Google Scholar 

  15. Li, G., Pong, T.K.: Calculus of the exponent of kurdyka-łojasiewicz inequality and its applications to linear convergence of first-order methods. Found. Comput. Math. 18(5), 1199–1232 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  16. Li, Q., Zhou, Y., Liang, Y., Varshney, P.K.: Convergence analysis of proximal gradient with momentum for nonconvex optimization. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2111–2119. JMLR. org, Sydney, Australia (2017)

  17. Lu, C., Wei, Y., Lin, Z., Yan, S.: Proximal iteratively reweighted algorithm with multiple splitting for nonconvex sparsity optimization. In: Twenty-Eighth AAAI Conference on Artificial Intelligence. AAAI Press, Québec, Canada (2014)

  18. Lu, Z.: Iterative reweighted minimization methods for \(\ell _p\) regularized unconstrained nonlinear programming. Math. Program. 147(1–2), 277–307 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  19. Luo, Z.Q., Pang, J.S., Ralph, D.: Mathematical Programs with Equilibrium Constraints. Cambridge University Press, Cambridge (1996)

    Book  MATH  Google Scholar 

  20. Portilla, J.: Image restoration through l0 analysis-based sparse optimization in tight frames. In: 2009 16th IEEE International Conference on Image Processing (ICIP), pp. 3909–3912. IEEE, IEEE Press, Cairo, Egypt (2009)

  21. Scardapane, S., Comminiello, D., Hussain, A., Uncini, A.: Group sparse regularization for deep neural networks. Neurocomputing 241, 81–89 (2017)

    Article  Google Scholar 

  22. Sun, T., Jiang, H., Cheng, L.: Global convergence of proximal iteratively reweighted algorithm. J. Global Optim. 68(4), 815–826 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  23. Wang, F.: Study on the kurdyka–łojasiewicz exponents of \(\ell _p\) regularization functions (in chinese). Master thesis, Southwestern University of Finance and Economics (2021)

  24. Wang, H., Zeng, H., Wang, J.: Relating lp regularization and reweighted l1 regularization. Optim. Lett. 15, 2639–2660 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  25. Wang, H., Zhang, F., Wu, Q., Hu, Y., Shi, Y.: Nonconvex and nonsmooth sparse optimization via adaptively iterative reweighted methods. arXiv preprint arXiv:1810.10167 (2018)

  26. Yin, P., Lou, Y., He, Q., Xin, J.: Minimization of 1–2 for compressed sensing. SIAM J. Sci. Comput. 37(1), A536–A563 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  27. Yu, P., Li, G., Pong, T.K.: Kurdyka-łojasiewicz exponent via inf-projection. Found. Comput. Math. pp. 1–47 (2021)

  28. Zhou, Y., Yu, Y., Dai, W., Liang, Y., Xing, E.: On convergence of model parallel proximal gradient algorithm for stale synchronous parallel system. In: Artificial Intelligence and Statistics, pp. 713–722. JMLR, Cadiz, Spain (2016)

Download references

Acknowledgements

Hao Wang was supported by the National Natural Science Foundation of China under Grant 12001367 and the Natural Science Foundation of Shanghai under Grant 21ZR1442800.

Author information

Authors and Affiliations

  1. School of Information Science and Technology, ShanghaiTech University, Shanghai, China

    Hao Wang & Hao Zeng

  2. Department of Mathematics, University of Washington, Seattle, USA

    Jiashan Wang

Authors
  1. Hao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hao Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiashan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hao Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, H., Zeng, H. & Wang, J. Convergence rate analysis of proximal iteratively reweighted \(\ell _1\) methods for \(\ell _p\) regularization problems. Optim Lett 17, 413–435 (2023). https://doi.org/10.1007/s11590-022-01907-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11590-022-01907-4

Keywords

Search

Navigation