Skip to main content
SpringerLink
Log in

Scaled Proximal Gradient Methods for Sparse Optimization Problems

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

Thresholding-based methods are widely used for sparse optimization problems in many applications including compressive sensing, image processing, and machine learning. However, the hard thresholding method may converge slowly or diverge for many practical problems. In this paper, we propose a scaled proximal gradient method for solving sparse optimization problems, where the scaled matrix can take a varying positive diagonal for connecting the residual reduction. The global convergence of the proposed method is established under some mild assumptions without the restricted isometry property condition. We also present a scaled proximal pursuit and a modified scaled proximal gradient method with global convergence under the restricted isometry property. Finally, some numerical tests are reported to illustrate the efficiency of the proposed methods over the classical thresholding-based methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data Availability

The datasets analysed during the current study were derived from the following public resources: http://vision.ucsd.edu/content/extended-yale-face-database-b-b.

Notes

  1. http://cvxr.com/cvx/.

  2. http://vision.ucsd.edu/content/extended-yale-face-database-b-b.

References

  1. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka–Łojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)

    MathSciNet  Google Scholar 

  2. Beck, A.: First-Order Methods in Optimization. SIAM, Philadelphia (2017)

    Google Scholar 

  3. Bertsimas, D., King, A., Mazumder, R.: Best subset selection via a modern optimization lens. Ann. Stat. 44, 813–852 (2016)

    MathSciNet  Google Scholar 

  4. Blumensath, T.: Accelerated iterative hard thresholding. Signal Process. 92, 752–756 (2012)

    Google Scholar 

  5. Blumensath, T., Davies, M.E.: Iterative hard thresholding for sparse approximation. J. Fourier Anal. Appl. 14, 629–654 (2008)

    MathSciNet  Google Scholar 

  6. Blumensath, T., Davies, M.E.: Iterative hard thresholding for compressed sensing. Appl. Comput. Harmon. Anal. 27, 265–274 (2009)

    MathSciNet  Google Scholar 

  7. Blumensath, T., Davies, M.E.: Normalized iterative hard thresholding: guaranteed stability and performance. IEEE J. Sel. Top. Signal Process. 4, 298–309 (2010)

    Google Scholar 

  8. Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146, 459–494 (2014)

    MathSciNet  Google Scholar 

  9. Bonnans, J.F., Gilbert, JCh., Lemaréchal, C., Sagastizábal, C.A.: A family of variable metric proximal methods. Math. Program. 68, 15–47 (1995)

    MathSciNet  Google Scholar 

  10. Candès, E.J.: Compressive sampling. In: Proceedings of the International Congress of Mathematicians, Madrid, Spain (2006)

  11. Candès, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52, 489–509 (2006)

    MathSciNet  Google Scholar 

  12. Candès, E.J., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 51, 4203–4215 (2005)

    MathSciNet  Google Scholar 

  13. Candès, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(\ell _1\)-minimization. J. Fourier Anal. Appl. 14, 877–905 (2008)

    MathSciNet  Google Scholar 

  14. Cevher, V.: On accelerated hard thresholding methods for sparse approximation. In: Proceedings of SPIE Optical Engineering + Applications, Wavelets and Sparsity XIV, 813811 (2011)

  15. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20, 33–61 (1998)

    MathSciNet  Google Scholar 

  16. Dai, W., Milenkovic, O.: Subspace pursuit for compressive sensing signal reconstruction. IEEE Trans. Inf. Theory 55, 2230–2249 (2009)

    MathSciNet  Google Scholar 

  17. Davis, G.M., Mallat, S.G., Zhang, Z.: Adaptive time-frequency decompositions. Opt. Eng. 33, 2183–2191 (1994)

    Google Scholar 

  18. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52, 1289–1306 (2006)

    MathSciNet  Google Scholar 

  19. Donoho, D.L.: De-noising by soft-thresholdinng. IEEE Trans. Inf. Theory 41, 613–627 (1995)

    Google Scholar 

  20. Elad, M.: Why simple shrinkage is still relevant for redundant representation? IEEE Trans. Inf. Theory 52, 5559–5569 (2006)

    MathSciNet  Google Scholar 

  21. Elad, M.: Sparse and redundant Representations: From Theory to Applications in Signal and Image Processing. Springer, New York (2010)

    Google Scholar 

  22. Eldar, Y.C., Kutyniok, G.: Compressed Sensing: Theory and Applications. Cambridge University Press, Cambridge (2012)

    Google Scholar 

  23. Foucart, S.: Hard thresholding pursuit: an algorithm for compressive sensing. SIAM J. Numer. Anal. 49, 2543–2563 (2011)

    MathSciNet  Google Scholar 

  24. Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Springer, New York (2013)

    Google Scholar 

  25. Han, Z., Li, H., Yin, W.: Compressive Sensing for Wireless Networks. Cambridge University Press, Cambridge (2013)

    Google Scholar 

  26. Lee, K.C., Ho, J., Kriegman, D.J.: Acquiring linear subspaces for face recognition under variable lighting. IEEE Trans. Pattern Anal. Mach. Intell. 27, 684–698 (2005)

    Google Scholar 

  27. Liu, H., Yue, M.C., So, A.M.C., Ma, W.K.: A discrete first-order method for large-scale MIMO detection with provable guarantees. In: Proceedings of the IEEE 18th International Workshop on Signal Processing Advances in Wireless Communications (2017)

  28. Mallat, S.G., Zhang, Z.: Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Process. 41, 3397–3415 (1993)

    Google Scholar 

  29. Meng, N., Zhao, Y.B.: Newton-step-based hard thresholding algorithms for sparse signal recovery. IEEE Trans. Signal Process. 68, 6594–6606 (2020)

    MathSciNet  Google Scholar 

  30. Miller, A.: Subset Selection in Regression, 2nd edn. Chapman and Hall, London (2002)

    Google Scholar 

  31. Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24, 227–234 (1995)

    MathSciNet  Google Scholar 

  32. Mordukhovich, B.S.: Variational Analysis and Generalized Differentiation I-Basic Theory. Springer, Berlin (2006)

    Google Scholar 

  33. Needell, D., Tropp, J.A.: CoSaMP: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmon. Anal. 26, 301–321 (2009)

    MathSciNet  Google Scholar 

  34. Parikh, N., Boyd, S.: Proximal algorithms. Found. Trends Optim. 1, 127–239 (2014)

    Google Scholar 

  35. Patel, V., Chellappa, R.: Sparse representations, compressive sensing and dictionaries for pattern recognition. In: Proceedings of The First Asian Conference on Pattern Recognition, pp. 325–329 (2011)

  36. Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Springer, Berlin (1998)

    Google Scholar 

  37. Starck, J., Murtagh, F., Fadili, J.: Sparse Image and Signal Processing: Wavelets, Curvelets, Morphological Diversity. Cambridge University Press, Cambridge (2010)

    Google Scholar 

  38. Tropp, J.A., Gilbert, A.C.: Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans. Inf. Theory 53, 4655–4666 (2007)

    MathSciNet  Google Scholar 

  39. Wright, S.J.: Coordinate descent algorithms. Math. Program. 151, 3–34 (2015)

    MathSciNet  Google Scholar 

  40. Xiao, G.Y., Bai, Z.J.: A geometric proximal gradient method for sparse least squares regression with probabilistic simplex constraint. J. Sci. Comput. 92, 22 (2022)

    MathSciNet  Google Scholar 

  41. Xu, Z., Chang, X., Xu, F., Zhang, H.: \(L_{1/2}\) regularization: a thresholding representation theory and a fast solver. IEEE Trans. Neural Netw. Learn. Syst. 23, 1013–1027 (2012)

    Google Scholar 

  42. Zhao, Y.B.: Sparse Optimization Theory and Methods. CRC Press, Boca Raton (2018)

    Google Scholar 

  43. Zhao, Y.B.: Optimal \(k\)-thresholding algorithms for sparse optimization problems. SIAM J. Optim. 30, 31–55 (2020)

    MathSciNet  Google Scholar 

  44. Zhao, Y.B., Li, D.: Reweighted \(\ell _1\)-minimization for sparse solutions to underdetermined linear systems. SIAM J. Optim. 22, 1065–1088 (2012)

    MathSciNet  Google Scholar 

  45. Zhao, Y.B., Luo, Z.Q.: Constructing new weighted \(\ell _1\)-algorithms for the sparsest points of polyhedral sets. Math. Oper. Res. 42, 57–76 (2017)

    MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the editor and the referees for their valuable comments.

Author information

Authors and Affiliations

  1. School of Mathematical Sciences, Fudan University, Shanghai, 200433, People’s Republic of China

    Guiyun Xiao & Shuqin Zhang

  2. Key Laboratory of Mathematics for Nonlinear Science (Fudan University), Ministry of Education, Shanghai, China

    Shuqin Zhang

  3. Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai, China

    Shuqin Zhang

  4. School of Mathematical Sciences, Xiamen University, Xiamen, 361005, People’s Republic of China

    Zheng-Jian Bai

Authors
  1. Guiyun Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuqin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zheng-Jian Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zheng-Jian Bai.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The research of Shuqin Zhang was partially supported by the National Key R &D Program of China Grant 2021YFA1003305. The research of Zheng-Jian Bai was partially supported by the National Natural Science Foundation of China Grant 12371382.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiao, G., Zhang, S. & Bai, ZJ. Scaled Proximal Gradient Methods for Sparse Optimization Problems. J Sci Comput 98, 2 (2024). https://doi.org/10.1007/s10915-023-02393-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10915-023-02393-1

Keywords

Mathematics Subject Classification

Search

Navigation