Skip to main content
SpringerLink
Log in

Conjugate Gradients Acceleration of Coordinate Descent for Linear Systems

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

This paper introduces a conjugate gradients (CG) acceleration of the coordinate descent algorithm (CD) for linear systems. It is shown that the Kaczmarz algorithm (KACZ) can simulate CD exactly, so CD can be accelerated by CG similarly to the CG acceleration of KACZ (Björck and Elfving in BIT 19:145–163, 1979). Experimental results were carried out on large sets of problems of reconstructing bandlimited functions from random sampling. The randomness causes extreme variance between different instances of these problems, thus causing extreme variance in the advantage of CGCD over CD. The reduction of the number of iterations by CGCD varies from about 50–90% and beyond. The implementation of CGCD is simple. CGCD can also be used for the parallel solution of linear systems derived from partial differential equations, and for the efficient solution of multiple right-hand-side problems and matrix inversion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availability

The link https://cs.haifa.ac.il~gordon/cgcd.c provides the C-program for the experiments. The text at the beginning of the program explains in detail how to run it.

Notes

  1. https://cs.haifa.ac.il~gordon/cgcd.c.

References

  1. Bertrand, Q., Massias, M.: Anderson acceleration of coordinate descent. In: Proceedings of International Conference on Artificial Intelligence and Statistics, vol. 130, no. 10 (2021)

  2. Björck, Å., Elfving, T.: Accelerated projection methods for computing pseudoinverse solutions of systems of linear equations. BIT 19, 145–163 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  3. Feichtinger, H., Gröchenig, K.: Theory and practice of irregular sampling. In: Frazier, M. (ed.) Wavelets: Mathematics and Applications, pp. 305–363. CRC Press, Boca Raton (1994)

    MATH  Google Scholar 

  4. Gordon, D.: A derandomization approach to recovering bandlimited signals across a wide range of random sampling rates. Numer. Algorithms 77(4), 1141–1157 (2018). https://doi.org/10.1007/s11075-017-0356-3

    Article  MathSciNet  MATH  Google Scholar 

  5. Gordon, D., Gordon, R.: CGMN revisited: robust and efficient solution of stiff linear systems derived from elliptic partial differential equations. ACM Trans. Math. Softw. 35(3), 18:1-18:27 (2008)

    Article  MathSciNet  Google Scholar 

  6. Gordon, D., Gordon, R.: CARP-CG: a robust and efficient parallel solver for linear systems, applied to strongly convection-dominated PDEs. Parallel Comput. 36(9), 495–515 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  7. Kaczmarz, S.: Angenäherte Auflösung von Systemen linearer Gleichungen. Bull. l’Acad. Pol. Sci. Lett. A35, 355–357 (1937)

    MATH  Google Scholar 

  8. Lee, Y.T., Sidford, A.: Efficient accelerated coordinate descent methods and faster algorithms for solving linear systems. In: Proceedings of 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, FOCS 2013, pp. 147—156. IEEE Computer Society, USA (2013)

  9. Nesterov, Y.E.: Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 22(2), 341–362 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  10. Nesterov, Y.E., Stich, S.U.: Efficiency of the accelerated coordinate descent method on structured optimization problems. SIAM J. Optim. 27(1), 110–123 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  11. Ruhe, A.: Numerical aspects of Gram–Schmidt orthogonalization of vectors. Linear Algebra Appl. 52(53), 591–601 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  12. Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd edn. SIAM, Philadelphia (2003)

    Book  MATH  Google Scholar 

  13. Shi, H.-J.M., Tu, S., Xu, Y., Yin, W.: A primer on coordinate descent algorithms (2016)

  14. Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15, 262–278 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  15. Wang, Q., Li, W., Bao, W., Zhang, F.: Accelerated randomized coordinate descent for solving linear systems. Mathematics 10(22), 4379 (2022)

    Article  Google Scholar 

  16. Wright, S.J.: Coordinate descent algorithms. Math. Program. 151(1), 3–34 (2015)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The author is grateful to the reviewers for their very useful comments.

Funding

The author declares that no funds, grants, or other support were received during the preparation of this manuscript.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Haifa, 34988, Haifa, Israel

    Dan Gordon

Authors
  1. Dan Gordon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dan Gordon.

Ethics declarations

Conflict of interest

The author declares that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gordon, D. Conjugate Gradients Acceleration of Coordinate Descent for Linear Systems. J Sci Comput 96, 86 (2023). https://doi.org/10.1007/s10915-023-02307-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10915-023-02307-1

Keywords

Mathematics Subject Classification

Search

Navigation