Abstract
This paper introduces a conjugate gradients (CG) acceleration of the coordinate descent algorithm (CD) for linear systems. It is shown that the Kaczmarz algorithm (KACZ) can simulate CD exactly, so CD can be accelerated by CG similarly to the CG acceleration of KACZ (Björck and Elfving in BIT 19:145–163, 1979). Experimental results were carried out on large sets of problems of reconstructing bandlimited functions from random sampling. The randomness causes extreme variance between different instances of these problems, thus causing extreme variance in the advantage of CGCD over CD. The reduction of the number of iterations by CGCD varies from about 50–90% and beyond. The implementation of CGCD is simple. CGCD can also be used for the parallel solution of linear systems derived from partial differential equations, and for the efficient solution of multiple right-hand-side problems and matrix inversion.
Similar content being viewed by others
Data Availability
The link https://cs.haifa.ac.il~gordon/cgcd.c provides the C-program for the experiments. The text at the beginning of the program explains in detail how to run it.
Notes
https://cs.haifa.ac.il~gordon/cgcd.c.
References
Bertrand, Q., Massias, M.: Anderson acceleration of coordinate descent. In: Proceedings of International Conference on Artificial Intelligence and Statistics, vol. 130, no. 10 (2021)
Björck, Å., Elfving, T.: Accelerated projection methods for computing pseudoinverse solutions of systems of linear equations. BIT 19, 145–163 (1979)
Feichtinger, H., Gröchenig, K.: Theory and practice of irregular sampling. In: Frazier, M. (ed.) Wavelets: Mathematics and Applications, pp. 305–363. CRC Press, Boca Raton (1994)
Gordon, D.: A derandomization approach to recovering bandlimited signals across a wide range of random sampling rates. Numer. Algorithms 77(4), 1141–1157 (2018). https://doi.org/10.1007/s11075-017-0356-3
Gordon, D., Gordon, R.: CGMN revisited: robust and efficient solution of stiff linear systems derived from elliptic partial differential equations. ACM Trans. Math. Softw. 35(3), 18:1-18:27 (2008)
Gordon, D., Gordon, R.: CARP-CG: a robust and efficient parallel solver for linear systems, applied to strongly convection-dominated PDEs. Parallel Comput. 36(9), 495–515 (2010)
Kaczmarz, S.: Angenäherte Auflösung von Systemen linearer Gleichungen. Bull. l’Acad. Pol. Sci. Lett. A35, 355–357 (1937)
Lee, Y.T., Sidford, A.: Efficient accelerated coordinate descent methods and faster algorithms for solving linear systems. In: Proceedings of 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, FOCS 2013, pp. 147—156. IEEE Computer Society, USA (2013)
Nesterov, Y.E.: Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 22(2), 341–362 (2012)
Nesterov, Y.E., Stich, S.U.: Efficiency of the accelerated coordinate descent method on structured optimization problems. SIAM J. Optim. 27(1), 110–123 (2017)
Ruhe, A.: Numerical aspects of Gram–Schmidt orthogonalization of vectors. Linear Algebra Appl. 52(53), 591–601 (1983)
Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd edn. SIAM, Philadelphia (2003)
Shi, H.-J.M., Tu, S., Xu, Y., Yin, W.: A primer on coordinate descent algorithms (2016)
Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15, 262–278 (2009)
Wang, Q., Li, W., Bao, W., Zhang, F.: Accelerated randomized coordinate descent for solving linear systems. Mathematics 10(22), 4379 (2022)
Wright, S.J.: Coordinate descent algorithms. Math. Program. 151(1), 3–34 (2015)
Acknowledgements
The author is grateful to the reviewers for their very useful comments.
Funding
The author declares that no funds, grants, or other support were received during the preparation of this manuscript.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The author declares that there is no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Gordon, D. Conjugate Gradients Acceleration of Coordinate Descent for Linear Systems. J Sci Comput 96, 86 (2023). https://doi.org/10.1007/s10915-023-02307-1
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10915-023-02307-1
Keywords
- Coordinate descent
- CD
- CGCD
- CGMN
- Conjugate gradients acceleration
- Gauss–Seidel
- Kaczmarz algorithm
- Linear systems
- Matrix inversion
- Multiple right-hand-sides
- Parallelism