Abstract
We review and compare several representative and effective randomized projection iteration methods, including the randomized Kaczmarz method, the randomized coordinate descent method, and their modifications and extensions, for solving the large, sparse, consistent or inconsistent systems of linear equations. We also anatomize, extract, and purify the asymptotic convergence theories of these iteration methods, and discuss, analyze, and summarize their advantages and disadvantages from the viewpoints of both theory and computations.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and materials
Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.
References
Bai, Z.-Z., Liu, X.-G.: On the Meany inequality with applications to convergence analysis of several row-action iteration methods. Numer. Math. 124, 215–236 (2013)
Bai, Z.-Z., Pan, J.-Y.: Matrix Analysis and Computations. SIAM, Philadelphia (2021)
Bai, Z.-Z., Wang, L.: On convergence rates of Kaczmarz-type methods with different selection rules of working rows. Appl. Numer. Math. 186, 289–319 (2023)
Bai, Z.-Z., Wu, W.-T.: On greedy randomized Kaczmarz method for solving large sparse linear systems. SIAM J. Sci. Comput. 40, A592–A606 (2018)
Bai, Z.-Z., Wu, W.-T.: On relaxed greedy randomized Kaczmarz methods for solving large sparse linear systems. Appl. Math. Lett. 83, 21–26 (2018)
Bai, Z.-Z., Wu, W.-T.: On convergence rate of the randomized Kaczmarz method. Linear Algebra Appl. 553, 252–269 (2018)
Bai, Z.-Z., Wu, W.-T.: On partially randomized extended Kaczmarz method for solving large sparse overdetermined inconsistent linear systems. Linear Algebra Appl. 578, 225–250 (2019)
Bai, Z.-Z., Wu, W.-T.: On greedy randomized coordinate descent methods for solving large linear least-squares problems. Numer. Linear Algebra Appl. 26(e2237), 1–15 (2019)
Bai, Z.-Z., Wu, W.-T.: On greedy randomized augmented Kaczmarz method for solving large sparse inconsistent linear systems. SIAM J. Sci. Comput. 43, A3892–A3911 (2021)
Bai, Z.-Z., Wang, L., Wu, W.-T.: On convergence rate of the randomized Gauss–Seidel method. Linear Algebra Appl. 611, 237–252 (2021)
Bai, Z.-Z., Wang, L., Muratova, G.V.: On relaxed greedy randomized augmented Kaczmarz methods for solving large sparse inconsistent linear systems. East Asian J. Appl. Math. 12, 323–332 (2022)
Byrne, C.: A unified treatment of some iterative algorithms in signal processing and image reconstruction. Inverse Probl. 20, 103–120 (2004)
Chang, K.-W., Hsieh, C.-J., Lin, C.-J.: Coordinate descent method for large-scale L2-loss linear support vector machines. J. Mach. Learn. Res. 9, 1369–1398 (2008)
Golub, G.H., Van Loan, C.F.: Matrix Computations, 4th edn. Johns Hopkins University Press, Baltimore (2013)
Hansen, P.C., Jørgensen, J.S., Lionheart, W.R.B.: Computed Tomography: Algorithms, Insight, and Just Enough Theory. SIAM, Philadelphia (2021)
Herman, G.T.: Fundamentals of Computerized Tomography: Image Reconstruction from Projections, 2nd edn. Springer, Dordrecht (2009)
Herman, G.T., Meyer, L.B.: Algebraic reconstruction techniques can be made computationally efficient. IEEE Trans. Med. Imaging 12, 600–609 (1993)
Kaczmarz, S.: Angenäherte Auflösung von Systemen linearer Gleichungen. Bull. Int. Acad. Polon. Sci. Lett. A 35, 355–357 (1937)
Leventhal, D., Lewis, A.S.: Randomized methods for linear constraints: Convergence rates and conditioning. Math. Oper. Res. 35, 641–654 (2010)
Ma, A., Needell, D., Ramdas, A.: Convergence properties of the randomized extended Gauss–Seidel and Kaczmarz methods. SIAM J. Matrix Anal. Appl. 36, 1590–1604 (2015)
Meany, R.K.: A matrix inequality. SIAM J. Numer. Anal. 6, 104–107 (1969)
Natterer, F.: The Mathematics of Computerized Tomography. SIAM, Philadelphia (2001)
Popa, C.: Convergence rates for Kaczmarz-type algorithms. Numer. Algorithms 79, 1–17 (2018)
Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15, 262–278 (2009)
Wu, W.-T.: On convergence of the partially randomized extended Kaczmarz method. East Asian J. Appl. Math. 12, 435–448 (2022)
Zouzias, A., Freris, N.M.: Randomized extended Kaczmarz for solving least squares. SIAM J. Matrix Anal. Appl. 34, 773–793 (2013)
Acknowledgements
The authors are very much indebted to the referees for their constructive suggestions and insightful comments, which greatly improved the original manuscript of this paper.
Funding
This work is supported by The National Natural Science Foundation of China (No. 12071472, No. 12001043), and The Beijing Institute of Technology Research Fund Program for Young Scholars, P.R. China.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Bai, ZZ., Wu, WT. Randomized Kaczmarz iteration methods: Algorithmic extensions and convergence theory. Japan J. Indust. Appl. Math. 40, 1421–1443 (2023). https://doi.org/10.1007/s13160-023-00586-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13160-023-00586-7
Keywords
- System of linear equations
- Randomized projection iteration
- Kaczmarz method
- Coordinate descent method
- Convergence property