A Randomized Kaczmarz Algorithm with Exponential Convergence



The Kaczmarz method for solving linear systems of equations is an iterative algorithm that has found many applications ranging from computer tomography to digital signal processing. Despite the popularity of this method, useful theoretical estimates for its rate of convergence are still scarce. We introduce a randomized version of the Kaczmarz method for consistent, overdetermined linear systems and we prove that it converges with expected exponential rate. Furthermore, this is the first solver whose rate does not depend on the number of equations in the system. The solver does not even need to know the whole system but only a small random part of it. It thus outperforms all previously known methods on general extremely overdetermined systems. Even for moderately overdetermined systems, numerical simulations as well as theoretical analysis reveal that our algorithm can converge faster than the celebrated conjugate gradient algorithm. Furthermore, our theory and numerical simulations confirm a prediction of Feichtinger et al. in the context of reconstructing bandlimited functions from nonuniform sampling.


Kaczmarz algorithm Randomized algorithm Random matrix Convergence rate 


  1. 1.
    Bass, R.F., Gröchenig, K.: Random sampling of multivariate trigonometric polynomials. SIAM J. Math. Anal. 36(3), 773–795 (2004/05) MATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Benedetto, J.J., Ferreira, P.J.S.G. (eds.): Modern Sampling Theory: Mathematics and Applications. Applied and Numerical Harmonic Analysis. Birkhäuser, Boston (2001) Google Scholar
  3. 3.
    Cenker, C., Feichtinger, H.G., Mayer, M., Steier, H., Strohmer, T.: New variants of the POCS method using affine subspaces of finite codimension, with applications to irregular sampling. In: Proc. SPIE: Visual Communications and Image Processing, pp. 299–310 (1992) Google Scholar
  4. 4.
    Censor, Y., Eggermont, P.P.B., Gordon, D.: Strong underrelaxation in Kaczmarz’s method for inconsistent linear systems. Numer. Math. 41, 83–92 (1983) MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Demmel, J.: The probability that a numerical analysis problem is difficult. Math. Comput. 50(182), 449–480 (1988) MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Deutsch, F.: Rate of convergence of the method of alternating projections. In: Parametric Optimization and Approximation (Oberwolfach, 1983). Internat. Schriftenreihe Numer. Math., vol. 72, pp. 96–107. Birkhäuser, Basel (1985) Google Scholar
  7. 7.
    Deutsch, F., Hundal, H.: The rate of convergence for the method of alternating projections. II. J. Math. Anal. Appl. 205(2), 381–405 (1997) MATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Edelman, A.: Eigenvalues and condition numbers of random matrices. SIAM J. Matrix Anal. Appl. 9(4), 543–560 (1988) MATHCrossRefMathSciNetGoogle Scholar
  9. 9.
    Edelman, A.: On the distribution of a scaled condition number. Math. Comput. 58(197), 185–190 (1992) MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Edelman, A., Sutton, B.D.: Tails of condition number distributions. SIAM J. Matrix Anal. Appl. 27(2), 547–560 (2005) MATHCrossRefMathSciNetGoogle Scholar
  11. 11.
    Feichtinger, H.G., Gröchenig, K.H.: Theory and practice of irregular sampling. In: Benedetto, J., Frazier, M. (eds.) Wavelets: Mathematics and Applications, pp. 305–363. CRC Press, Boca Raton (1994) Google Scholar
  12. 12.
    Feichtinger, H.G., Gröchenig, K., Strohmer, T.: Efficient numerical methods in non-uniform sampling theory. Numer. Math. 69, 423–440 (1995) MATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Frieze, A., Kannan, R., Vempala, S.: Fast Monte-Carlo algorithms for finding low-rank approximations. Proc. Found. Comput. Sci. 39, 378–390 (1998). Journal version in J. ACM 51, 1025–1041 (2004) Google Scholar
  14. 14.
    Galántai, A.: On the rate of convergence of the alternating projection method in finite dimensional spaces. J. Math. Anal. Appl. 310(1), 30–44 (2005) MATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    Geman, S.: Limit theorem for the norm of random matrices. Ann. Probab. 8(2), 252–261 (1980) MATHCrossRefMathSciNetGoogle Scholar
  16. 16.
    Golub, G.H., van Loan, C.F.: Matrix Computations, 3rd edn. Johns Hopkins, Baltimore (1996) MATHGoogle Scholar
  17. 17.
    Gröchenig, K.: Reconstruction algorithms in irregular sampling. Math. Comput. 59, 181–194 (1992) MATHCrossRefGoogle Scholar
  18. 18.
    Gröchenig, K.: Irregular sampling, Toeplitz matrices, and the approximation of entire functions of exponential type. Math. Comput. 68, 749–765 (1999) MATHCrossRefGoogle Scholar
  19. 19.
    Hanke, M.: Conjugate Gradient Type Methods for Ill-Posed Problems. Longman Scientific & Technical, Harlow (1995) MATHGoogle Scholar
  20. 20.
    Hanke, M., Niethammer, W.: On the acceleration of Kaczmarz’s method for inconsistent linear systems. Linear Algebra Appl. 130, 83–98 (1990) MATHCrossRefMathSciNetGoogle Scholar
  21. 21.
    Hanke, M., Niethammer, W.: On the use of small relaxation parameters in Kaczmarz’s method. Z. Angew. Math. Mech. 70(6), T575–T576 (1990). Bericht über die Wissenschaftliche Jahrestagung der GAMM (Karlsruhe, 1989) MathSciNetGoogle Scholar
  22. 22.
    Herman, G.T.: Image Reconstruction from Projections. The Fundamentals of Computerized Tomography. Computer Science and Applied Mathematics. Academic Press, New York (1980). Google Scholar
  23. 23.
    Herman, G.T., Meyer, L.B.: Algebraic reconstruction techniques can be made computationally efficient. IEEE Trans. Medical Imaging 12(3), 600–609 (1993) CrossRefGoogle Scholar
  24. 24.
    Herman, G.T., Lent, A., Lutz, P.H.: Relaxation methods for image reconstruction. Commun. Assoc. Comput. Mach. 21, 152–158 (1978) MATHMathSciNetGoogle Scholar
  25. 25.
    Hounsfield, G.N.: Computerized transverse axial scanning (tomography): Part I. Description of the system. Br. J. Radiol. 46, 1016–1022 (1973) Google Scholar
  26. 26.
    Kaczmarz, S.: Angenäherte Auflösung von Systemen linearer Gleichungen. Bull. Int. Acad. Polon. Sci. Lett. A 335–357 (1937) Google Scholar
  27. 27.
    Natterer, F.: The Mathematics of Computerized Tomography. Wiley, New York (1986) MATHGoogle Scholar
  28. 28.
    Rudelson, M., Vershynin, R.: Sampling from large matrices: an approach through geometric functional analysis. J. ACM 54(4), 21 (2006) CrossRefMathSciNetGoogle Scholar
  29. 29.
    Rudelson, M., Vershynin, R.: Invertibility of random matrices. I. The smallest singular value is of order n −1/2. Preprint Google Scholar
  30. 30.
    Rudelson, M., Vershynin, R.: Invertibility of random matrices. II. The Littlewood–Offord theory. Preprint Google Scholar
  31. 31.
    Sezan, K.M., Stark, H.: Applications of convex projection theory to image recovery in tomography and related areas. In: Stark, H. (ed.) Image Recovery: Theory and Application, pp. 415–462. Acad. Press, San Diego (1987) Google Scholar
  32. 32.
    Silverstein, J.W.: The smallest eigenvalue of a large-dimensional Wishart matrix. Ann. Probab. 13(4), 1364–1368 (1985) MATHCrossRefMathSciNetGoogle Scholar
  33. 33.
    Spielman, D.A., Teng, S.-H.: Smoothed analysis of algorithms. In: Proceedings of the International Congress of Mathematicians, vol. I, pp. 597–606. Higher Ed. Press, Beijing (2002). Google Scholar
  34. 34.
    Tao, T., Vu, V.: Inverse Littlewood–Offord theorems and the condition number of random discrete matrices. Ann. Math. (2008, to appear) Google Scholar
  35. 35.
    van der Sluis, A., van der Vorst, H.A.: The rate of convergence of conjugate gradients. Numer. Math. 48, 543–560 (1986) MATHCrossRefMathSciNetGoogle Scholar
  36. 36.
    Yeh, S., Stark, H.: Iterative and one-step reconstruction from nonuniform samples by convex projections. J. Opt. Soc. Am. A 7(3), 491–499 (1990) CrossRefGoogle Scholar

Copyright information

© Birkhäuser Boston 2008

Authors and Affiliations

  1. 1.Department of MathematicsUniversity of CaliforniaDavisUSA

Personalised recommendations