Encyclopedia of Applied and Computational Mathematics

2015 Edition
| Editors: Björn Engquist

Classical Iterative Methods

  • Owe Axelsson
Reference work entry
DOI: https://doi.org/10.1007/978-3-540-70529-1_242


Iterative solution methods to solve linear systems of equations were originally formulated as basic iteration methods of defect–correction type, commonly referred to as Richardson’s iteration method. These methods developed further into various versions of splitting methods, including the successive overrelaxation (SOR) method. Later, immensely important developments included convergence acceleration methods, such as the Chebyshev and conjugate gradient iteration methods, and preconditioning methods of various forms. A major strive has been to find methods with a total computational complexity of optimal order, that is, proportional to the degrees of freedom involved in the equation.

Methods that have turned out to have been particularly important for the further developments of linear equation solvers are surveyed.


In many applications of quite different types appearing in various sciences, engineering, and finance, large-scale linear algebraic systems of...

This is a preview of subscription content, log in to check access.


  1. 1.
    Axelsson, O.: A generalized SSOR method. BIT 13, 443–467 (1972)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Axelsson, O.: Solution of linear systems of equations: iterative methods. In: Barker, V.A. (ed.) Sparse Matrix Techniques. Lecture Notes in Mathematics, vol. 572, pp. 1–51. Springer, Berlin (1977)CrossRefGoogle Scholar
  3. 3.
    Axelsson, O.: A survey of preconditioned iterative methods for linear systems of algebraic equations. BIT 25, 166–187 (1985)MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Axelsson, O.: A generalized conjugate gradient, least square method. Numer. Math. 51, 209–227 (1987)MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Axelsson, O.: On the rate of convergence of the conjugate gradient method. In: Er-xiong, J. (ed.) Numerical Algebra: Proceedings of 92 Shanghai International Numerical Algebra and Its Application Conference. China Science and Technology Press, Shanghai (1992)Google Scholar
  6. 6.
    Axelsson, O.: Iterative Solution Methods. Cambridge University Press, New York (1994)zbMATHCrossRefGoogle Scholar
  7. 7.
    Axelsson, O.: Optimal preconditioners based on rate of convergence estimates for the conjugate gradient method. In: Mika, S., Brandner, M. (eds.) Lecture Notes of IMAMM 99, pp. 5–56. University of West Bohemia, Pilsen (1999)Google Scholar
  8. 8.
    Axelsson, O.: Condition numbers for the study of the rate of convergence of the conjugate gradient method. In: Margenov, S., Vassilevski, P.S. (eds.) Iterative Methods in Linear Algebra II, pp. 3–33, IMACS, Piscataway (1999)Google Scholar
  9. 9.
    Axelsson, O.: Stabilization of algebraic multilevel iteration methods: additive methods. Numer. Algorithms 21, 23–47 (1999)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Axelsson, O.: Review article, Milestones in the development of iterative solution methods. J. Electr. Comput. Eng. 2010, 1–33 (2010)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Axelsson, O., Barker, V.A.: Finite Element Solution of Boundary Value Problems: Theory and Computations. Academic, Orlando (1984). Reprinted as SIAM Classics in Applied Mathematics 35, Philadelphia (2001)Google Scholar
  12. 12.
    Axelsson, O., Blaheta, R.: Preconditioning of matrices partitioned in two by two block form: eigenvalue estimates and Schwarz DD for mixed FEM. Numer. Linear Algebra Appl. 17, 787–810 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Axelsson, O., Kaporin, I.: On the sublinear and superlinear rate of convergence of conjugate gradient methods. Numer. Algorithms 25, 1–22 (2000)MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Axelsson, O., Karátson, J.: Superlinearly convergent CG methods via equivalent preconditioning for nonsymmetric elliptic operators. Numer. Math. 99, 197–223 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    Axelsson, O., Karátson, J.: Equivalent operator preconditioning for elliptic problems. Numer. Algorithms 50, 297–380 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Axelsson, O., Kutcherov, A.: Real valued iterative methods for solving complex symmetric linear systems. Numer. Linear Algebra Appl. 7, 197–218 (2000)MathSciNetzbMATHCrossRefGoogle Scholar
  17. 17.
    Axelsson, O., Lindskog, G.: On the rate of convergence of the preconditioned conjugate gradient method. Numer. Math. 48, 499–523 (1986)MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Axelsson, O., Neytcheva, M.: Algebraic multilevel iteration method for Stieltjes matrices. Numer. Linear Algebra Appl. 1, 213–236 (1994)MathSciNetzbMATHCrossRefGoogle Scholar
  19. 19.
    Axelsson, O., Nikolova, M.: Conjugate gradient minimum residual method (GCG–MR) with variable preconditioners and a relation between residuals of the GCG–MR and GCG–OR methods. Commun. Appl. Anal. 1, 371–388 (1997)MathSciNetzbMATHGoogle Scholar
  20. 20.
    Axelsson, O., Padiy, A.: On the additive version of the algebraic multilevel iteration method for anisotropic elliptic problems. SIAM J. Sci. Comput. 20, 1807–1830 (1999)MathSciNetzbMATHCrossRefGoogle Scholar
  21. 21.
    Axelsson, O., Polman, B.: On approximate factorization methods for block matrices suitable for vector and parallel processors. Linear Algebra Appl. 77, 3–26 (1986)MathSciNetzbMATHCrossRefGoogle Scholar
  22. 22.
    Axelsson, O., Vassilevski, P.S.: Algebraic multilevel preconditioning methods I. Numer. Math. 56, 157–177 (1989)MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    Axelsson, O., Vassilevski, P.S.: Algebraic multilevel preconditioning methods II. SIAM J. Numer. Anal. 27, 1569–1590 (1990)MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    Axelsson, O., Brinkkemper, S., Il’in, V.P.: On some versions of incomplete block–matrix factorization iterative methods. Linear Algebra Appl. 58, 3–15 (1984)MathSciNetzbMATHCrossRefGoogle Scholar
  25. 25.
    Axelsson, O., Lu, H., Polman, B.: On the numerical radius of matrices and its application to iterative solution methods. Linear Multilinear Algebra 37, 225–238 (1994)MathSciNetzbMATHCrossRefGoogle Scholar
  26. 26.
    Axelsson, O., Blaheta, R., Neytcheva, M.: Preconditioning for boundary value problems using elementwise Schur complements. SIAM J. Matrix Anal. Appl. 31, 767–789 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  27. 27.
    Bakhvalov, N.S.: Numerical solution of a relaxation method with natural constraints on the elliptic operator. USSR Comput. Math. Math. Phys. 6, 101–135 (1966)CrossRefGoogle Scholar
  28. 28.
    Bank, R.E., Dupont, T.F., Yserentant, H.: The hierarchical basis multigrid method. Numer. Math. 52, 427–458 (1988)MathSciNetzbMATHCrossRefGoogle Scholar
  29. 29.
    Beauwens, R.: Factorization iterative methods, M–operators and H–operators. Numer. Math. 31, 335–357 (1979)MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    Beauwens, R., Ben Bouzid, M.: On sparse block factorization, iterative methods. SIAM J. Numer. Anal. 24, 1066–1076 (1987)MathSciNetzbMATHCrossRefGoogle Scholar
  31. 31.
    Birkhoff, G., Varga, R.S., Young, D.M.: Alternating direction implicit methods. In: Alt, F., Rubinoff, M. (eds.) Advances in Computers, vol. 3, pp. 189–273. Academic, New York (1962)Google Scholar
  32. 32.
    Braess, D.: Finite Elements: Theory, Fast Solvers, and Applications in Solid Mechanics, 2nd edn. Cambridge University Press, Cambridge (2001)zbMATHGoogle Scholar
  33. 33.
    Bramble, J.H.: Multigrid Methods. Pitman Research Notes in Mathematics Series, vol. 294. Longman Scientific and Technical, Harlow (1993)Google Scholar
  34. 34.
    Bramble, J.H., Pasciak, J.E., Xu, J.: Parallel multilevel preconditioners. Math. Comput. 55, 1–22 (1990)MathSciNetzbMATHCrossRefGoogle Scholar
  35. 35.
    Brandt, A.: Multi–level adaptive solution to boundary–value problems. Math. Comput. 31, 333–390 (1977)zbMATHCrossRefGoogle Scholar
  36. 36.
    Concus, P., Golub, G.H., O’Leary, D.P.: A generalized conjugate gradient method for the numerical solution of elliptic partial differential equations. In: Bunch, J.R., Rose, D.J. (eds.) Sparse Matrix Computations, pp. 309–332. Academic, New York (1976)Google Scholar
  37. 37.
    Concus, P., Golub, G.H., Meurant, G.: Block preconditioning for the conjugate gradient method. SIAM J. Sci. Stat. Comput. 6, 220–252 (1985)MathSciNetzbMATHCrossRefGoogle Scholar
  38. 38.
    Dax, A.: The convergence of linear stationary iterative process for solving singular unstructured systems of linear equations. SIAM Rev. 32, 611–635 (1990)MathSciNetzbMATHCrossRefGoogle Scholar
  39. 39.
    D’Yakonov, E.G.: On the iterative method for the solution of finite difference equations. Dokl. Akad. Nauk SSSR, 138, 522–525 (1961)MathSciNetGoogle Scholar
  40. 40.
    Erisman, A.M., Tinney, W.F.: On computing certain elements of the inverse of a sparse matrix. Commun. ACM 18, 177–179 (1975)MathSciNetzbMATHCrossRefGoogle Scholar
  41. 41.
    Fischer, B., Freund, R.: Chebyshev polynomials are not always optimal. J. Approx. Theory 65, 261–272 (1990)MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    Fedorenko, R.: The speed of convergence of one iterative process. USSR Comput. Math. Math. Phys. 4, 227–235 (1964)zbMATHCrossRefGoogle Scholar
  43. 43.
    Frankel, S.P.: Convergence rates of iterative treatments of partial differential equations. Math. Tables Aids Comput. 4, 65–75 (1950)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Freund, R.: On conjugate gradient type methods and polynomial preconditioners for a class of complex non–Hermitian matrices. Numer. Math. 57, 285–312 (1990)MathSciNetzbMATHCrossRefGoogle Scholar
  45. 45.
    Gauss, C.F.: Brief an Gerling vom 26 Dec.1823, Werke 9, 278–281 (1823). A translation by Forsythe, G.E., in MTAC 5, 255–258 (1950)Google Scholar
  46. 46.
    Golub, G.H., O’Leary, D.P.: Some history of the conjugate gradient and Lanczos algorithms: 1948–1976. SIAM Rev. 31, 50–102 (1989)MathSciNetzbMATHCrossRefGoogle Scholar
  47. 47.
    Golub, G.H., Varga, R.S.: Chebyshev semi–iterative methods, successive overrelaxation iterative methods, and second-order Richardson Iterative Methods, part I and II. Numer. Math. 3, 147–156, 157–168 (1961)MathSciNetzbMATHCrossRefGoogle Scholar
  48. 48.
    Greenbaum, A.: Comparison of splittings used with the conjugate gradient algorithm. Numer. Math. 33, 181–194 (1979)MathSciNetzbMATHCrossRefGoogle Scholar
  49. 49.
    Greenbaum, A.: Iterative Methods for Solving Linear Systems. Frontiers in Applied Mathematics, vol. 17. SIAM, Philadelphia (1997)Google Scholar
  50. 50.
    Greenbaum, A., Strakos, Z.: Predicting the behaviour of finite precision Lanczos and conjugate gradient computations. SIAM J. Matrix Anal. Appl. 13, 121–137 (1992)MathSciNetzbMATHCrossRefGoogle Scholar
  51. 51.
    Gunn, J.E.: The solution of elliptic difference equations by semi–explicit iterative techniques. SIAM J. Numer. Anal. Ser. B 2, 24–45 (1964)MathSciNetzbMATHGoogle Scholar
  52. 52.
    Gustafsson, I.: A class of first–order factorization methods. BIT 18, 142–156 (1978)MathSciNetzbMATHCrossRefGoogle Scholar
  53. 53.
    Hackbusch, W.: Multigrid Methods and Applications. Springer, Berlin/Heidelberg/New York (1985)zbMATHCrossRefGoogle Scholar
  54. 54.
    Hayes, R.M.: Iterative methods for solving linear problems in Hilbert spaces. In: Tausky, O. (ed.) Contributions to the Solutions of Systems of Linear Equations and the Determination of Eigenvalues. National Bureau of Standards Applied Mathematics Series, vol. 39, pp. 71–103. U.S. G.P.O., Washington, DC (1954)Google Scholar
  55. 55.
    Hestenes, M.R., Stiefel, E.: Methods of conjugate gradients for solving linear systems. J. Res. Nat. Bur. Stand. B 1449, 409–436 (1952)MathSciNetzbMATHCrossRefGoogle Scholar
  56. 56.
    Il’in, V.P.: Iterative Incomplete Factorization Methods. World Scientific, Singapore (1992)zbMATHCrossRefGoogle Scholar
  57. 57.
    Joubert, W.D., Young, D.M.: Necessary and sufficient conditions for the simplification of generalized conjugate–gradient algorithms. Linear Algebra Appl. 88/89, 449–485 (1987)Google Scholar
  58. 58.
    Kaporin, I.E.: New convergence results and preconditioning strategies for the conjugate gradient method. Numer. Linear Algebra Appl. 1, 179–210 (1994)MathSciNetzbMATHCrossRefGoogle Scholar
  59. 59.
    Kolotilina, L.Y., Yeremin A.Y.: On a family of two-level preconditionings of the incomplete block factorization type. Sovit J. Numer. Anal. Math. Model. 1, 292–320 (1986)MathSciNetGoogle Scholar
  60. 60.
    Kolotilina, L.Y., Yeremin, A.Y.: Factorized sparse approximate inverse preconditionings I. Theory. SIAM J. Matrix Anal. Appl. 14, 45–58 (1993)MathSciNetzbMATHCrossRefGoogle Scholar
  61. 61.
    Kraus, J.: Algebraic multilevel preconditioning of finite element matrices using local Schur complements. Numer. Linear Algebra Appl. 13, 49–70 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  62. 62.
    Mandel, J., McCormick, S.F., Ruge, J.: An algebraic theory for multigrid methods for variational problems. SIAM J. Numer. Anal. 25, 91–110 (1988)MathSciNetzbMATHCrossRefGoogle Scholar
  63. 63.
    Manteuffel, T.A.: The Tchebychev iteration for nonsymmetric linear systems. Numer. Math. 28, 307–327 (1977)MathSciNetzbMATHCrossRefGoogle Scholar
  64. 64.
    Mc Cormick, S.: Multilevel Adaptive Methods for Partial Differential Equations. Frontiers in Applied Mathematics, vol. 6. SIAM, Philadelphia (1989)Google Scholar
  65. 65.
    Meijerink, J.A., van der Vorst, H.A.: An iterative solution method for linear systems of which the coefficient matrix is a symmetric M–matrix. Math. Comput. 31, 148–162 (1979)MathSciNetzbMATHGoogle Scholar
  66. 66.
    Nevanlinna, O.: Convergence of Iterations for Linear Equations. Lectures in Mathematics, ETH Zürich. Birkhäuser, Basel (1993)Google Scholar
  67. 67.
    Nicolaides, R.: Deflation of conjugate gradients with application to boundary value problems. SIAM J. Numer. Anal. 24, 355–365 (1987)MathSciNetzbMATHCrossRefGoogle Scholar
  68. 68.
    Notay, Y.: On the convergence rate of the conjugate gradients in presence of rounding errors. Numer. Math. 65, 301–317 (1993)MathSciNetzbMATHCrossRefGoogle Scholar
  69. 69.
    Neytcheva, M.: On element–by–element Schur complement approximations. Linear Algebra Appl. 434, 2308–2324 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  70. 70.
    Paige, C.C., Saunders, M.A.: Solution of sparse indefinite systems of linear equations. SIAM J. Numer. Anal. 12, 617–629 (1975)MathSciNetzbMATHCrossRefGoogle Scholar
  71. 71.
    Peaceman, D.W., Rachford, H.H. Jr.: The numerical solution of parabolic and elliptic differential equations. J. Soc. Indust. Appl. Math. 3, 28–41 (1955)MathSciNetzbMATHCrossRefGoogle Scholar
  72. 72.
    Reid, J.K.: The use of conjugate gradients for systems of linear equations possessing “Property A”. SIAM J. Numer. Anal. 9, 325–332.8 (1972)MathSciNetzbMATHCrossRefGoogle Scholar
  73. 73.
    Richardson, L.F.: The approximate arithmetical solution by finite differences of physical problems involving differential equations, with an application to the stresses in a masonry dam. Trans. R. Soc. Lond A 210, 307–357 (1911)zbMATHCrossRefGoogle Scholar
  74. 74.
    Saad, Y.: A flexible inner–outer preconditioned GMRES algorithm. SIAM J. Sci. Comput. 14, 461–469 (1993)MathSciNetzbMATHCrossRefGoogle Scholar
  75. 75.
    Saad, Y.: Iterative Methods for Sparse Linear Systems. PWS, Boston (1996)zbMATHGoogle Scholar
  76. 76.
    Saad, Y., Schultz, M.H.: GMRES: a generalized minimum residual algorithm for solving nonsymmetric linear systems. SIAM J. Sci. Stat. Comput. 7, 856–869 (1986)MathSciNetzbMATHCrossRefGoogle Scholar
  77. 77.
    Schwarz, H.A.: Über einen Grenzübergang durch alternierendes Verfahren. Vierteljahrsschrift der Naturforschenden Gesellschaft in Zürich 15, 272–286 (1870)Google Scholar
  78. 78.
    Seidel, P.: Über ein Verfahren, die Gleichungen, auf welche die Methode der kleinsten Quadrate führt, sowie lineare Gleichungen überhaupt, durch successive Annäherung aufzulosen. Abh. Math. Phys. K. Bayerische Akad. Wiss. München 11, 81–108 (1814)zbMATHGoogle Scholar
  79. 79.
    Simoncini, V., Szyld, D.B.: Flexible inner–outer Krylov subspace methods. SIAM J. Numer. Anal. 40, 2219–2239 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  80. 80.
    Smith, B.F., Bjorstad, P.E., Gropp, W.D.: Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations. Cambridge University Press, Cambridge (1996)zbMATHGoogle Scholar
  81. 81.
    Takahishi, K., Fagan, J., Chen, M.S.: Formation of a sparse bus impedance matrix and its application to short circuit study. In: Proceedings of the 8th PICA Conference, Minneapolis, pp. 63–69 (1973)Google Scholar
  82. 82.
    Tanabe, K.: Characterization of linear stationary processes for solving singular system of linear equations. Numer. Math. 22, 349–359 (1974)MathSciNetzbMATHCrossRefGoogle Scholar
  83. 83.
    Tosseli, A., Widlund, O.B.: Domain Decomposition Methods: Algorithms and Theory. Springer, Berlin (2005)Google Scholar
  84. 84.
    van der Vorst, H.A.: The convergence behaviour of preconditioned CG and CG-S in the presence of rounding errors. In: Axelsson, O., Kolotilina, L.Y. (eds.) Preconditioned Conjugate Gradient Methods. Lecture Notes in Mathematics, vol. 1457, pp. 126–136. Springer, Berlin (1989)CrossRefGoogle Scholar
  85. 85.
    Varga, R.S.: Matrix Iterative Analysis. Prentice Hall, Englewood Cliffs (1962)Google Scholar
  86. 86.
    Vassilevski, P.S.: Multilevel Block Factorization Preconditioners: Matrix–Based Analysis and Algorithms for Solving Finite Element Equations. Springer, New York (2008)zbMATHGoogle Scholar
  87. 87.
    Wachspress, E.: Iterative Solution of Elliptic Systems and Applications to the Neutron Diffusion Equations of Reactor Physics. Prentice Hall, Englewood Cliffs(1966)zbMATHGoogle Scholar
  88. 88.
    Wesseling, P.: An Introduction to Multigrid Methods. Wiley, Chichester (1992)zbMATHGoogle Scholar
  89. 89.
    Wozniakowski, H.H.: Round off error analysis of a new class of conjugate gradient algorithms. Linear Algebra Appl. 29, 507–529 (1980)MathSciNetzbMATHCrossRefGoogle Scholar
  90. 90.
    Young, D.M.: Iterative methods for solving partial difference equations of elliptic type. Doctoral thesis, Harward University. Reprinted in http:www/sccm.stanford.edu/pub/sccm/ (1950)
  91. 91.
    Young, D.M.: Iterative Solution of Large Linear Systems. Academic, New York (1971)zbMATHGoogle Scholar
  92. 92.
    Young, D.M.: Second degree iterative methods for the solution of large linear systems. J. Approx. Theory 5, 137–148 (1972)MathSciNetzbMATHCrossRefGoogle Scholar
  93. 93.
    Yserentant, H.: Old and new convergence proofs for multigrid methods. Acta Numer. 2, 285–326 (1993)MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Owe Axelsson
    • 1
  1. 1.Division of Scientific Computing, Department of Information TechnologyUppsala UniversityUppsalaSweden