Iterative Schemes for Large Linear Systems

  • Efstratios Gallopoulos
  • Bernard Philippe
  • Ahmed H. Sameh
Part of the Scientific Computation book series (SCIENTCOMP)


Sparse linear systems occur in a multitude of applications in computational science and engineering. While the need for solving such linear systems is most prevalent in numerical simulations using mathematical models based on differential equations, it also arises in other areas. For example, the PageRank vector that was used by Google to order the nodes of a network based on its link structure can be interpreted as the solution of a linear system of order equal to the number of nodes (Brin and Page, The anatomy of a large-scale hypertextual web search engine. In: Proceedings of 7th Intnational Conference World Wide Web, (1998), Langville and Meyer, Google’s PageRank and Beyond: The Science of Search Engine Rankings, (2006) [1, 2]). The system can be very large and sparse so that parallel iterative methods become necessary; see e.g. (Gleich et al. Fast parallel pagerank: a linear system approach, (2004), Gleich et al. SIAM J. Sci. Comput. 32(1), (2010) [3, 4]). We note that we will not describe any parallel asynchronous iterative methods that are sometimes proposed for solving such systems when they are very large, possibly distributed across several computers that are connected by means of a relatively slow network; cf. (Bahi et al. Parallel Iterative Algorithms, (2008), Bertsekas and Tsitsiklis, Parallel and Distributed Computation, (1989), Kollias et al. PARCO, (2005), Ishii and Tempo, IEEE Trans. Autom. Control, 55(9), (2010)[5, 6, 7, 8]).


  1. 1.
    Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. In: Proceedings of 7th International Conference World Wide Web, pp. 107–117. Elsevier Science Publishers B.V, Brisbane (1998)Google Scholar
  2. 2.
    Langville, A., Meyer, C.: Google’s PageRank and Beyond: The Science of Search Engine Rankings. Princeton University Press, Princeton (2006)Google Scholar
  3. 3.
    Gleich, D., Zhukov, L., Berkhin., P.: Fast parallel PageRank: a linear system approach. Technical report, Yahoo Corporate (2004)Google Scholar
  4. 4.
    Gleich, D., Gray, A., Greif, C., Lau, T.: An inner-outer iteration for computing PageRank. SIAM J. Sci. Comput. 32(1), 349–371 (2010)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Bahi, J., Contassot-Vivier, S., Couturier, R.: Parallel Iterative Algorithms. Chapman & Hall/CRC, Boca Raton (2008)MATHGoogle Scholar
  6. 6.
    Bertsekas, D.P., Tsitsiklis, J.N.: Parallel and Distributed Computation. Prentice Hall, Englewood Cliffs (1989)MATHGoogle Scholar
  7. 7.
    Kollias, G., Gallopoulos, E., Szyld, D.: Asynchronous iterative computations with web information retrieval structures: The PageRank case. In: PARCO, pp. 309–316 (2005)Google Scholar
  8. 8.
    Ishii, H., Tempo, R.: Distributed randomized algorithms for the PageRank computation. IEEE Trans. Autom. Control 55(9), 1987–2002 (2010)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Kalnay, E., Takacs, L.: A simple atmospheric model on the sphere with 100% parallelism. Advances in Computer Methods for Partial Differential Equations IV (1981). Also published as NASA Technical Memorandum No. 83907, Laboratory for Atmospheric Sciences, Research Review 1980–1981, pp. 89–95, Goddard Sapce Flight Center, Maryland, December 1981
  10. 10.
    Gallopoulos, E.: Fluid dynamics modeling. In: Potter, J.L. (ed.) The Massively Parallel Processor, pp. 85–101. The MIT Press, Cambridge (1985)Google Scholar
  11. 11.
    Gallopoulos, E., McEwan, S.D.: Numerical experiments with the massively parallel processor. In: Proceedings of the 1983 International Conference on Parallel Processing, August 1983, pp. 29–35 (1983)Google Scholar
  12. 12.
    Potter, J. (ed.): The Massively Parallel Processor. MIT Press, Cambridge (1985)Google Scholar
  13. 13.
    Varga, R.S.: Matrix Iterative Analysis. Prentice Hall Inc., Englewood Cliffs (1962)Google Scholar
  14. 14.
    Wachspress, E.L.: Iterative Solution of Elliptic Systems. Prentice-Hall Inc., Englewood Cliffs (1966)MATHGoogle Scholar
  15. 15.
    Young, D.: Iterative Solution of Large Linear Systems. Academic Press, New York (1971)MATHGoogle Scholar
  16. 16.
    Golub, G., Van Loan, C.: Matrix Computations, 4th edn. Johns Hopkins (2013)Google Scholar
  17. 17.
    Golub, G.H., Varga, R.S.: Chebychev semi-iterative methods, successive overrelaxation iterative methods, and second order Richardson iterative methods: part I. Numer. Math. 3, 147–156 (1961)MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    O’Leary, D., White, R.: Multi-splittings of matrices and parallel solution of linear systems. SIAM J. Algebra Discret. Method 6, 630–640 (1985)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Neumann, M., Plemmons, R.: Convergence of parallel multisplitting iterative methods for M-matrices. Linear Algebra Appl. 88–89, 559–573 (1987)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Szyld, D.B., Jones, M.T.: Two-stage and multisplitting methods for the parallel solution of linear systems. SIAM J. Matrix Anal. Appl. 13, 671–679 (1992)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Hageman, L., Young, D.: Applied Iterative Methods. Academic Press, New York (1981)MATHGoogle Scholar
  22. 22.
    Keller, H.: On the solution of singular and semidefinite linear systems by iteration. J. Soc. Indus. Appl. Math. 2(2), 281–290 (1965)CrossRefMATHGoogle Scholar
  23. 23.
    Stiefel, E.L.: Kernel polynomials in linear algebra and their numerical approximations. U.S. Natl. Bur. Stand. Appl. Math. Ser. 49, 1–22 (1958)MathSciNetGoogle Scholar
  24. 24.
    Saad, Y., Sameh, A., Saylor, P.: Solving elliptic difference equations on a linear array of processors. SIAM J. Sci. Stat. Comput. 6(4), 1049–1063 (1985)MathSciNetCrossRefMATHGoogle Scholar
  25. 25.
    Rutishauser, H.: Refined iterative methods for computation of the solution and the eigenvalues of self-adjoint boundary value problems. In: Engli, M., Ginsburg, T., Rutishauser, H., Seidel, E. (eds.) Theory of Gradient Methods. Springer (1959)Google Scholar
  26. 26.
    Householder, A.S.: The Theory of Matrices in Numerical Analysis. Dover Publications, New York (1964)MATHGoogle Scholar
  27. 27.
    Dongarra, J., Duff, I., Sorensen, D., van der Vorst, H.: Numerical Linear Algebra for High-Performance Computers. SIAM, Philadelphia (1998)CrossRefMATHGoogle Scholar
  28. 28.
    Meurant, G.: Computer Solution of Large Linear Systems. Studies in Mathematics and its Applications. Elsevier Science (1999).
  29. 29.
    Saad, Y.: Iterative Methods for Sparse Linear Systems. SIAM, Philadelphia (2003)CrossRefMATHGoogle Scholar
  30. 30.
    van der Vorst, H.A.: Iterative Krylov Methods for Large Linear Systems. Cambridge University Press, Cambridge (2003).
  31. 31.
    Saad, Y., Schultz, M.H.: GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM J. Sci. Stat. Comput. 7(3), 856–869 (1986)MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Sidje, R.B.: Alternatives for parallel Krylov subspace basis computation. Numer. Linear Algebra Appl. 305–331 (1997)Google Scholar
  33. 33.
    Nuentsa Wakam, D., Erhel, J.: Parallelism and robustness in GMRES with the Newton basis and the deflated restarting. Electron. Trans. Linear Algebra (ETNA) 40, 381–406 (2013)MathSciNetMATHGoogle Scholar
  34. 34.
    Nuentsa Wakam, D., Atenekeng-Kahou, G.A.: Parallel GMRES with a multiplicative Schwarz preconditioner. J. ARIMA 14, 81–88 (2010)MathSciNetGoogle Scholar
  35. 35.
    Philippe, B., Reichel, L.: On the generation of Krylov subspace bases. Appl. Numer. Math. (APNUM) 62(9), 1171–1186 (2012)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Joubert, W.D., Carey, G.F.: Parallelizable restarted iterative methods for nonsymmetric linear systems. Part I: theory. Int. J. Comput. Math. 44, 243–267 (1992)CrossRefGoogle Scholar
  37. 37.
    Joubert, W.D., Carey, G.F.: Parallelizable restarted iterative methods for nonsymmetric linear systems. Part II: parallel implementation. Int. J. Comput. Math. 44, 269–290 (1992)CrossRefMATHGoogle Scholar
  38. 38.
    Parlett, B.: The Symmetric Eigenvalue Problem. SIAM, Philadelphia (1998)CrossRefMATHGoogle Scholar
  39. 39.
    Bai, Z., Hu, D., Reichel, L.: A Newton basis GMRES implementation. IMA J. Numer. Anal. 14, 563–581 (1994)MathSciNetCrossRefMATHGoogle Scholar
  40. 40.
    Reichel, L.: Newton interpolation at Leja points. BIT 30, 332–346 (1990)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  • Efstratios Gallopoulos
    • 1
  • Bernard Philippe
    • 2
  • Ahmed H. Sameh
    • 3
  1. 1.Computer Engineering and Informatics DepartmentUniversity of PatrasPatrasGreece
  2. 2.Campus de BeaulieuINRIA/IRISARennes CedexFrance
  3. 3.Department of Computer SciencePurdue UniversityWest LafayetteUSA

Personalised recommendations