50 Years of Time Parallel Time Integration

  • Martin J. GanderEmail author
Part of the Contributions in Mathematical and Computational Sciences book series (CMCS, volume 9)


Time parallel time integration methods have received renewed interest over the last decade because of the advent of massively parallel computers, which is mainly due to the clock speed limit reached on today’s processors. When solving time dependent partial differential equations, the time direction is usually not used for parallelization. But when parallelization in space saturates, the time direction offers itself as a further direction for parallelization. The time direction is however special, and for evolution problems there is a causality principle: the solution later in time is affected (it is even determined) by the solution earlier in time, but not the other way round. Algorithms trying to use the time direction for parallelization must therefore be special, and take this very different property of the time dimension into account.We show in this chapter how time domain decomposition methods were invented, and give an overview of the existing techniques. Time parallel methods can be classified into four different groups: methods based on multiple shooting, methods based on domain decomposition and waveform relaxation, space-time multigrid methods and direct time parallel methods. We show for each of these techniques the main inventions over time by choosing specific publications and explaining the core ideas of the authors. This chapter is for people who want to quickly gain an overview of the exciting and rapidly developing area of research of time parallel methods.


Multigrid Method Cyclic Reduction Multicore Architecture Waveform Relaxation Parareal Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The author is very thankful for the comments of Stefan Vandewalle, which greatly improved this manuscript and also made the content more complete. We thank the Bibliotheque de Geneve for granting permission to reproduce pictures from the original sources.


  1. 1.
    Al-Khaleel, M., Ruehli, A.E., Gander, M.J.: Optimized waveform relaxation methods for longitudinal partitioning of transmission lines. IEEE Trans. Circuits Syst. Regul. Pap. 56, 1732–1743 (2009)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Al-Khaleel, M.D., Gander, M.J., Ruehli, A.E.: Optimization of transmission conditions in waveform relaxation techniques for RC circuits. SIAM J. Numer. Anal. 52, 1076–1101 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Amodio, P., Brugnano, L.: Parallel solution in time of ODEs: some achievements and perspectives. Appl. Numer. Math. 59, 424–435 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Axelsson, A., Verwer, J.: Boundary value techniques for initial value problems in ordinary differential equations. Math. Comput. 45, 153–171 (1985)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Bastian, P., Burmeister, J., Horton, G.: Implementation of a parallel multigrid method for parabolic differential equations. In: Parallel Algorithms for Partial Differential Equations. Proceedings of the 6th GAMM seminar Kiel, pp. 18–27 (1990)Google Scholar
  6. 6.
    Bellen, A., Zennaro, M.: Parallel algorithms for initial-value problems for difference and differential equations. J. Comput. Appl. Math. 25, 341–350 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Bennequin, D., Gander, M.J., Halpern, L.: A homographic best approximation problem with application to optimized Schwarz waveform relaxation. Math. Comput. 78, 185–223 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Bjørhus, M.: On domain decomposition, subdomain iteration and waveform relaxation. Ph.D. thesis, University of Trondheim, Norway (1995)Google Scholar
  9. 9.
    Brugnano, L., Trigiante, D.: A parallel preconditioning technique for boundary value methods. Appl. Numer. Math. 13, 277–290 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Brugnano, L., Trigiante, D.: Convergence and stability of boundary value methods for ordinary differential equations. J. Comput. Appl. Math. 66, 97–109 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Brugnano, L., Trigiante, D.: Solving Differential Equations by Multistep Initial and Boundary Value Methods. CRC Press, Boca Raton (1998)zbMATHGoogle Scholar
  12. 12.
    Burrage, K.: Parallel and Sequential Methods for Ordinary Differential Equations. Clarendon Press, New York (1995)zbMATHGoogle Scholar
  13. 13.
    Chartier, P., Philippe, B.: A parallel shooting technique for solving dissipative ODEs. Computing 51, 209–236 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Christlieb, A.J., Macdonald, C.B., Ong, B.W.: Parallel high-order integrators. SIAM J. Sci. Comput. 32, 818–835 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Crann, D., Davies, A.J., Lai, C.-H., Leong, S.H.: Time domain decomposition for European options in financial modelling. Contemp. Math. 218, 486–491 (1998)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Deshpande, A., Malhotra, S., Schultz, M., Douglas, C.: A rigorous analysis of time domain parallelism. Parallel Algorithms Appl. 6, 53–62 (1995)CrossRefzbMATHGoogle Scholar
  17. 17.
    Douglas, J.Jr., Santos, J.E., Sheen, D., Bennethum, L.S.: Frequency domain treatment of one-dimensional scalar waves. Math. Models Methods Appl. Sci. 3, 171–194 (1993)Google Scholar
  18. 18.
    Douglas, C., Kim, I., Lee, H., Sheen, D.: Higher-order schemes for the Laplace transformation method for parabolic problems. Comput. Vis. Sci. 14, 39–47 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Emmett, M., Minion, M.L.: Toward an efficient parallel in time method for partial differential equations. Commun. Appl. Math. Comput. Sci. 7, 105–132 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Engquist, B., Ying, L.: Sweeping preconditioner for the Helmholtz equation: moving perfectly matched layers. Multiscale Model. Simul. 9, 686–710 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Falgout, R., Friedhoff, S., Kolev, T.V., MacLachlan, S., Schroder, J.B.: Parallel time integration with multigrid. SIAM J. Sci. Comput. 36, C635–C661 (2014)CrossRefzbMATHGoogle Scholar
  22. 22.
    Farhat, C., Cortial, J., Dastillung, C., Bavestrello, H.: Time-parallel implicit integrators for the near-real-time prediction of linear structural dynamic responses. Int. J. Numer. Methods Eng. 67, 697–724 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Franklin, M.A.: Parallel solution of ordinary differential equations. IEEE Trans. Comput. 100, 413–420 (1978)CrossRefGoogle Scholar
  24. 24.
    Gander, M.J.: Overlapping Schwarz for linear and nonlinear parabolic problems. In: Proceedings of the 9th International Conference on Domain Decomposition, pp. 97–104 (1996). ddm.orgGoogle Scholar
  25. 25.
    Gander, M.J.: A waveform relaxation algorithm with overlapping splitting for reaction diffusion equations. Numer. Linear Algebra Appl. 6, 125–145 (1998)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Gander, M.J.: Optimized Schwarz methods. SIAM J. Numer. Anal. 44, 699–731 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Gander, M.J., Güttel, S.: ParaExp: a parallel integrator for linear initial-value problems. SIAM J. Sci. Comput. 35, C123–C142 (2013)CrossRefzbMATHGoogle Scholar
  28. 28.
    Gander, M.J., Hairer, E.: Nonlinear convergence analysis for the parareal algorithm. In: Widlund, O.B., Keyes, D.E. (eds.) Domain Decomposition Methods in Science and Engineering XVII. Lecture Notes in Computational Science and Engineering, vol. 60, pp. 45–56. Springer, Berlin (2008)CrossRefGoogle Scholar
  29. 29.
    Gander, M.J., Halpern, L.: Absorbing boundary conditions for the wave equation and parallel computing. Math. Comput. 74, 153–176 (2004)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Gander, M.J., Halpern, L.: Optimized Schwarz waveform relaxation methods for advection reaction diffusion problems. SIAM J. Numer. Anal. 45, 666–697 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Gander, M.J., Halpern, L.: A direct solver for time parallelization. In: 22nd International Conference of Domain Decomposition Methods. Springer, Berlin (2014)Google Scholar
  32. 32.
    Gander, M.J., Neumüller, M.: Analysis of a new space-time parallel multigrid algorithm for parabolic problems (2015, submitted)Google Scholar
  33. 33.
    Gander, M.J., Petcu, M.: Analysis of a Krylov subspace enhanced parareal algorithm for linear problems. In: ESAIM: Proceedings. EDP Sciences, vol. 25, pp. 114–129 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Gander, M.J., Stuart, A.M.: Space-time continuous analysis of waveform relaxation for the heat equation. SIAM J. Sci. Comput. 19, 2014–2031 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Gander, M.J., Vandewalle, S.: Analysis of the parareal time-parallel time-integration method. SIAM J. Sci. Comput. 29, 556–578 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Gander, M.J., Halpern, L., Nataf, F.: Optimal convergence for overlapping and non-overlapping Schwarz waveform relaxation. In: Lai, C.-H., Bjørstad, P., Cross, M., Widlund, O. (eds.) Eleventh International Conference of Domain Decomposition Methods (1999). ddm.orgGoogle Scholar
  37. 37.
    Gander, M.J., Halpern, L., Nataf, F.: Optimal Schwarz waveform relaxation for the one dimensional wave equation. SIAM J. Numer. Anal. 41, 1643–1681 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Gander, M.J., Jiang, Y.-L., Li, R.-J.: Parareal Schwarz waveform relaxation methods. In: Widlund, O.B., Keyes, D.E. (eds.) Domain Decomposition Methods in Science and Engineering XX. Lecture Notes in Computational Science and Engineering, vol. 60, pp. 45–56. Springer, Berlin (2013)Google Scholar
  39. 39.
    Gander, M.J., Kwok, F., Mandal, B.: Dirichlet-Neumann and Neumann-Neumann waveform relaxation algorithms for parabolic problems (2015, submitted)Google Scholar
  40. 40.
    Gear, C.W.: Parallel methods for ordinary differential equations. Calcolo 25, 1–20 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Giladi, E., Keller, H.B.: Space time domain decomposition for parabolic problems. Numer. Math. 93, 279–313 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Güttel, S.: A parallel overlapping time-domain decomposition method for ODEs. In: Domain Decomposition Methods in Science and Engineering XX, pp. 459–466. Springer, Berlin (2013)Google Scholar
  43. 43.
    Hackbusch, W.: Parabolic multi-grid methods. In: Glowinski, R. Lions, J.-L. (eds.) Computing Methods in Applied Sciences and Engineering, VI. pp. 189–197. North-Holland, Amsterdam (1984)Google Scholar
  44. 44.
    Hoang, T.-P., Jaffré, J., Japhet, C., Kern, M., Roberts, J.E.: Space-time domain decomposition methods for diffusion problems in mixed formulations. SIAM J. Numer. Anal. 51, 3532–3559 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Horton, G., Vandewalle, S.: A space-time multigrid method for parabolic partial differential equations. SIAM J. Sci. Comput. 16, 848–864 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  46. 46.
    Horton, G., Vandewalle, S., Worley, P.: An algorithm with polylog parallel complexity for solving parabolic partial differential equations, SIAM J. Sci. Comput. 16, 531–541 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Keller, H.B.: Numerical Solution for Two-Point Boundary-Value Problems. Dover Publications Inc, New York (1992)Google Scholar
  48. 48.
    Kiehl, M.: Parallel multiple shooting for the solution of initial value problems. Parallel Comput. 20, 275–295 (1994)MathSciNetCrossRefzbMATHGoogle Scholar
  49. 49.
    Kogge, P.M., Stone, H.S.: A parallel algorithm for the efficient solution of a general class of recurrence equations. IEEE Trans. Comput. 100, 786–793 (1973)MathSciNetCrossRefGoogle Scholar
  50. 50.
    Kwok, F.: Neumann-Neumann waveform relaxation for the time-dependent heat equation. In: Domain Decomposition Methods in Science and Engineering, DD21. Springer, Berlin (2014)Google Scholar
  51. 51.
    Lai, C.-H.: On transformation methods and the induced parallel properties for the temporal domain. In: Magoulès, F. (ed.) Substructuring Techniques and Domain Decomposition Methods, pp. 45–70. Saxe-Coburg Publications, Scotland (2010)CrossRefGoogle Scholar
  52. 52.
    Lelarasmee, E., Ruehli, A.E., Sangiovanni-Vincentelli, A. L.: The waveform relaxation method for time-domain analysis of large scale integrated circuits. IEEE Trans. CAD IC Syst. 1, 131–145 (1982)CrossRefGoogle Scholar
  53. 53.
    Lindelöf, E.: Sur l’application des méthodes d’approximations successives à l’étude des intégrales réelles des équations différentielles ordinaires. J. de Math. Pures et Appl. 10, 117–128 (1894)zbMATHGoogle Scholar
  54. 54.
    Lions, J.-L., Maday, Y., Turinici, G.: A parareal in time discretization of PDEs. C.R. Acad. Sci. Paris, Serie I 332, 661–668 (2001)MathSciNetzbMATHGoogle Scholar
  55. 55.
    Lubich, C., Ostermann, A.: Multi-grid dynamic iteration for parabolic equations. BIT 27, 216–234 (1987)MathSciNetCrossRefzbMATHGoogle Scholar
  56. 56.
    Maday, Y., Rønquist, E.M.: Parallelization in time through tensor-product space–time solvers. C.R. Math. 346, 113–118 (2008)Google Scholar
  57. 57.
    Maday, Y., Turinici, G.: The parareal in time iterative solver: a further direction to parallel implementation. In: Domain Decomposition Methods in Science and Engineering, pp. 441–448. Springer, Berlin (2005)Google Scholar
  58. 58.
    Mandal, B.: A time-dependent Dirichlet-Neumann method for the heat equation. In: Domain Decomposition Methods in Science and Engineering, DD21. Springer, Berlin (2014)Google Scholar
  59. 59.
    Milne, W.E., Wiley, J.: Numerical Solution of Differential Equations. vol. 19(3). Wiley, New York (1953)Google Scholar
  60. 60.
    Minion, M.L.: A hybrid parareal spectral deferred corrections method. Commun. Appl. Math. Comput. Sci. 5, 265–301 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  61. 61.
    Miranker, W.L., Liniger, W.: Parallel methods for the numerical integration of ordinary differential equations. Math. Comput. 91, 303–320 (1967)MathSciNetCrossRefGoogle Scholar
  62. 62.
    Nataf, F., Rogier. F.: Factorization of the convection-diffusion operator and the Schwarz algorithm. M 3 AS 5, 67–93 (1995)Google Scholar
  63. 63.
    Neumüller, M.: Space-time methods: fast solvers and applications, Ph.D. thesis, University of Graz (2013)Google Scholar
  64. 64.
    Nievergelt, J.: Parallel methods for integrating ordinary differential equations. Commun. ACM 7, 731–733 (1964)MathSciNetCrossRefzbMATHGoogle Scholar
  65. 65.
    Picard, E.: Sur l’application des méthodes d’approximations successives à l’étude de certaines équations différentielles ordinaires. J. de Math. Pures et Appl. 9, 217–271 (1893)zbMATHGoogle Scholar
  66. 66.
    Ruprecht, D., Krause, R.: Explicit parallel-in-time integration of a linear acoustic-advection system. Comput. Fluids 59, 72–83 (2012)MathSciNetCrossRefGoogle Scholar
  67. 67.
    Saha, P., Stadel, J., Tremaine, S.: A parallel integration method for solar system dynamics. Astron. J. 114, 409–415 (1997)CrossRefGoogle Scholar
  68. 68.
    Saltz, J.H., Naik, V.K.: Towards developing robust algorithms for solving partial differential equations on MIMD machines. Parallel Comput. 6, 19–44 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  69. 69.
    Sameh, A.H., Brent, R.P.: Solving triangular systems on a parallel computer. SIAM J. Numer. Anal. 14, 1101–1113 (1977)MathSciNetCrossRefzbMATHGoogle Scholar
  70. 70.
    Schwarz, H.A.: Über einen Grenzübergang durch alternierendes Verfahren. Vierteljahrsschrift der Naturforschenden Gesellschaft in Zürich 15, 272–286 (1870)Google Scholar
  71. 71.
    Shampine, L., Watts, H.: Block implicit one-step methods. Math. Comput. 23, 731–740 (1969)MathSciNetCrossRefzbMATHGoogle Scholar
  72. 72.
    Sheen, D., Sloan, I., Thomée, V.: A parallel method for time-discretization of parabolic problems based on contour integral representation and quadrature. Math. Comput. Am. Math. Soc. 69, 177–195 (1999)CrossRefGoogle Scholar
  73. 73.
    Sheen, D., Sloan, I.H., Thomée, V.: A parallel method for time discretization of parabolic equations based on Laplace transformation and quadrature. IMA J. Numer. Anal. 23, 269–299 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  74. 74.
    Simoens, J., Vandewalle, S.: Waveform relaxation with fast direct methods as preconditioner. SIAM J. Sci. Comput. 21, 1755–1773 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  75. 75.
    Speck, R., Ruprecht, D., Krause, R., Emmett, M., Minion, M., Winkel, M., Gibbon, P.: A massively space-time parallel n-body solver. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, Salt Lake City, p. 92. IEEE Computer Society Press, Los Alamitos (2012)Google Scholar
  76. 76.
    Speck, R., Ruprecht, D., Emmett, M., Minion, M., Bolten, M., Krause, R.: A multi-level spectral deferred correction method. arXiv:1307.1312 (2013, arXiv preprint)Google Scholar
  77. 77.
    Thomée, V.: A high order parallel method for time discretization of parabolic type equations based on Laplace transformation and quadrature. Int. J. Numer. Anal. Model. 2, 121–139 (2005)Google Scholar
  78. 78.
    Vandewalle, S., Van de Velde, E.: Space-time concurrent multigrid waveform relaxation. Ann. Numer. Math. 1, 347–363 (1994)MathSciNetzbMATHGoogle Scholar
  79. 79.
    Womble, D.E.: A time-stepping algorithm for parallel computers. SIAM J. Sci. Stat. Comput. 11, 824–837 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  80. 80.
    Worley, P.: Parallelizing across time when solving time-dependent partial differential equations. In: Sorensen, D. (ed.) Proceedings of 5th SIAM Conference on Parallel Processing for Scientific Computing. SIAM, Houston (1991)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Section of MathematicsUniversity of GenevaGeneva 4Switzerland

Personalised recommendations