Journal of Scientific Computing

, Volume 61, Issue 2, pp 258–280 | Cite as

Parallel Domain Decomposition Methods with Mixed Order Discretization for Fully Implicit Solution of Tracer Transport Problems on the Cubed-Sphere

  • Haijian Yang
  • Chao Yang
  • Xiao-Chuan CaiEmail author


In this paper, a fully implicit finite volume Eulerian scheme and a corresponding scalable parallel solver are developed for some tracer transport problems on the cubed-sphere. To efficiently solve the large sparse linear system at each time step on parallel computers, we introduce a Schwarz preconditioned Krylov subspace method using two discretizations. More precisely speaking, the higher order method is used for the residual calculation and the lower order method is used for the construction of the preconditioner. The matrices from the two discretizations have similar sparsity pattern and eigenvalue distributions, but the matrix from the lower order method is a lot sparser, as a result, excellent scalability results (in total computing time and the number of iterations) are obtained. Even though Schwarz preconditioner is originally designed for elliptic problems, our experiments indicate clearly that the method scales well for this class of purely hyperbolic problems. In addition, we show numerically that the proposed method is highly scalable in terms of both strong and weak scalabilities on a supercomputer with thousands of processors.


Transport equation Cubed-sphere Fully implicit method  Domain decomposition Parallel scalability 



The authors would like to express their appreciations to the anonymous reviewers for the invaluable comments that greatly improved the quality of the manuscript. This work was supported in part by NSF grant CCF-1216314 and DOE grant DE-SC0001774. H. Yang was also supported in part by NSFC grants 91330111, 11201137 and 11272352. C. Yang was also supported in part by NSFC grants 61170075 and 91130023


  1. 1.
    Balay, S., Buschelman, K., Gropp, W.D., Kaushik, D., Knepley, M., McInnes, L.C., Smith, B.F., Zhang, H.: PETSc Users Manual. Argonne National Laboratory (2012)Google Scholar
  2. 2.
    Brown, P.N., Shumaker, D.E., Woodward, C.S.: Fully implicit solution of large-scale non-equilibrium radiation diffusion with high order time integration. J. Comput. Phys. 204, 760–783 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Cai, X.-C., Gropp, W.D., Keyes, D.E., Melvin, R.G., Young, D.P.: Parallel Newton-Krylov-Schwarz algorithms for the transonic full potential equation. SIAM J. Sci. Comput. 19, 246–265 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Cai, X.-C., Sarkis, M.: A restricted additive Schwarz preconditioner for general sparse linear systems. SIAM J. Sci. Comput. 21, 792–797 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Chen, C., Xiao, F.: Shallow water model on cubed-sphere by multi-moment finite volume method. J. Comput. Phys. 227, 5019–5044 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Evans, K.J., Knoll, D.A.: Temporal accuracy of phase change convection simulations using the JFNK-SIMPLE algorithm. Int. J. Num. Meth. Fluids. 55, 637–655 (2007)CrossRefzbMATHGoogle Scholar
  7. 7.
    Erath, C., Lauritzen, P.H., Garcia, J.H., Tufo, H.M.: Integrating a scalable and efficient semi-Lagrangian multi-tracer transport scheme in HOMME. Proc. Comput. Sci. 9, 994–1003 (2012)CrossRefGoogle Scholar
  8. 8.
    Harris, L.M., Lauritzen, P.H., Mittal, R.: A flux-form version of the conservative semi-Lagrangian multi-tracer transport scheme (CSLAM) on the cubed-sphere grid. J. Comput. Phys. 230, 1215–1237 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Jacobson, M.Z.: Fundamentals of Atmospheric Modeling. Cambridge University Press, New York (1999)Google Scholar
  10. 10.
    Knoll, D.A., Chacon, L., Margolin, L.G., Mousseau, V.A.: On balanced approximations for time integration of multiple time scale systems. J. Comput. Phys. 185, 583–611 (2003)CrossRefzbMATHGoogle Scholar
  11. 11.
    Lauritzen, P.H., Nair, R.D., Ullrich, P.A.: A conservative semi-Lagrangian multi-tracer transport scheme (CSLAM) on the cubed-sphere grid. J. Comput. Phys. 229, 1401–1424 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Lauritzen, P.H., Skamarock, W.C.: Test-case suite for 2D passive tracer transport: a proposal for the NCAR transport workshop. March (2011)Google Scholar
  13. 13.
    Lauritzen, P.H., Jablonowski, C., Taylor, M., Nair, R.: Numerical Techniques for Global Atmospheric Models. Lecture Notes in Computational Science and Engineering. Springer, Berlin (2011)CrossRefGoogle Scholar
  14. 14.
    Lauritzen, P.H., Ullrich, P.A., Nair, R.D.: Atmospheric transport schemes: desirable properties and a semi-Lagrangian view on finite-volume discretizations. In: Lecture Notes in Computational Science and Engineering (Tutorials), vol. 80, Springer, (2011)Google Scholar
  15. 15.
    Lauritzen, P.H., Skamarock, W.C., Prather, M.J., Taylor, M.A.: A standard test case suite for two-dimensional linear transport on the sphere. Geosci. Model Dev. Discuss. 5, 189–228 (2012)CrossRefGoogle Scholar
  16. 16.
    Lauritzen, P.H., Thuburn, J.: Evaluating advection/transport schemes using interrelated tracers, scatter plots and numerical mixing diagnostics. Q. J. Roy. Meteor. Soc. 138, 906–918 (2012)CrossRefGoogle Scholar
  17. 17.
    Nair, R.D., Thomas, S.J., Loft, R.D.: A discontinuous Galerkin global shallow water model. Mon. Weather Rev. 133, 876–888 (2005)CrossRefGoogle Scholar
  18. 18.
    Nair, R.D., Lauritzen, P.H.: A class of deformational-flow test cases for linear transport problems on the sphere. J. Comput. Phys. 229, 8868–8887 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Putman, W.M., Lin, S.-J.: Finite-volume transport on various cubed-sphere grids. J. Comput. Phys. 227, 55–78 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Rancic, M.R., Purser, J., Mesinger, F.: A global-shallow water model using an expanded spherical cube: Gnomonic versus conformal coordinates. Q. J. Roy. Meteor. Soc. 122, 959–982 (1996)CrossRefGoogle Scholar
  21. 21.
    Ronchi, C., Iacono, R., Paolucci, P.: The cubed sphere: a new method for the solution of partial differential equations in spherical geometry. J. Comput. Phys. 124, 93–114 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Sadourny, R., Arakawa, A., Mintz, Y.: Integration of the nondivergent barotropic vorticity equation with an icosahedralhexagonal grid for the sphere. Mon. Weather Rev. 96, 351–356 (1968)CrossRefGoogle Scholar
  23. 23.
    Sadourny, R.: Conservative finite-difference approximations of the primitive equations on quasi-uniform spherical grids. Mon. Weather Rev. 100, 211–224 (1972)CrossRefGoogle Scholar
  24. 24.
    Saad, Y.: Iterative Methods for Sparse Linear Systems. SIAM, Philadelphia (2003)CrossRefzbMATHGoogle Scholar
  25. 25.
    Shadid, J.N., Tuminaro, R.S., Devine, K.D., Hennigan, G.L., Lin, P.T.: Performance of fully coupled domain decomposition preconditioners for finite element transport/reaction simulations. J. Comput. Phys. 205, 24–47 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Smith, B., Bjørstad, P., Gropp, W.: Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations. Cambridge University Press, Cambridge (1996)zbMATHGoogle Scholar
  27. 27.
    Toselli, A., Widlund, O.: Domain Decomposition Methods-Algorithms and Theory. Springer, Berlin (2005)zbMATHGoogle Scholar
  28. 28.
    Van Albada, G.D., van Leer, B., Roberts, W.W.: A comparative study of computational methods in cosmic gas dynamics. Astron. Astrophys. 108, 95–103 (1982)Google Scholar
  29. 29.
    White III, J.B., Dongarra, J.J.: High-performance high-resolution semi-Lagrangian tracer transport on a sphere. J. Comput. Phys. 230, 6778–6799 (2011)CrossRefzbMATHGoogle Scholar
  30. 30.
    Williamson, D.L., Drake, J.B., Hack, J.J., Jakob, R., Swarztrauber, P.N.: A standard test set for numerical approximations to the shallow water equations in spherical geometry. J. Comput. Phys. 102, 211–224 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Wu, Y., Cai, X.-C., Keyes, D.E.: Additive Schwarz methods for hyperbolic equations. In: Mandel, J., Farhat, C., Cai, X.-C. (eds.) Proceedings of the 10th International Conference on Domain Decomposition Methods, AMS, pp. 513–521 (1998)Google Scholar
  32. 32.
    Yang, C., Cao, J., Cai, X.-C.: A fully implicit domain decomposition algorithm for shallow water equations on the cubed-sphere. SIAM J. Sci. Comput. 32, 418–438 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Yang, C., Cai, X.-C.: Parallel multilevel methods for implicit solution of shallow water equations with nonsmooth topography on cubed-sphere. J. Comput. Phys. 230, 2523–2539 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Yang, C., Cai, X.-C.: A scalable fully implicit compressible Euler solver for mesoscale nonhydrostatic simulation of atmospheric flows. SIAM J. Sci. Comput. To appearGoogle Scholar
  35. 35.
    Yang, H., Cai, X.-C.: Parallel two-grid semismooth Newton-Krylov-Schwarz method for nonlinear complementarity problems. J. Sci. Comput. 47, 258–280 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Yang, H., Prudencio, E., Cai, X.-C.: Fully implicit Lagrange-Newton-Krylov-Schwarz algorithms for boundary control of unsteady incompressible flows. Int. J. Numer. Meth. Eng. 91, 644–665 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Zhang, J., Wang, L.L., Rong, Z.: A prolate-element method for nonlinear PDEs on the sphere. J. Sci. Comput. 47, 73–92 (2011)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.College of Mathematics and EconometricsHunan UniversityHunanPeople’s Republic of China
  2. 2.Laboratory of Parallel Software and Computational Science, Institute of SoftwareChinese Academy of SciencesBeijingPeople’s Republic of China
  3. 3.State Key Laboratory of Computer Science, Institute of SoftwareChinese Academy of SciencesBeijingPeople’s Republic of China
  4. 4.Department of Computer ScienceUniversity of Colorado BoulderBoulderUSA

Personalised recommendations