Skip to main content
SpringerLink
Log in

A space-time parallel algorithm with adaptive mesh refinement for computational fluid dynamics

  • PinT 2019
  • Published:
Computing and Visualization in Science

Abstract

This paper describes a space-time parallel algorithm with space-time adaptive mesh refinement (AMR). AMR with subcycling is added to multigrid reduction-in-time (MGRIT) in order to provide solution efficient adaptive grids with a reduction in work performed on coarser grids. This algorithm is achieved by integrating two software libraries: XBraid (Parallel time integration with multigrid. https://computation.llnl.gov/projects/parallel-timeintegration-multigrid) and Chombo (Chombo software package for AMR applications—design document, 2014). The former is a parallel time integration library using multigrid and the latter is a massively parallel structured AMR library. Employing this adaptive space-time parallel algorithm is Chord (Comput Fluids 123:202–217, 2015), a computational fluid dynamics (CFD) application code for solving compressible fluid dynamics problems. For the same solution accuracy, speedups are demonstrated from the use of space-time parallelization over the time-sequential integration on Couette flow and Stokes’ second problem. On a transient Couette flow case, at least a \(1.5\times \) speedup is achieved, and with a time periodic problem, a speedup of up to \(13.7\times \) over the time-sequential case is obtained. In both cases, the speedup is achieved by adding processors and exploring additional parallelization in time. The numerical experiments show the algorithm is promising for CFD applications that can take advantage of the time parallelism. Future work will focus on improving the parallel performance and providing more tests with complex fluid dynamics to demonstrate the full potential of the algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

References

  1. AMReX: Software framework for block structured AMR. https://amrex-codes.github.io/amrex/. Accessed 3 Oct 2018

  2. CASTRO: An adaptive mesh, astrophysical radiation hydrodynamics simulation code. https://amrex-astro.github.io/Castro/. Accessed 16 May 2019

  3. Overture: An object-oriented toolkit for solving partial differential equations in complex geometry. https://www.overtureframework.org/. Accessed 16 May 2019

  4. p4est: Parallel adaptive mesh refinement on forests of octrees. http://www.p4est.org/. Accessed 16 May 2019

  5. SAMRAI: Structured adaptive mesh refinement application infrastructure. https://computation.llnl.gov/projects/samrai. Accessed 3 Oct 2018

  6. XBraid: Parallel time integration with multigrid. https://computation.llnl.gov/projects/parallel-time-integration-multigrid. Accessed 24 Sep 2018

  7. Adams, M., Colella, P., Graves, D.T., Johnson, J.N., Johansen, H.S., Keen, N.D., Ligocki, T.J., Martin, D.F., McCorquodale, P.W., Modiano, D., Schwartz, P.O., Sternberg, T.D., Van Straalen, B.: Chombo software package for AMR applications - design document. Technical Report LBNL-6616E, Lawrence Berkeley National Laboratory (2014)

  8. Anderson, J.: Modern Compressible Flow: With Historical Perspective, 3rd edn. McGraw-Hill Education, Boston (2002)

    Google Scholar 

  9. Batchelor, G.K.: An Introduction to Fluid Dynamics. Cambridge University Press, Cambridge (2000)

    Book  Google Scholar 

  10. Brandt, A.: Multi-level adaptive computations in fluid dynamics. In: Technical Report AIAA-79-1455. AIAA, Williamsburg, VA (1979)

  11. Briggs, W.L., Henson, V.E., McCormick, S.F.: A Multigrid Tutorial, 2nd edn. Society for Industrial and Applied Mathematics, Philadelphia (2000)

    Book  Google Scholar 

  12. Christopher, J., Gao, X., Guzik, S.M., Falgout, R., Schroder, J.: Parallel in time for a fully space-time adaptive mesh refinement algorithm. In: AIAA Scitech 2020 Forum. American Institute of Aeronautics and Astronautics, Orlando, FL (2020). https://doi.org/10.2514/6.2020-0340

  13. Christopher, J., Gao, X., Guzik, S.M., Falgout, R.D., Schroder, J.B.: Space-time adaptivity with multigrid reduction in time for the compressible navier-stokes equations. In: 19th Copper Mountain Conference On Multigrid Methods, pp. 10 (2019)

  14. Colella, P., Dorr, M.R., Hittinger, J.A.F., Martin, D.F.: High-order finite-volume methods in mapped coordinates. JCP 230, 2952–2976 (2011)

    MathSciNet  MATH  Google Scholar 

  15. Colella, P., Sekora, M.D.: A limiter for PPM that preserves accuracy at smooth extrema. J. Comput. Phys. 227(15), 7069–7076 (2008). https://doi.org/10.1016/j.jcp.2008.03.034

    Article  MathSciNet  MATH  Google Scholar 

  16. Day, M.S., Bell, J.B.: Numerical simulation of laminar reacting flows with complex chemistry. Combust. Theory Model. 4, 535–556 (2000)

    Article  Google Scholar 

  17. De Sterck, H., Falgout, R.D., Howse, A.J.M., MacLachlan, S.P., Schroder, J.B.: Parallel-in-time multigrid with adaptive spatial coarsening for the linear advection and inviscid burgers equations. SIAM J. Sci. Comput. 41, A538 (2017)

    MathSciNet  MATH  Google Scholar 

  18. Dubey, A., Almgren, A., Bell, J., Berzins, M., Brandt, S., Bryan, G., Colella, P., Graves, D., Lijewski, M., Löffler, F., O’Shea, B., Schnetter, E., Van Straalen, B., Weide, K.: A survey of high level frameworks in block-structured adaptive mesh refinement packages. J. Parallel Distrib. Comput. 74(12), 3217–3227 (2014). https://doi.org/10.1016/j.jpdc.2014.07.001

    Article  Google Scholar 

  19. Falgout, R.D., Friedhoff, S., Kolev, T.V., MacLachlan, S.P., Schroder, J.B.: Parallel time integration with multigrid. SIAM J. Sci. Comput 36, C635–C661 (2014)

    Article  MathSciNet  Google Scholar 

  20. Falgout, R.D., Friedhoff, S., Kolev, T.V., MacLachlan, S.P., Schroder, J.B.: Parallel time integration with multigrid. SIAM J. Sci. Comput. 36(6), C635–C661 (2014)

    Article  MathSciNet  Google Scholar 

  21. Falgout, R.D., Friedhoff, S., Vandewalle, S., Kolev, T.V., MacLachlan, S.P., Schroder, J.B.: Multigrid methods with space-time concurrency. Comput. Vis. Sci. 18(4–5), 123–143 (2017)

    Article  MathSciNet  Google Scholar 

  22. Falgout, R.D., Katz, A., Kolev, T.V., Schroder, J., Wissink, A.M., Yang, U.: Parallel time integration with multigrid reduction for a compressible fluid dynamics application. Technical report Lawrence Livermore National Lab (2014)

  23. Falgout, R.D., Katz, A., Kolev, T.V., Schroder, J.B., Wissink, A., Yang, U.M.: Parallel time integration with multigrid reduction for a compressible fluid dynamics application. Technical Report LLNL-JRNL-663416, Lawrence Livermore National Laboratory (2015)

  24. Falgout, R.D., Manteuffel, T.A., Schroder, J.B., Southworth, B.: Parallel-in-time for moving meshes. Technical Report LLNL-TR-681918, Lawrence Livermore National Laboratory (2016)

  25. Friedhoff, S., Hahne, J., Kulchytska-Ruchka, I., Schöps, S.: Exploring parallel-in-time approaches for eddy current problems. arXiv preprint arXiv:1810.13263 (2018)

  26. Fryxell, B., Olson, K., Ricker, P., Timmes, F.X., Zingale, M., Lamb, D.Q., MacNeice, P., Rosner, R., Truran, J.W., Tufo, H.: FLASH: an adaptive mesh hydrodynamics code for modeling astrophysical thermonuclear flashes. ApJS 131(1), 273 (2000). https://doi.org/10.1086/317361

    Article  Google Scholar 

  27. Gander, M.J.: 50 years of time parallel time integration. In: Carraro, T., Geiger, M., Körkel, S., Rannacher, R. (eds.) Multiple Shooting and Time Domain Decomposition, pp. 69–114. Springer, Berlin (2015)

    Chapter  Google Scholar 

  28. Gander, M.J., Neumüller, M.: Analysis of a new space-time parallel multigrid algorithm for parabolic problems. SIAM J. Sci. Comput. 38(4), A2173–A2208 (2016)

    Article  MathSciNet  Google Scholar 

  29. Gao, X.: A parallel solution-adaptive method for turbulent non-premixed combusting flows. Ph.D. thesis, University of Toronto (2008)

  30. Gao, X., Guzik, S.M., Olschanowsky, C.: A high-performance fourth-order finite-volume algorithm for compressible viscous flows on structured grids. Submitted to AIAA J

  31. Gao, X., Guzik, S.M.J.: A fourth-order scheme for the compressible navier-stokes equations. In: AIAA 2015-0298, 53rd AIAA Aerospace Sciences Meeting (2015)

  32. Gao, X., Guzik, S.M.J., Colella, P.: Fourth order boundary treatment for viscous fluxes on cartesian grid finite-volume methods. In: AIAA 2014-1277, 52nd AIAA Aerospace Sciences Meeting (2014)

  33. Gao, X., Northrup, S., Groth, C.: Parallel solution-adaptive method for two-dimensional non-premixed combusting flows. Progress Comput. Fluid. Dyn. Int. J. 11(2), 76–95 (2011)

    Article  MathSciNet  Google Scholar 

  34. Gao, X., Owen, L.D., Guzik, S.M.: A parallel adaptive numerical method with generalized curvilinear coordinate transformation for compressible navier-stokes equations. Int. J. Numer. Meth. Fluids, Accepted

  35. Gao, X., Owen, L.D., Guzik, S.M.: A high-order finite-volume method for combustion. In: AIAA 2016-1808, 54th AIAA Aerospace Sciences Meeting (2016)

  36. Garnier, E., Adams, N., Sagaut, P.: Large Eddy Simulation for Compressible Flows. Scientific Computation. Springer Netherlands (2009). https://doi.org/10.1007/978-90-481-2819-8

  37. Georgiadis, N.J., Rizzetta, D.P., Fureby, C.: Large-eddy simulation: Current capabilities, recommended practices, and future research pp. 27 (2009)

  38. Groth, C., De Zeeuw, D., Powell, K., Gombosi, T., Stout, Q.: A parallel solution-adaptive scheme for ideal magnetohydrodynamics. In: 14th Computational Fluid Dynamics Conference. American Institute of Aeronautics and Astronautics, Norfolk, VA, U.S.A. (1999). https://doi.org/10.2514/6.1999-3273

  39. Günther, S., Gauger, N.R., Schroder, J.B.: A non-intrusive parallel-in-time adjoint solver with the XBraid library. Comput. Vis. Sci. 19(3–4), 85–95 (2018)

    Article  MathSciNet  Google Scholar 

  40. Günther, S., Gauger, N.R., Schroder, J.B.: A non-intrusive parallel-in-time approach for simultaneous optimization with unsteady PDEs. Optim. Methods Softw. 34, 1306 (2018)

    Article  MathSciNet  Google Scholar 

  41. Günther, S., Ruthotto, L., Schroder, J., Cyr, E., Gauger, N.: Layer-parallel training of deep residual neural networks. SIAM Journal on Data Science (2019 (submitted)). ArXiv preprint arXiv:1812.04352

  42. Guzik, S.M., Gao, X., Olschanowsky, C.: A high-performance finite-volume algorithm for solving partial differential equations governing compressible viscous flows on structured grids. Comput. Math Appl. 72, 2098–2118 (2016)

    Article  MathSciNet  Google Scholar 

  43. Guzik, S.M., Gao, X., Owen, L.D., McCorquodale, P., Colella, P.: A freestream-preserving fourth-order finite-volume method in mapped coordinates with adaptive-mesh refinement. Comput. Fluids 123, 202–217 (2015)

    Article  MathSciNet  Google Scholar 

  44. Guzik, S.M.J., McCorquodale, P., Colella, P.: A freestream-preserving high-order finite-volume method for mapped grids with adaptive-mesh refinement. In: AIAA 2012-0574, 50th AIAA Aerospace Sciences Meeting (2012)

  45. Hessenthaler, A., Nordsletten, D., Röhrle, O., Schroder, J., Falgout, R.: Convergence of the multigrid-reduction-in-time algorithm for the linear elasticity equations. Numer. Linear Algebra Appl. 25(3), e2155 (2018)

    Article  MathSciNet  Google Scholar 

  46. Horton, G., Vandewalle, S.: A space-time multigrid method for parabolic partial differential equations. SIAM J. Sci. Comput. 16(4), 848–864 (1995)

    Article  MathSciNet  Google Scholar 

  47. Howse, A.J., Sterck, H.D., Falgout, R.D., MacLachlan, S., Schroder, J.: Parallel-in-time multigrid with adaptive spatial coarsening for the linear advection and inviscid burgers equations. SIAM J. Sci. Comput. 41(1), A538–A565 (2019)

    Article  MathSciNet  Google Scholar 

  48. Langer, U., Matculevich, S., Repin, S.: Guaranteed error bounds and local indicators for adaptive solvers using stabilised space-time IgA approximations to parabolic problems. Comput. Math. Appl. 78(8), 2641–2671 (2019). https://doi.org/10.1016/j.camwa.2019.04.009

    Article  MathSciNet  MATH  Google Scholar 

  49. Lecouvez, M., Falgout, R.D., Woodward, C.S., Top, P.: A parallel multigrid reduction in time method for power systems. In: Power and Energy Society General Meeting (PESGM) pp. 1–5 (2016)

  50. Lohner, R., Othmer, C., Mrosek, M., Figueroa, A., Degro, A.: Overnight industrial LES for external aerodynamics. In: AIAA Scitech 2020 Forum. American Institute of Aeronautics and Astronautics. https://doi.org/10.2514/6.2020-2031

  51. Margolin, L.G., Rider, W.J., Grinstein, F.F.: Modeling turbulent flow with implicit LES. J. Turbul. 7, 1–27 (2006). https://doi.org/10.1080/14685240500331595

    Article  MathSciNet  MATH  Google Scholar 

  52. Martin, D., Colella, P., Anghel, M., Alexander, F.: Adaptive mesh refinement for multiscale nonequilibrium physics. Comput. Sci. Eng. 7(3), 24–31 (2005). https://doi.org/10.1109/MCSE.2005.45

    Article  Google Scholar 

  53. Martin, D.F., Colella, P., Graves, D.: A cell-centered adaptive projection method for the incompressible Navier-Stokes equations in three dimensions. J. Comput. Phys. 227(3), 1863–1886 (2008). https://doi.org/10.1016/j.jcp.2007.09.032

    Article  MathSciNet  MATH  Google Scholar 

  54. McCorquodale, P., Colella, P.: A high-order finite-volume method for conservation laws on locally refined grids. Commun. App. Math. Comput. Sci. 6(1), 1–25 (2011)

    Article  MathSciNet  Google Scholar 

  55. Miniati, F., Colella, P.: Block structured adaptive mesh and time refinement for hybrid, hyperbolic + N-body systems. J. Comput. Phys. 227(1), 400–430 (2007). https://doi.org/10.1016/j.jcp.2007.07.035

    Article  MathSciNet  MATH  Google Scholar 

  56. Owen, L., Guzik, S., Gao, X.: A high-order adaptive algorithm for multispecies gaseous flows on mapped domains. Comput. Fluids 170, 249–260 (2018). https://doi.org/10.1016/j.compfluid.2018.05.010

    Article  MathSciNet  MATH  Google Scholar 

  57. Owen, L.D., Gao, X., Guzik, S.M.: Techniques for improving monotonicity in a fourth-order finite-volume algorithm solving shocks and detonations. J. Comput. Phys. (2020). https://doi.org/10.1016/j.jcp.2020.109515

    Article  MathSciNet  MATH  Google Scholar 

  58. Owen, L.D., Guzik, S.M., , Gao, X.: High-order CFD modeling of multispecies flows. In: Fall Meeting of the Western States Section of the Combustion Institute (2015). WSSCI 2015-134IE-0030

  59. Owen, L.D., Guzik, S.M., Gao, X.: A fourth-order finite-volume algorithm for compressible flow with chemical reactions on mapped grids (AIAA 2017-4498) (2017)

  60. Sagaut, P.: Large Eddy Simulation for Incompressible Flows: An Introduction, 3 edn. Scientific Computation. Springer, Berlin (2006). https://doi.org/10.1007/b137536

  61. Schroder, J.B., Lecouvez, M., Falgout, R.D., Woodward, C.S., Top, P.: Parallel-in-time solution of power systems with scheduled events. Power and Energy Society General Meeting (PESGM) pp. 1–5 (2018)

  62. Simon, H.D. (ed.): Parallel Computational Fluid Dynamics: Implementations and Results. MIT Press, Cambridge (1992)

    Google Scholar 

  63. Steinbach, O.: Space-time finite element methods for parabolic problems. Comput. Methods Appl. Math. (2015). https://doi.org/10.1515/cmam-2015-0026

    Article  MathSciNet  MATH  Google Scholar 

  64. Steinbach, O., Yang, H.: Comparison of algebraic multigrid methods for an adaptive space-time finite-element discretization of the heat equation in 3D and 4D: Comparison of AMG for adaptive space-time FEM in 3D and 4D. Numer. Linear Algebra Appl. 25(3), e2143 (2018). https://doi.org/10.1002/nla.2143

    Article  MATH  Google Scholar 

  65. Sterck, H.D., Falgout, R.D., Friedhoff, S., Krzysik, O.A., MacLachlan, S.P.: Optimizing mgrit and parareal coarse-grid operators for linear advection. arXiv:1910.03726. (2019)

  66. Van Straalen, B., Colella, P., Graves, D.T., Keen, N.: Petascale block-structured AMR applications without distributed meta-data. In: Proceedings, Part II. Lecture Notes in Computer Science. Euro-Par 2011 Parallel Processing—17th International Conference, Springer (2011). ISBN 978-3-642-23396-8

  67. Yang, Z.: Large-eddy simulation: past, present and the future. Chin. J. Aeronaut. 28(1), 11–24 (2015)

    Article  Google Scholar 

  68. Yue, X., Shu, S., Xu, X., Bu, W., Pan, K.: Parallel-in-time with fully finite element multigrid for 2-d space-fractional diffusion equations. arXiv preprint arXiv:1805.06688 (2018)

  69. Zhang, Q., Johansen, H., Colella, P.: A fourth-order accurate finite-volume method with structured adaptive mesh refinement for solving the advection-diffusion equation. SIAM J. Sci. Comput. 34(2), B179–B201 (2012). https://doi.org/10.1137/110820105

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This document was prepared as an account of work sponsored by an agency of the United States government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes.

Author information

Authors and Affiliations

  1. CFD & Propulsion Laboratory, Colorado State University, Fort Collins, USA

    Joshua Christopher, Stephen M. Guzik & Xinfeng Gao

  2. Center for Applied Scientific Computing, Lawrence Livermore National Laboratory, Livermore, USA

    Robert D. Falgout

  3. Department of Mathematics and Statistics, University of New Mexico, Albuquerque, USA

    Jacob B. Schroder

Authors
  1. Joshua Christopher
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert D. Falgout
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jacob B. Schroder
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stephen M. Guzik
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xinfeng Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joshua Christopher.

Additional information

Communicated by Robert Speck.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 (LLNL-JRNL-798697)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Christopher, J., Falgout, R.D., Schroder, J.B. et al. A space-time parallel algorithm with adaptive mesh refinement for computational fluid dynamics. Comput. Visual Sci. 23, 13 (2020). https://doi.org/10.1007/s00791-020-00334-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00791-020-00334-1

Keywords

Search

Navigation