Journal of Scientific Computing

, Volume 63, Issue 1, pp 23–48 | Cite as

Efficient GPU-Implementation of Adaptive Mesh Refinement for the Shallow-Water Equations

  • Martin L. Sætra
  • André R. Brodtkorb
  • Knut-Andreas Lie


The shallow-water equations model hydrostatic flow below a free surface for cases in which the ratio between the vertical and horizontal length scales is small and are used to describe waves in lakes, rivers, oceans, and the atmosphere. The equations admit discontinuous solutions, and numerical solutions are typically computed using high-resolution schemes. For many practical problems, there is a need to increase the grid resolution locally to capture complicated structures or steep gradients in the solution. An efficient method to this end is adaptive mesh refinement (AMR), which recursively refines the grid in parts of the domain and adaptively updates the refinement as the simulation progresses. Several authors have demonstrated that the explicit stencil computations of high-resolution schemes map particularly well to many-core architectures seen in hardware accelerators such as graphics processing units (GPUs). Herein, we present the first full GPU-implementation of a block-based AMR method for the second-order Kurganov–Petrova central scheme. We discuss implementation details, potential pitfalls, and key insights, and present a series of performance and accuracy tests. Although it is only presented for a particular case herein, we believe our approach to GPU-implementation of AMR is transferable to other hyperbolic conservation laws, numerical schemes, and architectures similar to the GPU.


Adaptive mesh refinement (AMR) GPU computing Shallow-water simulation Conservation law Domain decomposition 

Mathematics Subject Classification

35L65 65M08 65M50 76B15 68N19 



This work is supported in part by the Research Council of Norway through Grant No. 180023 (Parallel3D), and in part by the Centre of Mathematics for Applications at the University of Oslo (Sætra and Lie).


  1. 1.
    Berger, M., LeVeque, R.: Adaptive mesh refinement for two-dimensional hyperbolic systems and the AMRCLAW software. SIAM J. Numer. Anal. 35, 2298–2316 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  2. 2.
    Berger, M.J., Colella, P.: Local adaptive mesh refinement for shock hydrodynamics. J. Comput. Phys. 82, 64–84 (1989). doi: 10.1016/0021-9991(89)90035-1 CrossRefzbMATHGoogle Scholar
  3. 3.
    Berger, M.J., Oliger, J.: Adaptive mesh refinement for hyperbolic partial differential equations. J. Comput. Phys. 53(3), 484–512 (1984). doi: 10.1016/0021-9991(84)90073-1 CrossRefzbMATHMathSciNetGoogle Scholar
  4. 4.
    Brodtkorb, A., Dyken, C., Hagen, T., Hjelmervik, J., Storaasli, O.: State-of-the-art in heterogeneous computing. J. Sci. Program. 18(1), 1–33 (2010)Google Scholar
  5. 5.
    Brodtkorb, A.R., Hagen, T.R., Lie, K.A., Natvig, J.R.: Simulation and visualization of the Saint-Venant system using GPUs. Comput. Vis. Sci. 13(7), 341–353 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  6. 6.
    Brodtkorb, A.R., Hagen, T.R., Sætra, M.L.: Graphics processing unit (GPU) programming strategies and trends in GPU computing. J. Parallel Distrib. Comput. 73(1), 4–13 (2013). doi: 10.1016/j.jpdc.2012.04.003 CrossRefGoogle Scholar
  7. 7.
    Brodtkorb, A.R., Sætra, M.L.: Explicit shallow water simulations on GPUs: guidelines and best practices. In: XIX International Conference on Water Resources, CMWR 2012, June 17–22, 2012. University of Illinois at Urbana-Champaign (2012)Google Scholar
  8. 8.
    Brodtkorb, A.R., Sætra, M.L., Altinakar, M.: Efficient shallow water simulations on GPUs: implementation, visualization, verification, and validation. Comput. Fluids 55, 1–12 (2012). doi: 10.1016/j.compfluid.2011.10.012 CrossRefzbMATHMathSciNetGoogle Scholar
  9. 9.
    Burstedde, C., Ghattas, O., Gurnis, M., Isaac, T., Stadler, G., Warburton, T., Wilcox, L.: Extreme-scale AMR. In: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–12. IEEE Computer Society (2010)Google Scholar
  10. 10.
    Colella, P., Graves, D.T., Johnson, J.N., Johansen, H.S., Keen, N.D., Ligocki, T.J., Martin, D.F., McCorquodale, P.W., Modiano, D., Schwartz, P.O., Sternberg, T.D., Van Straalen, B.: Chombo software package for AMR applications design document. Technical report, Lawrence Berkeley National Laboratory (2012)Google Scholar
  11. 11.
    Delestre, O., Lucas, C., Ksinant, P.A., Darboux, F., Laguerre, C., Vo, T.N.T., James, F., Cordier, S.: SWASHES: a compilation of shallow water analytic solutions for hydraulic and environmental studies. Int. J. Numer. Methods Fluids 72(3), 269–300 (2013). doi: 10.1002/fld.3741 CrossRefMathSciNetGoogle Scholar
  12. 12.
    Goutal, N., Maurel, F.: Proceedings of the 2nd Workshop on Dam-Break Wave Simulation. Technical report, Groupe Hydraulique Fluviale, Département Laboratoire National d’Hydraulique, Electricité de France (1997)Google Scholar
  13. 13.
    Harris, M.: NVIDIA GPU computing SDK 4.1: Optimizing parallel reduction in CUDA (2011)Google Scholar
  14. 14.
    Holdahl, R., Holden, H., Lie, K.A.: Unconditionally stable splitting methods for the shallow water equations. BIT Numer. Math. 39(3), 451–472 (1999). doi: 10.1023/A:1022366502335 CrossRefzbMATHMathSciNetGoogle Scholar
  15. 15.
    Hornung, R.D., Kohn, S.R.: Managing application complexity in the SAMRAI object-oriented framework. Concurr. Comput. Pract. Exp. 14(5), 347–368 (2002). doi: 10.1002/cpe.652 CrossRefzbMATHGoogle Scholar
  16. 16.
    Jones, S.: Introduction to dynamic parallelism. In: GPU Technology Conference Presentation S0338 (2012)Google Scholar
  17. 17.
    Khronos Group: OpenCL.
  18. 18.
    Kurganov, A., Petrova, G.: A second-order well-balanced positivity preserving central-upwind scheme for the Saint-Venant system. Commun. Math. Sci. 5, 133–160 (2007)CrossRefzbMATHMathSciNetGoogle Scholar
  19. 19.
    Lawrence Berkeley National Laboratory, Center for Computational Sciences and Engineering: BoxLib.
  20. 20.
    LeVeque, R.J., George, D.L., Berger, M.J.: Tsunami modelling with adaptively refined finite volume methods. Acta Numer. 20, 211–289 (2011). doi: 10.1017/S0962492911000043 CrossRefzbMATHMathSciNetGoogle Scholar
  21. 21.
    Li, R., Wu, S.: H-adaptive mesh method with double tolerance adaptive strategy for hyperbolic conservation laws. J. Sci. Comput. 56(3), 616–636 (2013). doi: 10.1007/s10915-013-9692-1 CrossRefzbMATHMathSciNetGoogle Scholar
  22. 22.
    Lie, K.A., Noelle, S.: On the artificial compression method for second-order nonoscillatory central difference schemes for systems of conservation laws. SIAM J. Sci. Comput. 24(4), 1157–1174 (2003)CrossRefzbMATHMathSciNetGoogle Scholar
  23. 23.
    MacNeice, P., Olson, K.M., Mobarry, C., de Fainchtein, R., Packer, C.: PARAMESH: a parallel adaptive mesh refinement community toolkit. Comput. Phys. Commun. 126(3), 330–354 (2000). doi: 10.1016/S0010-4655(99)00501-9 CrossRefzbMATHGoogle Scholar
  24. 24.
    Meuer, H., Strohmaier, E., Dongarra, J., Simon, H.: Top 500 supercomputer sites (2013).
  25. 25.
    Nessyahu, H., Tadmor, E.: Non-oscillatory central differencing for hyperbolic conservation laws. J. Comput. Phys. 87(2), 408–463 (1990)CrossRefzbMATHMathSciNetGoogle Scholar
  26. 26.
    Nicholaeff, D., Davis, N., Trujillo, D., Robey, R.W.: Cell-based adaptive mesh refinement implemented with general purpose graphics processing units. Technical report, Los Alamos National Laboratory (2012)Google Scholar
  27. 27.
    NVIDIA: NVIDIA CUDA programming guide 5.0 (2012)Google Scholar
  28. 28.
    NVIDIA: NVIDIA GeForce GTX 680. Technical report, NVIDIA Corporation (2012)Google Scholar
  29. 29.
    NVIDIA: CUDA community showcase (2013).
  30. 30.
    Owens, J., Houston, M., Luebke, D., Green, S., Stone, J., Phillips, J.: GPU computing. Proc. IEEE 96(5), 879–899 (2008). doi: 10.1109/JPROC.2008.917757 CrossRefGoogle Scholar
  31. 31.
    Sætra, M.L.: Shallow water simulation on GPUs for sparse domains. In: A. Cangiani, R.L. Davidchack, E. Georgoulis, A.N. Gorban, J. Levesley, M.V. Tretyakov (eds.) Numerical Mathematics and Advanced Applications 2011, pp. 673–680. Springer, Berlin (2013). doi: 10.1007/978-3-642-33134-3_71.
  32. 32.
    Sætra, M.L., Brodtkorb, A.R.: Shallow water simulations on multiple GPUs. In: K. Jónasson (ed.) Applied Parallel and Scientific Computing, Lecture Notes in Computer Science, vol. 7134, pp. 56–66. Springer, Berlin (2012). doi: 10.1007/978-3-642-28145-7_6.
  33. 33.
    Schive, H.Y., Tsai, Y.C., Chiueh, T.: GAMER: a graphic processing unit accelerated adaptive-mesh-refinement code for astrophysics. Astrophys. J. Suppl. Ser. 186(2), 457–484 (2010). doi: 10.1088/0067-0049/186/2/457 CrossRefGoogle Scholar
  34. 34.
    Sweby, P.K.: High resolution schemes using flux limiters for hyperbolic conservation laws. SIAM J. Numer. Anal. 21(5), 995–1011 (1984)CrossRefzbMATHMathSciNetGoogle Scholar
  35. 35.
    The Enzo Project: Enzo.
  36. 36.
    van Leer, B.: Towards the ultimate conservative difference scheme. V. A second-order sequel to Godunov’s method. J. Comput. Phys. 32(1), 101–136 (1979). doi: 10.1016/0021-9991(79)90145-1 CrossRefGoogle Scholar
  37. 37.
    Wang, P., Abel, T., Kaehler, R.: Adaptive mesh fluid simulations on GPU. New Astron. 15(7), 581–589 (2010). doi: 10.1016/j.newast.2009.10.002 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Martin L. Sætra
    • 1
    • 2
  • André R. Brodtkorb
    • 3
  • Knut-Andreas Lie
    • 1
    • 3
  1. 1.Centre of Mathematics for ApplicationsUniversity of OsloOsloNorway
  2. 2.Norwegian Meteorological InstituteOsloNorway
  3. 3.Department of Applied MathematicsSINTEF ICTOsloNorway

Personalised recommendations