Abstract
The shallow-water equations model hydrostatic flow below a free surface for cases in which the ratio between the vertical and horizontal length scales is small and are used to describe waves in lakes, rivers, oceans, and the atmosphere. The equations admit discontinuous solutions, and numerical solutions are typically computed using high-resolution schemes. For many practical problems, there is a need to increase the grid resolution locally to capture complicated structures or steep gradients in the solution. An efficient method to this end is adaptive mesh refinement (AMR), which recursively refines the grid in parts of the domain and adaptively updates the refinement as the simulation progresses. Several authors have demonstrated that the explicit stencil computations of high-resolution schemes map particularly well to many-core architectures seen in hardware accelerators such as graphics processing units (GPUs). Herein, we present the first full GPU-implementation of a block-based AMR method for the second-order Kurganov–Petrova central scheme. We discuss implementation details, potential pitfalls, and key insights, and present a series of performance and accuracy tests. Although it is only presented for a particular case herein, we believe our approach to GPU-implementation of AMR is transferable to other hyperbolic conservation laws, numerical schemes, and architectures similar to the GPU.
Similar content being viewed by others
Notes
On the June 2013 list, there were 43 GPU-powered machines, and 12 machines using the Intel Xeon Phi co-processor.
It should be noted that some parts of the code, such as launching kernels on the GPU, is necessarily performed on the CPU.
Here, we use a midpoint integration rule, central-upwind numerical flux, and a piecewise-linear reconstruction with slopes limited by a generalized minmod function, see [18] for details. The weight is 0.5 for both steps of the particular stability-preserving, second-order Runge–Kutta scheme used herein. If other schemes are used, this weight must be altered accordingly.
Only the array containing boundary values for the end of the time-step series is filled in the initialization. The boundary values for the start of the time-step series is simply set to be the end-values from the previous time-step series.
References
Berger, M., LeVeque, R.: Adaptive mesh refinement for two-dimensional hyperbolic systems and the AMRCLAW software. SIAM J. Numer. Anal. 35, 2298–2316 (1998)
Berger, M.J., Colella, P.: Local adaptive mesh refinement for shock hydrodynamics. J. Comput. Phys. 82, 64–84 (1989). doi:10.1016/0021-9991(89)90035-1
Berger, M.J., Oliger, J.: Adaptive mesh refinement for hyperbolic partial differential equations. J. Comput. Phys. 53(3), 484–512 (1984). doi:10.1016/0021-9991(84)90073-1
Brodtkorb, A., Dyken, C., Hagen, T., Hjelmervik, J., Storaasli, O.: State-of-the-art in heterogeneous computing. J. Sci. Program. 18(1), 1–33 (2010)
Brodtkorb, A.R., Hagen, T.R., Lie, K.A., Natvig, J.R.: Simulation and visualization of the Saint-Venant system using GPUs. Comput. Vis. Sci. 13(7), 341–353 (2010)
Brodtkorb, A.R., Hagen, T.R., Sætra, M.L.: Graphics processing unit (GPU) programming strategies and trends in GPU computing. J. Parallel Distrib. Comput. 73(1), 4–13 (2013). doi:10.1016/j.jpdc.2012.04.003
Brodtkorb, A.R., Sætra, M.L.: Explicit shallow water simulations on GPUs: guidelines and best practices. In: XIX International Conference on Water Resources, CMWR 2012, June 17–22, 2012. University of Illinois at Urbana-Champaign (2012)
Brodtkorb, A.R., Sætra, M.L., Altinakar, M.: Efficient shallow water simulations on GPUs: implementation, visualization, verification, and validation. Comput. Fluids 55, 1–12 (2012). doi:10.1016/j.compfluid.2011.10.012
Burstedde, C., Ghattas, O., Gurnis, M., Isaac, T., Stadler, G., Warburton, T., Wilcox, L.: Extreme-scale AMR. In: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–12. IEEE Computer Society (2010)
Colella, P., Graves, D.T., Johnson, J.N., Johansen, H.S., Keen, N.D., Ligocki, T.J., Martin, D.F., McCorquodale, P.W., Modiano, D., Schwartz, P.O., Sternberg, T.D., Van Straalen, B.: Chombo software package for AMR applications design document. Technical report, Lawrence Berkeley National Laboratory (2012)
Delestre, O., Lucas, C., Ksinant, P.A., Darboux, F., Laguerre, C., Vo, T.N.T., James, F., Cordier, S.: SWASHES: a compilation of shallow water analytic solutions for hydraulic and environmental studies. Int. J. Numer. Methods Fluids 72(3), 269–300 (2013). doi:10.1002/fld.3741
Goutal, N., Maurel, F.: Proceedings of the 2nd Workshop on Dam-Break Wave Simulation. Technical report, Groupe Hydraulique Fluviale, Département Laboratoire National d’Hydraulique, Electricité de France (1997)
Harris, M.: NVIDIA GPU computing SDK 4.1: Optimizing parallel reduction in CUDA (2011)
Holdahl, R., Holden, H., Lie, K.A.: Unconditionally stable splitting methods for the shallow water equations. BIT Numer. Math. 39(3), 451–472 (1999). doi:10.1023/A:1022366502335
Hornung, R.D., Kohn, S.R.: Managing application complexity in the SAMRAI object-oriented framework. Concurr. Comput. Pract. Exp. 14(5), 347–368 (2002). doi:10.1002/cpe.652
Jones, S.: Introduction to dynamic parallelism. In: GPU Technology Conference Presentation S0338 (2012)
Khronos Group: OpenCL. http://www.khronos.org/opencl/
Kurganov, A., Petrova, G.: A second-order well-balanced positivity preserving central-upwind scheme for the Saint-Venant system. Commun. Math. Sci. 5, 133–160 (2007)
Lawrence Berkeley National Laboratory, Center for Computational Sciences and Engineering: BoxLib. https://ccse.lbl.gov/BoxLib/index.html
LeVeque, R.J., George, D.L., Berger, M.J.: Tsunami modelling with adaptively refined finite volume methods. Acta Numer. 20, 211–289 (2011). doi:10.1017/S0962492911000043
Li, R., Wu, S.: H-adaptive mesh method with double tolerance adaptive strategy for hyperbolic conservation laws. J. Sci. Comput. 56(3), 616–636 (2013). doi:10.1007/s10915-013-9692-1
Lie, K.A., Noelle, S.: On the artificial compression method for second-order nonoscillatory central difference schemes for systems of conservation laws. SIAM J. Sci. Comput. 24(4), 1157–1174 (2003)
MacNeice, P., Olson, K.M., Mobarry, C., de Fainchtein, R., Packer, C.: PARAMESH: a parallel adaptive mesh refinement community toolkit. Comput. Phys. Commun. 126(3), 330–354 (2000). doi:10.1016/S0010-4655(99)00501-9
Meuer, H., Strohmaier, E., Dongarra, J., Simon, H.: Top 500 supercomputer sites (2013). http://www.top500.org/
Nessyahu, H., Tadmor, E.: Non-oscillatory central differencing for hyperbolic conservation laws. J. Comput. Phys. 87(2), 408–463 (1990)
Nicholaeff, D., Davis, N., Trujillo, D., Robey, R.W.: Cell-based adaptive mesh refinement implemented with general purpose graphics processing units. Technical report, Los Alamos National Laboratory (2012)
NVIDIA: NVIDIA CUDA programming guide 5.0 (2012)
NVIDIA: NVIDIA GeForce GTX 680. Technical report, NVIDIA Corporation (2012)
NVIDIA: CUDA community showcase (2013). http://www.nvidia.com/object/cuda_showcase_html.html
Owens, J., Houston, M., Luebke, D., Green, S., Stone, J., Phillips, J.: GPU computing. Proc. IEEE 96(5), 879–899 (2008). doi:10.1109/JPROC.2008.917757
Sætra, M.L.: Shallow water simulation on GPUs for sparse domains. In: A. Cangiani, R.L. Davidchack, E. Georgoulis, A.N. Gorban, J. Levesley, M.V. Tretyakov (eds.) Numerical Mathematics and Advanced Applications 2011, pp. 673–680. Springer, Berlin (2013). doi:10.1007/978-3-642-33134-3_71.
Sætra, M.L., Brodtkorb, A.R.: Shallow water simulations on multiple GPUs. In: K. Jónasson (ed.) Applied Parallel and Scientific Computing, Lecture Notes in Computer Science, vol. 7134, pp. 56–66. Springer, Berlin (2012). doi:10.1007/978-3-642-28145-7_6.
Schive, H.Y., Tsai, Y.C., Chiueh, T.: GAMER: a graphic processing unit accelerated adaptive-mesh-refinement code for astrophysics. Astrophys. J. Suppl. Ser. 186(2), 457–484 (2010). doi:10.1088/0067-0049/186/2/457
Sweby, P.K.: High resolution schemes using flux limiters for hyperbolic conservation laws. SIAM J. Numer. Anal. 21(5), 995–1011 (1984)
The Enzo Project: Enzo. http://enzo-project.org/
van Leer, B.: Towards the ultimate conservative difference scheme. V. A second-order sequel to Godunov’s method. J. Comput. Phys. 32(1), 101–136 (1979). doi:10.1016/0021-9991(79)90145-1
Wang, P., Abel, T., Kaehler, R.: Adaptive mesh fluid simulations on GPU. New Astron. 15(7), 581–589 (2010). doi:10.1016/j.newast.2009.10.002
Acknowledgments
This work is supported in part by the Research Council of Norway through Grant No. 180023 (Parallel3D), and in part by the Centre of Mathematics for Applications at the University of Oslo (Sætra and Lie).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Sætra, M.L., Brodtkorb, A.R. & Lie, KA. Efficient GPU-Implementation of Adaptive Mesh Refinement for the Shallow-Water Equations. J Sci Comput 63, 23–48 (2015). https://doi.org/10.1007/s10915-014-9883-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10915-014-9883-4
Keywords
- Adaptive mesh refinement (AMR)
- GPU computing
- Shallow-water simulation
- Conservation law
- Domain decomposition