Skip to main content

Distributed Sparse Block Grids on GPUs

Part of the Lecture Notes in Computer Science book series (LNTCS,volume 12728)


We present a design and implementation of distributed sparse block grids that transparently scale from a single CPU to multi-GPU clusters. We support dynamic sparse grids as, e.g., occur in computer graphics with complex deforming geometries and in multi-resolution numerical simulations. We present the data structures and algorithms of our approach, focusing on the optimizations required to render them computationally efficient on CPUs and GPUs alike. We provide a scalable implementation in the OpenFPM software library for HPC. We benchmark our implementation on up to 16 Nvidia GTX 1080 GPUs and up to 64 Nvidia A100 GPUs showing state-of-the-art scalability (68% to 96% parallel efficiency) on three benchmark problems. On a single GPU, our implementation is 14 to 140-fold faster than on a multi-core CPU.


  • Sparse grid
  • Block grid
  • CUDA
  • GPU
  • Distributed data

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-78713-4_15
  • Chapter length: 19 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-78713-4
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.


  1. Adalsteinsson, D., Sethian, J.A.: The fast construction of extension velocities in level set methods. J. Comput. Phys. 148, 2–22 (1999)

    MathSciNet  CrossRef  Google Scholar 

  2. Sean Baxter. moderngpu 2.0 (2016)

    Google Scholar 

  3. Bayati, B., Chatelain, P., Koumoutsakos, P.: Adaptive mesh refinement for stochastic reaction-diffusion processes. J. Chem. Phys. 230(1), 13–26 (2011)

    MathSciNet  MATH  Google Scholar 

  4. Bergdorf, M., Cottet, G.-H., Koumoutsakos, P.: Multilevel adaptive particle methods for convection-diffusion equations. Multiscale Model. Simul. 4(1), 328–357 (2005)

    MathSciNet  CrossRef  Google Scholar 

  5. Bergdorf, M., Koumoutsakos, P.: A Lagrangian particle-wavelet method. Multiscale Model. Simul. 5(3), 980–995 (2006)

    MathSciNet  CrossRef  Google Scholar 

  6. Bergdorf, M., Sbalzarini, I.F., Koumoutsakos, P.: A Lagrangian particle method for reaction-diffusion systems on deforming surfaces. J. Math. Biol. 61, 649–663 (2010)

    MathSciNet  CrossRef  Google Scholar 

  7. Brun, E., Guittet, A., Gibou, F.: A local level-set method using a hash table data structure. J. Comput. Phys. 231(6), 2528–2536 (2012)

    MathSciNet  CrossRef  Google Scholar 

  8. Gupta, A., Incardona, P., Aydin, A.D., Gumhold, S., Gunther, U., Sbalzarini, I F.: An architecture for interactive in situ visualization and its transparent implementation in OpenFPM. In: In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV’20), pp. 20–26. ACM, New York (2020)

    Google Scholar 

  9. Hoetzlein. R.K.: GVDB: raytracing sparse voxel database structures on the GPU. In: Eurographics/ACM SIGGRAPH Symposium on High Performance Graphics (2016)

    Google Scholar 

  10. Houston, B., Nielsen, M.B., Batty, C., Nilsson, O., Museth, K.: Hierarchical RLE level set: a compact and versatile deformable surface representation. ACM Trans. Graph. 25(1), 151–175 (2006)

    CrossRef  Google Scholar 

  11. Incardona, P., Leo, A., Zaluzhnyi, Y., Ramaswamy, R., Sbalzarini, I.F.: OpenFPM: a scalable open framework for particle and particle-mesh codes on parallel computers. Comput. Phys. Commun. 241, 155–177 (2019)

    CrossRef  Google Scholar 

  12. Kretz, M., Lindenstruth, V.: Vc: A C++ library for explicit vectorization. Softw. Pract. Exper. 42(11), 1409–1430 (2012)

    CrossRef  Google Scholar 

  13. Merrill, D.: CUDA UnBound (CUB) library (2015)

    Google Scholar 

  14. Museth, K.: VDB: high-resolution sparse volumes with dynamic topology. ACM Trans. Graph. 32(3), 27 (2013)

    Google Scholar 

  15. Setaluri, R., Aanjaneya, M., Bauer, S., Sifakis. E.: SPGrid: a sparse paged grid structure applied to adaptive smoke simulation. ACM Trans. Graph. 33(6), 205 (2014)

    Google Scholar 

  16. Zhang, W., et al.: AMReX: a framework for block-structured adaptive mesh refinement. J. Open Source Softw. 4(37), 1370–1370 (2019)

    CrossRef  Google Scholar 

Download references


The authors are grateful to the Centre for Information Services and High Performance Computing (ZIH) of TU Dresden and to the Scientific Computing Facility of MPI-CBG for providing their facilities for the benchmarks. This work was supported by the Federal Ministry of Education and Research (Bundesministerium für Bildung und Forschung, BMBF) under funding codes 01/S18026A-F (competence center for Big Data and AI “ScaDS.AI Dresden/Leipzig”) and 031L0160 (project “SPlaT-DM – computer simulation platform for topology-driven morphogenesis”).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Ivo F. Sbalzarini .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Incardona, P., Bianucci, T., Sbalzarini, I.F. (2021). Distributed Sparse Block Grids on GPUs. In: Chamberlain, B.L., Varbanescu, AL., Ltaief, H., Luszczek, P. (eds) High Performance Computing. ISC High Performance 2021. Lecture Notes in Computer Science(), vol 12728. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-78712-7

  • Online ISBN: 978-3-030-78713-4

  • eBook Packages: Computer ScienceComputer Science (R0)