GPU Implementation of ConeTorre Algorithm for Fluid Dynamics Simulation

  • Vadim Levchenko
  • Andrey Zakirov
  • Anastasia PerepelkinaEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11657)


LRnLA algorithms allow simulation of large problems with performance that exceeds the memory-bound limit of the traditional stepwise algorithms, that is, algorithms without any kind of temporal blocking. We show how the ConeTorre LRnLA algorithm that was successfully implemented for various CPU codes may be ported to work with CUDA framework and implemented the Lattice-Boltzmann Method (LBM) for fluid dynamics. As the standard tools and guidelines do not comply with the LRnLA paradigm, we have performed manual optimization of the communication between main memory levels of GPU and reduce overhead for data access patterns. We have made the performance estimate of the LRnLA implementation with the use of the Roofline model. The computation remains memory-bound, but with the ConeTorre algorithm the operational intensity is increased several times, and the maximum achievable performance for the chosen algorithm parameters is 9 billion cell updates per second on Tesla V100. We have achieved more than 66% of the estimate. As a result, we have developed a fluid simulation code based on the Lattice-Boltzmann method with a performance that surpasses state-of-the-art solutions.


LRnLA Lattice-Boltzmann Temporal blocking Wavefront blocking 



The work is supported by Russian Science Foundation, grant # 18-71-10004.


  1. 1.
    Computer system K-60 (2018).
  2. 2.
    Bailey, P., Myre, J., Walsh, S.D., Lilja, D.J., Saar, M.O.: Accelerating lattice Boltzmann fluid flow simulations using graphics processors. In: International Conference on Parallel Processing, ICPP 2009, pp. 550–557. IEEE (2009)Google Scholar
  3. 3.
    Calore, E., Gabbana, A., Kraus, J., Pellegrini, E., Schifano, S.F., Tripiccione, R.: Massively parallel lattice-boltzmann codes on large GPU clusters. Parallel Comput. 58, 1–24 (2016)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Degenhardt, R.: Advanced Lattice Boltzmann Models for the Simulation of Additive Manufacturing Processes. doctoralthesis, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) (2017)Google Scholar
  5. 5.
    Endo, T., Midorikawa, H., Sato, Y.: Software technology that deals with deeper memory hierarchy in post-petascale era. In: Sato, M. (ed.) Advanced Software Technologies for Post-Peta Scale Computing, pp. 227–248. Springer, Singapore (2019). Scholar
  6. 6.
    Habich, J., Zeiser, T., Hager, G., Wellein, G.: Enabling temporal blocking for a lattice Boltzmann flow solver through multicore-aware wavefront parallelization. In: 21st International Conference on Parallel Computational Fluid Dynamics, pp. 178–182 (2009)Google Scholar
  7. 7.
    Levchenko, V., Perepelkina, A., Zakirov, A.: Diamondtorre algorithm for high-performance wave modeling. Computation 4(3), 29 (2016)CrossRefGoogle Scholar
  8. 8.
    Levchenko, V., Perepelkina, A.: Locally recursive non-locally asynchronous algorithms for stencil computation. Lobachevskii J. Math. 39(4), 552–561 (2018)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Malas, T., Hager, G., Ltaief, H., Stengel, H., Wellein, G., Keyes, D.: Multicore-optimized wavefront diamond blocking for optimizing stencil updates. SIAM J. Sci. Comput. 37(4), C439–C464 (2015)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Maruyama, N., Aoki, T.: Optimizing stencil computations for NVIDIA kepler GPUs. In: Proceedings of the 1st International Workshop on High-Performance Stencil Computations, Vienna, pp. 89–95 (2014)Google Scholar
  11. 11.
    Montessori, A., et al.: Chapter 20 - multicomponent lattice Boltzmann models for biological applications. In: Cerrolaza, M., Shefelbine, S.J., Garz-Alvarado, D. (eds.) Numerical Methods and Advanced Simulation in Biomechanics and Biological Processes, pp. 357–370. Academic Press (2018)., Scholar
  12. 12.
    Morton, G.M.: A computer oriented geodetic data base and a new technique in file sequencing (1966)Google Scholar
  13. 13.
    Nguyen, A., Satish, N., Chhugani, J., Kim, C., Dubey, P.: 3.5-D blocking optimization for stencil computations on modern CPUs and GPUs. In: High Performance Computing, Networking, Storage and Analysis (SC), pp. 1–13. IEEE (2010)Google Scholar
  14. 14.
    Niedermeier, C.A., Janßen, C.F., Indinger, T.: Massively-parallel multi-GPU simulations for fast and accurate automotive aerodynamics. In: 7th European Conference on Computational Fluid Dynamics (2018)Google Scholar
  15. 15.
    Perepelkina, A.Y., Levchenko, V.D., Goryachev, I.A.: Implementation of the kinetic plasma code with locally recursive non-locally asynchronous algorithms. J. Phys. Conf. Ser. 510, 012042 (2014)CrossRefGoogle Scholar
  16. 16.
    Perepelkina, A., Levchenko, V.: LRnLA algorithm ConeFold with non-local vectorization for LBM implementation. In: Voevodin, V., Sobolev, S. (eds.) RuSCDays 2018. CCIS, vol. 965, pp. 101–113. Springer, Cham (2019). Scholar
  17. 17.
    Perepelkina, A., Levchenko, V., Khilkov, S.: The DiamondCandy LRnLA algorithm: raising efficiency of the 3D cross-stencil schemes. J. Supercomputing (2018).
  18. 18.
    Perepelkina, A., Levchenko, V.: The DiamondCandy algorithm for maximum performance vectorized cross-stencil computation. Keldysh Institute Preprints (225) (2018)Google Scholar
  19. 19.
    Riesinger, C., Bakhtiari, A., Schreiber, M., Neumann, P., Bungartz, H.J.: A holistic scalable implementation approach of the lattice Boltzmann method for CPU/GPU heterogeneous clusters. Computation 5(4), 48 (2017)CrossRefGoogle Scholar
  20. 20.
    Shimokawabe, T., Endo, T., Onodera, N., Aoki, T.: A stencil framework to realize large-scale computations beyond device memory capacity on GPU supercomputers. In: 2017 IEEE International Conference on Cluster Computing (CLUSTER), pp. 525–529. IEEE (2017)Google Scholar
  21. 21.
    Succi, S.: The Lattice Boltzmann Equation: For Fluid Dynamics And Beyond. Oxford University Press, Oxford (2001)zbMATHGoogle Scholar
  22. 22.
    Tomczak, T., Szafran, R.G.: A new GPU implementation for lattice-Boltzmann simulations on sparse geometries. arXiv preprint arXiv:1611.02445 (2016)
  23. 23.
    Vizitiu, A., Itu, L., Niţă, C., Suciu, C.: Optimized three-dimensional stencil computation on Fermi and Kepler GPUs. In: 2014 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6. IEEE (2014)Google Scholar
  24. 24.
    Williams, S., Waterman, A., Patterson, D.: Roofline: an insightful visual performance model for multicore architectures. Commun. ACM 52(4), 65–76 (2009)CrossRefGoogle Scholar
  25. 25.
    Wittmann, M.: Hardware-effiziente, hochparallele implementierungen von lattice-boltzmann-verfahren für komplexe geometrien (2016)Google Scholar
  26. 26.
    Zakirov, A., Levchenko, V., Perepelkina, A., Zempo, Y.: High performance FDTD algorithm for GPGPU supercomputers. J. Phys. Conf. Ser. 759, 012100 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Keldysh Institute of Applied MathematicsMoscowRussia
  2. 2.Kintech Lab Ltd.MoscowRussia

Personalised recommendations