Skip to main content
SpringerLink
Log in

Lethe-DEM: an open-source parallel discrete element solver with load balancing

  • Published:
Computational Particle Mechanics Aims and scope Submit manuscript

Abstract

Approximately \({75}\%\) of the raw material and \({50}\%\) of the products in the chemical industry are granular materials. The discrete element method (DEM) provides detailed insights of phenomena at particle scale, and it is therefore often used for modeling granular materials. However, because DEM tracks the motion and contact of individual particles separately, its computational cost increases nonlinearly \(O(n_\mathrm{p}\log (n_\mathrm{p}))\) – \(O(n_\mathrm{p}^2)\) (depending on the algorithm) with the number of particles (\(n_\mathrm{p}\)). In this article, we introduce a new open-source parallel DEM software with load balancing: Lethe-DEM. Lethe-DEM, a module of Lethe, consists of solvers for two-dimensional and three-dimensional DEM simulations. Load balancing allows Lethe-DEM to significantly increase the parallel efficiency by \(\approx {25}\)–\({70}\%\) depending on the granular simulation. We explain the fundamental modules of Lethe-DEM, its software architecture, and the governing equations. Furthermore, we verify Lethe-DEM with several tests including analytical solutions and comparison with other software. Comparisons with experiments in a flat-bottomed silo, wedge-shaped silo, and rotating drum validate Lethe-DEM. We investigate the strong and weak scaling of Lethe-DEM with \({1}\le n_\mathrm{c} \le {192}\) and \({32}\le n_\mathrm{c} \le {320}\) processes, respectively, with and without load balancing. The strong-scaling analysis is performed on the wedge-shaped silo and rotating drum simulations, while for the weak-scaling analysis, we use a dam-break simulation. The best scalability of Lethe-DEM is obtained in the range of \({5000}\le n_\mathrm{p}/n_\mathrm{c} \le {15{,}000}\). Finally, we demonstrate that large-scale simulations can be carried out with Lethe-DEM using the simulation of a three-dimensional cylindrical silo with \(n_\mathrm{p}={4.3}\times 10^6\) on 320 cores.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

References

  1. Richard P, Nicodemi M, Delannay R, Ribiere P, Bideau D (2005) Slow relaxation and compaction of granular systems. Nat Mater 4(2):121–128

    Article  Google Scholar 

  2. Larsson S (2019) Particle methods for modelling granular material flow. Ph.D. thesis, Luleå University of Technology

  3. Nedderman RM (2005) Statics and kinematics of granular materials. Cambridge University Press, Cambridge

    Google Scholar 

  4. Golshan S, Zarghami R, Saleh K Modeling methods for gravity flow of granular solids in silos. Rev Chem Eng 1 (ahead-of-print)

  5. Blais B, Vidal D, Bertrand F, Patience GS, Chaouki J (2019) Experimental methods in chemical engineering: discrete element method-dem. Can J Chem Eng 97(7):1964–1973

    Article  Google Scholar 

  6. Golshan S, Sotudeh-Gharebagh R, Zarghami R, Mostoufi N, Blais B, Kuipers J (2020) Review and implementation of CFD-DEM applied to chemical process systems. Chem Eng Sci 221:115646

    Article  Google Scholar 

  7. Norouzi HR, Zarghami R, Sotudeh-Gharebagh R, Mostoufi N (2016) Coupled CFD-DEM modeling: formulation, implementation and application to multiphase flows. Wiley, London

    Book  Google Scholar 

  8. Dan H-C, Zhang Z, Chen J-Q, Wang H (2018) Numerical simulation of an indirect tensile test for asphalt mixtures using discrete element method software. J Mater Civ Eng 30(5):04018067

    Article  Google Scholar 

  9. Coetzee C (2017) Calibration of the discrete element method. Powder Technol 310:104–142

    Article  Google Scholar 

  10. Wolff MF, Salikov V, Antonyuk S, Heinrich S, Schneider GA (2013) Three-dimensional discrete element modeling of micromechanical bending tests of ceramic-polymer composite materials. Powder Technol 248:77–83

    Article  Google Scholar 

  11. Tavarez FA, Plesha ME (2007) Discrete element method for modelling solid and particulate materials. Int J Numer Methods Eng 70(4):379–404

    Article  MATH  Google Scholar 

  12. Cook BK, Jensen RP (2002) Discrete element methods. American Society of Civil Engineers, Reston

    Book  Google Scholar 

  13. Ismail Y, Sheng Y, Yang D, Ye J (2015) Discrete element modelling of unidirectional fibre-reinforced polymers under transverse tension. Compos Part B Eng 73:118–125

    Article  Google Scholar 

  14. Jerier J-F, Hathong B, Richefeu V, Chareyre B, Imbault D, Donze F-V, Doremus P (2011) Study of cold powder compaction by using the discrete element method. Powder Technol 208(2):537–541

    Article  Google Scholar 

  15. Ketterhagen WR, am Ende MT, Hancock BC (2009) Process modeling in the pharmaceutical industry using the discrete element method. J Pharm Sci 98(2):442–470

    Article  Google Scholar 

  16. Tijskens E, Ramon H, De Baerdemaeker J (2003) Discrete element modelling for process simulation in agriculture. J Sound Vib 266(3):493–514

    Article  Google Scholar 

  17. Boac JM, Ambrose RK, Casada ME, Maghirang RG, Maier DE (2014) Applications of discrete element method in modeling of grain postharvest operations. Food Eng Rev 6(4):128–149

    Article  Google Scholar 

  18. Blais B, Lassaigne M, Goniva C, Fradette L, Bertrand F (2016) Development of an unresolved cfd-dem model for the flow of viscous suspensions and its application to solid-liquid mixing. J Comput Phys 318:201–221

    Article  MathSciNet  MATH  Google Scholar 

  19. Bérard A, Patience GS, Blais B (2020) Experimental methods in chemical engineering: unresolved cfd-dem. Can J Chem Eng 98(2):424–440

    Article  Google Scholar 

  20. Kloss C, Goniva C, Hager A, Amberger S, Pirker S (2012) Models, algorithms and validation for opensource dem and cfd-dem. Prog Comput Fluid Dyn Int J 12(2–3):140–152

    Article  MathSciNet  Google Scholar 

  21. Weinhart T, Orefice L, Post M, van Schrojenstein-Lantman MP, Denissen IF, Tunuguntla DR, Tsang J, Cheng H, Shaheen MY, Shi H et al (2020) Fast, flexible particle simulations-an introduction to mercurydpm. Comput Phys Commun 249:107129

    Article  MathSciNet  Google Scholar 

  22. D. Solutions, Edem 23 user guide, Edinburgh, Scotland, UK

  23. Forgber T, Toson P, Madlmeir S, Kureck H, Khinast JG, Jajcevic D (2020) Extended validation and verification of xps/avl-fire\(^{{\rm TM}}\), a computational cfd-dem software platform. Powder Technol 361:880–893

    Article  Google Scholar 

  24. Blais B, Barbeau L, Bibeau V, Gauvin S, El Geitani T, Golshan S, Kamble R, Mikahori G, Chaouki J (2020) Lethe: an open-source parallel high-order adaptative cfd solver for incompressible flows. SoftwareX 12:100579

    Article  Google Scholar 

  25. Arndt D, Bangerth W, Blais B, Clevenger TC, Fehling M, Grayver AV, Heister T, Heltai L, Kronbichler M, Maier M et al (2020) The deal. II library, version 9.2. J Numer Math 28(3):131–146

    Article  MathSciNet  MATH  Google Scholar 

  26. Kronbichler M, Kormann K (2012) A generic interface for parallel cell-based finite element operator application. Comput Fluids 63:135–147

    Article  MathSciNet  MATH  Google Scholar 

  27. Burstedde C, Wilcox LC, Ghattas O (2011) p4est: scalable algorithms for parallel adaptive mesh refinement on forests of octrees. SIAM J Sci Comput 33(3):1103–1133

    Article  MathSciNet  MATH  Google Scholar 

  28. Burstedde C (2020) Parallel tree algorithms for AMR and non-standard data access. ACM Trans Math Softw

  29. Cundall PA, Strack OD (1979) A discrete numerical model for granular assemblies. Geotechnique 29(1):47–65

    Article  Google Scholar 

  30. Tsuji Y, Tanaka T, Ishida T (1992) Lagrangian numerical simulation of plug flow of cohesionless particles in a horizontal pipe. Powder Technol 71(3):239–250

    Article  Google Scholar 

  31. Zhou Y, Wright B, Yang R, Xu BH, Yu A-B (1999) Rolling friction in the dynamic simulation of sandpile formation. Physica A 269(2–4):536–553

    Article  Google Scholar 

  32. Brilliantov NV, Pöschel T (1998) Rolling friction of a viscous sphere on a hard plane. EPL (Europhys Lett) 42(5):511

    Article  Google Scholar 

  33. Delacroix B, Bouarab A, Fradette L, Bertrand F, Blais B (2020) Simulation of granular flow in a rotating frame of reference using the discrete element method. Powder Technol 369:146–161

    Article  Google Scholar 

  34. Geuzaine C, Remacle J-F (2009) Gmsh: a 3-d finite element mesh generator with built-in pre-and post-processing facilities. Int J Numer Methods Eng 79(11):1309–1331

    Article  MathSciNet  MATH  Google Scholar 

  35. Martin K, Hoffman B (2010) Mastering CMake: a cross-platform build system. Kitware, Clifton Park

    Google Scholar 

  36. Gropp W, Lusk E, Doss N, Skjellum A (1996) A high-performance, portable implementation of the mpi message passing interface standard. Parallel Comput 22(6):789–828

    Article  MATH  Google Scholar 

  37. Bangerth W, Burstedde C, Heister T, Kronbichler M (2012) Algorithms and data structures for massively parallel generic adaptive finite element codes. ACM Trans Math Softw

  38. Gassmöller R, Lokavarapu H, Heien E, Puckett EG, Bangerth W (2018) Flexible and scalable particle-in-cell methods with adaptive mesh refinement for geodynamic computations. Geochem Geophys Geosyst 19(9):3596–3604

    Article  Google Scholar 

  39. Garg R, Galvin J, Li T, Pannala S (2012) Open-source mfix-dem software for gas-solids flows: part I-verification studies. Powder Technol 220:122–137

    Article  Google Scholar 

  40. Norouzi HR, Zarghami R, Mostoufi N (2017) New hybrid CPU-GPU solver for CFD-DEM simulation of fluidized beds. Powder Technol 316:233–244

    Article  Google Scholar 

  41. Kruggel-Emden H, Wirtz S, Scherer V (2008) A study on tangential force laws applicable to the discrete element method (dem) for materials with viscoelastic or plastic behavior. Chem Eng Sci 63(6):1523–1541

    Article  Google Scholar 

  42. Balevičius R, Maknickas A, Sielamowicz I (2021) Experimental, continuum-and dem-based velocities in a flat-bottomed bin. Powder Technol 377:297–307

    Article  Google Scholar 

  43. Golshan S, Esgandari B, Zarghami R, Blais B, Saleh K (2020) Experimental and dem studies of velocity profiles and residence time distribution of non-spherical particles in silos. Powder Technol 373:510–521

  44. Alizadeh E, Dubé O, Bertrand F, Chaouki J (2013) Characterization of mixing and size segregation in a rotating drum by a particle tracking method. AIChE J 59(6):1894–1905

Download references

Acknowledgements

The authors would like to acknowledge support received by the deal.II community. Without such rigorously developed open source project such as deal.II, the present work could have never been achieved. The authors would like to acknowledge the support received from Calcul Québec and Compute Canada. Computations shown in this work were made on the supercomputer Beluga, Cedar, and Graham managed by Calcul Québec and Compute Canada. The operation of these supercomputers is funded by the Canada Foundation for Innovation (CFI), the ministère de l’Économie, de la science et de l’innovation du Québec (MESI), and the Fonds de recherche du Québec - Nature et technologies (FRQ-NT). This project was partially funded by the Natural Sciences and Engineering Research Council via NSERC Grant RGPIN-2020-04510. RG was supported by the Computational Infrastructure for Geodynamics (CIG), through the National Science Foundation under Award No. EAR-1550901, administered by The University of California-Davis.

Author information

Authors and Affiliations

  1. Research Unit for Industrial Flows Processes (URPEI), Department of Chemical Engineering, École Polytechique de Montréal, PO Box 6079, Stn Centre-Ville, Montréal, QC, H3C 3A7, Canada

    Shahab Golshan & Bruno Blais

  2. Continuum Simulations, Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon GmbH, Germany, Max-Planck-Str. 1, 21502, Geesthacht, Germany

    Peter Munch

  3. Institute for Computational Mechanics, Technical University of Munich, Boltzmannstr. 15, 85748, Garching, Germany

    Peter Munch & Martin Kronbichler

  4. Department of Geological Sciences, University of Florida, Gainesville, FL, USA

    Rene Gassmöller

  5. Division of Scientific Computing, Department of Information Technology, Uppsala University, Box 337, 751 05, Uppsala, Sweden

    Martin Kronbichler

Authors
  1. Shahab Golshan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Munch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rene Gassmöller
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Martin Kronbichler
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bruno Blais
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bruno Blais.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A. An example of the input files

Appendix A. An example of the input files

figure e
figure f

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Golshan, S., Munch, P., Gassmöller, R. et al. Lethe-DEM: an open-source parallel discrete element solver with load balancing. Comp. Part. Mech. 10, 77–96 (2023). https://doi.org/10.1007/s40571-022-00478-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40571-022-00478-6

Keywords

Search

Navigation