Computational Particle Mechanics

, Volume 5, Issue 4, pp 553–577 | Cite as

A comprehensive study of MPI parallelism in three-dimensional discrete element method (DEM) simulation of complex-shaped granular particles

  • Beichuan YanEmail author
  • Richard A. Regueiro


A three-dimensional (3D) DEM code for simulating complex-shaped granular particles is parallelized using message-passing interface (MPI). The concepts of link-block, ghost/border layer, and migration layer are put forward for design of the parallel algorithm, and theoretical scalability function of 3-D DEM scalability and memory usage is derived. Many performance-critical implementation details are managed optimally to achieve high performance and scalability, such as: minimizing communication overhead, maintaining dynamic load balance, handling particle migrations across block borders, transmitting C++ dynamic objects of particles between MPI processes efficiently, eliminating redundant contact information between adjacent MPI processes. The code executes on multiple US Department of Defense (DoD) supercomputers and tests up to 2048 compute nodes for simulating 10 million three-axis ellipsoidal particles. Performance analyses of the code including speedup, efficiency, scalability, and granularity across five orders of magnitude of simulation scale (number of particles) are provided, and they demonstrate high speedup and excellent scalability. It is also discovered that communication time is a decreasing function of the number of compute nodes in strong scaling measurements. The code’s capability of simulating a large number of complex-shaped particles on modern supercomputers will be of value in both laboratory studies on micromechanical properties of granular materials and many realistic engineering applications involving granular materials.


Parallelism Discrete element Granular materials Complex-shaped MPI Granularity 



We would like to acknowledge the support provided by ONR MURI Grant N00014-11-1-0691 and the DoD High Performance Computing Modernization Program (HPCMP) for granting us the computing resources required to conduct this work. This work also utilized the Janus supercomputer, which is supported by the National Science Foundation (Award Number CNS-0821794) and the University of Colorado Boulder.

Compliance with ethical standards

Conflict of interest

The authors declare that there is no conflict of interest.


  1. 1.
    Bardet JP (1997) Experimental soil mechanics. Prentice Hall, Upper Saddle RiverGoogle Scholar
  2. 2.
    Baugh JW Jr, Konduri R (2001) Discrete element modelling on a cluster of workstations. Eng Comput 17(1):1–15CrossRefGoogle Scholar
  3. 3.
    Cundall PA, Strack OD (1979) A discrete numerical model for granular assemblies. Geotechnique 29(1):47–65CrossRefGoogle Scholar
  4. 4.
    Delaney GW, Cleary PW, Sinnott MD, Morrison RD (2010) Novel application of dem to modelling comminution processes. In: IOP conference series: materials science and engineering, IOP Publishing, vol 10, p 012099Google Scholar
  5. 5.
    Foster I (1995) Designing and building parallel programs. Addison Wesley Publishing Company, ReadingzbMATHGoogle Scholar
  6. 6.
    Fu P, Dafalias YF (2011a) Fabric evolution within shear bands of granular materials and its relation to critical state theory. Int J Numer Anal Methods Geomech 35(18):1918–1948CrossRefGoogle Scholar
  7. 7.
    Fu P, Dafalias YF (2011b) Study of anisotropic shear strength of granular materials using dem simulation. Int J Numer Anal Methods Geomech 35(10):1098–1126CrossRefGoogle Scholar
  8. 8.
    Grest GS, Dünweg B, Kremer K (1989) Vectorized link cell fortran code for molecular dynamics simulations for a large number of particles. Comput Phys Commun 55(3):269–285CrossRefGoogle Scholar
  9. 9.
    Gustafson JL (1990) Fixed time, tiered memory, and superlinear speedup. In: Proceedings of the fifth distributed memory computing conference (DMCC5), pp 1255–1260Google Scholar
  10. 10.
    Henty DS (2000) Performance of hybrid message-passing and shared-memory parallelism for discrete element modeling. In: Proceedings of the 2000 ACM/IEEE conference on supercomputing, IEEE Computer Society, p 10Google Scholar
  11. 11.
    Hertz H (1882) Ueber die Berührung fester elastischer Körper [On the fixed elastic body contact]. Journal für die reine und angewandte Mathematik (Crelle) 92:156–171zbMATHGoogle Scholar
  12. 12.
    Jagadish HV, Ooi BC, Tan KL, Yu C, Zhang R (2005) iDistance: an adaptive B+-tree based indexing method for nearest neighbor search. ACM Trans Database Syst. (TODS) 30(2):364–397CrossRefGoogle Scholar
  13. 13.
    Karp AH, Flatt HP (1990) Measuring parallel processor performance. Commun. ACM 33(5):539–543CrossRefGoogle Scholar
  14. 14.
    Lim KW, Andrade JE (2014) Granular element method for three-dimensional discrete element calculations. Int. J. Numer. Anal. Methods Geomech. 38(2):167–188CrossRefGoogle Scholar
  15. 15.
    Lippman SB, Lajoie J, Moo BE (2005) C++ primer. Addison-Wesley, ReadingGoogle Scholar
  16. 16.
    Maknickas A, Kačeniauskas A, Kačianauskas R, Balevičius R, Džiugys A (2006) Parallel dem software for simulation of granular media. Informatica 17(2):207–224zbMATHGoogle Scholar
  17. 17.
    Michael JQ (2003) Parallel programming in C with MPI and OpenMP. McGraw-Hill Press, New YorkGoogle Scholar
  18. 18.
    Mindlin R (1949) Compliance of elastic bodies in contact. Trans ASME J App Mech 16(3):259–268MathSciNetzbMATHGoogle Scholar
  19. 19.
    Mindlin R, Deresiewicz H (1953) Elastic spheres in contact under varying oblique forces. Trans ASME J App Mech 20(3):327–344MathSciNetzbMATHGoogle Scholar
  20. 20.
    Muja M, Lowe DG (2009) Fast approximate nearest neighbors with automatic algorithm configuration. VISAPP 1(2):331–340Google Scholar
  21. 21.
    Munjiza A, Andrews K (1998) Nbs contact detection algorithm for bodies of similar size. Int J Num Methods Eng 43(1):131–149CrossRefGoogle Scholar
  22. 22.
    Munjiza A, Walther JH, Sbalzarini IF (2009) Large-scale parallel discrete element simulations of granular flow. Eng Comput 26(6):688–697CrossRefGoogle Scholar
  23. 23.
    Ng TT (1994) Numerical simulations of granular soil using elliptical particles. Comput Geotech 16(2):153–169CrossRefGoogle Scholar
  24. 24.
    Ng TT (2004) Triaxial test simulations with discrete element method and hydrostatic boundaries. J Eng Mech 130(10):1188–1194CrossRefGoogle Scholar
  25. 25.
    Peters JF, Hopkins MA, Kala R, Wahl RE (2009) A polyellipsoid particle for nonspherical discrete element method. Eng Comput 26(6):645–657CrossRefGoogle Scholar
  26. 26.
    Regueiro R, Duan Z, Yan B (2016) Overlapped-coupling between spherical discrete elements and micropolar finite elements in one dimension using a bridging-scale decomposition for statics. Eng Comput 33(1):28–63CrossRefGoogle Scholar
  27. 27.
    Stroustrup B (2013) The C++ programming language. Pearson Education, LondonzbMATHGoogle Scholar
  28. 28.
    Vedachalam V, Virdee D (2011) Discrete element modelling of granular snow particles using liggghts. M.Sc., University of EdinburghGoogle Scholar
  29. 29.
    Washington DW, Meegoda JN (2003) Micro-mechanical simulation of geotechnical problems using massively parallel computers. Int J Numer Anal Methods Geomech 27(14):1227–1234CrossRefGoogle Scholar
  30. 30.
    Wellmann C, Lillie C, Wriggers P (2008) A contact detection algorithm for superellipsoids based on the common-normal concept. Eng Comput 25(5):432–442CrossRefGoogle Scholar
  31. 31.
    Williams JR, Pentland AP (1992) Superquadrics and modal dynamics for discrete elements in interactive design. Eng Comput 9(2):115–127CrossRefGoogle Scholar
  32. 32.
    Williams JR, Perkins E, Cook B (2004) A contact algorithm for partitioning n arbitrary sized objects. Eng Comput 21(2/3/4):235–248CrossRefGoogle Scholar
  33. 33.
    Yan B, Regueiro RA (2018) Comparison Between \(O(n^2)\) and \(O(n)\) neighbor search algorithm and its influence on superlinear speedup in parallel 3D discrete element method (DEM) for complex-shaped particles (in review)Google Scholar
  34. 34.
    Yan B, Regueiro RA, Sture S (2010) Three dimensional ellipsoidal discrete element modeling of granular materials and its coupling with finite element facets. Eng Comput 27(4):519–550CrossRefGoogle Scholar

Copyright information

© OWZ 2018

Authors and Affiliations

  1. 1.Department of Civil, Environmental, and Architectural EngineeringUniversity of Colorado BoulderBoulderUSA

Personalised recommendations