A three-dimensional (3D) DEM code for simulating complex-shaped granular particles is parallelized using message-passing interface (MPI). The concepts of link-block, ghost/border layer, and migration layer are put forward for design of the parallel algorithm, and theoretical scalability function of 3-D DEM scalability and memory usage is derived. Many performance-critical implementation details are managed optimally to achieve high performance and scalability, such as: minimizing communication overhead, maintaining dynamic load balance, handling particle migrations across block borders, transmitting C++ dynamic objects of particles between MPI processes efficiently, eliminating redundant contact information between adjacent MPI processes. The code executes on multiple US Department of Defense (DoD) supercomputers and tests up to 2048 compute nodes for simulating 10 million three-axis ellipsoidal particles. Performance analyses of the code including speedup, efficiency, scalability, and granularity across five orders of magnitude of simulation scale (number of particles) are provided, and they demonstrate high speedup and excellent scalability. It is also discovered that communication time is a decreasing function of the number of compute nodes in strong scaling measurements. The code’s capability of simulating a large number of complex-shaped particles on modern supercomputers will be of value in both laboratory studies on micromechanical properties of granular materials and many realistic engineering applications involving granular materials.
Parallelism Discrete element Granular materials Complex-shaped MPI Granularity
This is a preview of subscription content, log in to check access.
We would like to acknowledge the support provided by ONR MURI Grant N00014-11-1-0691 and the DoD High Performance Computing Modernization Program (HPCMP) for granting us the computing resources required to conduct this work. This work also utilized the Janus supercomputer, which is supported by the National Science Foundation (Award Number CNS-0821794) and the University of Colorado Boulder.
Compliance with ethical standards
Conflict of interest
The authors declare that there is no conflict of interest.
Baugh JW Jr, Konduri R (2001) Discrete element modelling on a cluster of workstations. Eng Comput 17(1):1–15CrossRefGoogle Scholar
Cundall PA, Strack OD (1979) A discrete numerical model for granular assemblies. Geotechnique 29(1):47–65CrossRefGoogle Scholar
Delaney GW, Cleary PW, Sinnott MD, Morrison RD (2010) Novel application of dem to modelling comminution processes. In: IOP conference series: materials science and engineering, IOP Publishing, vol 10, p 012099Google Scholar
Foster I (1995) Designing and building parallel programs. Addison Wesley Publishing Company, ReadingzbMATHGoogle Scholar
Fu P, Dafalias YF (2011a) Fabric evolution within shear bands of granular materials and its relation to critical state theory. Int J Numer Anal Methods Geomech 35(18):1918–1948CrossRefGoogle Scholar
Fu P, Dafalias YF (2011b) Study of anisotropic shear strength of granular materials using dem simulation. Int J Numer Anal Methods Geomech 35(10):1098–1126CrossRefGoogle Scholar
Grest GS, Dünweg B, Kremer K (1989) Vectorized link cell fortran code for molecular dynamics simulations for a large number of particles. Comput Phys Commun 55(3):269–285CrossRefGoogle Scholar
Gustafson JL (1990) Fixed time, tiered memory, and superlinear speedup. In: Proceedings of the fifth distributed memory computing conference (DMCC5), pp 1255–1260Google Scholar
Henty DS (2000) Performance of hybrid message-passing and shared-memory parallelism for discrete element modeling. In: Proceedings of the 2000 ACM/IEEE conference on supercomputing, IEEE Computer Society, p 10Google Scholar
Hertz H (1882) Ueber die Berührung fester elastischer Körper [On the fixed elastic body contact]. Journal für die reine und angewandte Mathematik (Crelle) 92:156–171zbMATHGoogle Scholar
Jagadish HV, Ooi BC, Tan KL, Yu C, Zhang R (2005) iDistance: an adaptive B+-tree based indexing method for nearest neighbor search. ACM Trans Database Syst. (TODS) 30(2):364–397CrossRefGoogle Scholar
Mindlin R, Deresiewicz H (1953) Elastic spheres in contact under varying oblique forces. Trans ASME J App Mech 20(3):327–344MathSciNetzbMATHGoogle Scholar
Muja M, Lowe DG (2009) Fast approximate nearest neighbors with automatic algorithm configuration. VISAPP 1(2):331–340Google Scholar
Munjiza A, Andrews K (1998) Nbs contact detection algorithm for bodies of similar size. Int J Num Methods Eng 43(1):131–149CrossRefGoogle Scholar
Munjiza A, Walther JH, Sbalzarini IF (2009) Large-scale parallel discrete element simulations of granular flow. Eng Comput 26(6):688–697CrossRefGoogle Scholar
Ng TT (1994) Numerical simulations of granular soil using elliptical particles. Comput Geotech 16(2):153–169CrossRefGoogle Scholar
Ng TT (2004) Triaxial test simulations with discrete element method and hydrostatic boundaries. J Eng Mech 130(10):1188–1194CrossRefGoogle Scholar
Peters JF, Hopkins MA, Kala R, Wahl RE (2009) A polyellipsoid particle for nonspherical discrete element method. Eng Comput 26(6):645–657CrossRefGoogle Scholar
Regueiro R, Duan Z, Yan B (2016) Overlapped-coupling between spherical discrete elements and micropolar finite elements in one dimension using a bridging-scale decomposition for statics. Eng Comput 33(1):28–63CrossRefGoogle Scholar
Vedachalam V, Virdee D (2011) Discrete element modelling of granular snow particles using liggghts. M.Sc., University of EdinburghGoogle Scholar
Washington DW, Meegoda JN (2003) Micro-mechanical simulation of geotechnical problems using massively parallel computers. Int J Numer Anal Methods Geomech 27(14):1227–1234CrossRefGoogle Scholar
Wellmann C, Lillie C, Wriggers P (2008) A contact detection algorithm for superellipsoids based on the common-normal concept. Eng Comput 25(5):432–442CrossRefGoogle Scholar
Williams JR, Pentland AP (1992) Superquadrics and modal dynamics for discrete elements in interactive design. Eng Comput 9(2):115–127CrossRefGoogle Scholar
Williams JR, Perkins E, Cook B (2004) A contact algorithm for partitioning n arbitrary sized objects. Eng Comput 21(2/3/4):235–248CrossRefGoogle Scholar
Yan B, Regueiro RA (2018) Comparison Between \(O(n^2)\) and \(O(n)\) neighbor search algorithm and its influence on superlinear speedup in parallel 3D discrete element method (DEM) for complex-shaped particles (in review)Google Scholar
Yan B, Regueiro RA, Sture S (2010) Three dimensional ellipsoidal discrete element modeling of granular materials and its coupling with finite element facets. Eng Comput 27(4):519–550CrossRefGoogle Scholar