Skip to main content
SpringerLink
Log in

A comprehensive study of MPI parallelism in three-dimensional discrete element method (DEM) simulation of complex-shaped granular particles

  • Published:
Computational Particle Mechanics Aims and scope Submit manuscript

Abstract

A three-dimensional (3D) DEM code for simulating complex-shaped granular particles is parallelized using message-passing interface (MPI). The concepts of link-block, ghost/border layer, and migration layer are put forward for design of the parallel algorithm, and theoretical scalability function of 3-D DEM scalability and memory usage is derived. Many performance-critical implementation details are managed optimally to achieve high performance and scalability, such as: minimizing communication overhead, maintaining dynamic load balance, handling particle migrations across block borders, transmitting C++ dynamic objects of particles between MPI processes efficiently, eliminating redundant contact information between adjacent MPI processes. The code executes on multiple US Department of Defense (DoD) supercomputers and tests up to 2048 compute nodes for simulating 10 million three-axis ellipsoidal particles. Performance analyses of the code including speedup, efficiency, scalability, and granularity across five orders of magnitude of simulation scale (number of particles) are provided, and they demonstrate high speedup and excellent scalability. It is also discovered that communication time is a decreasing function of the number of compute nodes in strong scaling measurements. The code’s capability of simulating a large number of complex-shaped particles on modern supercomputers will be of value in both laboratory studies on micromechanical properties of granular materials and many realistic engineering applications involving granular materials.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

References

  1. Bardet JP (1997) Experimental soil mechanics. Prentice Hall, Upper Saddle River

    Google Scholar 

  2. Baugh JW Jr, Konduri R (2001) Discrete element modelling on a cluster of workstations. Eng Comput 17(1):1–15

    Article  Google Scholar 

  3. Cundall PA, Strack OD (1979) A discrete numerical model for granular assemblies. Geotechnique 29(1):47–65

    Article  Google Scholar 

  4. Delaney GW, Cleary PW, Sinnott MD, Morrison RD (2010) Novel application of dem to modelling comminution processes. In: IOP conference series: materials science and engineering, IOP Publishing, vol 10, p 012099

  5. Foster I (1995) Designing and building parallel programs. Addison Wesley Publishing Company, Reading

    MATH  Google Scholar 

  6. Fu P, Dafalias YF (2011a) Fabric evolution within shear bands of granular materials and its relation to critical state theory. Int J Numer Anal Methods Geomech 35(18):1918–1948

    Article  Google Scholar 

  7. Fu P, Dafalias YF (2011b) Study of anisotropic shear strength of granular materials using dem simulation. Int J Numer Anal Methods Geomech 35(10):1098–1126

    Article  Google Scholar 

  8. Grest GS, Dünweg B, Kremer K (1989) Vectorized link cell fortran code for molecular dynamics simulations for a large number of particles. Comput Phys Commun 55(3):269–285

    Article  Google Scholar 

  9. Gustafson JL (1990) Fixed time, tiered memory, and superlinear speedup. In: Proceedings of the fifth distributed memory computing conference (DMCC5), pp 1255–1260

  10. Henty DS (2000) Performance of hybrid message-passing and shared-memory parallelism for discrete element modeling. In: Proceedings of the 2000 ACM/IEEE conference on supercomputing, IEEE Computer Society, p 10

  11. Hertz H (1882) Ueber die Berührung fester elastischer Körper [On the fixed elastic body contact]. Journal für die reine und angewandte Mathematik (Crelle) 92:156–171

    MATH  Google Scholar 

  12. Jagadish HV, Ooi BC, Tan KL, Yu C, Zhang R (2005) iDistance: an adaptive B+-tree based indexing method for nearest neighbor search. ACM Trans Database Syst. (TODS) 30(2):364–397

    Article  Google Scholar 

  13. Karp AH, Flatt HP (1990) Measuring parallel processor performance. Commun. ACM 33(5):539–543

    Article  Google Scholar 

  14. Lim KW, Andrade JE (2014) Granular element method for three-dimensional discrete element calculations. Int. J. Numer. Anal. Methods Geomech. 38(2):167–188

    Article  Google Scholar 

  15. Lippman SB, Lajoie J, Moo BE (2005) C++ primer. Addison-Wesley, Reading

    Google Scholar 

  16. Maknickas A, Kačeniauskas A, Kačianauskas R, Balevičius R, Džiugys A (2006) Parallel dem software for simulation of granular media. Informatica 17(2):207–224

    MATH  Google Scholar 

  17. Michael JQ (2003) Parallel programming in C with MPI and OpenMP. McGraw-Hill Press, New York

    Google Scholar 

  18. Mindlin R (1949) Compliance of elastic bodies in contact. Trans ASME J App Mech 16(3):259–268

    MathSciNet  MATH  Google Scholar 

  19. Mindlin R, Deresiewicz H (1953) Elastic spheres in contact under varying oblique forces. Trans ASME J App Mech 20(3):327–344

    MathSciNet  MATH  Google Scholar 

  20. Muja M, Lowe DG (2009) Fast approximate nearest neighbors with automatic algorithm configuration. VISAPP 1(2):331–340

    Google Scholar 

  21. Munjiza A, Andrews K (1998) Nbs contact detection algorithm for bodies of similar size. Int J Num Methods Eng 43(1):131–149

    Article  Google Scholar 

  22. Munjiza A, Walther JH, Sbalzarini IF (2009) Large-scale parallel discrete element simulations of granular flow. Eng Comput 26(6):688–697

    Article  Google Scholar 

  23. Ng TT (1994) Numerical simulations of granular soil using elliptical particles. Comput Geotech 16(2):153–169

    Article  Google Scholar 

  24. Ng TT (2004) Triaxial test simulations with discrete element method and hydrostatic boundaries. J Eng Mech 130(10):1188–1194

    Article  Google Scholar 

  25. Peters JF, Hopkins MA, Kala R, Wahl RE (2009) A polyellipsoid particle for nonspherical discrete element method. Eng Comput 26(6):645–657

    Article  Google Scholar 

  26. Regueiro R, Duan Z, Yan B (2016) Overlapped-coupling between spherical discrete elements and micropolar finite elements in one dimension using a bridging-scale decomposition for statics. Eng Comput 33(1):28–63

    Article  Google Scholar 

  27. Stroustrup B (2013) The C++ programming language. Pearson Education, London

    MATH  Google Scholar 

  28. Vedachalam V, Virdee D (2011) Discrete element modelling of granular snow particles using liggghts. M.Sc., University of Edinburgh

  29. Washington DW, Meegoda JN (2003) Micro-mechanical simulation of geotechnical problems using massively parallel computers. Int J Numer Anal Methods Geomech 27(14):1227–1234

    Article  Google Scholar 

  30. Wellmann C, Lillie C, Wriggers P (2008) A contact detection algorithm for superellipsoids based on the common-normal concept. Eng Comput 25(5):432–442

    Article  Google Scholar 

  31. Williams JR, Pentland AP (1992) Superquadrics and modal dynamics for discrete elements in interactive design. Eng Comput 9(2):115–127

    Article  Google Scholar 

  32. Williams JR, Perkins E, Cook B (2004) A contact algorithm for partitioning n arbitrary sized objects. Eng Comput 21(2/3/4):235–248

    Article  Google Scholar 

  33. Yan B, Regueiro RA (2018) Comparison Between \(O(n^2)\) and \(O(n)\) neighbor search algorithm and its influence on superlinear speedup in parallel 3D discrete element method (DEM) for complex-shaped particles (in review)

  34. Yan B, Regueiro RA, Sture S (2010) Three dimensional ellipsoidal discrete element modeling of granular materials and its coupling with finite element facets. Eng Comput 27(4):519–550

    Article  Google Scholar 

Download references

Acknowledgements

We would like to acknowledge the support provided by ONR MURI Grant N00014-11-1-0691 and the DoD High Performance Computing Modernization Program (HPCMP) for granting us the computing resources required to conduct this work. This work also utilized the Janus supercomputer, which is supported by the National Science Foundation (Award Number CNS-0821794) and the University of Colorado Boulder.

Author information

Authors and Affiliations

  1. Department of Civil, Environmental, and Architectural Engineering, University of Colorado Boulder, Boulder, CO, USA

    Beichuan Yan & Richard A. Regueiro

Authors
  1. Beichuan Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Richard A. Regueiro
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Beichuan Yan.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, B., Regueiro, R.A. A comprehensive study of MPI parallelism in three-dimensional discrete element method (DEM) simulation of complex-shaped granular particles. Comp. Part. Mech. 5, 553–577 (2018). https://doi.org/10.1007/s40571-018-0190-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40571-018-0190-y

Keywords

Search

Navigation