Abstract
In the past, the number of CPU cores/threads was usually less than 8/16; now, the maximum number is 128/256. As a CPU-based parallel method, OpenMP has an increasing advantage with the increase in CPU cores and threads. A parallel combined finite-discrete element method (FDEM) for modeling underground excavation and rock reinforcement using OpenMP is implemented. Its computational performance is validated in the two advanced CPUs: AMD Ryzen Threadripper PRO 5995WX (64/128 cores/threads); and 2 × AMD EPYC 7T83 (128/256 cores/threads). Then, its ability in simulating tunnel excavation under rockbolt-shotcrete-grouting support is implemented using the novel solid bolt model, which can explicitly capture the interaction between bolt, grout, and rock. The parallel performance validation of the uniaxial compression test shows: (i) for the speedup ratio, the OpenMP-based parallel FDEM obtains maximum speedup ratios of 30 (33 k elements) and 41 (3304 k elements) on the Threadripper, and 31 and 43 on the 2 × EPYC, respectively; (ii) for the scalability of speedup ratio, when the number of threads used is less than 128, the speedup ratio is always increasing with the increase of the number of threads; (iii) for the stability of speedup ratio, it has a stable speedup ratio, regardless of whether the rock is pre- or post-fractured.
Highlights
-
A parallel FDEM for modeling underground excavation and rock reinforcement using OpenMP is implemented.
-
Its computational efficiency is validated in the two advanced CPUs: AMD Ryzen Threadripper PRO 5995WX; and 2 × AMD EPYC 7T83 (128/256 cores/threads).
-
Its ability in simulating rockbolt-shotcrete-grouting support is implemented using the novel solid bolt model.
-
It obtains maximum speedup ratios of 30 (33 k elements) and 41 (3304 k elements) on the Threadripper, and 31 and 43 on the 2 × EPYC.
-
It has good scalability and stability of speedup ratio, regardless of whether the rock is pre- or post-fractured.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
An HM, Liu HY, Han H, Zheng X, Wang XG (2017) Hybrid finite-discrete element modelling of dynamic fracture and resultant fragment casting and muck-piling by rock blast. Comput Geotech 81(1):322–345. https://doi.org/10.1016/j.compgeo.2016.09.007
Barla M, Piovano G, Grasselli G (2012) Rock slide simulation with the combined finite-discrete element method. Int J Geomech 12(6):711–721. https://doi.org/10.1061/(ASCE)GM.1943-5622.0000204
Batinić M, Smoljanović H, Munjiza A, Mihanović A (2017) GPU based parallel FDEM for analysis of cable structures. Građevinar 69(12):1085–1092
Cai W, Gao K, Wu S, Long W (2023) Moment tensor-based approach for acoustic emission simulation in brittle rocks using combined finite-discrete element method (FDEM). Rock Mech Rock Eng 56(6):3903–3925. https://doi.org/10.1007/s00603-023-03261-y
Carranza-Torres C (2009) Analytical and numerical study of the mechanics of rockbolt reinforcement around tunnels in rock masses. Rock Mech Rock Eng 42(2):175–228. https://doi.org/10.1007/s00603-009-0178-2
Chen SH, Qiang S, Chen SF, Egger P (2004) Composite element model of the fully grouted rock bolt. Rock Mech Rock Eng 37(3):193–212. https://doi.org/10.1007/s00603-003-0006-z
Chen Y, Ma G, Zhou W, Wei D, Zhao Q, Zou Y, Grasselli G (2021) An enhanced tool for probing the microscopic behavior of granular materials based on X-ray micro-CT and FDEM. Comput Geotech 132:103974. https://doi.org/10.1016/j.compgeo.2020.103974
Deng P, Liu Q (2020) Influence of the softening stress path on crack development around underground excavations: Insights from 2D-FDEM modelling. Comput Geotech 117(1):103239. https://doi.org/10.1016/j.compgeo.2019.103239
Deng P, Liu Q, Huang X, Bo Y, Liu Q, Li W (2021) Sensitivity analysis of fracture energies for the combined finite-discrete element method (FDEM). Eng Fract Mech 251:107793. https://doi.org/10.1016/j.engfracmech.2021.107793
Deng P, Liu Q, Huang X, Pan Y, Wu J (2022) FDEM numerical modeling of failure mechanisms of anisotropic rock masses around deep tunnels. Comput Geotech 142:104535. https://doi.org/10.1016/j.compgeo.2021.104535
Elmo D, Stead D, Eberhardt E, Vyazmensky A (2013) Applications of finite/discrete element modeling to rock engineering problems. Int J Geomech 13(5):565–580. https://doi.org/10.1061/(ASCE)GM.1943-5622.0000238
Farsi A, Bedi A, Latham JP, Bowers K (2019) Simulation of fracture propagation in fibre-reinforced concrete using FDEM: an application to tunnel linings. Comput Particle Mech 7(5):961–974. https://doi.org/10.1007/s40571-019-00305-5
Fukuda D, Mohammadnejad M, Liu H, Dehkhoda S, Chan A, Cho SH, Min GJ, Han H, Kodama J, Fujii Y (2019) Development of a GPGPU-parallelized hybrid finite-discrete element method for modeling rock fracture. Int J Numer Anal Methods Geomech 43(10):1797–1824
Fukuda D, Mohammadnejad M, Liu H, Zhang Q, Zhao J, Dehkhoda S, Chan A, Kodama J-I, Fujii Y (2020) Development of a 3D hybrid finite-discrete element simulator based on GPGPU-parallelized computation for modelling rock fracturing under quasi-static and dynamic loading conditions. Rock Mech Rock Eng 53:1079–1112. https://doi.org/10.1007/s00603-019-01960-z
Fukuda D, Liu H, Zhang Q, Zhao J, Kodama J-I, Fujii Y, Chan AHC (2021) Modelling of dynamic rock fracture process using the finite-discrete element method with a novel and efficient contact activation scheme. Int J Rock Mech Min Sci 138(1):104645. https://doi.org/10.1016/j.ijrmms.2021.104645
Ha J, Tatone B, Gaspari G, Grasselli G (2021) Simulating tunnel support integrity using FEM and FDEM based on laboratory test data. Tunn Undergr Space Technol 111:103848. https://doi.org/10.1016/j.tust.2021.103848
Han H, Fukuda D, Liu H, Salmi EF, Sellers E, Liu T, Chan A (2020) Combined finite-discrete element modelling of rock fracture and fragmentation induced by contour blasting during tunnelling with high horizontal in-situ stress. Int J Rock Mech Min Sci 127(1):104214. https://doi.org/10.1016/j.ijrmms.2020.104214
Han H, Fukuda D, Liu H, Fathi Salmi E, Sellers E, Liu T, Chan A (2021) Combined finite-discrete element modellings of rockbursts in tunnelling under high in-situ stresses. Comput Geotech 137(1):104261. https://doi.org/10.1016/j.compgeo.2021.104261
Hoek E (1998) Tunnel support in weak rock. In: Keynote address, symposium of sedimentary rock engineering, Taipei, Taiwan
Huang G-H, Xu Y-Z, Chen X-F, Xia M, Zhang S, Yi X-W (2020) A new C++ programming strategy for three-dimensional sphere discontinuous deformation analysis. Int J Geomech 20(10):04020175. https://doi.org/10.1061/(ASCE)GM.1943-5622.0001811
Itasca (2005) FLAC version 5.0: theory and background. Itasca Consulting Group Inc., Minneapolis, Minnesota
Itasca (2014) UDEC version 6.0: theory and background. Itasca Consulting Group Inc., Minneapolis, Minnesota
Itasca (2016) PFC 2D version 3.1: theory and background. Itasca Consulting Group Inc., Minneapolis, Minnesota
Joulin C, Xiang J, Latham J-P (2020) A novel thermo-mechanical coupling approach for thermal fracturing of rocks in the three-dimensional FDEM. Comput Particle Mech 7(5):935–946. https://doi.org/10.1007/s40571-020-00319-4
Knight EE, Rougier E, Lei Z, Euser B, Chau V, Boyce SH, Gao K, Okubo K, Froment M (2020) HOSS: an implementation of the combined finite-discrete element method. Comput Particle Mech 7(5):765–787. https://doi.org/10.1007/s40571-020-00349-y
Lei Z, Rougier E, Knight EE, Munjiza A (2014) A framework for grand scale parallelization of the combined finite discrete element method in 2D. Comput Particle Mech 1(3):307–319. https://doi.org/10.1007/s40571-014-0026-3
Liang D, Zhang N, Liu H, Fukuda D, Rong H (2021) Hybrid finite-discrete element simulator based on GPGPU-parallelized computation for modelling crack initiation and coalescence in sandy mudstone with prefabricated cross-flaws under uniaxial compression. Eng Fract Mech 247(1):107658. https://doi.org/10.1016/j.engfracmech.2021.107658
Lisjak A, Liu Q, Zhao Q, Mahabadi OK, Grasselli G (2013) Numerical simulation of acoustic emission in brittle rocks by two-dimensional finite-discrete element analysis. Geophys J Int 195(1):423–443. https://doi.org/10.1093/gji/ggt221
Lisjak A, Garitte B, Grasselli G, Müller HR, Vietor T (2015) The excavation of a circular tunnel in a bedded argillaceous rock (opalinus clay): short-term rock mass response and FDEM numerical analysis. Tunn Undergr Space Technol 45:227–248. https://doi.org/10.1016/j.tust.2014.09.014
Lisjak A, Kaifosh P, He L, Tatone BSA, Mahabadi OK, Grasselli G (2017) A 2D, fully-coupled, hydro-mechanical, FDEM formulation for modelling fracturing processes in discontinuous, porous rock masses. Comput Geotech 81(1):1–18. https://doi.org/10.1016/j.compgeo.2016.07.009
Lisjak A, Mahabadi OK, He L, Tatone BSA, Kaifosh P, Haque SA, Grasselli G (2018) Acceleration of a 2D/3D finite-discrete element code for geomechanical simulations using general purpose GPU computing. Comput Geotech 100:84–96. https://doi.org/10.1016/j.compgeo.2018.04.011
Lisjak A, Young-Schultz T, Li B, He L, Tatone BSA, Mahabadi OK (2020) A novel rockbolt formulation for a GPU-accelerated, finite-discrete element method code and its application to underground excavations. Int J Rock Mech Min Sci 134:104410. https://doi.org/10.1016/j.ijrmms.2020.104410
Liu Q, Wang W, Ma H (2019a) Parallelized combined finite-discrete element (FDEM) procedure using multi-GPU with CUDA. Int J Numer Anal Methods Geomech 44:208–238. https://doi.org/10.1002/nag.3011
Liu Q, Deng P, Bi C, Li W, Liu J (2019b) FDEM numerical simulation of the fracture and extraction process of soft surrounding rock mass and its rockbolt-shotcrete-grouting reinforcement methods in the deep tunnel. Rock and Soil Mechanics 40(10):4065–4083
Liu Q, Xu X, Wu Z (2020a) A GPU-based numerical manifold method for modeling the formation of the excavation damaged zone in deep rock tunnels. Comput Geotech 118:103351. https://doi.org/10.1016/j.compgeo.2019.103351
Liu X, Mao J, Zhao L, Shao L, Li T (2020b) The distance potential function-based finite-discrete element method. Comput Mech 66(6):1477–1495. https://doi.org/10.1007/s00466-020-01913-2
Liu H, Liu Q, Ma H, Fish J (2021) A novel GPGPU-parallelized contact detection algorithm for combined finite-discrete element method. Int J Rock Mech Min Sci 144:104782. https://doi.org/10.1016/j.ijrmms.2021.104782
Liu G, Ma F, Zhang M, Guo J, Jia J (2022) Y-Mat: an improved hybrid finite-discrete element code for addressing geotechnical and geological engineering problems. Eng Comput 39(5):1962–1983. https://doi.org/10.1108/EC-12-2020-0741
Lukas T, Schiava D’Albano GG, Munjiza A (2014) Space decomposition based parallelization solutions for the combined finite-discrete element method in 2D. J Rock Mech Geotech Eng 6(6):607–615. https://doi.org/10.1016/j.jrmge.2014.10.001
Ma H, Wang W, Liu Q, Tian Y, Jiang Y, Liu H, Huang D (2022) Extremely large deformation of tunnel induced by rock mass fracture using GPGPU parallel FDEM. Int J Numer Anal Methods Geomech 2022:1–26
Mahabadi OK, Lisjak A, Munjiza A, Grasselli G (2012) Y-geo: new combined finite-discrete element numerical code for geomechanical applications. Int J Geomech 12(6):676–688. https://doi.org/10.1061/(asce)gm.1943-5622.0000216
Miglietta PC, Bentz EC, Grasselli G (2017) Finite/discrete element modelling of reversed cyclic tests on unreinforced masonry structures. Eng Struct 138(1):159–169. https://doi.org/10.1016/j.engstruct.2017.02.019
Munjiza A (2004) The combined finite-discrete element method. Wiley, New York
Munjiza A, Andrews KRF (1998) NBS contact detection algorithm for bodies of similar size. Int J Numer Methods Eng 43(1):131–149. https://doi.org/10.1002/(SICI)1097-0207(19980915)43:1%3c131::AID-NME447%3e3.0.CO;2-S
Munjiza A, Andrews KRF (2000) Penalty function method for combined finite-discrete element systems comprising large number of separate bodies. Int J Numer Methods Eng 49(11):1377–1396. https://doi.org/10.1002/1097-0207(20001220)49:11%3c1377::AID-NME6%3e3.0.CO;2-B
Munjiza A, Andrews KRF, White JK (1999) Combined single and smeared crack model in combined finite-discrete element analysis. Int J Numer Methods Eng 44(1):41–57. https://doi.org/10.1002/(SICI)1097-0207(19990110)44:1%3c41::AID-NME487%3e3.0.CO;2-A
Munjiza A, Knight EE, Rougier E (2011) Computational mechanics of discontinua. Wiley, New York
Munjiza A, Knight EE, Rougier E (2014) Large strain finite element method: a practical course. Wiley, New York
Munjiza A, Rougier E, Lei Z, Knight EE (2020) FSIS: a novel fluid–solid interaction solver for fracturing and fragmenting solids. Comput Particle Mech 7(5):789–805
Munjiza A, Rougier E, Knight EE, Lei Z (2023a) Discrete and combined finite discrete element methods for computational mechanics of discontinua. Comprehens Struct Integr 2023:V3-408-V403-428
Munjiza A, Rougier E, Lei Z, Euser B, Knight EE (2023b) Towards FDEM based hybrid simulation tools for AI driven virtual experimentation in science and engineering. Elsevier, London
Peng X, Chen G, Yu P, Zhang Y, Zhang H, Guo L (2020) A full-stage parallel architecture of three-dimensional discontinuous deformation analysis using OpenMP. Comput Geotech 118:103346. https://doi.org/10.1016/j.compgeo.2019.103346
Schiava D’Albano GG (2014) Computational and algorithmic solutions for large scale combined finite-discrete elements simulations. University of London, Queen Mary
Sharafisafa M, Aliabadian Z, Sato A, Shen L (2023) Coupled thermo-hydro-mechanical simulation of hydraulic fracturing in deep reservoirs using finite-discrete element method. Rock Mech Rock Eng 56(7):5039–5075. https://doi.org/10.1007/s00603-023-03325-z
Shen B, Stephansson O, Rinne M (2020) Modelling rock fracturing processes with FRACOD. In: Shen B, Stephansson O, Rinne M (eds) Modelling rock fracturing processes: theories, methods, and applications. Springer, Cham, pp 105–134
Smoljanović H, Živaljić N, Nikolić Ž, Munjiza A (2018) Numerical analysis of 3D dry-stone masonry structures by combined finite-discrete element method. Int J Solids Struct 136–137(1):150–167. https://doi.org/10.1016/j.ijsolstr.2017.12.012
Sun L, Grasselli G, Liu Q, Tang X (2019) Coupled hydro-mechanical analysis for grout penetration in fractured rocks using the finite-discrete element method. Int J Rock Mech Min Sci 124:104138. https://doi.org/10.1016/j.ijrmms.2019.104138
Tao Z, Luo S, Li M, Shulin R, Manchao H (2020) Optimization of large deformation control parameters of layered slate tunnels based on numerical simulation and field test. Chin J Rock Mech Eng 39(3):491–506
Tatone BSA, Grasselli G (2015) A calibration procedure for two-dimensional laboratory-scale hybrid finite-discrete element simulations. Int J Rock Mech Min Sci 75:56–72. https://doi.org/10.1016/j.ijrmms.2015.01.011
Tatone B, Lisjak A, Mahabadi O, Vlachopoulos N (2015) Incorporating rock reinforcement elements into numerical analyses based on the hybrid finite-discrete element method (FDEM). In: Proceedings of the 13th ISRM congress: innovations in applied and theoretical rock mechanics, Canada
Ulusay R, Hudson JA (2012) Suggested methods for rock failure criteria: general introduction. Rock Mech Rock Eng 45(6):971. https://doi.org/10.1007/s00603-012-0273-7
Wang Z, Liu Q (2021) Failure criterion for soft rocks considering intermediate principal stress. Int J Min Sci Technol 31(4):565–575. https://doi.org/10.1016/j.ijmst.2021.05.005
Wang W, Liu Q, Ma H, Lu H, Wang Z (2020) Numerical analysis of material modeling rock reinforcement in 2D FDEM and parameter study. Comput Geotech 126(1):103767. https://doi.org/10.1016/j.compgeo.2020.103767
Wang Z, Liu Q, Wang Y (2021) Thermo-mechanical FDEM model for thermal cracking of rock and granular materials. Powder Technol 393:807–823. https://doi.org/10.1016/j.powtec.2021.08.030
Wang X, Wang Q, An B, He Q, Wang P, Wu J (2022) A GPU parallel scheme for accelerating 2D and 3D peridynamics models. Theoret Appl Fract Mech 121:103458. https://doi.org/10.1016/j.tafmec.2022.103458
Wei D, Zhao B, Dias-da-Costa D, Gan Y (2019) An FDEM study of particle breakage under rotational point loading. Eng Fract Mech 212:221–237. https://doi.org/10.1016/j.engfracmech.2019.03.036
Wu Z, Cui W, Weng L, Liu Q (2023) Modeling geothermal heat extraction-induced potential fault activation by developing an FDEM-based THM coupling scheme. Rock Mech Rock Eng 56(5):3279–3299. https://doi.org/10.1007/s00603-023-03218-1
Xiang J, Munjiza A, Latham J-P (2009) Finite strain, finite rotation quadratic tetrahedral element for the combined finite-discrete element method. Int J Numer Methods Eng 79(8):946–978. https://doi.org/10.1002/nme.2599
Xiang J, Latham J-P, Farsi A (2017) Algorithms and capabilities of solidity to simulate interactions and packing of complex shapes. In: Li X, Feng Y, Mustoe G (eds) Proceedings of the 7th international conference on discrete element methods. Springer, Singapore, pp 139–149
Xu D, Liu X, Jiang Q, Li S, Zhou Y, Qiu S, Yan F, Zheng H, Huang X (2022) A local homogenization approach for simulating the reinforcement effect of the fully grouted bolt in deep underground openings. Int J Min Sci Technol 32(2):247–259. https://doi.org/10.1016/j.ijmst.2022.01.003
Yan C, Zheng H, Sun G, Ge X (2014) Parallel analysis of two-dimensional finite-discrete element method based on OpenMP. Rock Soil Mech 35(9):2717–2724. https://doi.org/10.16285/j.rsm.2014.09.002
Yan C, Ma H, Tang Z, Ke W (2022a) A two-dimensional moisture diffusion continuous model for simulating dry shrinkage and cracking of soil. Int J Geomech 22(10):04022172. https://doi.org/10.1061/(ASCE)GM.1943-5622.0002570
Yan C, Wang T, Gao Y, Ke W, Wang G (2022b) A three-dimensional grouting model considering hydromechanical coupling based on the combined finite-discrete element method. Int J Geomech 22(11):04022189. https://doi.org/10.1061/(ASCE)GM.1943-5622.0002448
Yao F, Ma G, Guan S, Chen Y, Liu Q, Feng C (2020) Interfacial shearing behavior analysis of rockfill using FDEM simulation with irregularly shaped particles. Int J Geomech 20(3):04019193. https://doi.org/10.1061/(ASCE)GM.1943-5622.0001590
Yu P, Peng X, Chen G, Guo L, Zhang Y (2020) OpenMP-based parallel two-dimensional discontinuous deformation analysis for large-scale simulation. Int J Geomech 20(7):04020083. https://doi.org/10.1061/(ASCE)GM.1943-5622.0001705
Zhang L, Quigley SF, Chan AHC (2013) A fast scalable implementation of the two-dimensional triangular discrete element method on a GPU platform. Adv Eng Softw 60–61:70–80. https://doi.org/10.1016/j.advengsoft.2012.10.006
Zhou B, Wei D, Ku Q, Wang J, Zhang A (2020) Study on the effect of particle morphology on single particle breakage using a combined finite-discrete element method. Comput Geotech 122:103532. https://doi.org/10.1016/j.compgeo.2020.103532
Zivaljic N, Smoljanović H, Nikolić Ž (2013) A combined finite-discrete element model for RC structures under dynamic loading. Eng Comput Int J Comput Aided Eng. https://doi.org/10.1108/EC-03-2012-0066
Acknowledgements
We appreciate the comments of our anonymous reviewers to improve the quality of our manuscript.
Funding
This work was funded by the National Natural Science Foundation of China (Grant No. 52378309), Youth Science and Technology Innovation Fund of BGRIMM Technology Group (Grant No. 04-2349), Shenzhen Science and Technology Program (Grant No. KQTD20180412181337494), China Postdoctoral Science Foundation (Grant Nos. 2022TQ0218 and 2022M722187), and Visiting Researcher Fund Program of State Key Laboratory of Water Resources Engineering and Management (Grant No. 2022SGG05).
Author information
Authors and Affiliations
Contributions
ZW: Conceptualization, Methodology, Software, Writing—original draft. FL: Supervision, Validation, Writing—review and editing. GM: Writing—review and editing.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, Z., Li, F. & Mei, G. OpenMP Parallel Finite-Discrete Element Method for Modeling Excavation Support with Rockbolt and Grouting. Rock Mech Rock Eng 57, 3635–3657 (2024). https://doi.org/10.1007/s00603-023-03746-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00603-023-03746-w