Skip to main content
Log in

Deep-learning-enhanced model reconstruction of realistic 3D rock particles by intelligent video tracking of 2D random particle projections

  • Research Paper
  • Published:
Acta Geotechnica Aims and scope Submit manuscript

Abstract

Rock assemblies, such as ballasts and cobbles, are typically composed of granular particles of various sizes and shapes that significantly affect their mechanical behavior. Currently, it remains challenging to generate realistic and identical three-dimensional (3D) models for rock particles with a limited number of their individual two-dimensional (2D) projections. This study proposes a novel and effective deep-learning-enhanced approach to achieve the realistic reconstruction of 3D particle models, which includes the following sequential steps: (1) automatically detecting 2D projected images of each 3D particle individually during dynamic scanning of a rock particle assembly; (2) predicting 3D particle size indexes based on 2D images for each particle; (3) generating a corresponding 3D skeleton network from 2D outlines; and (4) reconstructing the particle surface of individual particle by spatial interpolations. Sensitivity analysis and verification are finally performed on both randomized superball particles and real cobbles/ballasts. The results demonstrated that the proposed approach is robust and efficient in reconstructing 3D rock particle models and offers a rapid and low-cost practical solution for convincible numerical studies of granular materials.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Data availability

All data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. Alhani IJ, Noor MJBM, Al-Bared MAM, Harahap ISH, Albadri WM (2020) Mechanical response of saturated and unsaturated gravels of different sizes in drained triaxial testing. Acta Geotechnica 15(11):3075

    Google Scholar 

  2. Bewley A, Ge Z, Ott L, Ramos F, Upcroft B (2016) Simple online and realtime tracking. In: 2016 IEEE international conference on image processing (ICIP), Phoenix, USA, 25–28 September, pp 3464–3468

  3. Brechbühler C, Gerig G, Kübler O (1995) Parametrization of closed surfaces for 3D shape description. Comput Vis Image Underst 61(2):154–170

    Google Scholar 

  4. Chen Z, Han Z, Hao J, Zhu Q, Soh YC (2015) Fusion of wifi, smartphone sensors and landmarks using the kalman filter for indoor localization. Sensors 15:715–732

    Google Scholar 

  5. Chen M, Li M, Li Y (2021) Rock particle motion information detection based on video instance segmentation. Sensors 21(12):4108

    Google Scholar 

  6. Cho GC, Dodds J, Santamarina JC (2006) Particle shape effects on packing density, stiffness, and strength: natural and crushed sands. J Geotechn Geoenviron Eng 132(5):591–602

    Google Scholar 

  7. Dan HC, Bai GW, Zhu ZH (2021) Application of deep learning-based image recognition technology to asphalt–aggregate mixtures. Methodol Constr Build Mater 297:123770

    Google Scholar 

  8. Deng J, Guo J, Xue N, Zafeiriou S (2019) Arcface: additive angular margin loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Long Beach, CA, USA, 16–20 June, pp 4690–4699

  9. Farber L, Tardos G, Michaels JN (2003) Use of X-ray tomography to study the porosity and morphology of granules. Powder Technol 132(1):57–63

    Google Scholar 

  10. Galiyawala H, Raval MS, Dave S (2019) Visual appearance based person retrieval in unconstrained environment videos. Image Vis Comput 92:103816

    Google Scholar 

  11. Ganju E, Kılıç M, Prezzi M, Salgado R, Parab N, Chen W (2021) Effect of particle characteristics on the evolution of particle size, particle morphology, and fabric of sands loaded under uniaxial compression. Acta Geotech 16(11):3489–3516

    Google Scholar 

  12. Jiang H, Bian X, Cheng C, Chen Y, Chen R (2016) Simulating train moving loads in physical model testing of railway infrastructure and its numerical calibration. Acta Geotech 11(2):231–242

    Google Scholar 

  13. Kim Y, Ma J, Lim SY, Song JY, Yun TS (2022) Determination of shape parameters of sands: a deep learning approach. Acta Geotech 17:1–11

    Google Scholar 

  14. Kuo CY, Frost JD, Lai JS, Wang LB (1996) Three-dimensional image analysis of aggregate particles from orthogonal projections. Transp Res Rec 1526(1):98–103

    Google Scholar 

  15. Lai Z, Chen Q (2019) Reconstructing granular particles from X-ray computed tomography using the TWS machine learning tool and the level set method. Acta Geotech 14(1):1–18

    Google Scholar 

  16. Liang Z, Nie Z, An A, Gong J, Wang X (2019) A particle shape extraction and evaluation method using a deep convolutional neural network and digital image processing. Powder Technol 353:156–170

    Google Scholar 

  17. List J, Köhler U, Witt W (2011) Dynamic image analysis extended to fine and coarse particles. Part Syst Anal 2011:1–5

    Google Scholar 

  18. Lu M, McDowell GR (2007) The importance of modelling ballast particle shape in the discrete element method. Gr Matter 9(1–2):69

    Google Scholar 

  19. Mahalanobis PC (1936) On the generalized distance in statistics. National Institute of Science of India

  20. Mahawish A, Bouazza A, Gates WP (2018) Effect of particle size distribution on the bio-cementation of coarse aggregates. Acta Geotech 13(4):1019–1025

    Google Scholar 

  21. Marsaglia G (1972) Choosing a point from the surface of a sphere. Ann Math Stat 43(2):645–646

    MATH  Google Scholar 

  22. Nguyen H, Bui X-N, Tran Q-H, Nguyen D-A, Hoa LTT, Le Q-T et al (2021) Predicting blast-induced ground vibration in open-pit mines using different nature-inspired optimization algorithms and deep neural network. Nat Resour Res. https://doi.org/10.1007/s11053-021-09896-4

    Article  Google Scholar 

  23. Nie Z, Liang Z, Wang X (2018) A three-dimensional particle roundness evaluation method. Gr Matter 20(2):32

    Google Scholar 

  24. Nurzynska K, Iwaszenko S (2020) Application of texture features and machine learning methods to grain segmentation in rock material images. Image Anal Stereol 39(2):73–90

    MathSciNet  MATH  Google Scholar 

  25. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Chintala S (2019) Pytorch: An imperative style, high-performance deep learning library. Adv Neural Inf Process Syst 32:1

    Google Scholar 

  26. Rahman MM, Dafalias YF (2021) Modelling undrained behaviour of sand with fines and fabric anisotropy. Acta Geotech 17:2305

    Google Scholar 

  27. Rao C, Tutumluer E, Kim IT (2002) Quantification of coarse aggregate angularity based on image analysis. Transp Res Rec 1787(1):117–124

    Google Scholar 

  28. Russell BC, Torralba A, Murphy KP, Freeman WT (2008) LabelMe: a database and web-based tool for image annotation. Int J Comput Vis 77(1):157–173

    Google Scholar 

  29. Su D, Wang X, Yang H, Hong C (2019) Roughness analysis of general-shape particles, from 2D closed outlines to 3D closed surfaces. Powder Technol 356:423

    Google Scholar 

  30. Su D, Yan WM (2018) 3D characterization of general-shape sand particles using microfocus X-ray computed tomography and spherical harmonic functions, and particle regeneration using multivariate random vector. Powder Technol 323:8–23

    Google Scholar 

  31. Su D, Yan WM (2019) Prediction of 3D size and shape descriptors of irregular granular particles from projected 2D images. Acta Geotech 15:1–23

    Google Scholar 

  32. Sun Z, Wang C, Hao X, Li W, Zhang X (2020) Quantitative evaluation for shape characteristics of aggregate particles based on 3D point cloud data. Constr Build Mater 263:120156

    Google Scholar 

  33. Sun Q, Zheng Y, Li B, Zheng J, Wang Z (2019) Three-dimensional particle size and shape characterisation using structural light. Géotech Lett 9(1):72–78

    Google Scholar 

  34. Ueda T, Oki T, Koyanaka S (2019) 2D–3D conversion method for assessment of multiple characteristics of particle shape and size. Powder Technol 343:287–295

    Google Scholar 

  35. Wang X, Gong J, An A, Zhang K, Nie Z (2019) Random generation of convex granule packing based on weighted Voronoi tessellation and cubic-polynomial-curve fitting. Comput Geotech 113:103088

    Google Scholar 

  36. Wang ZY, Gu DM, Zhang WG (2020) Influence of excavation schemes on slope stability: a DEM study. J Mt Sci 17(6):1509–1522

    Google Scholar 

  37. Wang X, Tian K, Su D, Zhao J (2019) Superellipsoid-based study on reproducing 3D particle geometry from 2D projections. Comput Geotech 114:103–131

    Google Scholar 

  38. Wang Z, Wang L, Zhang W (2019) A random angular bend algorithm for two-dimensional discrete modeling of granular materials. Materials 12(13):2169

    Google Scholar 

  39. Wang X, Yin ZY, Su D, Wu X, Zhao JD (2021) A novel approach of random packing generation of complex-shaped 3D particles with controllable sizes and shapes. Acta Geotechnica 17:1–22

    Google Scholar 

  40. Wang X, Yin ZY, Su D, Xiong H, Feng YT (2021) A novel Arcs-based discrete element modeling of arbitrary convex and concave 2D particles. Comput Methods Appl Mech Eng 386:114071

    MathSciNet  MATH  Google Scholar 

  41. Wang X, Yin ZY, Xiong H, Su D, Feng YT (2021) A spherical-harmonic-based approach to discrete element modeling of 3D irregular particles. Int J Numer Methods Eng 122(20):5626–5655

    MathSciNet  Google Scholar 

  42. Williams JR, Pentland AP (1992) Superquadrics and modal dynamics for discrete elements in interactive design. Eng Comput 9(2):115–127

    Google Scholar 

  43. Wojke N, Bewley A, Paulus D (2017) Simple online and realtime tracking with a deep association metric. In: 2017 IEEE international conference on image processing (ICIP), Beijing, China, 17–20 Sep, pp 3645–3649

  44. Xu YR, Xu Y (2021) Numerical simulation of direct shear test of rockfill based on particle breaking. Acta Geotechnica 16:1–12

    Google Scholar 

  45. Yamamoto KI, Inoue T, Miyajima T, Doyama T, Sugimoto M (2002) Measurement and evaluation of three-dimensional particle shape under constant particle orientation with a tri-axial viewer. Adv Powder Technol 13(2):181–200

    Google Scholar 

  46. Yan WM, Su D (2018) Inferring 3D particle size and shape characteristics from projected 2D images: lessons learned from ellipsoids. Comput Geotech 104:281–287

    Google Scholar 

  47. Yan WM, Su D (2018) Evaluation of three-dimensional particle shape index from projected two-dimensional image. Géotech Lett 8(4):336–343

    Google Scholar 

  48. Yang HW, Lourenço SD, Baudet BA, Choi CE, Ng CW (2019) 3D Analysis of gravel surface texture. Powder Technol 346:414–424

    Google Scholar 

  49. Yang L, Nguyen H, Bui X-N, Nguyen-Thoi T, Zhou J, Huang J (2021) Prediction of gas yield generated by energy recovery from municipal solid waste using deep neural network and moth-flame optimization algorithm. J Clean Prod 311:127672

    Google Scholar 

  50. Yang L, Fan Y, Xu N (2019) Video instance segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 5188–5197

  51. Zhang H, Nguyen H, Bui X-N, Nguyen-Thoi T, Bui T-T, Nguyen N et al (2020) Developing a novel artificial intelligence model to estimate the capital cost of mining projects using deep neural network-based ant colony optimization algorithm. Resour Policy 66:101604

    Google Scholar 

  52. Zhang H, Nguyen H, Bui X-N, Pradhan B, Asteris PG, Costache R et al (2021) A generalized artificial intelligence model for estimating the friction angle of clays in evaluating slope stability using a deep neural network and Harris Hawks optimization algorithm. Eng Comput. https://doi.org/10.1007/s00366-020-01272-9

    Article  Google Scholar 

  53. Zhao S, Zhou X (2017) Effects of particle asphericity on the macro-and micro-mechanical behaviors of granular assemblies. Gr Matter 19(2):38

    Google Scholar 

  54. Zhou W, Liu J, Ma G, Chang X (2017) Three-dimensional DEM investigation of critical state and dilatancy behaviors of granular materials. Acta Geotech 12(3):527–540

    Google Scholar 

  55. Zhou B, Wang J (2017) Generation of a realistic 3D sand assembly using X-ray micro-computed tomography and spherical harmonic-based principal component analysis. Int J Numer Anal Methods Geomech 41(1):93–109

    MathSciNet  Google Scholar 

  56. Zhou B, Wang J, Zhao B (2015) Micromorphology characterization and reconstruction of sand particles using micro X-ray tomography and spherical harmonics. Eng Geol 184:126–137

    Google Scholar 

  57. Zou Y, Ma G, Mei J, Zhao J, Zhou W (2021) Microscopic origin of shape-dependent shear strength of granular materials: a granular dynamics perspective. Acta Geotechnica. https://doi.org/10.1007/s11440-021-01403-6

    Article  Google Scholar 

Download references

Acknowledgements

This research was financially supported by the Research Grants Council (RGC) of Hong Kong Special Administrative Region Government (HKSARG) of China (Grant Nos. 15220221), Natural Science Foundation of Shenzhen, China (No. JCYJ20210324094607020) and the National Natural Science Foundation of China (Grant No. 51878416).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zhen-Yu Yin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1

To generate the 3D model for a superball particle, the polar representation is used in this study. Given the local spherical coordinate \(\left(\theta ,\varphi \right)\) of a point on the surface, the corresponding local Cartesian coordinate \(\left(x,y,z\right)\) is expressed as:

$$\left\{ {\begin{array}{*{20}l} {x\left( {\theta ,\varphi } \right) = {\text{Sign}}\left( {{\text{cos}}\left( \theta \right)} \right)a\left| {{\text{cos}}\left( \theta \right)} \right|^{\varepsilon } \left| {{\text{cos}}\left( \varphi \right)} \right|^{\varepsilon } } \hfill \\ {y\left( {\theta ,\varphi } \right) = {\text{Sign}}\left( {{\text{sin}}\left( \theta \right)} \right)b\left| {{\text{sin}}\left( \theta \right)} \right|^{\varepsilon } \left| {{\text{cos}}\left( \varphi \right)} \right|^{\varepsilon } } \hfill \\ {z\left( {\theta ,\varphi } \right) = {\text{Sign}}\left( {{\text{sin}}\left( \varphi \right)} \right){\text{c}}\left| {{\text{sin}}\left( \varphi \right)} \right|^{\varepsilon } } \hfill \\ \end{array} } \right.$$
(45)

where term \(\mathrm{Sign}\left(x\right)\) is the signum function defined as:

$$\mathrm{Sign}\left(x\right)=\left\{\begin{array}{l}-1\\ 0 \\ 1\end{array}\right. \begin{array}{l}\quad \mathrm{if} \quad x<0,\\ \quad\mathrm{if}\quad x=0,\\\quad \mathrm{if} \quad x>0.\end{array}$$
(46)

Appendix 2

The \(\theta\) and \(\varphi\) for a random unit vector in 3D space can be easily generated through the following steps:

  • Step 1: Generate a pair of random variables \({\xi }_{1}\) and \({\xi }_{2}\) that are uniformly distributed on [− 1,1].

  • Step 2: Calculate \({r}^{2}\) by Eq. (47). If \({r}^{2}<1\), abandon this pair and repeat step 1.

    $${r}^{2}={{\xi }_{1}}^{2}+{{\xi }_{2}}^{2}$$
    (47)
  • Step 3: Calculate \({n}_{x},{n}_{y},{n}_{z}\) by the following expression

    $$\left\{ {\begin{array}{*{20}l} {n_{x} = 2\xi _{1} \sqrt[2]{{1 - r^{2} }}} \hfill \\ {n_{y} = 2\xi _{2} \sqrt[2]{{1 - r^{2} }}} \hfill \\ {n_{x} = 1 - r^{2} } \hfill \\ \end{array} } \right.$$
    (48)

Appendix 3

The process to search all the surface points on the 3D particle that corresponds to the projected points in plane M is implemented based on the following steps:

  1. 1.

    Randomly select one point \({Q}_{1}\) from plane M such that vector \(\mathop \rightharpoonup \limits_{{OQ_{1} }} \left({x}_{q1}, {y}_{q1},{z}_{q1}\right)\) is perpendicular to the projection vector \(\mathbf{O}\mathbf{V}\), as shown in Fig. 6b.

  2. 2.

    Find the surface point \({P}_{1}\), as shown in Fig. 6c that has a normal vector \(\mathop \rightharpoonup \limits_{{N_{1} }}\) parallel to vector \(\mathop \rightharpoonup \limits_{{OQ_{1} }}\) as follows: Given a normal vector \(\mathop \rightharpoonup \limits_{{N_{1} }}\left({n}_{x},{n}_{y},{n}_{z}\right)\) on the surface, the corresponding local spherical coordinates \(\left(\theta ,\varphi \right)\) of \({P}_{1}\) are obtained using the following equations:

    $$\theta =\mathrm{atan}2\left[{\mathrm{Sign}\left({n}_{y}\right)\left|b{n}_{y}\right|}^{\frac{1}{2-{\varepsilon }_{1}}},{\mathrm{Sign}\left({n}_{x}\right)\left|a{n}_{x}\right|}^{\frac{1}{2-{\varepsilon }_{1}}}\right]$$
    (49)
    $$\varphi =\mathrm{atan}2\left[{\mathrm{Sign}\left({n}_{z}\right)\left|c{n}_{z}\right|\mathrm{cos}\theta }^{\left|2-{\varepsilon }_{2}\right|\frac{1}{2-{\varepsilon }_{2}}},{\left|a{n}_{x}\right|}^{\frac{1}{2-{\varepsilon }_{2}}}\right]$$
    (50)

    where the term \(\mathrm{atan}2\left(x,y\right)\) is the arctangent function of \(\frac{x}{y}\) producing results in the range \((-\uppi ,\uppi ]\). After the local spherical coordinates \(\left(\theta ,\varphi \right)\) of \({P}_{1}\) are obtained, the corresponding local Cartesian coordinates \(\left(x,y,z\right)\) of \({P}_{1}\) can be computed using Eq. (45).

  3. 3.

    Rotate \(\mathop \rightharpoonup \limits_{{OQ_{1} }}\) at a discretized angle \(\Delta\updelta =\uppi /\mathrm{m}\) around axis \(\mathbf{O}\mathbf{V}\) to obtain a new vector \(\mathop \rightharpoonup \limits_{{OQ_{i}^{\prime } }}\) and find the corresponding surface point \({P}_{i}\) with \(\mathop \rightharpoonup \limits_{{N_{i} }} \left(\updelta \right)\) parallel to \(\mathop \rightharpoonup \limits_{{OQ_{i}^{\prime } }}\);

  4. 4.

    Repeat (3) until \(\updelta =2\uppi\), and connect all acquired points, in turn, to form the full projected outline, as shown in Fig. 6d.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, X., Zhang, H., Yin, ZY. et al. Deep-learning-enhanced model reconstruction of realistic 3D rock particles by intelligent video tracking of 2D random particle projections. Acta Geotech. 18, 1407–1430 (2023). https://doi.org/10.1007/s11440-022-01616-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11440-022-01616-3

Keywords

Navigation