Skip to main content
SpringerLink
Log in

Scalable multiscale modeling of platelets with 100 million particles

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

We developed the core components of the AI-aided multiple time stepping algorithm for multiscale modeling of cell dynamics. This algorithm was implemented and analyzed on two supercomputer architectures with an application of simulating the aggregation of 250 platelets, or 102 million particles. To scale on these computers with complex memory and network architectures with GPUs, we devised a biomechanics-informed task mapping scheme to optimize load imbalance, communications, and memory utilization. Our simulations, scaling well up to 192 nodes on a Summit-like supercomputer with a peak speed of 11 petaflops, achieved a rate of 423 \(\upmu \)s/day which is 500 times faster than the conventional algorithm using static time step and this has enabled studies of record size blood clots at record spatial–temporal resolutions. Additionally, we discovered the sensitive dependence of the scalability and execution time on the methods of decomposition, CPU–GPU coupling, and task mapping.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

All data generated or analyzed during this study are included in this published article.

References

  1. Dror RO, Dirks RM, Grossman JP, Xu H, Shaw DE (2012) Biomolecular simulation: a computational microscope for molecular biology. Annu Rev Biophys 41:429–52. https://doi.org/10.1146/annurev-biophys-042910-155245

    Article  Google Scholar 

  2. Karplus M, McCammon JA (2002) Molecular dynamics simulations of biomolecules. Nat Struct Biol 9(9):646–652. https://doi.org/10.1038/nsb0902-646

    Article  Google Scholar 

  3. Monaghan JJ (1992) Smoothed particle hydrodynamics. Annu Rev Astron Astrophys 30(1):543–574. https://doi.org/10.1146/annurev.aa.30.090192.002551

    Article  Google Scholar 

  4. Marrink SJ, De Vries AH, Mark AE (2004) Coarse grained model for semiquantitative lipid simulations. J Phys Chem B 108(2):750–760. https://doi.org/10.1021/jp036508g

    Article  Google Scholar 

  5. Groot RD, Warren PB (1997) Dissipative particle dynamics: bridging the gap between atomistic and mesoscopic simulation. J Chem Phys 107(11):4423–4435. https://doi.org/10.1063/1.474784

    Article  Google Scholar 

  6. Hughes TJR (2012) The finite element method: linear static and dynamic finite element analysis. Cour Corp. Cour Corp. Englewood Cliffs, New Jersey 4:245

  7. Ayton GS, Noid WG, Voth GA (2007) Multiscale modeling of biomolecular systems: in serial and in parallel. Curr Opin Struct Biol 17(2):192–198. https://doi.org/10.1016/j.sbi.2007.03.004

    Article  Google Scholar 

  8. Tuckerman ME, Berne BJ, Martyna GJ (1991) Molecular dynamics algorithm for multiple time scales: Systems with long range forces. J Chem Phys 94(10):6811–6815. https://doi.org/10.1063/1.460259

    Article  Google Scholar 

  9. Shaw DE, Deneroff MM, Dror RO, Kuskin JS, Larson RH, Salmon JK, Young C, Batson B, Bowers KJ, Chao JC (2008) Anton, a special-purpose machine for molecular dynamics simulation. Commun ACM 51(7):91–97. https://doi.org/10.1145/1364782.1364802

    Article  Google Scholar 

  10. Shaw DE, Grossman J, Bank JA, Batson B, Butts JA, Chao JC, Deneroff MM, Dror RO, Even A, Fenton CH (2014) Anton 2: raising the bar for performance and programmability in a special-purpose molecular dynamics supercomputer. In: SC’14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, IEEE. https://doi.org/10.1109/SC.2014.9

  11. Kumar S, Huang C, Zheng G, Bohm E, Bhatele A, Phillips JC, Yu H, Kalé LV (2008) Scalable molecular dynamics with namd on the ibm blue gene/l system. IBM J Res Dev 52(1.2):177–188. https://doi.org/10.1147/rd.521.0177

    Article  Google Scholar 

  12. Fu H, Liao J, Yang J, Wang L, Song Z, Huang X, Yang C, Xue W, Liu F, Qiao F (2016) The sunway taihulight supercomputer: system and applications. Sci China Inf Sci 59(7):072001. https://doi.org/10.1007/s11432-016-5588-7

    Article  Google Scholar 

  13. Hérault A, Bilotta G, Dalrymple RA (2010) Sph on gpu with cuda. J Hydraul Res 48(sup1):74–79. https://doi.org/10.1080/00221686.2010.9641247

    Article  Google Scholar 

  14. Yang K, Bai Z, Su J, Guo H (2014) Efficient and large-scale dissipative particle dynamics simulations on gpu. Soft Mater 12(2):185–196. https://doi.org/10.1080/1539445X.2013.858359

    Article  Google Scholar 

  15. Jia W, Wang H, Chen M, Lu D, Lin L, Car R, Weinan E, Zhang L (2020) SC20: Pushing the limit of molecular dynamics with ab initio accuracy to 100 million atoms with machine learning. Int Conf High Perform Comput Netw Storage Anal. https://doi.org/10.1109/SC41405.2020.0000

    Article  Google Scholar 

  16. Strohmaier E, Dongarra J, Simon H, Meuer M, Meuer H (2021) TOP500 Supercomputing Sites. https://www.top500.org/lists/top500

  17. Zhang P, Gao C, Zhang N, Slepian MJ, Deng Y, Bluestein D (2014) Multiscale particle-based modeling of flowing platelets in blood plasma using dissipative particle dynamics and coarse grained molecular dynamics. Cell Mol Bioeng 7(4):552–574. https://doi.org/10.1007/s12195-014-0356-5

    Article  Google Scholar 

  18. Gupta P, Zhang P, Sheriff J, Bluestein D, Deng Y (2019) A multiscale model for recruitment aggregation of platelets by correlating with in vitro results. Cell Mol Bioeng 12(4):327–343. https://doi.org/10.1007/s12195-019-00583-2

    Article  Google Scholar 

  19. Bluestein D (2004) Research approaches for studying flow-induced thromboembolic complications in blood recirculating devices. Exp Rev Med Devices 1(1):65–80. https://doi.org/10.1586/17434440.1.1.65

    Article  Google Scholar 

  20. Tsao CW, Aday AW, Almarzooq ZI, Alonso A, Beaton AZ, Bittencourt MS, Boehme AK, Buxton AE, Carson AP, Commodore-Mensah Y (2022) Heart disease and stroke statistics-2022 update: a report from the american heart association. Circulation 145:153–639. https://doi.org/10.1161/CIR.0000000000001052

    Article  Google Scholar 

  21. Wang W, King MR (2012) Multiscale modeling of platelet adhesion and thrombus growth. Annal Biomed Eng 40:2345–2354. https://doi.org/10.1007/s10439-012-0558-8

    Article  Google Scholar 

  22. Fedosov DA, Caswell B, Karniadakis GE (2010) A multiscale red blood cell model with accurate mechanics, rheology, and dynamics. Biophys J 98:2215–2225. https://doi.org/10.1016/j.bpj.2010.02.002

    Article  Google Scholar 

  23. Insley JA, Grinberg L, Fedosov DA, Morozov V, Caswell B, Papka ME, Karniadakis GE (2011) Blood flow: multi-scale modeling and visualization. In: Proceedings of the 2011 Companion on High Performance Computing Networking, Storage and Analysis Companion, p 139–140. https://doi.org/10.1145/2148600.2148673

  24. Mody NA, King MR (2008) Platelet adhesive dynamics. part i: characterization of platelet hydrodynamic collisions and wall effects. Biophys J 95:2539–2555. https://doi.org/10.1529/biophysj.107.127670

    Article  Google Scholar 

  25. Mody NA, King MR (2008) Platelet adhesive dynamics. part ii: High shear-induced transient aggregation via gpib\(\alpha \)-vwf-gpib\(\alpha \) bridging. Biophys J 95:2556–2574. https://doi.org/10.1529/biophysj.107.128520

    Article  Google Scholar 

  26. Pivkin IV, Richardson PD, Karniadakis G (2006) Blood flow velocity effects and role of activation delay time on growth and form of platelet thrombi. Proceed Nat Acad Sci 103:17164–17169. https://doi.org/10.1073/pnas.0608546103

    Article  Google Scholar 

  27. Zheng X, Yazdani A, Li H, Humphrey JD, Karniadakis GE (2020) A three-dimensional phase-field model for multiscale modeling of thrombus biomechanics in blood vessels. PLoS Comput Biol 16:1007709. https://doi.org/10.1371/journal.pcbi.1007709

    Article  Google Scholar 

  28. Zhu Y, Zhang P, Han C, Cong G, Deng Y (2021) Enabling ai-accelerated multiscale modeling of thrombogenesis at millisecond and molecular resolutions on supercomputers High performance computing. Springer International Publishing, Germany, pp 237–254. https://doi.org/10.1007/978-3-030-78713-4_13

    Book  Google Scholar 

  29. Zhang Z, Zhang P, Han C, Cong G, Yang C-C, Deng Y (2021) Online machine learning for accelerating molecular dynamics modeling of cells. Front Mol Biosci. https://doi.org/10.3389/fmolb.2021.81224

    Article  Google Scholar 

  30. Hanson WA (2019) The coral supercomputer systems. IBM J Res Dev 64(3/4):1. https://doi.org/10.1147/JRD.2019.2960220

    Article  Google Scholar 

  31. Han C, Zhang P, Bluestein D, Cong G, Deng Y (2020) Artificial intelligence for accelerating time integrations in multiscale modeling. J Comput Phys 427:110053. https://doi.org/10.1016/j.jcp.2020.110053

    Article  MathSciNet  MATH  Google Scholar 

  32. Plimpton S (1995) Fast parallel algorithms for short-range molecular dynamics. J Comput Phys 117(1):1–19. https://doi.org/10.1006/jcph.1995.1039

    Article  MATH  Google Scholar 

  33. Deng Y, Peierls RF, Rivera C (2000) An adaptive load balancing method for parallel molecular dynamics simulations. J Comput Phys 161(1):250–263. https://doi.org/10.1006/jcph.2000.6501

    Article  MathSciNet  MATH  Google Scholar 

  34. Phillips JC, Braun R, Wang W, Gumbart J, Tajkhorshid E, Villa E, Chipot C, Skeel RD, Kale L, Schulten K (2005) Scalable molecular dynamics with namd. J Comput Chem 26(16):1781–1802. https://doi.org/10.1002/jcc.20289

    Article  Google Scholar 

  35. Hess B, Kutzner C, van der Spoel D, Lindahl E (2008) Gromacs 4: algorithms for highly efficient, load-balanced, and scalable molecular simulation. J Chem Theory Comput 4(3):435–447. https://doi.org/10.1021/ct700301q

    Article  Google Scholar 

  36. Anderson DA, Tannehill JC, Pletcher RH, Ramakanth M, Shankar V (2020) Computational fluid mechanics and heat transfer. CRC Press. https://doi.org/10.1201/9781351124027

    Article  Google Scholar 

  37. Zhang P, Zhang N, Deng Y, Bluestein D (2015) A multiple time stepping algorithm for efficient multiscale modeling of platelets flowing in blood plasma. J Comput Phys 284:668–686. https://doi.org/10.1016/j.jcp.2015.01.004

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This project is supported by the SUNY-IBM Consortium Award, IPDyna: Intelligent Platelet Dynamics, FP00004096 (PI: Y. Deng). The simulations in this study were conducted on the AiMOS supercomputer at Rensselaer Polytechnic Institute and the SeaWulf Cluster at Stony Brook University (PIs: Y. Deng and P. Zhang).

Author information

Author notes

  1. Changnian Han, and, Peng Zhang, and, Yicong Zhu These authors contributed equally to this work.

Authors and Affiliations

  1. Department of Applied Mathematics and Statistics, Stony Brook University, 100 Nicolls Road, Stony Brook, 11794, NY, USA

    Changnian Han, Peng Zhang, Yicong Zhu & Yuefan Deng

  2. Oak Ridge National Laboratory, P.O. Box 2008, Oak Ridge, 37831, TN, USA

    Guojing Cong

  3. IBM Thomas J. Watson Research Center, Yorktown Heights, 10598, NY, USA

    James R. Kozloski, Chih Chieh Yang & Leili Zhang

  4. Mathematics, Division of Science, New York University, Abu Dhabi, PO Box 129188, UAE

    Yuefan Deng

Authors
  1. Changnian Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yicong Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guojing Cong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. James R. Kozloski
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chih Chieh Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Leili Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yuefan Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuefan Deng.

Ethics declarations

Conflict of interest

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, C., Zhang, P., Zhu, Y. et al. Scalable multiscale modeling of platelets with 100 million particles. J Supercomput 78, 19707–19724 (2022). https://doi.org/10.1007/s11227-022-04648-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04648-4

Keywords

Search

Navigation