Implementation of scalable bidomain-based 3D cardiac simulations on a graphics processing unit cluster

  • Ehsan EsmailiEmail author
  • Ali Akoglu
  • Salim Hariri
  • Talal Moukabary


Computational models of the human cardiac cells provide detailed properties of human ventricular cells. The execution time for a realistic 3D heart simulation based on these models is a major barrier for physicians to study and understand the heart diseases, and evaluate hypotheses rapidly toward developing treatments. Graphics processing unit (GPU)-based parallelization efforts to this date have been shown to be more effective than parallelization efforts on the CPU-based clusters in terms of addressing the 3D cardiac simulation time challenge. In this paper, we review all GPU-based studies and investigate both the cardiac cell models and cardiac tissue models in 3D space. We propose algorithmic optimizations based on red black successive over-relaxation method for reducing the number of simulation iterations and convergence method for dependence elimination between neighboring cells of the heart tissue. We investigate data transfer reduction and 2D mesh partitioning strategies, evaluate their impact on thread utilization, and propose a strongly scalable cardiac simulation. Our implementation results with reducing the execution time by a factor of five compared to the state-of-the-art baseline implementation. More importantly, our implementation is an important step toward achieving real-time cardiac simulations as it achieves the strongest scalability among all other cluster-based implementations.


Cardiac simulation Bidomain model Graphics processing unit (GPU) GPU cluster 



This material is based upon the work supported by the National Science Foundation under Grant No. CNS 1624668 I/UCRC: Industry/University Cooperative Research Center for Cloud and Autonomic Computing.


  1. 1.
    Desai AS, Stevenson LW (2012) Rehospitalization for heart failure: predict or prevent? Circulation 126(4):501–506CrossRefGoogle Scholar
  2. 2.
    Cheng A, Dalal D, Butcher B, Norgard S, Zhang Y, Dickfeld T, Eldadah ZA, Ellenbogen KA, Guallar E, Tomaselli GF (2013) Prospective observational study of implantable cardioverter-defibrillators in primary prevention of sudden cardiac death: study design and cohort description. J Am Heart Assoc 2(1):e000083CrossRefGoogle Scholar
  3. 3.
    Nimmagadda VK, Akoglu A, Hariri S, Moukabary T (2012) Cardiac simulation on multi-GPU platform. J Supercomput 59(3):1360–1378CrossRefGoogle Scholar
  4. 4.
    Biffard R, Leon LJ (2003) Cardiac tissue simulation using graphics hardware. In: Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2003. IEEE, vol 3, pp 2838–2840Google Scholar
  5. 5.
    Rocha BM, Campos FO, Amorim RM, Plank G, dos Santos RW, Liebmann M, Haase G (2011) Accelerating cardiac excitation spread simulations using graphics processing units. Concurr Comput Pract Exp 23(7):708–720CrossRefGoogle Scholar
  6. 6.
    Vigmond EJ, Boyle PM, Leon LJ, Plank G (2009) Near-real-time simulations of biolelectric activity in small mammalian hearts using graphical processing units. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2009. EMBC 2009. IEEE, pp 3290–3293Google Scholar
  7. 7.
    Amorim R, Haase G, Liebmann M, Dos Santos RW (2009) Comparing CUDA and OpenGL implementations for a Jacobi iteration. In: HPCS’09. International Conference on High Performance Computing & Simulation, 2009. IEEE, pp 22–32Google Scholar
  8. 8.
    Amorim RM, Rocha BM, Campos FO, dos Santos RW (2010) Automatic code generation for solvers of cardiac cellular membrane dynamics in GPUs. In: 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, pp 2666–2669Google Scholar
  9. 9.
    Bartocci E, Cherry EM, Glimm J, Grosu R, Smolka SA, Fenton FH (2011) Toward real-time simulation of cardiac dynamics. In: Proceedings of the 9th International Conference on Computational Methods in Systems Biology. ACM, pp 103–112Google Scholar
  10. 10.
    Garcia VM, Liberos A, Climent AM, Vidal A, Millet J, Gonzalez A (2011) An adaptive step size GPU ODE solver for simulating the electric cardiac activity. In: Computing in Cardiology, 2011. IEEE, pp 233–236Google Scholar
  11. 11.
    García-Molla VM, Liberos A, Vidal A, Guillem M, Millet J, Gonzalez A, Martínez-Zaldívar FJ, Climent AM (2014) Adaptive step ODE algorithms for the 3D simulation of electric heart activity with graphics processing units. Comput Biol Med 44:15–26CrossRefGoogle Scholar
  12. 12.
    Jararweh Y, Jarrah M, Hariri S (2012) Exploiting GPUs for compute-intensive medical applications. In: 2012 International Conference on Multimedia Computing and Systems (ICMCS). IEEE, pp 29–34Google Scholar
  13. 13.
    Esmaili E, Akoglu A, Ditzler G, Hariri S, Moukabary T, Szep J (2017) Autonomic management of 3D cardiac simulations. In: 2017 International Conference on Cloud and Autonomic Computing (ICCAC). IEEE, pp 1–9Google Scholar
  14. 14.
    Yu D, Du D, Yang H, Tu Y (2014) Parallel computing simulation of electrical excitation and conduction in the 3D human heart. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, pp 4315–4319Google Scholar
  15. 15.
    Chai J, Wen M, Wu N, Huang D, Yang J, Cai X, Zhang C, Yang Q (2013) Simulating cardiac electrophysiology in the era of GPU-cluster computing. IEICE Trans Inf Syst 96(12):2587–2595CrossRefGoogle Scholar
  16. 16.
    Neic A, Liebmann M, Hoetzl E, Mitchell L, Vigmond EJ, Haase G, Plank G (2012) Accelerating cardiac bidomain simulations using graphics processing units. IEEE Trans Biomed Eng 59(8):2281–2290CrossRefGoogle Scholar
  17. 17.
    Higham J, Aslanidi O, Zhang H (2011) Large speed increase using novel GPU based algorithms to simulate cardiac excitation waves in 3D rabbit ventricles. In: Computing in Cardiology, 2011. IEEE, pp 9–12Google Scholar
  18. 18.
    Zhang L, Wang K, Zuo W, Gai C (2014) G-Heart: a GPU-based system for electrophysiological simulation and multi-modality cardiac visualization. J Comput 9(2):360–368Google Scholar
  19. 19.
    Xia Y, Wang K, Zhang H (2015) Parallel optimization of 3D cardiac electrophysiological model using GPU. Comput Math Methods Med 2015:1–10CrossRefzbMATHGoogle Scholar
  20. 20.
    Mirams GR, Arthurs CJ, Bernabeu MO, Bordas R, Cooper J, Corrias A, Davit Y, Dunn SJ, Fletcher AG, Harvey DG et al (2013) Chaste: an open source C++ library for computational physiology and biology. PLoS Comput Biol 9(3):e1002970MathSciNetCrossRefGoogle Scholar
  21. 21.
    Yang J, Chai J, Wen M, Wu N, Zhang C (2013) Solving the cardiac model using multi-core CPU and many integrated cores (MIC). In: 2013 IEEE 10th International Conference on High Performance Computing and Communications & 2013 IEEE International Conference on Embedded and Ubiquitous Computing (HPCC\_EUC). IEEE, pp 1009–1015Google Scholar
  22. 22.
    Langguth J, Lan Q, Gaur N, Cai X, Wen M, Zhang CY (2016) Enabling tissue-scale cardiac simulations using heterogeneous computing on Tianhe-2. In: 2016 IEEE 22nd International Conference on Parallel and Distributed Systems (ICPADS). IEEE, pp 843–852Google Scholar
  23. 23.
    Arevalo HJ, Vadakkumpadan F, Guallar E, Jebb A, Malamas P, Wu KC, Trayanova NA (2016) Arrhythmia risk stratification of patients after myocardial infarction using personalized heart models. Nat Commun 7:11437CrossRefGoogle Scholar
  24. 24.
    Karma A (1994) Electrical alternans and spiral wave breakup in cardiac tissue. Chaos: an Interdisciplinary. J Nonlinear Sci 4(3):461–472Google Scholar
  25. 25.
    Iyer V, Mazhari R, Winslow RL (2004) A computational model of the human left-ventricular epicardial myocyte. Biophys J 87(3):1507–1525CrossRefGoogle Scholar
  26. 26.
    Ten Tusscher K, Noble D, Noble PJ, Panfilov AV (2004) A model for human ventricular tissue. Am J Physiol-Heart Circ Physiol 286(4):H1573–H1589CrossRefGoogle Scholar
  27. 27.
    Ten Tusscher KH, Panfilov AV (2006) Alternans and spiral breakup in a human ventricular tissue model. Am J Physiol-Heart Circ Physiol 291(3):H1088–H1100CrossRefGoogle Scholar
  28. 28.
    Luo Ch, Rudy Y (1991) A model of the ventricular cardiac action potential. Depolarization, repolarization, and their interaction. Circ Res 68(6):1501–1526CrossRefGoogle Scholar
  29. 29.
    Mahajan A, Shiferaw Y, Sato D, Baher A, Olcese R, Xie LH, Yang MJ, Chen PS, Restrepo JG, Karma A et al (2008) A rabbit ventricular action potential model replicating cardiac dynamics at rapid heart rates. Biophys J 94(2):392–410CrossRefGoogle Scholar
  30. 30.
    Bondarenko VE, Szigeti GP, Bett GC, Kim SJ, Rasmusson RL (2004) Computer model of action potential of mouse ventricular myocytes. Am J Physiol-Heart Circ Physiol 287(3):H1378–H1403CrossRefGoogle Scholar
  31. 31.
    Shannon TR, Wang F, Puglisi J, Weber C, Bers DM (2004) A mathematical treatment of integrated Ca dynamics within the ventricular myocyte. Biophys J 87(5):3351–3371CrossRefGoogle Scholar
  32. 32.
    Nickerson DP, Hunter PJ (2010) Cardiac cellular electrophysiological modeling. Cardiac electrophysiology methods and models. Springer, Boston, pp 135–158CrossRefGoogle Scholar
  33. 33.
    Majumder R, Nayak AR, Pandit R (2011) Scroll-wave dynamics in human cardiac tissue: lessons from a mathematical model with inhomogeneities and fiber architecture. PLOS ONE 6(4):e18052CrossRefGoogle Scholar
  34. 34.
    Majumder R, Nayak AR, Pandit R (2012) Nonequilibrium arrhythmic states and transitions in a mathematical model for diffuse fibrosis in human cardiac tissue. PLoS ONE 7(10):e45040CrossRefGoogle Scholar
  35. 35.
    Nayak AR, Shajahan T, Panfilov A, Pandit R (2013) Spiral-wave dynamics in a mathematical model of human ventricular tissue with myocytes and fibroblasts. PloS ONE 8(9):e72950CrossRefGoogle Scholar
  36. 36.
    Smaill BH, Hunter PJ (2010) Computer modeling of electrical activation: from cellular dynamics to the whole heart. Cardiac electrophysiology methods and models. Springer, Boston, pp 159–185CrossRefGoogle Scholar
  37. 37.
    Morton KW, Mayers DF (2005) Numerical solution of partial differential equations: an introduction. Cambridge University Press, CambridgeCrossRefzbMATHGoogle Scholar
  38. 38.
    Cuellar AA, Lloyd CM, Nielsen PF, Bullivant DP, Nickerson DP, Hunter PJ (2003) An overview of CellML 1.1, a biological model description language. Simulation 79(12):740–747CrossRefGoogle Scholar
  39. 39.
    Nvidia Corporation (2017) Nvidia CUDA C programming guide, version 8.0. Accessed June 2017
  40. 40.
    Eager DL, Zahorjan J, Lazowska ED (1989) Speedup versus efficiency in parallel systems. IEEE Trans Comput 38(3):408–423CrossRefGoogle Scholar
  41. 41.
    Arioli M (2004) A stopping criterion for the conjugate gradient algorithm in a finite element method framework. Numer Math 97(1):1–24MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Zhang C, Lan H, Ye Y, Estrade BD (2005) Parallel SOR iterative algorithms and performance evaluation on a Linux cluster. Technical report, Naval Research Laboratory Stennis Space Center MS Oceanography DivisionGoogle Scholar
  43. 43.
    Hadjidimos A (2000) Successive overrelaxation (SOR) and related methods. J Comput Appl Math 123(1–2):177–199MathSciNetCrossRefzbMATHGoogle Scholar
  44. 44.
    Hackbusch W (1994) Iterative solution of large sparse systems of equations, vol 95. Springer, New YorkzbMATHGoogle Scholar
  45. 45.
    Marsh M (2012) An assessment of numerical methods for cardiac simulation. Ph.D. thesis, University of SaskatchewanGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Ehsan Esmaili
    • 1
    Email author
  • Ali Akoglu
    • 1
  • Salim Hariri
    • 1
  • Talal Moukabary
    • 2
  1. 1.Department of Electrical and Computer EngineeringUniversity of ArizonaTucsonUSA
  2. 2.Carondelet Heart and Vascular Institute - Cardiology WestTucsonUSA

Personalised recommendations