Skip to main content
Log in

Performance of preconditioned iterative linear solvers for cardiovascular simulations in rigid and deformable vessels

  • Original Paper
  • Published:
Computational Mechanics Aims and scope Submit manuscript

Abstract

Computing the solution of linear systems of equations is invariably the most time consuming task in the numerical solutions of PDEs in many fields of computational science. In this study, we focus on the numerical simulation of cardiovascular hemodynamics with rigid and deformable walls, discretized in space and time through the variational multiscale finite element method. We focus on three approaches: the problem agnostic generalized minimum residual and stabilized bi-conjugate gradient (BICGS) methods, and a recently proposed, problem specific, bi-partitioned (BIPN) method. We also perform a comparative analysis of several preconditioners, including diagonal, block-diagonal, incomplete factorization, multigrid, and resistance based methods. Solver performance and matrix characteristics (diagonal dominance, symmetry, sparsity, bandwidth and spectral properties) are first examined for an idealized cylindrical geometry with physiologic boundary conditions and then successively tested on several patient-specific anatomies representative of realistic cardiovascular simulation problems. Incomplete factorization preconditioners provide the best performance and results in terms of both strong and weak scalability. The BIPN method was found to outperform other methods in patient-specific models with rigid walls. In models with deformable walls, BIPN was outperformed by BICG with diagonal and incomplete LU preconditioners.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. High performance computer applications 6th international conference, vol. 595, Germany, 1 2016

  2. Bazilevs Y, Calo VM, Hughes TJR, Zhang Y (2008) Isogeometric fluid-structure interaction: theory, algorithms and computations. Comput Mech 43:3–37

    Article  MathSciNet  MATH  Google Scholar 

  3. Bazilevs Y, Hsu MC, Benson DJ, Sankaran S, Marsden AL (2009) Computational fluid-structure interaction: methods and application to a total cavopulmonary connection. Comput Mech 45(1):77–89

    Article  MathSciNet  MATH  Google Scholar 

  4. Benzi M (2002) Preconditioning techniques for large linear systems: a survey. J Comput Phys 182:418–477

    Article  MathSciNet  MATH  Google Scholar 

  5. Benzi M, Szyld DB, van Duin A (1999) Orderings for incomplete factorization preconditioning of nonsymmetric problems. SIAM J Sci Comput 20(5):1652–1670

    Article  MathSciNet  MATH  Google Scholar 

  6. Chen W, Poirier B (2006) Parallel implementation of efficient preconditioned linear solver for grid-based applications in chemical physics. I: block jacobi diagonalization. J Comput Phys 219(1):185–197

    Article  MathSciNet  MATH  Google Scholar 

  7. Corsini C, Cosentino D, Pennati G, Dubini G, Hsia T-Y, Migliavacca F (2011) Multiscale models of the hybrid palliation for hypoplastic left heart syndrome. J Biomech 44:767–770

    Article  Google Scholar 

  8. Deparis S, Forti D, Grandperrin G, Quarteroni A (2016) Facsi: a block parallel preconditioner for fluid-structure interaction in hemodynamics. J Comput Phys 327:700–718

    Article  MathSciNet  MATH  Google Scholar 

  9. dos Santos RW, Plank G, Bauer S, Vigmond EJ (2004) Parallel multigrid preconditioner for the cardiac bidomain model. IEEE Trans Biomed Eng 51(11):1960–1967

    Article  Google Scholar 

  10. Esmaily-Moghadam M, Bazilevs Y, Marsden AL (2013) A new preconditioning technique for implicitly coupled multidomain simulations with applications to hemodynamics. Comput Mech 52(5):1141–1152

    Article  MathSciNet  MATH  Google Scholar 

  11. Esmaily-Moghadam M, Bazilevs Y, Marsden AL (2015) A bi-partitioned iterative algorithm for solving linear systems arising from incompressible flow problems. Comput Methods Appl Mech Eng 286:40–62

    Article  MathSciNet  MATH  Google Scholar 

  12. Esmaily-Moghadam M, Bazilevs Y, Marsden AL (2015) Impact of data distribution on the parallel performance of iterative linear solvers with emphasis on cfd of incompressible flows. Comput Mech 55:93–103

    Article  MathSciNet  MATH  Google Scholar 

  13. Figueroa CA, Vignon-Clementel IE, Jansen KE, Hughes TJ, Taylor CA (2006) A coupled momentum method for modeling blood flow in three-dimensional deformable arteries. Comput Methods Appl Mech Eng 195(41–43):5685–5706

    Article  MathSciNet  MATH  Google Scholar 

  14. Heroux MA, Bartlett RA, Howle VE, Hoekstra RJ, Hu JJ, Kolda TG, Lehoucq RB, Long KR, Pawlowski RP, Phipps ET, Salinger AG, Thornquist HK, Tuminaro RS, Willenbring JM, Williams A, Stanley KS (2005) An overview of the trilinos project. ACM Trans Math Softw 31(3):397–423

    Article  MathSciNet  MATH  Google Scholar 

  15. Hughes TJ, Liu WK, Zimmermann TK (1981) Lagrangian-eulerian finite element formulation for incompress- ible viscous flows. Comput Methods Appl Mech Eng 29(3):329–349

    Article  MATH  Google Scholar 

  16. Jansen KE, Whiting CH, Hulbert GM (2000) A generalized-\(\alpha \) method for integrating the filtered Navier–Stokes equations with a stabilized finite element method. Comput Methods Appl Mech Eng 190(3–4):305–319

    Article  MathSciNet  MATH  Google Scholar 

  17. Kim HJ, Vignon-Clementel IE, Figueroa CA, LaDisa JF, Jansen KE, Feinstein JA, Taylor CA (2009) On coupling a lumped parameter heart model and a three-dimensional finite element aorta model. Annu Biomed Eng 37(11):2153–2169

    Article  Google Scholar 

  18. Lagana K, Dubini G, Migliavacca F, Pietrabissa R, Pennati G, Veneziani A, Quarteroni A (2002) Multiscale modelling as a tool to prescribe realistic boundary conditions for the study of surgical procedures. Biorheology 39:359–364

    Google Scholar 

  19. Lan H, Updegrove A, Wilson NM, Maher GD, Shadden SC, Marsden AL (2017) A re-engineered software interface and workflow for the open source simvascular cardiovascular modeling. J Biomech Eng 140(2):024501:1–11

    Google Scholar 

  20. Long CC (2013) Fluid–structure interaction: physiologic simulation of pulsatile ventricular assist devices using isogeometric analysis. Dissertation in University of California, San Diego

  21. Manguoglu M, Takizawa K, Sameh AH, Tezduyar TE (2010) Solution of linear systems in arterial fluid mechanics computations with boundary layer mesh refinement. Comput Mech 46:83–89

    Article  MATH  Google Scholar 

  22. Marsden AL (2014) Optimization in cardiovascular modeling. Annu Rev Fluid Mech 46:519–546

    Article  MathSciNet  MATH  Google Scholar 

  23. Marsden AL, Esmaily-Moghadam M (2015) Multiscale modeling of cardiovascular flows for clinical decision support. Appl Mech Rev 67:030804

    Article  Google Scholar 

  24. Marsden AL, Feinstein JA, Taylor CA (2008) A computational framework for derivative-free optimization of cardiovascular geometries. Comput Methods Appl Mech Eng 197:1890–905

    Article  MathSciNet  MATH  Google Scholar 

  25. Moghadam ME, Vignon-Clementel I, Figliola R, Marsden A (2013) A modular numerical method for implicit 0d/3d coupling in cardiovascular finite element simulations. J Comput Phys 244:63–79

    Article  MathSciNet  MATH  Google Scholar 

  26. Nesbitt WS, Westein E, Tovar-Lopez FJ, Tolouei E, Mitchell A, Fu J, Carberry J, Fouras A, Jackson SP (2009) A shear gradient-dependent platelet aggregation mechanism drives thrombus formation. Nat Med 15(6):665–673

    Article  Google Scholar 

  27. Ramachandra AB, Kahn AM, Marsden AL (2016) Patient-specific simulations reveal significant differences in mechanical stimuli in venous and arterial coronary grafts. J Cardiovasc Trans Res 9(4):279–290

    Article  Google Scholar 

  28. Saad Y (2003) Iterative methods for sparse linear systems. Society for industrial and applied mathematics, ISBN: 978-0-89871-534-7

  29. Sankaran S, Kim HJ, Choi G, Taylor CA (2016) Uncertainty quantification in coronary blood flow simulations: Impact of geometry, boundary conditions and blood viscosity. J Biomech 49:2540–2547

    Article  Google Scholar 

  30. Schiavazzi DE, Arbia G, Baker C, Hlavacek AM, Hsia TY, Marsden AL, Vignon-Clementel IE, of Congenital Hearts Alliance (MOCHA) Investigators TM (2016) Uncertainty quantification in virtual surgery hemodynamics predictions for single ventricle palliation. Int J Numer Methods Biomed Eng e02737:1–25

  31. Schiavazzi DE, Baretta A, Pennati G, Hsia T-Y, Marsden AL (2017) Patient-specific parameter estimation in single-ventricle lumped circulation models under uncertainty. Int J Numer Methods Biomed Eng 33(3):e02799

    Article  Google Scholar 

  32. Schiavazzi DE, Doostan A, Iaccarino G, Marsden A (2017) A generalized multi-resolution expansion for uncertainty propagation with application to cardiovascular modeling. Comput Methods Appl Mech Eng 314:196–221

    Article  MathSciNet  Google Scholar 

  33. Sengupta D, Kahn AM, Burns JC, Sankaran S, Shadden SC, Marsden AL (2012) Image-based modeling of hemodynamics in coronary artery aneurysms caused by kawasaki disease. Biomech Model Mechanobiol 11:915–932

    Article  Google Scholar 

  34. Taylor CA, Fonte TA, Min JK (2013) Computational fluid dynamics applied to cardiac computed tomography for noninvasive quantification of fractional flow reserve. J Am College Cardiol 61(22):2233–41

    Article  Google Scholar 

  35. Towns J, Cockerill T, Dahan M, Foster I, Gaither K, Grimshaw A, Hazlewood V, Lathrop S, Lifka D, Peterson GD, Roskies R, Scott JR, Wilkins-Diehr N (2014) Xsede: accelerating scientific discovery. Comput Sci Eng 16(5):62–74

    Article  Google Scholar 

  36. Tran JS, Schiavazzi DE, Ramachandra AB, Kahn AM, Marsden AL (2017) Automated tuning for parameter identification and uncertainty quantification in multi-scale coronary simulations. Comput Fluids 142:128–138

    Article  MathSciNet  MATH  Google Scholar 

  37. Trefethen LN, Bau D (1997) Numerical Linear Algebra. SIAM

  38. Updegrove A, Wilson NM, Merkow J, Lan H, Marsden AL, Shadden SC (2016) Simvascular: an open source pipeline for cardiovascular simulation. Annl Biomed Eng 45(3):525–541

    Article  Google Scholar 

  39. Vedula V, Lee J, Xu H, Kuo C-CJ, Hsiai TK, Marsden AL (2017) A method to quantify mechanobiologic forces during zebrafish cardiac development using 4-d light sheet imaging and computational modeling. PLoS Comput Biol 13(10):e1005828

    Article  Google Scholar 

  40. Yang W, Marsden AL, Ogawa MT, Sakarovitch C, Hall KK, Rabinovitch M, Feinstein JA (2018) Right ventricular stroke work correlates with outcomes in pediatric pulmonary arterial hypertension. Pulm Circ

Download references

Acknowledgements

This work was supported by NIH grant (NIH R01-EB018302), NSF SSI grants 1663671 and 1339824, and NSF CDSE CBET 1508794. This work used the Extreme Science and Engineering Discovery Environment (XSEDE) [35], which is supported by National Science Foundation grant number ACI-1548562. We thank Mahidhar Tatineni for assisting on building Trilinos on Comet cluster, which was made possible through the XSEDE Extended Collaborative Support Service (ECSS) program [1]. The authors also thank Michael Saunders, Michael Heroux, Mahdi Esmaily, Ju Liu, and Vijay Vedula, for fruitful discussions that helped in the preparation of this paper. The authors would like to thank the two anonymous reviewers whose comments greatly contributed to improve the completeness of the present study. We also acknowledge support from the open source SimVascular project at www.simvascular.org.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jongmin Seo.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix

A GMRES restart

We tested different GMRES restart numbers in the pipe benchmark problems. In Fig. 20, we plot compute times for preconditioned GMRES using a pipe model with rigid wall and a pipe with deformable wall. Our test shows that decreasing restart number increases the compute time of linear solver in the rigid pipe model. The FSI model does not show a notable difference.

Fig. 20
figure 20

Compute times for GMRES and preconditioners using (top) a rigid pipe model, and (bottom) a pipe model with deformable wall using different the GMRES restart numbers

B Choice of smoother and subsmoother for ML

In the ML package, multiple options are available for the smoother and the subsmoother. As shown in Fig. 21, the Gauss–Seidel smoother works best among Chebyshev, symmetric Gauss–Seidel, and ILUT. For the subsmoother, the symmetric Gauss–Seidel is the best among Chebyshev and MLS (Fig. 22).

Fig. 21
figure 21

Compute times for linear solvers preconditioned with ML using a a rigid and b an FSI pipe model with tolerance \(\epsilon =10^{-3}\), with different smoothers. The symmteric Gauss–Seidel subsmoother is used. For the rigid wall model, 38 cores are used. For the FSI model, 48 cores are used

Fig. 22
figure 22

Compute times for linear solvers preconditioned with ML using a a rigid and b FSI pipe model with tolerance \(\epsilon =10^{-3}\), with different subsmoothers. The Gauss–Seidel smoother is used. For the rigid wall model, 38 cores are used. For the FSI model, 48 cores are used

C Effect of reordering in ILU

We evaluated and compared compute times of linear solvers with different reordering methods. RCM and METIS reordering for ILUT via the Trilinos IFPACK are implemented. We use 2 level fill-in and \(10^{-2}\) dropping tolerance for this test. Figure 23 shows performance differences between ILUT with different reordering schemes. From the testing, we confirm that the RCM is the fastest method against METIS and no reordering. The superior performance of RCM is notable when GMRES is used with ILUT.

Fig. 23
figure 23

Compute times for linear solvers preconditioned with ILUT using a a rigid and b FSI pipe model with tolerance \(\epsilon =10^{-3}\), with different reorderings. For the rigid wall, 38 cores are used. For the FSI, 48 cores are used

D Eigenvalue spectrums of the local and global matrix.

In this section we compare the spectrum of eigenvalues in the local and global matrices and investigate how our analysis on local eigenvalues can be generalized to the global matrix. We use a pipe model in the same dimension shown in Fig. 1 meshed with 24,450 elements with \(N_{nd}=5462\). We use one core to extract the global matrix, and four cores to examine local matrices. As shown in Fig. 24, the eigenvalue distributions of the global and local matrices are similar. Although the eigenvalues from the global and local matrices are not exactly the same, the distribution of eigenvalues of local matrices is a good approximations to the distribution of eigenvalues in the global matrix.

Fig. 24
figure 24

Spectrums of eigenvalues for a rigid pipe model with Neumann BC at the outlet. (top) eigenvalues obtained from four local matrices. Different colors are used to represent eigenvalues from different local matrices. (bottom) eigenvalues obtained from the global matrix

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Seo, J., Schiavazzi, D.E. & Marsden, A.L. Performance of preconditioned iterative linear solvers for cardiovascular simulations in rigid and deformable vessels. Comput Mech 64, 717–739 (2019). https://doi.org/10.1007/s00466-019-01678-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00466-019-01678-3

Keywords

Navigation