Skip to main content
SpringerLink
Log in

A scalable framework for the partitioned solution of fluid–structure interaction problems

  • Original Paper
  • Published:
Computational Mechanics Aims and scope Submit manuscript

Abstract

In this work, we present a scalable and efficient parallel solver for the partitioned solution of fluid–structure interaction problems through multi-code coupling. Two instances of an in-house parallel software, TermoFluids, are used to solve the fluid and the structural sub-problems, coupled together on the interface via the preCICE coupling library. For fluid flow, the Arbitrary Lagrangian–Eulerian form of the Navier–Stokes equations is solved on an unstructured conforming grid using a second-order finite-volume discretization. A parallel dynamic mesh method for unstructured meshes is used to track the moving boundary. For the structural problem, the nonlinear elastodynamics equations are solved on an unstructured grid using a second-order finite-volume method. A semi-implicit FSI coupling method is used which segregates the fluid pressure term and couples it strongly to the structure, while the remaining fluid terms and the geometrical nonlinearities are only loosely coupled. A robust and advanced multi-vector quasi-Newton method is used for the coupling iterations between the solvers. Both the fluid and the structural solver use distributed-memory parallelism. The intra-solver communication required for data update in the solution process is carried out using non-blocking point-to-point communicators. The inter-code communication is fully parallel and point-to-point, avoiding any central communication unit. Inside each single-physics solver, the load is balanced by dividing the computational domain into fairly equal blocks for each process. Additionally, a load balancing model is used at the inter-code level to minimize the overall idle time of the processes. Two practical test cases in the context of hemodynamics are studied, demonstrating the accuracy and computational efficiency of the coupled solver. Strong scalability test results show a parallel efficiency of 83% on 10,080 CPU cores.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Bazilevs Y, Calo VM, Zhang Y, Hughes TJR (2006) Isogeometric fluid–structure interaction analysis with applications to arterial blood flow. Comput Mech 38(4–5):310–322. https://doi.org/10.1007/s00466-006-0084-3

    Article  MathSciNet  MATH  Google Scholar 

  2. Takizawa K, Bazilevs Y, Tezduyar TE (2012) Space-time and ALE-VMS techniques for patient-specific cardiovascular fluid–structure interaction modeling. Arch Comput Methods Eng 19(2):171–225. https://doi.org/10.1007/s11831-012-9071-3

    Article  MathSciNet  MATH  Google Scholar 

  3. Degroote J (2013) Partitioned simulation of fluid–structure interaction. Arch Comput Methods Eng 20:185–238. https://doi.org/10.1007/s11831-013-9085-5

    Article  MathSciNet  MATH  Google Scholar 

  4. Hou G, Wang J, Layton A (2012) Numerical methods for fluid–structure interaction—a review. Commun Comput Phys 12(2):337–377. https://doi.org/10.4208/cicp.291210.290411s

    Article  MathSciNet  MATH  Google Scholar 

  5. Kong F, Cai X-C (2017) A scalable nonlinear fluid–structure interaction solver based on a Schwarz preconditioner with isogeometric unstructured coarse spaces in 3d. J Comput Phys 340:498–518. https://doi.org/10.1016/j.jcp.2017.03.043

    Article  MathSciNet  MATH  Google Scholar 

  6. Deparis S, Forti D, Grandperrin G, Quarteroni A (2016) FaCSI: a block parallel preconditioner for fluid–structure interaction in hemodynamics. J Comput Phys 327:700–718. https://doi.org/10.1016/j.jcp.2016.10.005

    Article  MathSciNet  MATH  Google Scholar 

  7. Kong F, Kheyfets V, Finol E, Cai X-C (2019) Simulation of unsteady blood flows in a patient-specific compliant pulmonary artery with a highly parallel monolithically coupled fluid–structure interaction algorithm. Int J Numer Methods Biomed Eng 35(7):e3208. https://doi.org/10.1002/cnm.3208

    Article  MathSciNet  Google Scholar 

  8. Kataoka S, Minami S, Kawai H, Yamada T, Yoshimura S (2014) A parallel iterative partitioned coupling analysis system for large-scale acoustic fluid–structure interactions. Comput Mech 53(6):1299–1310. https://doi.org/10.1007/s00466-013-0973-1

    Article  MathSciNet  MATH  Google Scholar 

  9. Cajas J, Houzeaux G, Vázquez M, Garcia M, Casoni E, Calmet H, Artigues A, Borrell R, Lehmkuhl O, Pastrana D et al (2018) Fluid–structure interaction based on HPC multicode coupling. SIAM J Sci Comput 40(6):C677–C703. https://doi.org/10.1137/17M1138868

    Article  MathSciNet  MATH  Google Scholar 

  10. Hewitt S, Margetts L, Revell A, Pankaj P, Levrero-Florencio F (2019) OpenFPCI: a parallel fluid–structure interaction framework. Comput Phys Commun 244:469–482. https://doi.org/10.1016/j.cpc.2019.05.016

    Article  Google Scholar 

  11. Larson J, Jacob R, Ong E (2005) The model coupling toolkit: a new Fortran90 toolkit for building multiphysics parallel coupled models. Int J High Perform Comput Appl 19(3):277–292. https://doi.org/10.1177/1094342005056115

    Article  Google Scholar 

  12. Bungartz H-J, Lindner F, Gatzhammer B, Mehl M, Scheufele K, Shukaev A, Uekermann B (2016) preCICE—a fully parallel library for multi-physics surface coupling. Comput Fluids 141:250–258. https://doi.org/10.1016/j.compfluid.2016.04.003

    Article  MathSciNet  MATH  Google Scholar 

  13. Thomas D, Cerquaglia ML, Boman R, Economon TD, Alonso JJ, Dimitriadis G, Terrapon VE (2019) CUPyDO—an integrated Python environment for coupled fluid–structure simulations. Adv Eng Softw 128:69–85

    Article  Google Scholar 

  14. Bungartz H-J, Lindner F, Mehl M, Scheufele K, Shukaev A, Uekermann B (2016) Partitioned fluid–structure–acoustics interaction on distributed data: coupling via precice. In: Bungartz HJ, Neumann P, Nagel WE (eds) Software for exascale computing—SPPEXA 2013–2015. Springer, Cham, pp 239–266. https://doi.org/10.1007/978-3-319-40528-5-11

    Chapter  MATH  Google Scholar 

  15. Cerquaglia ML, Thomas D, Boman R, Terrapon V, Ponthot J-P (2019) A fully partitioned Lagrangian framework for fsi problems characterized by free surfaces, large solid deformations and displacements, and strong added-mass effects. Comput Methods Appl Mech Eng 348:409–442. https://doi.org/10.1016/j.cma.2019.01.021

    Article  MathSciNet  MATH  Google Scholar 

  16. Causin P, Gerbeau JF, Nobile F (2005) Added-mass effect in the design of partitioned algorithms for fluid–structure problems. Comput Methods Appl Mech Eng 194:4506–4527. https://doi.org/10.1016/j.cma.2004.12.005

    Article  MathSciNet  MATH  Google Scholar 

  17. Förster C, Wall WA, Ramm E (2007) Artificial added mass instabilities in sequential staggered coupling of nonlinear structures and incompressible viscous flows. Comput Methods Appl Mech Eng 196:1278–1293. https://doi.org/10.1016/j.cma.2006.09.002

    Article  MathSciNet  MATH  Google Scholar 

  18. Fernández MA, Gerbeau J-F, Grandmont C (2007) A projection semi-implicit scheme for the coupling of an elastic structure with an incompressible fluid. Int J Numer Meth Eng 69(4):794–821. https://doi.org/10.1002/nme.1792

    Article  MathSciNet  MATH  Google Scholar 

  19. Naseri A, Lehmkuhl O, Gonzalez I, Bartrons E, Pérez-Segarra CD, Oliva A (2018) A semi-implicit coupling technique for fluid–structure interaction problems with strong added-mass effect. J Fluids Struct 80:94–112. https://doi.org/10.1016/j.jfluidstructs.2018.03.012

    Article  Google Scholar 

  20. Naseri A, Gonzalez I, Amani A, Pérez-Segarra CD, Oliva A (2019) A second-order time accurate semi-implicit method for fluid–structure interaction problems. J Fluids Struct 86:135–155. https://doi.org/10.1016/j.jfluidstructs.2019.02.007

    Article  Google Scholar 

  21. TermoFluids SL (2020) http://www.termofluids.com

  22. Rodriguez I, Borell R, Lehmkuhl O, Perez Segarra CD, Oliva A (2011) Direct numerical simulation of the flow over a sphere at Re = 3700. J Fluid Mech 679:263–287. https://doi.org/10.1017/jfm.2011.136

    Article  MATH  Google Scholar 

  23. Rodrguez I, Lehmkuhl O, Chiva J, Borrell R, Oliva A (2015) On the flow past a circular cylinder from critical to super-critical reynolds numbers: wake topology and vortex shedding. Int J Heat Fluid Flow 55:91–103. https://doi.org/10.1016/j.ijheatfluidflow.2015.05.009

    Article  Google Scholar 

  24. Balcázar N, Jofre L, Lehmkuhl O, Castro J, Rigola J (2014) A finite-volume/level-set method for simulating two-phase flows on unstructured grids. Int J Multiph Flow 64:55–72. https://doi.org/10.1016/j.ijmultiphaseflow.2014.04.008

    Article  MathSciNet  MATH  Google Scholar 

  25. Gutiérrez E, Favre F, Balcazar N, Amani A, Rigola J (2018) Numerical approach to study bubbles and drops evolving through complex geometries by using a level set-moving mesh-immersed boundary method. Chem Eng J 349:662–682. https://doi.org/10.1016/j.cej.2018.05.110

    Article  Google Scholar 

  26. Galione P, Lehmkuhl O, Rigola J, Oliva A (2015) Fixed-grid numerical modeling of melting and solidification using variable thermo-physical properties—application to the melting of n-octadecane inside a spherical capsule. Int J Heat Mass Transf 86:721–743. https://doi.org/10.1016/j.ijheatmasstransfer.2015.03.033

    Article  Google Scholar 

  27. Bartrons E, Oliet C, Gutierrez E, Naseri A, Pérez-Segarra CD (2018) A finite volume method to solve the frost growth using dynamic meshes. Int J Heat Mass Transf 124:615–628. https://doi.org/10.1016/j.ijheatmasstransfer.2018.03.104

    Article  Google Scholar 

  28. Colomer G, Borrell R, Trias FX, Rodríguez I (2013) Parallel algorithms for Sn transport sweeps on unstructured meshes. J Comput Phys 232(1):118–135. https://doi.org/10.1016/j.jcp.2012.07.009

    Article  Google Scholar 

  29. Jofre L, Borrell R, Lehmkuhl O, Oliva A (2015) Parallel load balancing strategy for volume-of-fluid methods on 3-d unstructured meshes. J Comput Phys 282:269–288. https://doi.org/10.1016/j.jcp.2014.11.009

    Article  MathSciNet  MATH  Google Scholar 

  30. Borrell R, Chiva J, Lehmkuhl O, Oyarzun G, Rodríguez I, Oliva A (2016) Optimising the termofluids CFD code for petascale simulations. Int J Comput Fluid Dyn 30(6):425–430. https://doi.org/10.1080/10618562.2016.1221503

    Article  MathSciNet  Google Scholar 

  31. Verstappen R, Veldman A (2003) Symmetry-preserving discretization of turbulent flow. J Comput Phys 187(1):343–368. https://doi.org/10.1016/S0021-9991(03)00126-8

    Article  MathSciNet  MATH  Google Scholar 

  32. Trias FX, Lehmkuhl O, Oliva A, Pérez-Segarra CD, Verstappen R (2014) Symmetry-preserving discretization of Navier–Stokes equations on collocated unstructured grids. J Comput Phys 258:246–267. https://doi.org/10.1016/j.jcp.2013.10.031

    Article  MathSciNet  MATH  Google Scholar 

  33. Jofre L, Lehmkuhl O, Ventosa J, Trias FX, Oliva A (2014) Conservation properties of unstructured finite-volume mesh schemes for the Navier–Stokes equations. Numer Heat Transf Part B Fundam 65(1):53–79. https://doi.org/10.1080/10407790.2013.836335

    Article  Google Scholar 

  34. Thomas P, Lombard C (1979) Geometric conservation law and its application to flow computations on moving grids. AIAA J 17(10):1030–1037. https://doi.org/10.2514/3.61273

    Article  MathSciNet  MATH  Google Scholar 

  35. Lesoinne M, Farhat C (1996) Geometric conservation laws for flow problems with moving boundaries and deformable meshes, and their impact on aeroelastic computations. Comput Methods Appl Mech Eng 134(1–2):71–90. https://doi.org/10.1016/0045-7825(96)01028-6

    Article  MATH  Google Scholar 

  36. Estruch O, Lehmkuhl O, Borrell R, Segarra CDP, Oliva A (2013) A parallel radial basis function interpolation method for unstructured dynamic meshes. Comput Fluids 80:44–54. https://doi.org/10.1016/j.compfluid.2012.06.015

    Article  MathSciNet  MATH  Google Scholar 

  37. Cardiff P, Demirdžić I Thirty years of the finite volume method for solid mechanics. arXiv preprint arXiv:1810.02105

  38. Jasak H (1996) Error analysis and estimation for the finite volume method with applications to fluid flows. PhD thesis, Imperial College London (University of London)

  39. Macleod AJ (1986) Acceleration of vector sequences by multidimensional \(\varDelta ^2\) methods. Commun Appl Numer Methods 2(4):385–392

    Article  Google Scholar 

  40. Tuković Ž, Ivanković A, Karač A (2013) Finite-volume stress analysis in multi-material linear elastic body. Int J Numer Meth Eng 93(4):400–419

    Article  MathSciNet  Google Scholar 

  41. Cardiff P, Tuković H, Jasak A, Ivanković A (2016) Block-coupled finite volume methodology for linear elasticity and unstructured meshes. Comput Struct 175:100–122. https://doi.org/10.1016/j.compstruc.2016.07.004

    Article  Google Scholar 

  42. Chandrashekar P, Garg A (2013) Vertex-centroid finite volume scheme on tetrahedral grids for conservation laws. Comput Math Appl 65(1):58–74

    Article  MathSciNet  Google Scholar 

  43. González I, Naseri A, Chiva J, Rigola J, Pérez-Segarra CD (2018) An enhanced finite volume based solver for thermoelastic materials in fluid–structure coupled problems. In: 6th European conference on computational mechanics (ECCM 6), 7th European conference on computational fluid dynamics (ECFD 7), Glasgow, UK, vol 15, pp 1115–1127

  44. Küttler U, Wall WA (2008) Fixed-point fluid–structure interaction solvers with dynamic relaxation. Comput Mech 43:61–72. https://doi.org/10.1007/s00466-008-0255-5

    Article  MATH  Google Scholar 

  45. Gerbeau JF, Vidrascu M (2003) A quasi-Newton algorithm based on a reduced model for fluid–structure interaction problems in blood flows. ESAIM Math Model Numer Anal 37:631–647. https://doi.org/10.1051/m2an:2003049

    Article  MathSciNet  MATH  Google Scholar 

  46. Michler C, Brummelen EHV, Borst RD (2005) An interface Newton–Krylov solver for fluid–structure interaction. Int J Numer Meth Fluids 47(10–11):1189–1195. https://doi.org/10.1002/fld.850

    Article  MATH  Google Scholar 

  47. Scheufele K, Mehl M (2017) Robust multisecant quasi-Newton variants for parallel fluid-structure simulations-and other multiphysics applications. SIAM J Sci Comput 39(5):S404–S433. https://doi.org/10.1137/16M1082020

    Article  MathSciNet  MATH  Google Scholar 

  48. Bungartz H-J, Lindner F, Mehl M, Uekermann B (2015) A plug-and-play coupling approach for parallel multi-field simulations. Comput Mech 55(6):1119–1129. https://doi.org/10.1007/s00466-014-1113-2

    Article  MathSciNet  MATH  Google Scholar 

  49. Mehl M, Uekermann B, Bijl H, Blom D, Gatzhammer B, Van Zuijlen A (2016) Parallel coupling numerics for partitioned fluid–structure interaction simulations. Comput Math Appl 71(4):869–891. https://doi.org/10.1016/j.camwa.2015.12.025

    Article  MathSciNet  Google Scholar 

  50. Oyarzun G, Borrell R, Gorobets A, Oliva A (2014) MPI-CUDA sparse matrix-vector multiplication for the conjugate gradient method with an approximate inverse preconditioner. Comput Fluids 92:244–252. https://doi.org/10.1016/j.compfluid.2013.10.035

    Article  MathSciNet  MATH  Google Scholar 

  51. Oyarzun G, Borrell R, Gorobets A, Oliva A (2017) Portable implementation model for CFD simulations. Application to hybrid CPU/GPU supercomputers. Int J Comput Fluid Dyn 31(9):396–411. https://doi.org/10.1080/10618562.2017.1390084

    Article  MathSciNet  Google Scholar 

  52. Oyarzun G, Borrell R, Gorobets A, Mantovani F, Oliva A (2018) Efficient CFD code implementation for the ARM-based Mont-Blanc architecture. Future Gener Comput Syst 79:786–796. https://doi.org/10.1016/j.future.2017.09.029

    Article  Google Scholar 

  53. Karypis G, Kumar V (1998) A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM J Sci Comput 20(1):359–392. https://doi.org/10.1137/S1064827595287997

    Article  MathSciNet  MATH  Google Scholar 

  54. The HDF Group, Hierarchical Data Format, version 5, http://www.hdfgroup.org/HDF5/ (1997–2019)

  55. Totounferoush A, Ebrahimi Pour N, Schroder J, Roller S, Mehl M (2019) A new load balancing approach for coupled multi-physics simulations. In: IEEE international parallel and distributed processing symposium workshops (IPDPSW). IEEE 2019, pp 676–682. https://doi.org/10.1109/IPDPSW.2019.00115

  56. Calotoiu A, Beckinsale D, Earl CW, Hoefler T, Karlin I, Schulz M, Wolf F (2016) Fast multi-parameter performance modeling. In: 2016 IEEE international conference on cluster computing (CLUSTER). IEEE, pp. 172–181. https://doi.org/10.1109/CLUSTER.2016.57

  57. Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities , https://www.lrz.de (2019)

  58. Formaggia L, Gerbeau JF, Nobile F, Quarteroni A (2001) On the coupling of 3D and 1D Navier–Stokes equations for flow problems in compliant vessels. Comput Methods Appl Mech Eng 191(6–7):561–582. https://doi.org/10.1016/S0045-7825(01)00302-4

    Article  MathSciNet  MATH  Google Scholar 

  59. Fernández MA, Moubachir M (2005) A Newton method using exact Jacobians for solving fluid–structure coupling. Comput Struct 83:127–142. https://doi.org/10.1016/j.compstruc.2004.04.021

    Article  Google Scholar 

  60. Eken A, Sahin M (2016) A parallel monolithic algorithm for the numerical simulation of large-scale fluid structure interaction problems. Int J Numer Meth Fluids 80(12):687–714. https://doi.org/10.1002/fld.4169

    Article  MathSciNet  Google Scholar 

  61. Lozovskiy A, Olshanskii MA, Vassilevski YV (2019) Analysis and assessment of a monolithic FSI finite element method. Comput Fluids 179:277–288. https://doi.org/10.1016/j.compfluid.2018.11.004

    Article  MathSciNet  MATH  Google Scholar 

  62. 2nd CFD challenge predicting patient-specific hemodynamics at rest and stress through an aortic coarctation, http://www.vascularmodel.org/miccai2013/ (2013)

  63. Fernández MA, Landajuela M, Vidrascu M (2015) Fully decoupled time-marching schemes for incompressible fluid/thin-walled structure interaction. J Comput Phys 297:156–181. https://doi.org/10.1016/j.jcp.2015.05.009

    Article  MathSciNet  MATH  Google Scholar 

  64. Westerhof N, Lankhaar J-W, Westerhof BE (2009) The arterial Windkessel. Med Biol Eng Comput 47(2):131–141. https://doi.org/10.1007/s11517-008-0359-2

    Article  Google Scholar 

  65. Pant S, Fabrèges B, Gerbeau J-F, Vignon-Clementel I (2014) A methodological paradigm for patient-specific multi-scale CFD simulations: from clinical measurements to parameter estimates for individual analysis. Int J Numer Methods Biomed Eng 30(12):1614–1648. https://doi.org/10.1002/cnm.2692

    Article  Google Scholar 

  66. Tuković Ž, Karač A, Cardiff P, Jasak H, Ivanković A (2018) OpenFOAM finite volume solver for fluid–solid interaction. Trans FAMENA 42(3):1–31. https://doi.org/10.21278/TOF.42301

    Article  Google Scholar 

  67. Seo J, Schiavazzi DE, Marsden AL (2019) Performance of preconditioned iterative linear solvers for cardiovascular simulations in rigid and deformable vessels. Comput Mech 64(3):717–739. https://doi.org/10.1007/s00466-019-01678-3

    Article  MathSciNet  MATH  Google Scholar 

  68. Jodlbauer D, Langer U, Wick T (2019) Parallel block-preconditioned monolithic solvers for fluid–structure interaction problems. Int J Numer Meth Eng 117(6):623–643. https://doi.org/10.1002/nme.5970

    Article  MathSciNet  Google Scholar 

  69. Forti D, Quarteroni A, Deparis S et al (2017) A parallel algorithm for the solution of large-scale nonconforming fluid–structure interaction problems in hemodynamics. J Comput Math 35(3):363–380. https://doi.org/10.4208/jcm.1702-m2016-0630

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was financially supported by—Ministerio de Economía y Competitividad, Secretaría de Estado de Investigación, Desarrollo e Innovación, Spain (ENE2017-88697-R),—priority program 1648—Software for Exascale Computing 214 (ExaFSA - Exascale Simulation of Fluid–Structure–Acoustics Interactions) of the German Research Foundation,—and a FI Ph.D. scholarship by the Agència de Gestió d’Ajuts Universitaris i de Recerca (AGAUR) of Generalitat de Catalunya (Spain). The performance measurements were carried out on the SuperMUC supercomputer at Leibniz Rechenzentrum (LRZ) der Bayerischen Akademie der Wissenschaften. The authors wish to thank LRZ for the computing time and the technical support.

Author information

Authors and Affiliations

  1. Heat and Mass Transfer Technological Center (CTTC), Universitat Politécnica de Catalunya - BarcelonaTech (UPC), ESEIAAT, C/ Colom 11, 08222, Terrassa, Barcelona, Spain

    Alireza Naseri, Ignacio González & Carlos David Pérez-Segarra

  2. Institute for Parallel and Distributed Systems (IPVS), University of Stuttgart, Universitätsstraße 38, 70569, Stuttgart, Germany

    Amin Totounferoush & Miriam Mehl

Authors
  1. Alireza Naseri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amin Totounferoush
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ignacio González
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Miriam Mehl
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Carlos David Pérez-Segarra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carlos David Pérez-Segarra.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Naseri, A., Totounferoush, A., González, I. et al. A scalable framework for the partitioned solution of fluid–structure interaction problems. Comput Mech 66, 471–489 (2020). https://doi.org/10.1007/s00466-020-01860-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00466-020-01860-y

Keywords

Search

Navigation