## Abstract

Various biological processes such as transport of oxygen and nutrients, thrombus formation, vascular angiogenesis and remodeling are related to cellular/subcellular level biological processes, where mesoscopic simulations resolving detailed cell dynamics provide a key to understanding and identifying the cellular basis of disease. However, the intrinsic stochastic effects can play an important role in mesoscopic processes, while the time step allowed in a mesoscopic simulation is restricted by rapid cellular/subcellular dynamic processes. These challenges significantly limit the timescale that can be reached by mesoscopic simulations even with high-performance computing. To break this bottleneck and achieve a biologically meaningful timescale, we propose a multiscale parareal algorithm in which a continuum-based solver supervises a mesoscopic simulation in the time-domain. Using an iterative prediction-correction strategy, the parallel-in-time mesoscopic simulation supervised by its continuum-based counterpart can converge fast. The effectiveness of the proposed method is first verified in a time-dependent flow with a sinusoidal flowrate through a Y-shaped bifurcation channel. The results show that the supervised mesoscopic simulations of both Newtonian fluids and non-Newtonian bloods converge to reference solutions after a few iterations. Physical quantities of interest including velocity, wall shear stress and flowrate are computed to compare against those of reference solutions, showing a less than 1% relative error on flowrate in the Newtonian flow and a less than 3% relative error in the non-Newtonian blood flow. The proposed method is then applied to a large-scale mesoscopic simulation of microvessel blood flow in a zebrafish hindbrain for temporal acceleration. The three-dimensional geometry of the vasculature is constructed directly from the images of live zebrafish under a confocal microscope, resulting in a complex vascular network with 95 branches and 57 bifurcations. The time-dependent blood flow from heartbeats in this realistic vascular network of zebrafish hindbrain is simulated using dissipative particle dynamics as the mesoscopic model, which is supervised by a one-dimensional blood flow model (continuum-based model) in multiple temporal sub-domains. The computational analysis shows that the resulting microvessel blood flow converges to the reference solution after only two iterations. The proposed method is suitable for long-time mesoscopic simulations with complex fluids and geometries. It can be readily combined with classical spatial decomposition for further acceleration.

This is a preview of subscription content, access via your institution.

## References

Al-hayanni MAN, Xia F, Rafiev A, Romanovsky A, Shafik R, Yakovlev A (2020) Amdahl’s law in the context of heterogeneous many-core systems-a survey. IET Comput Digital Tech 14(4):133–148

Alastruey J, Parker KH, Peiró J, Sherwin SJ (2008) Lumped parameter outflow models for 1-D blood flow simulations: effect on pulse waves and parameter estimation. Commun Comput Phys 4(2):317–336

Bernabeu MO, Jones ML, Nielsen JH, Kruger T, Nash RW, Groen D, Schmieschek S, Hetherington J, Gerhardt H, Franco CA, Coveney PV (2014) Computer simulations reveal complex distribution of haemodynamic forces in a mouse retina model of angiogenesis. J R Soc Interface 11(99):20140543

Bian X, Li Z, Adams NA (2018) A note on hydrodynamics from dissipative particle dynamics. Appl Math Mech (Engl Ed) 39(1):63–82

Blumers AL, Li Z, Karniadakis GE (2019) Supervised parallel-in-time algorithm for long-time lagrangian simulations of stochastic dynamics: Application to hydrodynamics. J Comput Phys 393:214–228

Blumers AL, Tang YH, Li Z, Li X, Karniadakis GE (2017) Gpu-accelerated red blood cells simulations with transport dissipative particle dynamics. Comput Phys Commun 217:171–179

Cai S, Mao Z, Wang Z, Yin M, Karniadakis GE (2021) Physics-informed neural networks (pinns) for fluid mechanics: A review. arXiv preprint arXiv:2105.09506

Chang HY, Li X, Karniadakis GE (2017) Modeling of biomechanics and biorheology of red blood cells in type 2 diabetes mellitus. Biophys J 113(2):481–490

Espanol P, Warren P (1995) Statistical mechanics of dissipative particle dynamics. EPL (Europhysics Letters) 30(4):191

Fedosov DA, Caswell B, Karniadakis GE (2010) A multiscale red blood cell model with accurate mechanics, rheology, and dynamics. Biophys J 98(10):2215–2225

Fedosov DA, Pan W, Caswell B, Gompper G, Karniadakis GE (2011) Predicting human blood viscosity in silico. Proc Nat Acad Sci 108(29):11772–11777

Formaggia L, Gerbeau JF, Nobile F, Quarteroni A (2002) Numerical treatment of defective boundary conditions for the Navier–Stokes equations. SIAM J Numer Anal 40(1):376–401

Formaggia L, Lamponi D, Quarteroni A (2003) One-dimensional models for blood flow in arteries. J Eng Math 47(3–4):251–276

Franke RP, Gräfe M, Schnittler H, Seiffge D, Mittermayer C, Drenckhahn D (1984) Induction of human vascular endothelial stress fibres by fluid shear stress. Nature 307(5952):648–649

Gander MJ, Vandewalle S (2007) Analysis of the parareal time-parallel time-integration method. SIAM J Sci Comput 29(2):556–578

Grinberg L, Cheever E, Anor T, Madsen JR, Karniadakis G (2011) Modeling blood flow circulation in intracranial arterial networks: a comparative 3d/1d simulation study. Ann Biomed Eng 39(1):297–309

Groot RD, Warren PB (1997) Dissipative particle dynamics: Bridging the gap between atomistic and mesoscopic simulation. J Chem Phys 107(11):4423–4435

Hoogerbrugge P, Koelman J (1992) Simulating microscopic hydrodynamic phenomena with dissipative particle dynamics. EPL (Europhysics Letters) 19(3):155

Huang C, Sheikh F, Hollander M, Cai C, Becker D, Chu PH, Evans S, Chen J (2003) Embryonic atrial function is essential for mouse embryogenesis, cardiac morphogenesis and angiogenesis. Development 130(24):6111–6119

Karniadakis GE, Kevrekidis IG, Lu L, Perdikaris P, Wang S, Yang L (2021) Physics-informed machine learning. Nat Rev Phys pp. 1–19

Kopp R, Schwerte T, Pelster B (2005) Cardiac performance in the zebrafish breakdance mutant. J Exp Biol 208(11):2123–2134

Lee J, Chou TC, Kang D, Kang H, Chen J, Baek KI, Wang W, Ding Y, Carlo DD, Tai YC, Hsiai TK (2017) A rapid capillary-pressure driven micro-channel to demonstrate newtonian fluid behavior of zebrafish blood at high shear rates. Sci Rep 7:1980

Li A, Chen R, Farimani AB, Zhang YJ (2020) Reaction diffusion system prediction based on convolutional neural network. Sci Rep 10(1):1–9

Li A, Farimani AB, Zhang YJ (2021) Deep learning of material transport in complex neurite networks. Sci Rep 11(1):1–13

Li Z, Bian X, Tang YH, Karniadakis G (2018) A dissipative particle dynamics method for arbitrarily complex geometries. J Comput Phys 355:534–547

Lions JL, Maday Y, Turinici G (2001) Résolution d’EDP par un schéma en temps pararéel. Comptes Rendus l’Academie des Sci Ser I Math 332(7):661–668

Liu Y, Liu WK (2006) Rheology of red blood cell aggregation by computer simulation. J Comput Phys 220(1):139–154

Mirzapou-shafiyi F, Kametani Y, Hikita T, Hasegawa Y, Nakayama M (2020) Numerical evaluation reveals the effect of branching morphology on vessel transport properties during angiogenesis. bioRxiv Oct 13, pp 1–28

Monaghan JJ (2012) Smoothed particle hydrodynamics and its diverse applications. Annu Rev Fluid Mech 44:323–346

Mynard J, Nithiarasu P (2008) A 1d arterial blood flow model incorporating ventricular pressure, aortic valve and regional coronary flow using the locally conservative galerkin (lcg) method. Commun Numer Methods Eng 24(5):367–417

Peng Z, Li X, Pivkin IV, Dao M, Karniadakis GE, Suresh S (2013) Lipid bilayer and cytoskeletal interactions in a red blood cell. Proc Nat Acad Sci 110(33):13356–13361

Peters A, Melchionna S, Kaxiras E, Lätt J, Sircar J, Bernaschi M, Bison M, Succi S (2010) Multiscale simulation of cardiovascular flows on the ibm bluegene/p: Full heart-circulation system at red-blood cell resolution. In: SC’10: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, IEEE, pp 1–10

Pindera MZ, Ding H, Athavale MM, Chen Z (2009) Accuracy of 1D microvascular flow models in the limit of low Reynolds numbers. Microvasc Res 77(3):273–280

Pivkin IV, Karniadakis GE (2008) Accurate coarse-grained modeling of red blood cells. Phys Rev Lett 101(11):118105

Quarteroni A, Veneziani A (2003) Analysis of a geometrical multiscale model based on the coupling of ODE and PDE for blood flow simulations. Multiscale Model. Simul. 1(2):173–195

Raghu R, Vignon-Clementel IE, Figueroa CA, Taylor CA (2011) Comparative study of viscoelastic arterial wall models in nonlinear one-dimensional finite element simulations of blood flow. J Biomech Eng 133(8)

Ray D, Hesthaven JS (2018) An artificial neural network as a troubled-cell indicator. J Comput Phys 367:166–191

Resnick N, Gimbrone MA (1995) Hemodynamic forces are complex regulators of endothelial gene expression. FASEB J. 9(10):874–882

Risau W (1997) Mechanisms of angiogenesis. Nature 386(6626):671–674

Sherwin S, Franke V, Peiró J, Parker K (2003) One-dimensional modelling of a vascular network in space-time variables. J Eng Math 47(3–4):217–250

Sherwin SJ, Formaggia L, Peiro J, Franke V (2003) Computational modelling of 1d blood flow with variable mechanical properties and its application to the simulation of wave propagation in the human arterial system. Int J Numer Meth Fluids 43(6–7):673–700

Symeonidis V, Karniadakis GE, Caswell B (2005) Dissipative particle dynamics simulations of polymer chains: scaling laws and shearing response compared to dna experiments. Phys Rev Lett 95(7):076001

Tang YH, Kudo S, Bian X, Li Z, Karniadakis GE (2015) Multiscale universal interface: a concurrent framework for coupling heterogeneous solvers. J Comput Phys 297:13–31

Wakayama Y, Fukuhara S, Ando K, Matsuda M, Mochizuki N (2015) Cdc42 mediates bmp-induced sprouting angiogenesis through fmnl3-driven assembly of endothelial filopodia in zebrafish. Dev Cell 32(1):109–122

Wang S, Li Z, Pan W (2019) Implicit-solvent coarse-grained modeling for polymer solutions via Mori-Zwanzig formalism. Soft Matter 15(38):7567–7582

Wang S, Ye T, Li G, Zhang X, Shi H (2021) Margination and adhesion dynamics of tumor cells in a real microvascular network. PLoS Comput Biol 17(2):e1008746

Wang Y, Li Z, Xu J, Yang C, Karniadakis GE (2019) Concurrent coupling of atomistic simulation and mesoscopic hydrodynamics for flows over soft multi-functional surfaces. Soft Matter 15(8):1747–1757

Westerhof N, Lankhaar JW, Westerhof BE (2009) The arterial windkessel. Med Biol Eng comput 47(2):131–141

Xia Y, Blumers A, Li Z, Luo L, Tang YH, Kane J, Goral J, Huang H, Deo M, Andrew M (2020) A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics. Comput Phys Commun 247:106874

Yazdani A, Deng Y, Li H, Javadi E, Li Z, Jamali S, Lin C, Humphrey JD, Mantzoros CS, Em Karniadakis G (2021) Integrating blood cell mechanics, platelet adhesive dynamics and coagulation cascade for modelling thrombus formation in normal and diabetic blood. J R Soc Interface 18(175):20200834

Yin M, Yazdani A, Karniadakis GE (2019) One-dimensional modeling of fractional flow reserve in coronary artery disease: Uncertainty quantification and bayesian optimization. Comput Methods Appl Mech Eng 353:66–85

Yu J, Hesthaven JS (2019) Flowfield reconstruction method using artificial neural network. Aiaa J 57(2):482–498

## Acknowledgements

*Tg(kdrl:GFP)* and *Tg(gata1:DsRed)* fish lines were provided by D.Y. Stainier at Max Planck Institute for Heart and Lung Research. The work is partially supported by grant U01 HL142518 from the National Institute of Health. Y.H. gratefully acknowledges the supports from Fostering Joint International Research (17KK0128) and Grant-in-Aid for Challenging Research (Pioneering: 20K20532) by MEXT (Ministry of Education, Culture, Sports, Science and Technology), Japan.

## Author information

### Authors and Affiliations

### Corresponding author

## Ethics declarations

### Conflict of interest

The authors declare that they have no conflict of interest.

## Additional information

### Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Appendix

### Appendix

### 1.1 dissipative particle dynamics

Each DPD particle is represented explicitly by its position and velocity. Its motion is governed by Newton’s equations of motion [18]:

where *t*, \(\mathbf {r}_i\), \(\mathbf {v}_i\) and \(\mathbf {F}_i\) denote time, position, velocity, and force, respectively. The force comprises three components: conservative force \(\mathbf {F}^C_{ij}\), dissipative force \(\mathbf {F}^D_{ij}\), and corresponding random force \(\mathbf {F}^R_{ij}\) from particle *j* within a radial cutoff \(r_c\) of particle *i*. They are expressed as [17]:

where \(\mathbf {e}_{ij}=\mathbf {r}_{ij} / r_{ij}\) is the unit vector between particles *i* and *j*, and \(\mathbf {v}_{ij}\) is the velocity difference. \(\Delta t_f\) is the time step of the fine propagator, and \(\xi \) is a symmetric Gaussian random variable with zero mean and unit variance [17]. The strengths of the conservative, dissipative, and random force are \(\alpha _{ij}\), \(\gamma _{ij}\) and \(\sigma _{ij}\), respectively. The interaction between pairs of particles are regulated by weighting functions \(\omega _C(r_{ij})\), \(\omega _D(r_{ij})\) and \(\omega _R(r_{ij})\). Most importantly, the balance between dissipation and thermal fluctuation is maintained by the fluctuation-dissipation theorem [9]:

where \(k_B T\) is the Boltzmann energy unit. A common choice for the weight functions is \(\omega _C(r)=1-r/r_c\) and \(\omega _D(r)=\omega ^2_R(r)=(1-r/r_c)^2\) for \(r\le r_c\) and zero for \(r>r_c\). For simplicity, \(k_BT\) and the mass of a particle are taken as the energy unit and mass unit, and their values are set to one.

#### 1.1.1 RBC and solvent

The particle-based blood flow model is based on the DPD approach [10, 34]. In this model, the blood is composed of solvents and red blood cells. The solvent is modeled with single-particles subject to only pairwise forces described in Sect. 5.1, and red blood cells are modeled discretely with bonded-particles. Membrane viscosity, elasticity and bending stiffness can be recovered with a spring-network that resembles a triangular mesh on a 2D surface. The shape of the viscoelastic membrane is maintained by a potential derived from a combination of bond, angle and dihedral interactions. The bonds between particles have an attractive and a repulsive component mimicking springs. The attractive potential adopts the form of the wormlike chain potential and is given by [10, 34]

where \(k_BT\) is the energy per unit mass, \(h_m\) is the maximum spring extension, and *p* is the persistence length. The repulsive potential adopts the form of a power function given by

where *m* is a positive coefficient, and \(k_p\) is force coefficient. Additionally a viscous component damping the springs is realized by a dissipative force, as well as the corresponding random force, given as

where \(\gamma ^T\) and \(\gamma ^C\) are dissipative coefficients, and \(v_{ij}\) is the relative velocity. \(d\mathbf {W}^S_{ij}\) is the symmetric component of a random matrix of independent Wiener increments \(d\mathbf {W}_{ij}\).

RBC’s structural stability is maintained by the area and volume constraints given by

where *j* is the triangle index. \(k_g\), \(k_l\), and \(k_v\) are global area, local area, and global volume constraints coefficients, respectively. The instantaneous area and volume of a RBC are denoted by \(A^t\) and \(V^t\), whereas \(A^t_0\) and \(V^t_0\) represent the equilibrium area and volume. \(A^t_0\) is calculated by summing the area of each triangle. \(V^t_0\) is found according to scaling relationship \(V^t_0/(A^t_0)^{3/2} = V^R / (A^R)^{3/2}\), where \(V^R\) and \(A^R\) are the experimental measurements. Lastly, the bending resistance of the membrane is modeled by

where \(k_b\) is the bending constant, and \(\theta _j\) is the angle between two neighboring triangles with the common edge *j*. We refer interested readers to references [10, 34] for more details.

### 1.2 1D blood flow model

Under the assumptions of axial symmetry and radial displacement of the vessel wall (Fig. 21a) [4, 40], the modeling of blood flow in a compliant vessel can be reduced from full-order Navier–Stokes equations to an 1D model. The 1D model has been widely used for blood flow simulations at a moderate [40] and low [33] Reynolds number. In our implementation, it serves as the low-fidelity model supervising the high-fidelity particle model because of its balance between its speed and accuracy.

The 1D model builds on mass conservation and momentum equations [40]:

where *x* is the axial coordinate along the vessel, and *A*(*x*, *t*) and *p*(*x*, *t*) are the cross-sectional area and intra-luminal pressure, respectively. \(K_r\) is the friction loss coefficient representing the viscous resistance of flow per unit length along the vessel. It can be expressed as \(K_r = \frac{2\alpha \mu \pi }{\alpha -1}\), where \(\alpha = 4/3\) assumes a laminar blood flow with parabolic velocity profile. We assume no tapering effect such as stenosis or atherosclerosis by setting \(S = 0\).

The system is closed with Laplace’s law on pressure-area:

The pressure *p* is the summation of the external pressure \(p_{ext}\) and pressure variation, which is characterized by arterial wall compliance \(\beta \) and square root of area difference. The Poisson ratio \(\nu \) is taken to be 0.5, and *h* and \(A_{o}\) are the vessel wall thickness and reference cross-sectional area. *E* is the Young’s modulus of the vessel wall.

We use the second-order Adams-Bashforth scheme to discretize and compute the time integration. For spatial discretization, the whole arterial network is decomposed into \(\textit{N}\) non-overlapping domains. Each domain can be further subdivided into elements. Within each element, the solution is formulated as a linear combination of orthogonal Legendre polynomials. At element interfaces, the solver uses the discontinuous Galerkin method to render the discontinuous numerical solutions. We refer interested readers to [40] for details.

The imposed boundary condition must satisfy conservation of mass, which constrains the flow rate at outlets. For the DPD model, we can impose Dirichlet type boundary conditions at inlets and outlets. However, for the 1D system, this would lead to a defective boundary condition [12] and a problem with well-posedness [35]. An alternative to the defective boundary condition is the Windkessel model [48], which is analogous to open-loop circuits. This approach captures the effective resistance (R) and compliance (C) of the neglected downstream vessels. In particular, we apply a three element “Windkessel” RCR boundary conditions to each of the outlets, as shown in Fig. 21(b). The RCR boundary conditions are given as:

where \(R_{p}\), \(R_{d}\), *C*, \(A^{*}\) and \(q_{in}\) represent proximal and distal resistance, capacitance, area and outflow at 1D outlets. We tune the values of each resistor and capacitor so that the outflow rate and pressure match those of experimental measurements. For more detail, we refer interested readers to [2].

### 1.3 Mapping to physical units

For the DPD model, we set the length scale \([L^{\text {DPD}}] = 1 \times 10^{-6}\)m. This means that one unit length in the DPD system is equivalent to \(10^{-6}\) meters of physical length. By matching the viscosity of real blood to that of the DPD fluid, the time scale \([T^{\text {DPD}}]\) can be evaluated with relation:

where \(\mu \) is the RBC membrane shear modulus, and \(\nu \) is the viscosity. The superscripts *M* and *P* denote the model and physical units, respectively. For the blood simulation, we use the viscosity of plasma \(\nu ^{\text {P}} = 1.2 \times 10^{-3}\) Pa s, and the RBC membrane shear modulus \(\mu ^{\text {P}}_{\text {s}} = 4.5 \times 10^{-6}\) Nm\(^{-1}\):

Here, we set \(\mu ^{\text {M}} = 100\) and measured \(\nu ^{\text {M}}=21.8132\) from running benchmark simulations.

#### 1.3.1 Parallel implementation

The continuum and particle solvers are, by themselves, stand-alone solvers with complex code structures. Implementing codes into the solvers for coupling, while complying with the structure of the software, is a challenging task. To minimize code refactoring, we use Multiscale Universal Interface (MUI). MUI is a library that facilitates the concurrent coupling of heterogeneous solvers [43]. It integrates MIMD (Multiple instruction streams, multiple data streams) and an asynchronous communication protocol to handle inter-solver information exchange, irrespective of intra-solver MPI implementation. Coupling via MUI does not pollute the MPI communicator space, and therefore requires less code refactoring.

We illustrate the information-exchange pipeline in Fig. 22a. In this step, CS sends the volume-flux at various times to the FSs as initial conditions. Each FS receives its initial condition and resets its state to match the condition. Because there is no temporal overlap among the fine-propagations, they can be carried out concurrently. For each FS, its simulation domain is decomposed into multiple sub-domains for spatial acceleration, which creates an additional layer of parallelism. The results are then sent back to the coarse-solver.

The dual level of parallelism (i.e. spatial and temporal) is achieved owing to the clever arrangement of intra-solver and inter-solver communicators. The inter-solver communication is managed by MUI through MPI inter-communicators, which establish links between the CS and each of the FSs, as illustrated in Fig. 22b. In contrast, inter-solver communication is enabled by MPI intra-communicators. The master of each propagator shares the incoming information among the associated worker processes, each of which handles a sub-domain. At the end of a fine-propagation, the master pools information from the worker processes. The gathered data are then sent back through the inter-solver communicator.

## Rights and permissions

## About this article

### Cite this article

Blumers, A.L., Yin, M., Nakajima, H. *et al.* Multiscale parareal algorithm for long-time mesoscopic simulations of microvascular blood flow in zebrafish.
*Comput Mech* **68**, 1131–1152 (2021). https://doi.org/10.1007/s00466-021-02062-w

Received:

Accepted:

Published:

Issue Date:

DOI: https://doi.org/10.1007/s00466-021-02062-w

### Keywords

- Multiscale modeling
- Zebrafish
- Vascular network
- Parallel-in-time
- Dissipative particle dynamics
- 1D blood flow modeling