Multiscale parareal algorithm for long-time mesoscopic simulations of microvascular blood flow in zebrafish

Abstract

Various biological processes such as transport of oxygen and nutrients, thrombus formation, vascular angiogenesis and remodeling are related to cellular/subcellular level biological processes, where mesoscopic simulations resolving detailed cell dynamics provide a key to understanding and identifying the cellular basis of disease. However, the intrinsic stochastic effects can play an important role in mesoscopic processes, while the time step allowed in a mesoscopic simulation is restricted by rapid cellular/subcellular dynamic processes. These challenges significantly limit the timescale that can be reached by mesoscopic simulations even with high-performance computing. To break this bottleneck and achieve a biologically meaningful timescale, we propose a multiscale parareal algorithm in which a continuum-based solver supervises a mesoscopic simulation in the time-domain. Using an iterative prediction-correction strategy, the parallel-in-time mesoscopic simulation supervised by its continuum-based counterpart can converge fast. The effectiveness of the proposed method is first verified in a time-dependent flow with a sinusoidal flowrate through a Y-shaped bifurcation channel. The results show that the supervised mesoscopic simulations of both Newtonian fluids and non-Newtonian bloods converge to reference solutions after a few iterations. Physical quantities of interest including velocity, wall shear stress and flowrate are computed to compare against those of reference solutions, showing a less than 1% relative error on flowrate in the Newtonian flow and a less than 3% relative error in the non-Newtonian blood flow. The proposed method is then applied to a large-scale mesoscopic simulation of microvessel blood flow in a zebrafish hindbrain for temporal acceleration. The three-dimensional geometry of the vasculature is constructed directly from the images of live zebrafish under a confocal microscope, resulting in a complex vascular network with 95 branches and 57 bifurcations. The time-dependent blood flow from heartbeats in this realistic vascular network of zebrafish hindbrain is simulated using dissipative particle dynamics as the mesoscopic model, which is supervised by a one-dimensional blood flow model (continuum-based model) in multiple temporal sub-domains. The computational analysis shows that the resulting microvessel blood flow converges to the reference solution after only two iterations. The proposed method is suitable for long-time mesoscopic simulations with complex fluids and geometries. It can be readily combined with classical spatial decomposition for further acceleration.

Appendix

1.1 dissipative particle dynamics

Each DPD particle is represented explicitly by its position and velocity. Its motion is governed by Newton’s equations of motion [18]:

$$\begin{aligned}&\frac{d\mathbf {r}_i}{dr} = v_i , \end{aligned}$$

(7)

$$\begin{aligned}&\frac{d\mathbf {v}_i}{dt} = \mathbf {F}_i = \sum _{j \ne i} \big (\mathbf {F}^C_{ij} + \mathbf {F}^D_{ij} + \mathbf {F}^R_{ij} \big ), \end{aligned}$$

(8)

where t, $\mathbf {r}_i$, $\mathbf {v}_i$ and $\mathbf {F}_i$ denote time, position, velocity, and force, respectively. The force comprises three components: conservative force $\mathbf {F}^C_{ij}$, dissipative force $\mathbf {F}^D_{ij}$, and corresponding random force $\mathbf {F}^R_{ij}$ from particle j within a radial cutoff $r_c$ of particle i. They are expressed as [17]:

$$\begin{aligned}&\mathbf {F}^C_{ij} = \alpha _{ij} \; \omega _C(r_{ij}) \; \mathbf {e}_{ij}, \end{aligned}$$

(9)

$$\begin{aligned}&\mathbf {F}^D_{ij} = -\gamma _{ij} \; \omega _D(r_{ij}) \; ( \mathbf {e}_{ij} \cdot \mathbf {v}_{ij} ) \; \mathbf {e}_{ij}, \end{aligned}$$

(10)

$$\begin{aligned}&\mathbf {F}^R_{ij} = \sigma _{ij} \; \omega _R(r_{ij}) \; \xi _{ij} \; \Delta t_f^{-1/2} \; \mathbf {e}_{ij}, \end{aligned}$$

(11)

where $\mathbf {e}_{ij}=\mathbf {r}_{ij} / r_{ij}$ is the unit vector between particles i and j, and $\mathbf {v}_{ij}$ is the velocity difference. $\Delta t_f$ is the time step of the fine propagator, and $\xi$ is a symmetric Gaussian random variable with zero mean and unit variance [17]. The strengths of the conservative, dissipative, and random force are $\alpha _{ij}$, $\gamma _{ij}$ and $\sigma _{ij}$, respectively. The interaction between pairs of particles are regulated by weighting functions $\omega _C(r_{ij})$, $\omega _D(r_{ij})$ and $\omega _R(r_{ij})$. Most importantly, the balance between dissipation and thermal fluctuation is maintained by the fluctuation-dissipation theorem [9]:

$$\begin{aligned} \sigma ^2_{ij} = 2 \gamma _{ij} k_B T , \;\;\;\;\;\; \omega _D(r_{ij}) = \omega _R^2(r_{ij}), \end{aligned}$$

(12)

where $k_B T$ is the Boltzmann energy unit. A common choice for the weight functions is $\omega _C(r)=1-r/r_c$ and $\omega _D(r)=\omega ^2_R(r)=(1-r/r_c)^2$ for $r\le r_c$ and zero for $r>r_c$. For simplicity, $k_BT$ and the mass of a particle are taken as the energy unit and mass unit, and their values are set to one.

1.1.1 RBC and solvent

The particle-based blood flow model is based on the DPD approach [10, 34]. In this model, the blood is composed of solvents and red blood cells. The solvent is modeled with single-particles subject to only pairwise forces described in Sect. 5.1, and red blood cells are modeled discretely with bonded-particles. Membrane viscosity, elasticity and bending stiffness can be recovered with a spring-network that resembles a triangular mesh on a 2D surface. The shape of the viscoelastic membrane is maintained by a potential derived from a combination of bond, angle and dihedral interactions. The bonds between particles have an attractive and a repulsive component mimicking springs. The attractive potential adopts the form of the wormlike chain potential and is given by [10, 34]

$$\begin{aligned} V_{WLC} = \frac{k_BTh_m}{4p}\frac{\frac{h}{h_m}^2(3-2\frac{h}{h_m})}{1-\frac{h}{h_m}}, \end{aligned}$$

(13)

where $k_BT$ is the energy per unit mass, $h_m$ is the maximum spring extension, and p is the persistence length. The repulsive potential adopts the form of a power function given by

$$\begin{aligned} V_{POW} = {\left\{ \begin{array}{ll} \frac{k_p}{(m-1)h^{m-1}}, &{} m \ne 1, \\ -k_p\log (h), &{} m=1, \end{array}\right. } \end{aligned}$$

(14)

where m is a positive coefficient, and $k_p$ is force coefficient. Additionally a viscous component damping the springs is realized by a dissipative force, as well as the corresponding random force, given as

$$\begin{aligned}&\mathbf {F}^D_{ij} = -\gamma ^T \mathbf {v}_{ij} - \gamma ^C (\mathbf {v}_{ij} \cdot \mathbf {e}_{ij}) \mathbf {e}_{ij}, \end{aligned}$$

(15)

$$\begin{aligned}&\mathbf {F}^R_{ij} = \sqrt{2k_BT} \Big (\sqrt{2\gamma ^T} (d\mathbf {W}^S_{ij} - tr[d\mathbf {W}^S_{ij}]\mathbf {I}/3)\nonumber \\&\quad \quad + \frac{\sqrt{3\gamma ^C-\gamma ^T}}{3} tr[d\mathbf {W}_{ij}] \mathbf {I} \Big ), \end{aligned}$$

(16)

where $\gamma ^T$ and $\gamma ^C$ are dissipative coefficients, and $v_{ij}$ is the relative velocity. $d\mathbf {W}^S_{ij}$ is the symmetric component of a random matrix of independent Wiener increments $d\mathbf {W}_{ij}$.

RBC’s structural stability is maintained by the area and volume constraints given by

$$\begin{aligned}&V_{area} = \frac{k_g(A^t-A^t_0)^2}{2A^t_0} + \sum _j \frac{k_l(A_j - A_{0,j})^2}{2A_{0,j}}, \end{aligned}$$

(17)

$$\begin{aligned}&V_{volume} = \frac{k_v(V^t-V^t_0)^2}{2V^t_0}, \end{aligned}$$

(18)

where j is the triangle index. $k_g$, $k_l$, and $k_v$ are global area, local area, and global volume constraints coefficients, respectively. The instantaneous area and volume of a RBC are denoted by $A^t$ and $V^t$, whereas $A^t_0$ and $V^t_0$ represent the equilibrium area and volume. $A^t_0$ is calculated by summing the area of each triangle. $V^t_0$ is found according to scaling relationship $V^t_0/(A^t_0)^{3/2} = V^R / (A^R)^{3/2}$, where $V^R$ and $A^R$ are the experimental measurements. Lastly, the bending resistance of the membrane is modeled by

$$\begin{aligned} V_{bending} = \sum _j k_b \Big ( 1-\cos (\theta _j-\theta _0) \Big ), \end{aligned}$$

(19)

where $k_b$ is the bending constant, and $\theta _j$ is the angle between two neighboring triangles with the common edge j. We refer interested readers to references [10, 34] for more details.

1.2 1D blood flow model

Under the assumptions of axial symmetry and radial displacement of the vessel wall (Fig. 21a) [4, 40], the modeling of blood flow in a compliant vessel can be reduced from full-order Navier–Stokes equations to an 1D model. The 1D model has been widely used for blood flow simulations at a moderate [40] and low [33] Reynolds number. In our implementation, it serves as the low-fidelity model supervising the high-fidelity particle model because of its balance between its speed and accuracy.

The 1D model builds on mass conservation and momentum equations [40]:

$$\begin{aligned}&\frac{\partial A}{\partial t} + \frac{\partial AU}{\partial x} = 0 \end{aligned}$$

(20)

$$\begin{aligned}&\frac{\partial U}{\partial t} + \frac{1}{2}\frac{\partial U^2}{\partial x} + \frac{1}{\rho }\frac{\partial p}{\partial x} = -\frac{K_rU}{\rho A}, \end{aligned}$$

(21)

where x is the axial coordinate along the vessel, and A(x, t) and p(x, t) are the cross-sectional area and intra-luminal pressure, respectively. $K_r$ is the friction loss coefficient representing the viscous resistance of flow per unit length along the vessel. It can be expressed as $K_r = \frac{2\alpha \mu \pi }{\alpha -1}$, where $\alpha = 4/3$ assumes a laminar blood flow with parabolic velocity profile. We assume no tapering effect such as stenosis or atherosclerosis by setting $S = 0$.

The system is closed with Laplace’s law on pressure-area:

$$\begin{aligned} p= & {} p_{ext} + \beta \big ( \root \of {A}-\root \of {A_{o}} \big ), \end{aligned}$$

(22)

$$\begin{aligned} \beta= & {} \frac{\sqrt{\pi }h E}{(1-\nu ^2)A_{o}}, \end{aligned}$$

(23)

The pressure p is the summation of the external pressure $p_{ext}$ and pressure variation, which is characterized by arterial wall compliance $\beta$ and square root of area difference. The Poisson ratio $\nu$ is taken to be 0.5, and h and $A_{o}$ are the vessel wall thickness and reference cross-sectional area. E is the Young’s modulus of the vessel wall.

Fig. 22

Pipeline of the parallel implementation. a CS sends the volume-flux to the FSs as initial conditions. The fine-solvers then propagate the systems concurrently. For each FS, its simulation domain is decomposed into multiple sub-domains for additional layer of parallelism. b The dual level of parallelism (i.e. spatial and temporal) is achieved owing to the clever arrangement of intra-solver and inter-solver MPI communicators

We use the second-order Adams-Bashforth scheme to discretize and compute the time integration. For spatial discretization, the whole arterial network is decomposed into $\textit{N}$ non-overlapping domains. Each domain can be further subdivided into elements. Within each element, the solution is formulated as a linear combination of orthogonal Legendre polynomials. At element interfaces, the solver uses the discontinuous Galerkin method to render the discontinuous numerical solutions. We refer interested readers to [40] for details.

The imposed boundary condition must satisfy conservation of mass, which constrains the flow rate at outlets. For the DPD model, we can impose Dirichlet type boundary conditions at inlets and outlets. However, for the 1D system, this would lead to a defective boundary condition [12] and a problem with well-posedness [35]. An alternative to the defective boundary condition is the Windkessel model [48], which is analogous to open-loop circuits. This approach captures the effective resistance (R) and compliance (C) of the neglected downstream vessels. In particular, we apply a three element “Windkessel” RCR boundary conditions to each of the outlets, as shown in Fig. 21(b). The RCR boundary conditions are given as:

$$\begin{aligned} q_{in} = \frac{P(A^{*}) - P_{m}}{R_{p}} = C\frac{dP_{c}}{dt} + \frac{P_{m} - P_{out}}{R_{d}} \end{aligned}$$

(24)

where $R_{p}$, $R_{d}$, C, $A^{*}$ and $q_{in}$ represent proximal and distal resistance, capacitance, area and outflow at 1D outlets. We tune the values of each resistor and capacitor so that the outflow rate and pressure match those of experimental measurements. For more detail, we refer interested readers to [2].

1.3 Mapping to physical units

For the DPD model, we set the length scale $[L^{\text {DPD}}] = 1 \times 10^{-6}$m. This means that one unit length in the DPD system is equivalent to $10^{-6}$ meters of physical length. By matching the viscosity of real blood to that of the DPD fluid, the time scale $[T^{\text {DPD}}]$ can be evaluated with relation:

$$\begin{aligned} {[}T^{\text {DPD}}] = [L^{\text {DPD}}] \frac{\nu ^{\text {P}}}{\nu ^{\text {M}}} \frac{\mu ^{\text {M}}}{\mu ^{\text {P}}} (\text {s}), \end{aligned}$$

(25)

where $\mu$ is the RBC membrane shear modulus, and $\nu$ is the viscosity. The superscripts M and P denote the model and physical units, respectively. For the blood simulation, we use the viscosity of plasma $\nu ^{\text {P}} = 1.2 \times 10^{-3}$ Pa s, and the RBC membrane shear modulus $\mu ^{\text {P}}_{\text {s}} = 4.5 \times 10^{-6}$ Nm$^{-1}$:

$$\begin{aligned}{}[T^{\text {DPD}}] = 10^{-6} \frac{1.2 \times 10^{-3}}{21.8132} \frac{100}{4.5 \times 10^{-6}} = 1.2225 \times 10^{-3} \text {s}.\nonumber \\ \end{aligned}$$

(26)

Here, we set $\mu ^{\text {M}} = 100$ and measured $\nu ^{\text {M}}=21.8132$ from running benchmark simulations.

1.3.1 Parallel implementation

The continuum and particle solvers are, by themselves, stand-alone solvers with complex code structures. Implementing codes into the solvers for coupling, while complying with the structure of the software, is a challenging task. To minimize code refactoring, we use Multiscale Universal Interface (MUI). MUI is a library that facilitates the concurrent coupling of heterogeneous solvers [43]. It integrates MIMD (Multiple instruction streams, multiple data streams) and an asynchronous communication protocol to handle inter-solver information exchange, irrespective of intra-solver MPI implementation. Coupling via MUI does not pollute the MPI communicator space, and therefore requires less code refactoring.

We illustrate the information-exchange pipeline in Fig. 22a. In this step, CS sends the volume-flux at various times to the FSs as initial conditions. Each FS receives its initial condition and resets its state to match the condition. Because there is no temporal overlap among the fine-propagations, they can be carried out concurrently. For each FS, its simulation domain is decomposed into multiple sub-domains for spatial acceleration, which creates an additional layer of parallelism. The results are then sent back to the coarse-solver.

The dual level of parallelism (i.e. spatial and temporal) is achieved owing to the clever arrangement of intra-solver and inter-solver communicators. The inter-solver communication is managed by MUI through MPI inter-communicators, which establish links between the CS and each of the FSs, as illustrated in Fig. 22b. In contrast, inter-solver communication is enabled by MPI intra-communicators. The master of each propagator shares the incoming information among the associated worker processes, each of which handles a sub-domain. At the end of a fine-propagation, the master pools information from the worker processes. The gathered data are then sent back through the inter-solver communicator.