The FDA nozzle
A cross-sectional view of the FDA nozzle is shown in Fig. 1. The flow direction is from left to right implying a sudden expansion. If the flow is applied from right to left, the same geometry will act as a canonical diffuser, which is not studied in this work. The 3D model of this nozzle was downloaded from the FDA website.Footnote 1 The outlet of the model was extended up to z = 0.253 m and the inlet was extended up to z = − 0.14 m. The radial profiles were recorded at 12 locations depicted in Fig. 1 to enable a comparison against the particle image velocimetry (PIV) experimental data released by the FDA.
Simulations were conducted with three different spatial and temporal resolutions in order to explore the capabilities of an off the shelf LBM on a desktop machine to a federal supercomputer. Based on previous observations on resolution requirements in LBM simulations of transitional flow [14], a spatial resolution of about 80μm was predicted as a minimum requirement for simulation of Reth = 2000 and accurate analysis of flow characteristics thereof. This resolution resulted in \(\sim 45\) million lattice sites representing fluid inside the nozzle and 50 lattice cells across the height of the nozzle throat. This is referred to as normal resolution (NR) hereon. For Reth = 3500 however, the flow was expected to transition and a high resolution (HR) of 40μm was thus chosen, which resulted in about 357 million lattice cells. As would be seen in Section 3, HR resolution sufficed for accurate flow simulations for both the Reynolds numbers. To further ensure mesh independence, assess the flow in accordance with Kolmogorov theory, and especially because it was not known whether LBM simulations will transition to turbulence, an additional set of simulations were conducted with an extreme resolution (XR) of 20μm resulting in about 2.88 billion lattice cells in the fluid domain.
Table 2 lists the employed spatial and temporal resolutions, the resulting number of lattice cells, the utilized CPUs, and total execution time of the simulation.
Table 2 Three different spatial and temporal resolutions and the corresponding number of lattice cells inside the fluid domain Setup of a simulation using the lattice Boltzmann method
To shed light on the aforementioned choice of resolutions and LB parameters, a brief explanation is provided here. The LBM is based on the mesoscopic representation of movement of fictitious particles. The particles have discrete velocities and they collide and stream to relax towards a thermodynamic equilibrium. The LB equation recovers the NSE under the continuum limits of low Mach and Knudsen numbers. Evolution of the particle probability distribution functions over time is described by the lattice Boltzmann equation with the MRT collision matrix:
$$ \begin{array}{@{}rcl@{}} f_{i}(\mathbf{r} + \mathbf{c}_{i}\delta t, t + \delta t) = f_{i}(\mathbf r,t) + {\Omega}_{ij} \left( {f_{i}^{e}}(\mathbf r,t) - f_{i}(\mathbf r,t) \right) \end{array} $$
(1)
where fi represents the density distributions of particles which are moving with discrete velocity ci at a position r at time t. The indices which run from i = 1 … Q denote the links per element, i.e., the discrete directions, depending on the chosen stencil (D3Q19 in our case). The collision matrix Ωij defines relaxation of various modes of the distribution functions fi towards an equilibrium \({f_{i}^{e}}\):
$$ \begin{array}{@{}rcl@{}} {f_{i}^{e}} = w_{i} \rho \Bigg(1 + \frac{\mathbf{c}_{i} \cdot \mathbf{u}}{{c_{s}^{2}}} - \frac{\mathbf{u}^{2}}{2{c_{s}^{2}}} + \frac{1}{2} \frac{(\mathbf{c}_{i} \cdot \mathbf{u})^{2}}{{c_{s}^{4}}} \Bigg) \end{array} $$
(2)
where wi are the weights for each discrete link, cs is the reference speed of sound in LBM obtained by integration of the discrete Boltzmann equation along characteristics, and u is the fluid velocity. The time step in LBM is coupled with the grid size by \(\delta t \sim \delta x^{2}\) due to diffusive scaling which is employed to recover the incompressible NSE. Details on the computation of macroscopic quantities from LBM can be found elsewhere [28]; here we restrict ourselves on the process of setting up a flow simulation.
Prescription of flow physics
Quantities like the mean velocity, \(\bar {u}\), on which the Reynolds number is based, fluid density ρ, and the kinematic viscosity ν are firstly chosen for the fluid to be simulated. The Reynolds number is then calculated as:
$$ \begin{array}{@{}rcl@{}} \text{Re} = \frac{\bar{u} d}{\nu} \end{array} $$
(3)
where d is the characteristic length equal to the throat diameter (0.004m) in this case. In this study, the Reynolds number is based on the nozzle throat.
The LB method requires the translation of these quantities into lattice units. Under diffusive scaling, the principal relaxation parameter Ω is fixed and can have a maximum value of 2.0 for stability. The lattice viscosity, νlattice, is then calculated using the relation:
$$ \begin{array}{@{}rcl@{}} \nu_{lattice} = {c_{s}^{2}} \bigg(\frac{1}{\Omega} - \frac{1}{2} \bigg) \end{array} $$
(4)
where \({c_{s}^{2}}\) is the speed of sound squared at reference state and is equal to \(\frac {1}{3}\) for the chosen stencil D3Q19. The time step is then calculated as:
$$ \begin{array}{@{}rcl@{}} \delta t = \frac{\nu_{lattice} \delta x^{2}}{\nu} \end{array} $$
(5)
Finally, the lattice velocity is obtained as:
$$ \begin{array}{@{}rcl@{}} \bar{u}_{lattice} = \bar{u} \delta t / \delta x \end{array} $$
(6)
The MRT collision operator allows for different relaxation rates of various moments of the distribution function, and it was employed throughout for the DNS reported in this manuscript. The principal relaxation parameter that defines the kinematic viscosity was set to Ω = 1.90. It is required that the \(\bar {u}_{lattice}\) in Eq. 6 is always less than 0.15 to enforce the Mach number limit of the LBM. The Ω can be adjusted in order to fine tune the lattice velocity and beyond its limit, the grid has to be refined (reduce the δx), which is also the principle limitation of the lattice Boltzmann method. Thus, the δt is controlled by the δx and the Ω, and it is further constrained by the fluid velocity. To assert correct prescription of parameters, if the characteristic length in Eq. 3 is replaced by the number of fluid cells along that particular characteristic length and \(\bar {u}\) and ν are replaced by \(\bar {u}_{lattice}\) and νlattice, respectively, the same Reynolds number must be obtained as that obtained from Eq. 3, which uses physical values.
The simulation framework
The employed simulation tool-chain is contained in the end-to-end parallel framework APES (adaptable poly engineering simulator) [17, 24].Footnote 2 Meshes are created using the mesh generator Seeder [9] and computations are carried out using the LBM solver Musubi [10]. Musubi writes out binary files containing physics information to the disk. These files are then converted to the visualization toolkit (VTK) format by the post-processing tool Harvester, which is contained within the APES framework. The open-source visualization tool ParaviewFootnote 3 is then used to visualize the physics of flow. The data for plots is written out by Musubi as ASCII files that are plotted using the Matplotlib plotting library within the Python programming language.
The 3D model of the nozzle in STL format was read by the mesh generator Seeder and volume meshes for LBM computations were saved on the disk. A higher order wall boundary condition described by Bouzidi et al. [3] was prescribed at the walls of the nozzle to reduce the influence of staircase artifacts in LBM and ensure rotational symmetry of the setup. The Musubi LBM solver then computed these meshes (see Table 2) on the SuperMUC-NG petascale system installed at the Leibniz Supercomputing Center in Munich, Germany. The number of utilized cores on SuperMUC-NG ranged from 38,016 to 304,128 (the whole system) based on our previous evaluations which have shown that this choice of CPUs results in minimal communication overhead among CPUs thereby ensuring the most optimal utilization of compute resources [17, 23].
Initial and boundary conditions
Initial conditions were set to zero pressure and velocity and a no-slip boundary condition was maintained at the walls. The flow rates at inlet were characterized based on the throat Reynolds number. The fluid itself was chosen to be Newtonian, with a fluid density of 1056kg/m3 and dynamic viscosity of 0.0035Ns/m2.
A parabolic velocity profile was prescribed at the inlet by quadratically interpolating the velocity at lattice nodes, and a zero pressure was maintained at the outlet. No synthetic turbulence was prescribed at the inlet unlike some previous studies. The outflow in such flow regimes can be critical resulting in back flow or instabilities in the LBM algorithm for which an extrapolation boundary condition was employed [16].
Following Fehn et al. [7], the simulated time interval was chosen on the basis of a characteristic flow-through time. The length of the throat, Lth, section and the mean velocity, \(\bar {u}\), were chosen as characteristic length scale and the reference velocity respectively. This resulted in a flow through time given by:
$$ \begin{array}{@{}rcl@{}} T_{ft} = \frac{L_{th}}{\bar{u}} \end{array} $$
(7)
For the Reth = 2000 case, this time was approximately 0.21 s whereas for the Reth = 3500 case, it was 0.12 s. Flow was allowed to develop for initial 2 s implying about 10 and 20 flow throughs for the Reth = 2000 and Reth = 3500 case respectively. Within the computational bounds, a total of 10 physical seconds were simulated for each case. A total of 50,000 samples were gathered every second for the analysis of flow characteristics.
Flow characterization
For the analysis of turbulent characteristics, the three-dimensional velocity field was decomposed into a mean and a fluctuating component as:
$$ \begin{array}{@{}rcl@{}} u_{i}(x,t) = \bar{u}_{i}(x) + u_{i}^{\prime}(x,t) \end{array} $$
(8)
The turbulent kinetic energy (TKE) was derived from the fluctuating components of the velocity in three directions as:
$$ \begin{array}{@{}rcl@{}} k = \frac{1}{2}\Big({u_{x}^{\prime 2} + u_{y}^{\prime 2} + u_{z}^{\prime 2}} \Big) \end{array} $$
(9)
The dimensionless Strouhal number, St, is defined as:
$$ \begin{array}{@{}rcl@{}} St = \frac{f d}{\bar{u}} \end{array} $$
(10)
where f is the frequency of flow fluctuations and d and \(\bar {u}\) are the characteristic length and mean velocities. A power spectral density of the TKE was computed as a function of the Strouhal number at various centerline locations using the Welch’s periodogram method to analyze the inertial and viscous subranges.
Kolmogorov microscales
The smallest structures that can exist in a turbulent flow can be estimated on the basis of Kolmogorov’s theory [22]—used in this work to assess the quality of the resolutions in DNS. Viscosity dominates and the TKE is dissipated into heat at the Kolmogorov scale [22]. The Kolmogorov theory, in general, applies to fully developed turbulent flows with Reynolds numbers much higher than those studied in this work. The reference to Kolmogorov scales and theory here is thus indicative for the assessment of mesh independence as has been done in various such studies at low Reynolds numbers [11, 12].
The Kolmogorov scales, non-dimensionalized with respect to the velocity scale \(\bar {u}\), and the length scale d (throat diameter) are computed from the fluctuating component of the non-dimensional strain rate defined as:
$$ \begin{array}{@{}rcl@{}} s^{\prime}_{ij} = \frac{1}{2}\bigg(\frac{\partial u_{i}^{\prime}}{\partial x_{j}} + \frac{\partial u_{j}^{\prime}}{\partial x_{i}}\bigg) \end{array} $$
(11)
The Kolmogorov length, time, and velocity scales are then respectively computed as:
$$ \begin{array}{@{}rcl@{}} \eta = \bigg(\frac{1}{Re^{2}} \frac{1}{2 s^{\prime}_{ij}s^{\prime}_{ij}} \bigg)^{1/4} \end{array} $$
(12)
$$ \begin{array}{@{}rcl@{}} \tau_{\eta} = \bigg(\frac{1}{2 s^{\prime}_{ij}s^{\prime}_{ij}}\bigg)^{1/2} \end{array} $$
(13)
$$ \begin{array}{@{}rcl@{}} u_{\eta} = \bigg(\frac{2s^{\prime}_{ij}s^{\prime}_{ij}}{Re^{2}} \bigg)^{1/4} \end{array} $$
(14)
Based on these scales, the quality of the spatial and temporal resolutions employed in a simulation can be estimated as the ratio of δx and δt against the corresponding Kolmogorov scales, i.e.,
$$ \begin{array}{@{}rcl@{}} l^{+} = \frac{\delta x}{\eta} t^{+} = \frac{\delta t}{\tau_{\eta}} \end{array} $$
(15)
The Kolmogorov scales in this study were computed along various locations along the centerline. The fluctuating strain rate was averaged between locations z8 and z12 (Fig. 1) due to variations in TKE caused by jet breakdown in these locations, and the Kolmogorov scales were computed from this averaged strain rate.
Table 3 Kolmogorov scales computed from NR, HR, and XR resolutions for Reth = 2000 and Reth = 3500 Comparison with experiments
The radial velocity profiles in the streamwise direction and the mean centerline velocity at 12 locations shown in Fig. 1 were plotted against corresponding data from 5 PIV experiments. Instantaneous centerline velocities and pressure were analyzed at 120 locations each 0.002 m apart along the length of the nozzle. The 5 FDA experiments were conducted by 3 independent laboratories where one laboratory ran three trials resulting in 5 data sets [8]. The axial component of the velocity uz was plotted directly against the data from experiments.Footnote 4
Pressure was probed and averaged at 12 circular cross sections across the length of the nozzle (z1 − z12). To enable comparison against experiments, the mean pressure difference was normalized with respect to the average velocity at the throat \(\bar {u}\):
$$ \begin{array}{@{}rcl@{}} {\Delta} p^{norm} = \frac{p_{z} - p_{z_{0}}}{0.5 \rho \bar{u}^{2}} \end{array} $$
(16)
where ρ is the fluid density and \(\bar {u}\) is the mean velocity at the nozzle throat, on which the Reth was based (see Eq. 3).
The shear stress was computed from the LBM simulations as it is one of the quantities of interest to predict hemolysis, and was directly compared with the experimental data.
The differences between a particular LBM simulation case and an experimental dataset were quantified on the basis of simple relative percentage errors, computed as:
$$ \begin{array}{@{}rcl@{}} \delta = \left|~\frac{U_{ref} - U_{h}}{U_{ref}}~\right| \times 100 \end{array} $$
(17)
where Uref denotes the reference solution (experiments in our case) and Uh denotes the LBM computed solution; the corresponding solutions can be for pressure, velocity, or shear stress.