Assessment of CFD Performance in Simulations of an Idealized Medical Device: Results of FDA’s First Computational Interlaboratory Study

Abstract

While computational fluid dynamics (CFD) is commonly used for medical device development, its usefulness for demonstrating device safety has not been proven. Reliable standardized methods for this specialized need are lacking and are inhibiting the use of computational methods in the regulatory review of medical devices. To meet this need, participants from academia, industry, and the U.S. Food and Drug Administration recently completed a computational interlaboratory study to determine the suitability and methodology for simulating fluid flow in an idealized medical device. A technical working committee designed the study, defined the model geometry and flow conditions, and identified comparison metrics. The model geometry was a 0.012 m diameter cylindrical nozzle with a conical collector and sudden expansion on either side of a 0.04 m long, 0.004 m diameter throat, which is able to cause hemolysis under certain flow conditions. Open invitations to participate in the study were extended through professional societies and organizations. Twenty-eight groups from around the world submitted simulation results for five flow rates, spanning laminar, transitional, and turbulent flows. Concurrently, three laboratories generated experimental validation data on geometrically similar physical models using particle image velocimetry. The simulations showed considerable variation from each other and from experiment. One main source of error involved turbulence model underestimations of the centerline velocities in the inlet and throat regions, because the flow was laminar in these regions. Turbulence models were also unable to accurately predict velocities and shear stresses in the recirculation zones downstream of the sudden expansion. The wide variety in results suggest that CFD studies used to assess safety in medical device submissions to the FDA require careful experimental validation. Better transitional models are needed, as many medical devices operate in the transitional regime. It is imperative that the community undertake and publish quality validation cases of biofluid dynamics and blood damage that include complications such as pulsatility, secondary flows, and short and/or curved inlets and outlets. The results of this interlaboratory study will be available in a benchmark database to help develop improved modeling techniques, and consensus standards and guidelines for using CFD in the evaluation of medical devices.

Appendix: Best Practices

CFD “Best Practices” identified in this study are listed below. These are not necessarily all-inclusive, nor do they guarantee an accurate simulation.

• (1) Identify the physics of the problem and choose the appropriate physical model.

• (2) Choose software carefully so that the physics of the problem can be properly captured.

• (3) If a turbulence model is used, justify the choice of the model and parameters used.

• (4) Correctly identify and quantify the necessary boundary conditions, initial conditions, material properties and other problem specifications.

• (5) Verify that the boundaries of the simulation do not interfere with the natural development and physical character of the flow.

• (6) Confirm that the simulation has numerically converged via residual reductions and/or monitoring of some physically relevant fluid flow quantity at a relevant probe point (or points) and/or surface location(s). For example, residuals of the algebraic solver for each dependent variable should be plotted. For steady simulations, three orders of magnitude drop is typically sufficient for an engineering analysis. Corresponding iterative convergence of both point and integral metrics should be plotted to determine the iterative error. For unsteady simulations, iterative convergence of algebraic solvers must be assessed, as well as the impact of outer correctors, and non-orthogonal correctors on the time-accuracy.

• (7) Perform a proper grid refinement study, preferably by doubling and quadrupling the grid density over the production grid. Choose reasonable variables of interest for convergence monitoring and a logical rationale for choosing the “best” mesh. In general, grid refinement studies are beneficial, but insufficient to ensure good results.

• (8) Validate the simulation with trustworthy experimental measurements of both local and global variables, using quantitative measures rather than side-by-side comparisons of contour plots. A global performance metric (e.g., pressure drop across a device) is rarely by itself a useful validation metric.

• (9) Validation should be performed against like or similar types of models.

• (10) Anisotropic boundary layer meshing used to resolve viscous effects should have mesh spacing that can be justified via physical reasoning.

• (11) Mesh cell type (e.g., triangular, tetrahedral, hexahedral) should be chosen with knowledge that different cell types have different numerical characteristics (e.g., artificial dissipation, limited order of spatial accuracies, etc.)

• (12) Portions of the flow domain meshed with a higher density than other regions should be chosen carefully with guidance from the intermediate solutions.

• (13) Make sure the results are reasonable and/or intuitive from a fluid mechanics point of view.

• (14) Ensure that basic conservation laws are obeyed. For example, in this study, the conservation of mass was checked by integrating velocity profiles. Other methods may need to be developed in other problems.

• (15) If available, use existing theoretical solutions (e.g., steady and unsteady pipe flow) and benchmark validation data (e.g., backward facing step) to check accuracy, simulation protocol, selection of parameters, etc. For example, the present sudden expansion case produces similar results to those found previously.8

• (16) When reporting results, provide plots with legends and labels. Contour plots may not be the best format for showing experimental validation or mesh convergence; quantitative results should be provided.

• (17) Some of the elementary errors made in this study suggest that training and experience are essential components to performing accurate simulations.