A virtual test facility for the efficient simulation of solid material response under strong shock and detonation wave loading
 1.5k Downloads
 37 Citations
Abstract
A virtual test facility (VTF) for studying the threedimensional dynamic response of solid materials subject to strong shock and detonation waves has been constructed as part of the research program of the Center for Simulating the Dynamic Response of Materials at the California Institute of Technology. The compressible fluid flow is simulated with a Cartesian finite volume method and treating the solid as an embedded moving body, while a Lagrangian finite element scheme is employed to describe the structural response to the hydrodynamic pressure loading. A temporal splitting method is applied to update the position and velocity of the boundary between time steps. The boundary is represented implicitly in the fluid solver with a level set function that is constructed onthefly from the unstructured solid surface mesh. Blockstructured mesh adaptation with time step refinement in the fluid allows for the efficient consideration of disparate fluid and solid time scales. We detail the design of the employed objectoriented mesh refinement framework AMROC and outline its effective extension for fluid–structure interaction problems. Further, we describe the parallelization of the most important algorithmic components for distributed memory machines and discuss the applied partitioning strategies. As computational examples for typical VTF applications, we present the dynamic deformation of a tantalum cylinder due to the detonation of an interior solid explosive and the impact of an explosioninduced shock wave on a multimaterial soft tissue body.
Keywords
Detonation Wave Ghost Cell Signed Distance Function Fluid Process Cartesian Mesh1 Introduction
The virtual test facility (VTF) is a software environment for coupling solvers for compressible computational fluid dynamics (CFD) with solvers for computational solid dynamics (CSD). The CFD solvers facilitate the computation of flows with strong shocks as well as fluid mixing. The CSD solvers provide capabilities for simulation of dynamic response in solids such as large plastic deformations, fracture and fragmentation. In addition, the VTF can be used to simulate highly coupled fluid–structure interaction problems, such as the high rate deformation of metallic solid targets forced by the loading from the detonation of energetic materials, or the rupture and fragmentation of brittle materials under shock wave impact. At present, all VTF solvers use timeexplicit shockcapturing schemes.
In order to implement fluid–structure coupling in the VTF, we apply a loosely coupled, partitioned approach. The technique operates as follows: one assumes disjoint fluid and solid domains and that the interaction takes place only at the fluid–solid interface. In this way, one can apply algorithms that are intrinsically suited for simulation of phenomena such as shock propagation, detonation or fluid mixing in the fluid solver, while algorithms similarly optimized for phenomena such as highrate plastic deformation or fracture can be employed in the solid solver. For example, a Lagrangian representation is most suitable to account numerically for large solid deformations, contact and fracture, while the governing equations of compressible fluid motion are most effectively solved in an Eulerian frame of reference [1]. In the loose fluid–structure coupling adopted, the information exchange is reduced to communicating the velocities and the geometry of the solid surface to the Eulerian fluid and imposing the hydrostatic pressure onto the Lagrangian solid as a force acting on its exterior [1, 2, 3, 4, 5, 6, 7]. This approach offers several advantages. Firstly, it allows for solver reuse (see [8] or [2] for details on the idea of modularization). Secondly, it becomes straightforward to take advantage of recent advances in multiscale constitutive modeling to describe the dynamic response of both solid and fluid. Such modeling typically also employs a Lagrangian description for solids and an Eulerian description for the fluid.
A key issue that arises with the proposed approach is how to represent the evolving surface geometry on the Eulerian fluid mesh. The application of bodyconforming meshes is somewhat cumbersome, because the fluid equations first need to be cast into a local arbitrary Lagrangian–Eulerian (ALE) frame of reference [9]. At each step, the mesh topology would have to be reconstructed and the solution reinterpolated. While this is possible (and successfully implemented in many presentday codes), the issues of mesh tangling and the requirements of frequent remeshing in the case of large deformations remain a challenge. The need to remesh is also an inherent bottleneck in massively parallel simulations [6].
An alternative to the use of bodyaligned fluid grids is the application of Cartesian meshes with immersed or embedded irregular boundaries. Here, there are two basic approaches: “cutcell” techniques that construct smaller cells by intersecting the Cartesian mesh exactly with the (triangulated) boundary and techniques that “diffuse” the boundary within one cell [10]. Cutcell methods have the advantage that they can represent accurately the boundary flux and thus facilitate the implementation of discretely conservative fluid solvers. However, the proposed numerical circumventions of the severe time step restriction in timeexplicit schemes [11, 12], which can result from very small cells created by the boundary intersection, are logically quite complicated. Most approaches have not yet been extended successfully to three spatial dimensions even for pure fluid flow problems. In the VTF, we therefore employ a diffused boundary technique in which some interior cells are used directly to enforce the embedded boundary conditions in the vicinity of the solid surface [4, 13]. This has been called the “ghost fluid” approach. One advantage of this approach is that the numerical stencil is not modified, thus ensuring optimal parallel scalability. We minimize conservation errors as well as possible numerical “staircase” artifacts at the embedded boundary by using blockstructured dynamic adaptation of the fluid mesh. As the solid deforms, the solidfluid boundary is represented implicitly with a scalar level set variable that is updated onthefly using an efficient algorithm described in more detail below. An important additional advantage of this approach is the ability to cope with topological transitions such as fracture or penetration.
The present paper details the implementation of the VTF approach and also describes its application to fluid–solid interaction problems wherein detonation and shock waves impinge on thick threedimensional solid materials. An extension of the basic, nonadaptive fluid–solid coupling algorithm used herein to thin, open structures has recently been presented in [14]. In Sect. 2, a Cartesian dynamically adaptive finite volume fluid solver for Euler equations with onestep chemistry is described. We detail the design of the underlying mesh refinement framework and discuss its parallelization. Section 3 outlines the parallel Lagrangian finite element solver for solid materials subjected to highintensity shock loadings. In Sect. 4, we describe a highly efficient algorithm to transform a triangulated surface mesh into a signed distance function on a hierarchical Cartesian mesh as a prerequisite for coupling. The fluid–structure interaction methodology, highlighting particular its incorporation into the adaptive fluid mesh refinement framework, and an efficient intersolver communication library are detailed in Sect. 5. Section 6 provides two threedimensional computational examples. In Sect. 6.1, we simulate the impact event of a strong hydrodynamic shock wave on a body comprised of softtissue; in Sect. 6.2 the propagation of a detonation wave in HMX through a plastic tantalum cylinder is described. Both computations were run on distributed memory machines and we comment briefly on the overall computational efficiency of the approach.
2 Eulerian fluid dynamics
In this section, we are concerned with the construction of an Eulerian fluid solver framework suitable for efficient fluid–structure coupling. Although the presentation is tailored to the Euler equations with simple onestep reaction, the concepts are equally applicable to general conservation laws with arbitrary source terms. Within the Center for Simulating the Dynamic Response of Materials at the California Institute of Technology, the same framework is also used successfully with solvers for the compressible Favreaveraged Navier–Stokes equations with a largeeddy turbulence model [15] and for detonation simulation in thermally perfect gas mixtures with detailed chemical kinetics [16, 17, 18, 19, 20].
2.1 Governing equations
2.2 Cartesian finite volume method with embedded boundaries
Following Clarke et al. [22], we apply the method of fractional steps to decouple the chemical reaction and hydrodynamic transport numerically. The homogeneous system of Eqs. 1–4 and the scalar ordinary differential equation ∂_{ t }λ = ψ(λ) are solved successively with the data of the preceding step as initial conditions. As the homogeneous system Eqs. 1–4 is a hyperbolic conservation law that admits discontinuous solutions (cf. [22]), we use a timeexplicit finite volume discretization that achieves a proper upwinding in all characteristic fields. The scheme is based on a straightforward generalization of the Roe scheme for the purely hydrodynamic Euler equations 1–3 and is extended to a multidimensional Cartesian scheme via the method of fractional steps (cf. [23]). To circumvent the intrinsic problem of unphysical total densities and internal energies near vacuum due to the Roe linearization (cf. [24]), the scheme has the possibility to switch to the simple, but extremely robust HartenLaxVan Leer (HLL) Riemann solver [17, 18, 19]. The MUSCLHancock variable extrapolation technique of Van Leer [23] is employed to achieve secondorder accuracy in regions where the solution is smooth.
Geometrically complex moving boundaries are considered within the Cartesian method outlined above by utilizing some of the finite volume cells as ghost cells to enforce immersed boundary conditions [10, 25]. The ghost cell values are set immediately before the original numerical update to model moving embedded walls. The boundary geometry is mapped onto the Cartesian mesh by employing a scalar level set function ϕ that stores the signed distance to the boundary surface and allows the efficient evaluation of the boundary outer normal in every mesh point as \(\mathbf{n}=\nabla\phi/\nabla\phi\) [26]. In coupled Eulerian–Lagrangian simulations, ϕ is updated onthefly by calling the closestpointtransform algorithm described in detail in Sect. 4. A cell is considered to be a valid fluid cell within the interior, if the distance ϕ in the cell midpoint is positive, and is treated as exterior otherwise. The unavoidable staircase approximation of the boundary with this approach is alleviated by using the dynamic mesh adaptation technique described in the next section to also refine the Cartesian mesh appropriately along the boundary.
After the application of the numerical scheme, the cells that have been used to impose the internal boundary conditions are set to the entire state vector of the nearest cell in the interior. This operation achieves a constant value extrapolation and ensures proper values in case such a cell becomes a regular interior cell in the next step due to boundary movement. Note that the boundary velocity \(\mathbf{w}\) gets automatically considered through the velocity modification in the internal ghost cells and the usual stability condition for timeexplicit methods for system 1–4 also ensures that the embedded boundary propagates at most one cell further in every time step.
2.3 Structured adaptive mesh refinement
The numerical scheme is applied on level l by calling a singlegrid routine in a loop over all subgrids. The subgrids get computationally decoupled by employing additional ghost cells around each computational grid. Three different types of ghost cells have to be considered: cells outside of the root domain are used to implement physical boundary conditions. Ghost cells overlaid by a grid on level l have a unique interior cell analogue and are set by copying the data value from the grid, where the interior cell is contained (synchronization). On the root level no further boundary conditions need to be considered, but for l > 0 internal boundaries can also occur. They are set by a conservative timespace interpolation from two previously calculated time steps of level l − 1.
Besides a general tree data structure that stores the topology of the hierarchy (cf. Fig. 2), the SAMR method requires at most two regular arrays assigned to each subgrid. They contain the discrete vector of state for the actual and updated time step. The regularity of the data allows high performance on vector and superscalar processors that allow cache optimizations. Small data arrays are effectively avoided by leaving coarse level data structures untouched when higher level grids are created. Values of cells covered by finer subgrids are subsequently overwritten by averaged fine grid values. This operation leads to a modification of the numerical stencil on the coarse mesh and requires a special flux correction in cells abutting a fine grid. The correction replaces the coarse grid flux along the fine grid boundary by a sum of fine grid fluxes and ensures the discrete conservation property of the hierarchical method (at least for purely Cartesian problems without embedded boundaries; see [27] or [19] for details).
2.4 Parallel implementation
Up to now, various reliable implementations of the SAMR method for single processor computers have been developed. Even the usage of parallel computers with shared memory is straightforward, because timeexplicit methods allow the parallel calculation of the gridwise numerical update [28]. But the question of an efficient parallelization strategy becomes more delicate for distributed memory architectures, because on such machines the costs for communication cannot be neglected. Due to the technical difficulties of implementing dynamical adaptive methods in distributed memory environments only a few parallelization strategies have been considered in practice to date (cf. [29, 30]).
2.5 Design of the AMROC framework
A salient feature of AMROC compared to other available SAMR implementations, e.g. [29, 34], is the realization of objectoriented framework concepts in C++ on all levels of software design. This allows for effective code reuse in implementing parallel SAMR algorithms and extensive capability for customization through subclass derivation.
The intermediate AMROC design level naturally utilizes classes of the base level that provide hierarchical data structures. The base level is divided into elementary functionality for single grid patches and the implementation of various lists that store these patches hierarchically. A common design for the base level (see also [29]) involves a Box class to specify a single rectangular box in global integer index space. Methods for geometric operations on boxes like concatenation or intersection are available. A Patch class adds consecutive data storage to a Box. In AMROC, the geometrical description of all refinement areas is stored in hierarchical lists of Box objects inside a single GridHierarchy. The templatized class GridFunction creates Patch objects for various, possibly complex, data types following exactly the Box lists of GridHierarchy. As the refinement lists in GridHierarchy evolve and are dynamically distributed to an evolving set of processors, the Patch objects in GridFunction are automatically recreated, including parallel redistribution and synchronization.
3 Lagrangian formulation of solid dynamics
We adopt a conventional Lagrangian formulation [36] for describing the large, dynamic deformations of solid materials subject to highintensity shock loadings. The formulation accounts for finite kinematics, inertia and general constitutive behavior, including strength.
3.1 Governing equations
3.2 Shockcapturing material model
3.3 Finiteelement temporal integration
3.4 Parallel implementation

The material library, which provides the material response to a given deformation history for a set of complex constitutive material models.

The mesh manager, which is responsible for the construction of the elements, connectivity tables and for the attribution of the properties to the elements.

The mechanics library, which manages the space and time integration of the finite element discretization.

The partitioning library, which is responsible for distributing the mesh to the processors of a parallelized application.
In Adlib, different types of applications are realized by providing a specific main program that calls the required library functionality. Problemspecific frontend routines, e.g. for boundary conditions, are linked to enable detailed customization. Although more classical than the concepts discussed in Sect. 2.5, we achieve effective code reuse in the same application domain. For instance, the extensive VTF material library is shared among Adlib and the VTF thinshell finite element solver with fracture capability (see [50] for a computational example).
For parallel fluid–structure interaction simulations, all the above libraries are linked to the coupled code. A solid update step begins as usual by applying boundary conditions (here tractions produced by the hydrodynamic pressure), and proceeds by computing the predictor configuration of the solid and performing the necessary constitutive updates at each element quadrature point as part of the assembly of the global residual vector. At this point, a pointtopoint communication operation is necessary to exchange the incomplete residuals at partition boundary nodes. After this step, the corrected nodal accelerations and velocities can be obtained locally by applying the unmodified corrector step. As the communication operations involve only partition boundary data, and retain a surface to volume character, the solid update algorithm shows excellent scalability properties.
4 Closestpointtransform algorithm
In Sect. 2, we have introduced the concept of a signed distance function as a natural way to represent a complex embedded boundary on a Cartesian mesh. While signed distance functions are easily prescribed for single elementary geometric objects, their evaluation can be extremely cumbersome for complex shapes. In coupled Eulerian–Lagrangian simulations, this complex shape is defined by the boundary of the solid mesh. Since the solid mesh is tetrahedral (cf. Sect. 3), the interface is a triangle mesh. In the following, we outline the specific algorithm that we have developed to effectively transform the explicit description of a triangulated surface mesh into a signed distance function. The problem is equivalent to finding for every discrete point on the Cartesian SAMR grid the nearest or closest point on this surface mesh. The algorithm is therefore named the closest point transform (CPT). Without loss of generality, we assume a single uniform Cartesian grid in the following discussion.
4.1 Problem description
Let, \(\phi(\mathbf{x}), \mathbf{x} \in \mathbb{R}^n,\) be the distance from the point \(\mathbf{x}\) to a manifold \(\mathcal{I}.\) If \(\dim(\mathcal{I}) = n  1\) and the manifold is closed, (for example, curves in 2D or surfaces in 3D), then the distance may be signed. The orientation of the manifold determines the sign of the distance. We adopt the convention that the outward normal points in the positive direction. In order for the distance to be well defined, the manifold must be orientable and have a consistent orientation. A Klein bottle in 3D, for example, is not orientable. Two concentric circles in 2D have consistent orientations only if the normals of the inner circle point “inward” and the normals of the outer circle point “outward”, or vice versa. Otherwise, the distance would be illdefined in the region between the circles. For manifolds which are not closed, the signed distance is illdefined in any neighborhood of the boundary. However, the distance is welldefined in neighborhoods of the manifold which do not contain the boundary.
In computing the distance and closest point to a triangle mesh surface, one can consider each component of the mesh (face, edge or vertex) separately. For each entity, there are simple geometric formulas for computing the distance and closest point. For signed distance, one needs to use the surface normals to determine the sign of the distance. The surface normal along an edge is in the direction of the average of the incident face normals. The surface normal at a vertex is a weighted average of the incident face normals. The weighting is proportional to the angle in the face at that vertex.
For the fluid–solid coupling, we only need to determine the distance and closest point information in a narrow band around the interface as the information is only utilized in the ghost cells. Let δ be the Cartesian distance such that all ghost cells are within that distance of the interface surface. If the distance is computed up to δ, then one can flood fill the distance to determine which of the remaining grid points are inside or outside the solid. ^{1}
4.2 Iteration and tree data structures
The algorithm has computational complexity \(\mathcal{O}( M G ),\) where M is the number of components in the mesh and G is the number of grid points. Since timeexplicit finite volume methods basically have complexity \(\mathcal{O}( G )\) for a single time step, this naive algorithm is not suitable for computing the CPT during the course of a simulation.
An alternative could be to store the mesh in a data structure that supports minimum distance queries, like a bounding box tree [60]. However, the average expected complexity of a single distance or closest point computation in this approach would still be \(\mathcal{O}(\log\,M).\) This implies an overall complexity of \(\mathcal{O}( G \log\,M )\) for the CPT algorithm. By taking advantage of the fact that the grid points of the Cartesian mesh form a lattice, we have been able to develop a CPT algorithm tailored especially for our purposes with better computational complexity.
4.3 The CSC algorithm
One can efficiently compute the distance and closest point on a grid by solving the eikonal equation with the method of characteristics and utilizing polyhedron scan conversion. This is called the characteristics/scan conversion (CSC) algorithm [61]. For a given grid point, the closest point on the triangle mesh lies on one of the primitives (faces, edges and vertices) that comprise the surface. The characteristics emanating from each of these primitives form polyhedral shapes, which we call characteristic polyhedra. A characteristic polyhedron contains all of the points which are possibly closest to its corresponding face, edge or vertex. We determine the grid points that lie within each of these polyhedra, then use simple geometric formulas to calculate distance and closest points for the primitive.
4.4 Computational complexity
Consider computing the CPT up to a distance of δ. If δ is small and the surface is smooth, the computational complexity of the algorithm is linear in both the size of the mesh and the number of grid points within δ of the surface. Thus, it has the optimal complexity.
Let the Cartesian grid have N points within a distance δ of the surface, and let ν be the ratio of the sum of the volumes of all the scan converted polyhedra divided by the volume of the domain within a distance δ of the surface. The ratio ν depends on the shape of the surface and the distance δ. If the surface is jagged and δ is relatively large, then ν will be large. If the surface is smooth and δ is relatively small, then ν will be close to unity. The total computational complexity of the algorithm is \(\mathcal{O}(\nu N + M).\) The \(\mathcal{O}(\nu N)\) term again comes from scan conversion and the closest point and distance computations for the grid points. The \(\mathcal{O}(M)\) term represents the construction of the characteristic polyhedra. Since we expect both M and N to be small compared to the total number of grid points G, the CSC algorithm is suitable for computing the CPT during the course of a simulation.
The CSC algorithm stores the grids for distance and closest point and the mesh for which the CPT is computed. Beyond these data structures, which define the problem, it does not require significant additional storage. The components of the mesh (i.e. the faces, edges and vertices) are dealt with one at a time. The memory required to scan convert a single polyhedron is insignificant compared to the memory needs of the grid and the mesh. Thus, the CSC algorithm essentially has the minimum storage requirements for the CPT problem.
4.5 Concurrency and SAMR
If the solid mesh (and hence the solid boundary) is distributed over multiple processes, the pieces must be assembled into a cohesive triangle mesh before computing the CPT. This is because one needs to know the incident faces of edges and vertices in order to compute the correct sign of the distance. We accomplish this by maintaining global identifiers for the nodes in the solid mesh.
In the course of a simulation with the SAMR framework sketched in Sect. 2.4, each fluid process performs the sequential CSC algorithm. As we use a rigorous domain decomposition to partition the SAMR hierarchy, only those components of the triangulated surface meshes that are within a distance δ of the local domain need to be considered for this computation. The necessary clipping operation is best performed before sending the distributed parts of the solid surface mesh to the receiving fluid processes. The details of our implementation are outlined in Sect. 5.2.
In order to make efficient use of the CPT algorithm within the SAMR method, we have organized our CPT implementation such that the algorithm can be called once for a multitude of subgrids that effectively lie within the same lattice. This arrangement ensures that each characteristic polyhedron is constructed and scan converted only once for each level in the SAMR hierarchy and guarantees computational performance that is basically independent of the number of subgrids.
5 Fluid–structure coupling
We have implemented this algorithm with an adhoc partitioning into dedicated fluid and solid processes that communicate to exchange the data along \(\mathcal{I}.\) In the following subsections we will outline some of the specifics of our approach that make the VTF a highly efficient framework for fluid–structure simulation on distributed memory machines.
5.1 Coupled simulations with Eulerian SAMR
Unsteady compressible fluid flows typically show a wide range of temporal and spatial scales. While the correct numerical representation of supersonic shock and detonation waves usually requires very fine resolution only in a small band around the phenomenon of interest, a considerably coarser resolution is often sufficient in the majority of the fluid domain. This is in particular true in our case of strong pressure waves arising from the detonation of highly energetic materials; one is interested mainly in the stress waves produced by shock impact in the solid target materials and the resulting material response. Hence, incoming fluid waves and the nearbody fluid–structure interaction have to be captured with high accuracy, but resolution can be reduced for outgoing fluid phenomena and in the far field. We achieve the required solution adaptation in the fluid by applying the dynamic mesh adaptation algorithm described in Sect. 2.3. The fluid–solid interface \(\mathcal{I}\) is treated herein as a discontinuity with apriori refinement at least up to the coupling level l _{ c }. As the wave phenomena in solid materials are usually at least as fast as the waves in compressible fluids, the coupling level l _{ c } will usually be the highest computationally permissible choice in order to ensure an accurate wave transmission. But special care is required to initiate the data exchange according to the above basic coupling method in a way that is compatible with the recursive SAMR algorithm.
5.2 Efficient intersolver communication
Critical to the performance of the coupled algorithms are the intersolver communication routines send_interface_data() and receive_interface_data(). In order to ensure good communication performance, we have implemented InterSolverCommunication as a dedicated asynchronous communication module that sets up detailed pointtopoint communication patterns between the fluid and solid processes and avoids assembling global data structures.
The domain decomposition of the solid mesh across the solid processes also partitions the triangle surface mesh (cf. Sect. 3.4). A simple approach to coupling is to assemble the global boundary of the solid (i.e. gather the pieces from each process and merge them into a single mesh) and broadcast it to all fluid processes. Each fluid process then uses the relevant portion of the global boundary for the CPT and supplies pressure information for a portion of the boundary. Next, the pressure information is merged and broadcast to each solid process. Unfortunately, this simple strategy is not efficient for large solid meshes owing to the costs of assembling, storing, and communicating the global boundary and global pressure data structures.
A better approach is to use pointtopoint communications between the solid and fluid processes. Each solid process sends its portion of the boundary only to those fluid processes which require it on order to compute the CPT. Consider a single fluid process whose grids lie in a domain Ω and suppose that the CPT will be computed to a distance of δ. Then the fluid process needs only those portions of the interface that are within a distance δ of Ω. If the fluid process receives only the relevant portions of the interface, it can assemble them into a local triangle mesh that is sufficient for computing the CPT and setting the boundary conditions in the ghost cells.
We determine the pointtopoint communication pattern with bounding box information. Each solid process constructs a Cartesian bounding box around its portion of the interface. Likewise, each fluid process constructs a Cartesian bounding box around its domain and enlarges it by δ. The solid bounding boxes define which portion of the interface the solid process has; the fluid bounding boxes define which portion of the interface the fluid process needs. These bounding boxes are gathered to root fluid and solid processes. The root processes then exchange their sets of bounding boxes and broadcast the set to either the fluid or solid processes. Now each fluid process has all of the solid bounding boxes and vice versa. Each process intersects its own bounding box with the received set of bounding boxes to set up communication data structures that consider only those portions of the surface mesh, and the data defined on it, that are relevant to the local process. The communication between solid and fluid is nonblocking to enable overlapping communication and computation.
This strategy is much more efficient than assembling the global boundary. It is far less costly to gather and broadcast bounding boxes than to gather, assemble, and broadcast the boundary itself. Intersecting the bounding boxes to determine the communication pattern is also inexpensive. Note that this pattern is computed anew for each intersolver communication and will in general change as the simulation progresses.
It is worth mentioning that the efficiency of the above pointtopoint communication scheme necessarily relies on the fact that the fluid and solid meshes by themselves are reasonably partitioned. One could easily construct pathological cases where each is partitioned into long, thin pieces and each fluid process needs to communicate with each solid process. However, with the localitypreserving partitioning strategies employed in AMROC (generalized spacefilling curve) and the parallel solid solver (graphpartitioning provided by Metis), this never occurs in practice.
6 Examples of application
In this section, we present two examples of application of the computational framework described above. The first example corresponds to the simulation of the effects of a blast wave on the human body. The second example is a simulation of a detonation wave confined in a detonation tube and impacting a tantalum target. These applications illustrate the robustness and versatility of the coupled computational approach as well as the computational performance on parallel machines of moderate size.
6.1 Shock wave impact on softtissue body
Material properties of soft tissues
 Liver  Torso 

Young modulus  5 MPa  100 MPa 
Poisson coefficient  0.3  0.3 
Density  1,500 kg/m^{3}  950 kg/m^{3} 
The solid mesh used for this simulation has 19,562 nodes, while the SAMR mesh in the fluid has 100 × 100 × 100 cells at the root level and employs two additional levels, both refined by a factor of 2 (cf. Fig. 13). The fluid domain is 5 m × 5 m × 5 m. Scaled gradients of pressure and density are used as refinement criteria to resolve the incoming pressure wave accurately. The coupling level is set to l _{ c } = 1, and we use K = 20 subiterations in the solid solver. The distance δ within which the exact signed distance information around the interface mesh is evaluated by the CPT algorithm is set to three times the diagonal of a finite volume cell.^{3} This setting is sufficient to allow the construction of two internal ghost cells according to Sect. 2.2. With target CFL conditions of 0.3 in the fluid and 1.0 in the solid, we calculate 420 fluid root level steps to reach a final time of 6 ms, which involves 16,800 update steps in the solid solver. The simulation is completed in about 10 h real time on ten dualprocessor 2 GHz G5 nodes of a Linux Beowulf cluster connected with Myrinet. In this computation, 14 and 6 processes were used for the fluid and solid solvers, respectively. The size of the fluid SAMR mesh increases during the simulation from approximately 1,045,000 cells initially to about 2,026,000 cells, when the pressure wave impacts the torso.
Finally, we discuss briefly the effects of the blast wave on the body. One of the main mechanisms of internal injury due to blast is related to the impedance mismatch between air and fluidfilled organs in the human body. This significantly affects the propagation of stress waves transmitted by the blast and causes stress concentrations and localized deformations at high rates which are responsible for tissue failure and injury. The mitigating effect of the fluid–solid interaction which reduces the amount of impulse transmitted to the body is not well understood, especially when strong compressibility effects are important, as is the case in air blasts. Figures 15, 16 and 17 show different snapshots of the transmitted stress waves propagating in the torso and their interactions with the liver as measured by the elastic strain energy density. Despite the significant idealizations of this simulation, it is clear that this approach provides a viable strategy for exploring material systems for blastinjury mitigation.
6.2 HMX detonation in a tantalum cylinder
The second and final simulation we want to discuss is the propagation of a detonation wave in a highenergy explosive material confined in a thickwalled solid cylinder closed at one end. The properties of the explosive correspond to HMX (C_{4}H_{8}N_{8}O_{8}) and those of the cylinder material to tantalum.
Material parameters corresponding to the plastic response of tantalum (SI units)
\({\epsilon}^p_0\)  5 × 10^{−4} 
\(\dot{\epsilon}^p_0\)  3 × 10^{−3} 
m  12.5 
n  5 
σ_{ y }  5 × 10^{8} 
According to Mader [70], unreacted HMX has a density of ρ_{0} = 1,900 kg/m^{3}, and we assume atmospheric pressure p _{0} = 100 kPa in the unreacted material. The detonation velocity for a freely propagating Chapman–Jouguet detonation (cf. [21]) in HMX is experimentally known to be approximately 9,100 m/s and the entire hydrodynamic flow can be described with reasonable accuracy with a constant adiabatic exponent of γ = 3 [70]. The rate factor T _{ R } is unknown; we therefore set it to T _{ R } = 1 μs, which is a reasonable value for most solid explosives [21].
The above values specify the process of steady onedimensional detonation propagation completely. A detonation wave consists of a leading hydrodynamic shock wave followed by a region of decaying continuous combustion toward chemical equilibrium. The simplified Chapman–Jouguet theory can be used to evaluate the energy release of our configuration to be q = 5,176 kJ/kg and to predict the hydrodynamic values in the equilibrium state p _{CJ}≈ 39.3 GPa and ρ_{CJ}≈ 2,533 kg/m^{3}. The steady internal structure can be calculated with the theory of Zeldovich, Neumann, and Döring (ZND), which constructs an analytic solution of Eqs. 1–6. Detailed derivations of the ZND solution can be found in the book by Fickett and Davis [21] or, for instance, in [32]. According to the ZND solution, the peak values at the head of the detonation are p _{vN}≈ 78.7 GPa and ρ_{vN}≈ 3,800 kg/m^{3}. We use the analytic ZND solution as our hydrodynamic initial conditions. Figure 18 displays the initial pressure distribution and its steady propagation in a onedimensional simulation on a uniform mesh with 960 finite volume cells. At considerably coarser resolutions, the reaction front is not resolved with sufficient accuracy, resulting in an incorrect speed of propagation and a significant underestimation of the peak value p _{vN}. However, this high resolution is necessary only inside the reaction zone, which makes the application of very effective dynamic mesh adaptation possible.
Throughout the simulation, the SAMR mesh increases from an initial size of approximately 706 k cells on level 1 and 6.5 M on level 2 to about 930 k and 10.0 M, respectively. The number of grids on both levels varies between 400 and 1,000. Compared with a uniform fluid mesh of 480 × 480 × 960≃ 221 M cells, that would otherwise be necessary to capture the detonation with similar accuracy, mesh adaptation clearly provides enormous savings. Figure 21 shows the highly localized fluid mesh refinement in the midplane for the second snapshot shown in Fig. 19.
The simulation ran on four nodes of a Pentium4 2.4 GHz dualprocessor system connected with Quadrics interconnect for about 63 h real time. Six processes were dedicated to the adaptive fluid simulation, while two were used for the smaller solid problem. The signed distance calculation with the CPT algorithm takes only 0.8% of the computational costs on the fluid nodes, which impressively confirms the practical applicability of the idea of implicit geometry representation for evolving surface meshes of moderate size.
Snapshots of the simulation displaying a cut through the hydrodynamic pressure distribution and the normal stress in the axial direction are shown in the Figs. 19 and 20. The graphics show several salient features of this coupled problem: the superseismic loading of the lowerimpedance cylinder walls leading to an inclined shock front in the solid (Fig. 19), the ensuing large deformations of the cylinder wall (Fig. 20 and lower graphic of Fig. 19), the reflection of the shocks in the solid at the constrained outer cylinder wall (Fig. 20), and the transmission of a highintensity shock into the solid target (lower graphic of Fig. 20). In the last shown time step, the HMX is fully depleted and a nonreactive, purely hyrdodynamic, shock wave, caused by the reflection of the detonation wave at the target, can be seen to propagate upstream in the fluid.
7 Conclusions
A loosely coupled fluid–structure interaction method for the timeaccurate simulation of solid materials responding dynamically to strong shock and detonation waves arising from the detonation of highenergetic materials has been presented. The approach utilizes a Lagrangian finite element solver for large deformations and a Cartesian dynamically adaptive finite volume solver with the capability to deal with moving embedded boundaries via a ghost fluid approach. Both solvers have been parallelized for distributed memory machines utilizing domain decomposition, and an effective intersolver communication module has been outlined. An algorithm has been presented that transforms the triangular solid surface mesh very efficiently into a signed distance function on the Cartesian grid. The application of the methodology to two distinct fluid–structure interaction problems has also been described. We note that the combined approach can lead to high efficiencies in the solution of coupled problems. Particular, our second computational example, in which a detonation wave in a highenergetic material impinges on a dynamically deforming tantalum cylinder, demonstrates the enormous savings in computational costs that can be obtained through structured dynamic mesh adaptation in the fluid for the considered problem class. This calculation required only 504 h CPU, whereas a simulation with an equivalent fluid unigrid mesh can be expected to be in the range of >10,000 h CPU.
The overall coupling method has been realized as a natural extension of the objectoriented C++ framework on which the adaptive finite volume fluid solver with embedded boundary capability has been built. We have discussed its design in detail and highlighted particular the implementation efficiency that we have gained by utilizing framework concepts. Further, all newly developed components are now generically available within the fluid framework, which already has enabled new, industrystrength, noncoupled fluid applications in which very complicate surface meshes derived from CAD drawings have been used as complex embedded boundaries.
Future development efforts will focus on the implementation of dynamic adaptation and mesh smoothing techniques for the solid solver, as well as investigations of the integration of timeimplicit methods for both fluid and solid into the VTF.
Footnotes
References
 1.Aivazis M, Goddard W, Meiron D, Ortiz M, Pool J, Shepherd J (2000) A virtual test facility for simulating the dynamic response of materials. Comput Sci Eng 2(2):42–53CrossRefGoogle Scholar
 2.Cummings J, Aivazis M, Samtaney R, Radovitzky R, Mauch S, Meiron D (2002) A virtual test facility for the simulation of dynamic response in materials. J Supercomput 23:39–50zbMATHCrossRefGoogle Scholar
 3.Mauch S, Meiron D, Radovitzky R, Samtaney R (2003) Coupled Eulerian–Lagrangian simulations using a level set method. In: Bathe K (ed) 2nd M.I.T. conference on computational fluid and solid mechanics, Cambridge, MA, June 17–20Google Scholar
 4.Arienti M, Hung P, Morano E, Shepherd J (2003) A level set approach to Eulerian–Lagrangian coupling. J Comput Phys 185:213–251zbMATHCrossRefGoogle Scholar
 5.Cirak F, Radovitzky R (2003) A general algorithm for coupling Lagrangianshell with Eulerianfluid formulations. In: Proceedings of the IUTAM symposium on integrated modeling of fully coupled fluid–structure interactions using analysis, computations and experiments, New Brunswick, NJ, June 1–6, 2003. International Union of Theoretical and Applied MechanicsGoogle Scholar
 6.Löhner R, Baum J, Charman C, Pelessone D (2003) Fluidstructure interaction simulations using parallel computers. In: Lecture notes in computer science, vol 2565. Springer, Berlin Heidelberg New York, pp 3–23Google Scholar
 7.Cirak F, Radovitzky R (2005) A Lagrangian–Eulerian shellfluid coupling algorithm based on level sets. Comput Struct 83:491–498CrossRefGoogle Scholar
 8.Löhner R, Cebral J, Yang C, Baum J, Mestreau E, Charman C, Pelessone D, (2004) Largescale fluidstructure interaction simulations. Comput Sci Eng 6(3):27–37CrossRefGoogle Scholar
 9.Hu H, Patankar N, Zhu M (2001) Direct numerical simulations of fluidsolid systems using the arbitrary Lagrangian–Eulerian technique. J Comput Phys 169(2):427–462zbMATHCrossRefMathSciNetGoogle Scholar
 10.Mittal R, Iaccarino G (2005) Immersed boundary methods. Annu Rev Fluid Mech 37:239–261CrossRefMathSciNetGoogle Scholar
 11.Quirk J (1994) An alternative to unstructured grids for computing gas dynamics flows around arbitrarily complex twodimensional bodies. Comput Fluids 23:125–142zbMATHCrossRefGoogle Scholar
 12.Berger M, Helzel C (2002) Grid aligned Hbox methods for conservation laws in complex geometries. In: Proceedings of the 3rd international symposium on finite volumes for complex applications, PorquerollesGoogle Scholar
 13.Fedkiw R (2002) Coupling an Eulerian fluid calculation to a Lagrangian solid calculation with the ghost fluid method. J Comput Phys 175:200–224zbMATHCrossRefGoogle Scholar
 14.Tam D, Radovitzky R, Samtaney R (2005) An algorithm for modelling the interaction of a flexible rod with a twodimensional highspeed flow. Int J Numer Methods Eng 64(8):1057–1077zbMATHCrossRefGoogle Scholar
 15.Pantano C, Deiterding R, Hill D, Pullin D (2005) A lownumerical dissipation patchbased adaptive mesh refinement method for largeeddy simulation of compressible flows. In: Proceedings of the Cyprus international symposium on complex effects in large eddy simulations, University of Cyprus, NicosiaGoogle Scholar
 16.Deiterding R (2005a) Detonation structure simulation with AMROC. In: Lecture notes in computer science, vol 3726. Springer, Berlin Heidelberg New York, pp 916–927Google Scholar
 17.Deiterding R, Bader G (2005) Highresolution simulation of detonations detailed chemistry. In: Warnecke G (ed) Analysis and numerics of conservation laws. Springer, Berlin Heidelberg New York, pp 69–91CrossRefGoogle Scholar
 18.Deiterding R (2005b) A highresolution method for realistic detonation structure simulation. In: Asakura F et al (eds) Proceegins of the 10th international conference on hyperbolic problems: theory, numerics, applications, Yokohama PublishersGoogle Scholar
 19.Deiterding R (2003) Parallel adaptive simulation of multidimensional detonation structures. PhD Thesis, Brandenburgische Technische Universität CottbusGoogle Scholar
 20.Deiterding R (2002) Efficient simulation of multidimensional detonation phenomena. In: Handlikova A et al (eds) Proceedings of the ALGORITMY 2002, 16th conference on scientific computing. Slovak University of Technology, Bratislava, pp 94–101Google Scholar
 21.Fickett W, Davis W (1979) Detonation. University of California Press, Berkeley, Los AngelesGoogle Scholar
 22.Clarke J, Karni S, Quirk J, Roe P, Simmonds L, Toro E (1993) Numerical computation of twodimensional unsteady detonation waves in high energy solids. J Comput Phys 106:215–233zbMATHGoogle Scholar
 23.Toro E (1999) Riemann solvers and numerical methods for fluid dynamics. Springer, Berlin Heidelberg New YorkzbMATHGoogle Scholar
 24.Einfeldt B, Munz C, Roe P, Sjögreen B (1991) On Godunovtype methods near low densities. J Comput Phys 92:273–295zbMATHCrossRefMathSciNetGoogle Scholar
 25.Fedkiw R, Aslam T, Merriman B, Osher S (1999) A nonoscillatory Eulerian approach to interfaces in multimaterial flows (the ghost fluid method). J Comput Phys 152:457–492zbMATHCrossRefMathSciNetGoogle Scholar
 26.Osher S, Fedkiw R (2003) Level set methods and dynamic implicit surfaces. Springer, Berlin Heidelberg New YorkzbMATHGoogle Scholar
 27.Berger M, Colella P (1988) Local adaptive mesh refinement for shock hydrodynamics. J Comput Phys 82:64–84CrossRefGoogle Scholar
 28.Bell J, Berger M, Saltzman J, Welcome M (1994) Threedimensional adaptive mesh refinement for hyperbolic conservation laws. SIAM J Sci Comp 15(1):127–138zbMATHCrossRefMathSciNetGoogle Scholar
 29.Rendleman C, Beckner V, Lijewski M, Crutchfield W, Bell J (2000) Parallelization of structured, hierarchical adaptive mesh refinement algorithms. Comput Vis Sci 3:147–157zbMATHCrossRefGoogle Scholar
 30.Parashar M, Browne J (1997) System engineering for high performance computing software: the HDDA/DAGH infrastructure for implementation of parallel structured adaptive mesh refinement. In: Structured adaptive mesh refinement grid methods, IMA volumes in mathematics and its applications. Springer, Berlin Heidelberg New YorkGoogle Scholar
 31.Deiterding R (2005c) AMROC—blockstructured adaptive mesh refinement in objectoriented C++. http://www.amroc.sourceforge.net
 32.Deiterding R (2005d) Construction and application of an AMR algorithm for distributed memory computers. In: Plewa T et al (eds) Lecture notes in computational science and engineering, vol 41. Springer, Berlin Heidelberg New York, pp 361–372Google Scholar
 33.Parashar M, Browne J (1996) On partitioning dynamic adaptive grid hierarchies. In: Proceedings of the 29th annual Hawaii international conference on system sciencesGoogle Scholar
 34.MacNeice P, Olson K, Mobarry C, deFainchtein R, Packer C (2000) PARAMESH: a parallel adaptive mesh refinement community toolkit. Comput Phys Commun 126:330–354zbMATHCrossRefGoogle Scholar
 35.Booch G, Rumbaugh J, Jacobsen I (1999) The unified modeling language user guide. AddisonWesley, ReadingGoogle Scholar
 36.Marsden J, Hughes T (1993) Mathematical foundations of elasticity. Dover Publications, New YorkGoogle Scholar
 37.Lew A, Radovitzky R, Ortiz M (2002) An artificialviscosity method for the Lagrangian analysis of shocks in solids with strength on unstructured, arbitraryorder tetrahedral meshes. J ComputAided Mater Des 8:213–231CrossRefGoogle Scholar
 38.Von Neumann J, Richtmyer R (1950) A method for the numerical calculation of hydrodynamic shocks. J Appl Phys 21:232–243zbMATHCrossRefMathSciNetGoogle Scholar
 39.Lubliner J (1972) On the thermodynamic foundations of nonlinear solid mechanics. Int J NonLinear Mech 7:237–254CrossRefGoogle Scholar
 40.Radovitzky R, Ortiz M (1999) Error estimation and adaptive meshing in strongly nonlinear dynamic problems. Comput Methods Appl Mech Eng 172:203–240zbMATHCrossRefMathSciNetGoogle Scholar
 41.Ortiz M, Stainier L (1999) The variational formulation of viscoplastic constitutive updates. Comput Methods Appl Mech Eng 171:419–444zbMATHCrossRefMathSciNetGoogle Scholar
 42.Cuitino A, Ortiz M (1992) Materialindependent method for extending stress update algorithms from smallstrain plasticity to finite plasticity with multiplicative kinematics. Eng Comput 9(4):437–451Google Scholar
 43.Stainier L, Cuitino A, Ortiz M (2002) A micromechanical model of hardening, rate sensitivity and thermal softening in BCC single crystals. J Mech Phys Solids 50:1511–1545zbMATHCrossRefGoogle Scholar
 44.Kambouchev N, Fernandez J, Radovitzky R (2005) Polyconvex model for materials with cubic anisotropy. http://www.arxiv.org/abs/condmat/0505178
 45.Benson D (1992) Computational methods in Lagrangian and Eulerian hydrocodes. Comput Methods Appl Mech Eng 99:235–394CrossRefMathSciNetGoogle Scholar
 46.Ciarlet P (1976) Numerical analysis of the finite element method. Les Presses de L’Universite de Montreal, QuebeczbMATHGoogle Scholar
 47.Belytschko T (1983) An overview of semidiscretization and time integration procedures. In: Belytschko T, Hughes T (eds) Computational methods for transient analysis. NorthHolland, Amsterdam, pp 1–65Google Scholar
 48.Hughes T (1983) Analysis of transient algorithms with particular reference to stability behavior. In: Belytschko T, Hughes T (eds) Computational methods for transient analysis. NorthHolland, Amsterdam, pp 67–155Google Scholar
 49.Kane C, Marsden J, Ortiz M, West M (2000) Variational integrators and the Newmark algorithm for conservative and dissipative mechanical systems. Int J Numer Methods Eng 49:1295–1325zbMATHCrossRefMathSciNetGoogle Scholar
 50.Deiterding R, Cirak F, Mauch S, Meiron D (2006) A virtual test facility for simulating detonationinduced fracture of thin flexible shells. In: Alexandrov V et al (eds) Lecture notes in computer science, vol 3992. Springer, Berlin Heidelberg New York, pp 122–130Google Scholar
 51.Karypis G (2001) METIS—family of multilevel partitioning algorithms. http://www.users.cs.umn.edu/∼karypis/metis
 52.Liu A, Joe B (1996) Quality local refinement of tetrahedral meshes based on 8subtetrahedron subdivision. Math Comput 65:1183–1200zbMATHCrossRefMathSciNetGoogle Scholar
 53.Stainier L, Radovitzky R, Ortiz M (1999) Constitutive modeling of Tantalum crystals and polycrystals. In: Croitoro E (ed) Proceedings of the 1st Canadian conference on nonlinear solid mechanics, Victoria, BC, pp 203–213Google Scholar
 54.Radovitzky R, Cuitino A (2003) Direct numerical simulation of polycrystals. In: Proceedings of the 44th AIAA/ASME/ASCE/AHS structures, structural dynamics, and materials conference, Norfolk, AIAAGoogle Scholar
 55.Zhao Z, Radovitzky R, Kuchnicki S, Cuitino A (2003) Dynamical evolution of texture in polycrystalline metals under high strainrate conditions. In: Proceedings of the international symposium on plasticity and its current applications, Quebec CityGoogle Scholar
 56.Cuitino A, Radovitzky R (eds) (2004) Multiscale material modeling and simulation. In: Sandia NM Proceedings of the symposia on multiscale material modeling and simulation, Cambridge, MA. Modelling Simul Mater Sci Eng, vol 12, issue 4, Institute of Physics PublishingGoogle Scholar
 57.Zhao Z, Radovitzky R, Cuitino A (2004) A study of surface roughening in fcc metals using direct numerical simulation. Acta Mater 52:5791–5804CrossRefGoogle Scholar
 58.Kuchnicki S, Cuitino A, Radovitzky R (2005) Efficient and robust constitutive integrators for singlecrystal plasticity modeling. Int J Plast 22:1988–2011CrossRefGoogle Scholar
 59.Sethian J (1999) Level set methods and fast marching methods. Cambridge University Press, CambridgezbMATHGoogle Scholar
 60.Johnson D, Cohen E (1998) In: Proceedings of the IEEE international conference on robotics and automation, Leuven, pp 3678–3684Google Scholar
 61.Mauch S (2003) Efficient algorithms for solving static Hamilton–Jacobi equations. PhD Thesis, California Institute of TechnologyGoogle Scholar
 62.Foley J, van Dam A, Feiner S, Hughes J (1996) Computer graphics: principles and practice. AddisonWesley, ReadingzbMATHGoogle Scholar
 63.Watt A (1993) 3D computer graphics. AddisonWesley, ReadingGoogle Scholar
 64.Park K, Felippa C, Deruntz J (1997) Stabilization of staggered solution procedures for fluidstructure interaction analysis. In: Belytschko T, Geers T (eds) Computational methods for fluid–structure interaction problems, New York, pp 94–124Google Scholar
 65.Zhang Q, Hisada T (2004) Studies of the strong coupling and weak coupling methods in FSI analysis. Int J Numer Methods Eng 60:2013–2029zbMATHCrossRefGoogle Scholar
 66.Taylor G (1950) The formation of a blast wave by a very intense explosion, I theorical discussion. Proc R Soc Ser A 201(1065):159–174Google Scholar
 67.Brode H (1955) Numerical solution of spherical blast waves. J Appl Phys 26(6):766–775CrossRefMathSciNetGoogle Scholar
 68.Cohen R, Gülseren O (2001) Thermal equation of state of tantalum. Phys Rev B 63:3363CrossRefGoogle Scholar
 69.Steinberg D, Cochran S, Guinan M (1980) A constitutive model for metals applicale at highstrain rate. J Appl Phys 51:1498–1504CrossRefGoogle Scholar
 70.Mader C (1979) Numerical modeling of detonations. University of California Press, Berkeley, Los AngeleszbMATHGoogle Scholar