An implicit parallel UGKS solver for flows covering various regimes
 83 Downloads
Abstract
This paper presents an engineeringoriented UGKS solver package developed in China Aerodynamics Research and Development Center (CARDC). The solver is programmed in Fortran language and uses structured bodyfitted mesh, aiming for predicting aerodynamic and aerothermodynamics characteristics in flows covering various regimes on complex threedimensional configurations. The conservative discrete ordinate method and implicit implementation are incorporated. Meanwhile, a local mesh refinement technique in the velocity space is developed. The parallel strategies include MPI and OpenMP. Test cases include a wedge, a cylinder, a 2D blunt cone, a sphere, and a X38like vehicle. Good agreements with experimental or DSMC results have been achieved.
Keywords
Unified gas kinetic scheme Conservative discrete ordinate method Implicit algorithm Mesh refinement MPI OpenMP Application1 Introduction
During the reentry process, vehicles may encounter different flow regimes such as free molecular, transitional, near continuum, and continuum regime. The determination of aerodynamic forces and heat loads has great impact on the design of vehicles [1]. In the noncontinuum regimes, traditional macroscopic methods, such as Euler, NavierStokes and Burnett equations, may become invalid. The following methods are mainly used for the nonequilibrium flow simulations. The first kind of method is based on probabilistic modeling. The most popular one is the direct simulation Monte Carlo (DSMC) method. DSMC was first proposed by Bird [2] more than half a century ago. It follows the evolution of representative particles with uncoupled transport and collision process. The DSMC has been fully validated for providing physical solutions through its comparison with the experiments measurements [3, 4]. It has played a key role in the design and flight analysis of vehicles in the rarefied environment. Some of the most cited DSMC codes in literature are DS2V/3 V [5], DAC [6], SMILE [7], MONACO [8], and DSMCFOAM [9]. The main differences among these codes are in the treatment of collision selection methods and mesh topology.
Another kind of approach is the deterministic method. Deterministic method mainly concerns the Boltzmann equation. Due to the complexity of the Boltzmann collision term, researchers usually choose the simplified collision model, such as BGK model [10], Shakhov model [11], Rykov model [12]. Titarev [13] has developed an implicit solver named Nesvetay3D on unstructured mesh. Threedimensional TVD method is applied for the numerical discretization. Both spatial and velocity mesh decomposition are used in the parallelization. A total number of 6.9 × 10^{9} mesh points in the sixdimensional space is used for the supersonic flow simulation around a reentry space vehicle. Wadsworth [14] has developed a parallel, finite volume 2D/axisymmetric code SMOKE which is based on conservative numerical schemes developed by Mieussens [15]. In Baranger’s team, a 3D code [16] has been used in the past years for rarefied flow simulations. This code can handle polyatomic gases. It uses block structured mesh and hybrid parallelization, i.e., space domain decomposition with MPI and inner parallelization with OpenMP. Furthermore, the code is equipped with velocity mesh refinement technique which improves the code in both CPU time saving and memory storage. Li’s team has developed a 3D code based on the model equation with the name gaskinetic unified algorithm (GKUA) [17, 18]. Threedimensional hypersonic flows around sphere and spacecraft with different Knudsen numbers and Mach numbers have been studied. The total sixdimensional mesh for a complex wingbody configuration reaches 7.3 × 10^{11} and 23,800 CPU cores [19] have been used in the computation.
However, the above deterministic methods share a common feature. They decouple the particle transport and collision. Therefore, the cell size and time step in these numerical schemes are limited by the particle mean free path and mean collision time in order to provide accurate numerical solutions. When the flow regime is close to continuum or near continuum, the time step and cell size limitations are rather severe and make these methods extremely timeconsuming and inefficient.
Another distinguishable deterministic method, which is named unified gas kinetic scheme (UGKS), was proposed by Xu et al. [20, 21, 22]. UGKS is a multiscale method with coupled particle transport and collision in its numerical flux modeling. It is based on an integral solution of the gaskinetic model equation. It can recover the flow physics from the kinetic particle transport and collision to the hydrodynamic wave propagation. Moreover, the time step is determined only by the CFL condition, which is not limited by the mean collision time. So the scheme becomes more efficient in various flow regimes, especially when the local Knudsen number is low. Applying UGKS to analyze aerodynamic and aerothermodynamics on flying vehicles in near space flight is our long term objective.
This paper is organized in the following. Section 2 is about the introduction of UGKS and some techniques to accelerate convergence. Section 3 is a simple description of the framework. Section 4 is some 2D and 3D validation test cases. The last section is the conclusion.
2 Method
2.1 Unified gas kinetic scheme
\( \left(x,y,z\right)=\left(\overline{x},\overline{y},\overline{z}\right)/\overline{L} \), \( t=\overline{t}/\left(\overline{L}/{\overline{U}}_{\infty}\right) \), \( \left(u,v,w\right)=\left(\overline{u},\overline{v},\overline{w}\right)/{\overline{U}}_{\infty } \), \( \rho =\overline{\rho}/{\overline{\rho}}_{\infty } \)
\( p=\overline{p}/\left({\overline{\rho}}_{\infty }{\overline{U}}_{\infty}^2\right) \), \( \tau =\overline{\tau}/\left(\overline{L}/{\overline{U}}_{\infty}\right) \), \( \mu =\overline{\mu}/{\overline{\mu}}_{\infty } \), \( \lambda =\overline{\lambda}/\left(1/{\overline{U}}_{\infty}^2\right) \), \( f=\overline{f}/\left({\overline{\rho}}_{\infty }/{\overline{U}}_{\infty}^3\right) \)
\( {f}^{+}={\overline{f}}^{+}/\left({\overline{\rho}}_{\infty }/{\overline{U}}_{\infty}^3\right) \), \( {g}_M={\overline{g}}_M/\left({\overline{\rho}}_{\infty }/{\overline{U}}_{\infty}^3\right) \).
f^{+} can be given in the form, f^{+} = g_{M} + g^{+}
The flux across a cell interface is based on the integral solution of the model equation. Discontinuous spatial reconstruction with nonlinear limiter is used to introduce artificial dissipation for UGKS once the scheme becomes a shock capturing method when the dissipative flow structure cannot be well resolved by the cell size. Details can be found in [20]. In this paper, we use van Leer limiter in the reconstruction. Due to the discreteness of the velocity space, numerical quadrature should be used to calculate various integrals. In this paper, composite NewtonCote’s (N − C) quadrature is adopted.
The Rykov model [12] for diatomic gases is also implemented in our UGKS code package. The corresponding details are omitted.
2.2 Conservative discrete ordinate method [23]
Here Err is the numerical error introduced by the numerical quadrature. Err can be reduced by increasing the velocity space mesh in a certain extent but will finally stay in some level, which is determined by the intrinsic nature of numerical quadrature.
This numerical error results in a source term in the governing Eq. (5). The source term can be expressed in the form \( {\int}_{t^{\zeta}}^{t^{\zeta +1}}\left[\frac{1}{\tau}\mathbf{Err}\left(NC\right)\right] dt \)
From Eq. (7) and Eq. (8) we can see that SS is related to freestream condition and numerical quadrature.
The first five equations in (9) represent conservation of mass, momentum and energy during collision process. In discretised velocity space, the multiple integral is replaced by numerical quadratures. If the equilibrium distribution function remains in the form given in section 2.1, Eq. (9) no longer holds due to numerical error of quadratures. In other words, the conservation property will not be maintained.
Here the symbol ∑ indicates that numerical quadratures are used. With the discrete f^{+} determined by the above group of variables, the conservation property holds and the numerical source term Err goes to machine zero, which has been validated in numerical experiments.
The UGKS in Section 2.1 has a secondorder of accuracy. What we do in this section only changes the form of the heat flux modified equilibrium state. The spatial reconstruction and the evaluation of the numerical flux remain unchanged. Thus, CDOM does not affect the spatial accuracy and the coupling of particle transport and collision.
2.3 Implicit UGKS [25]
where Δt_{min} is the minimum time step in the whole field determined by the CFL condition.
In structured meshes, (Δf)_{l,m,n} can be obtained after backward and forward substitution and f^{ζ + 1} can be got subsequently.
In the above procedure, the gain term f^{+} in the collision term is treated explicitly. Since UGKS is a multiscale hybrid method with both macroscopic and microscopic variable updates. The macroscopic variables can be updated implicitly first to give a preevaluating f^{+}, resulting in a complete implicit implementation [26] for the collision term. This is very useful for continuum or near continuum flows.
2.4 Local refinement in the velocity mesh
Generic adaptive mesh refinement (AMR) [27, 28] in velocity can greatly decrease the CPU time and memory requirements for UGKS. However, the resulting velocity meshes are usually different for different spatial cells, making it rather difficult to apply the implicit technique.
In our UGKS solver package, we combine the merits of both methods through the following procedure. First, the bounds and interval of a global uniform velocity mesh are calculated according to numerical experiences or a preconducted NavierStokes simulation results. Obviously, the lower and upper limits of the velocity mesh in each direction are determined by the highest temperature which usually appears in the shock layer. While the mesh interval Δv is determined by the lowest temperature in the whole field. Second, a global uniform velocity mesh is generated which we call background mesh. The interval of this mesh is a • Δv where a is larger than one. Then we give a patch on the background velocity mesh for the spatial cells whose velocity mesh interval should be less than a • Δv. The location of the patch can be determined by the precalculated NavierStokes results or even by the UGKS results with the background velocity mesh. The resulting velocity mesh is still structured. The implicit method can be applied without any difficulties.
Up to now, the only difficulty arising may be the interpolation of distribution functions from the background mesh to the patch. We use the following conservative method. Take 1D case for example, the composite NewtonCote’s quadrature requires that the total number of velocity points is 4 N + 1, where N is a positive integer. We can get an interpolation polynomial from the five distribution functions which is equally spaced on a small block of four successive intervals on the velocity mesh. Since NewtonCote’s quadrature coefficients are derived from this polynomial, they are consistent. It can be easily proved that the conservations of mass, momentum and energy hold if we extend the original 5 points equally spaced mesh to a 9 points equally spaced mesh. For 2D or 3D cases, extending a block mesh of 5 × 5 or 5 × 5 × 5 to 9 × 9 or 9 × 9 × 9 can be done in the same way. Proof of the conservation law can be verificated through some mathematical software such as MAPLE.
In this case, if we use global uniform mesh, the total mesh will be 241 × 241. With the local refinement technique, the total mesh is 121 × 121 + 9 × (9 × 9  5 × 5) = 15,145 which is only 1/3.8 of the former.
2.5 Parallelization
At present, hybrid parallelization similar to that in [16] is used. The space mesh is decomposed and parallelized with MPI which has been broadly applied in many traditional CFD software. In every MPI process, several threads are used with OpenMP. However, due to the architecture change of our new super cluster, three space dimensions and one velocity dimension decomposition technique is under developing, allowing for a larger parallel scale up to 10,000 cores in the near future.
3 Code framework

2D and 3D bodyfitted structured multiblock mesh

Steady and unsteady simulations

Explicit and implicit methods

Conservative discrete ordinate method

Local refinement in velocity mesh

Shakhov model for monatomic gases

Rykov model for diatomic gases

Diffuse or specular reflection wall boundaries, freestream boundary, outflow boundary, symmetrical boundary

Several models for the viscosity calculation such as hard sphere model, variable hard sphere model [30] or the Sutherland model

Hybrid parallelization with MPI and OpenMP
4 Validation cases
Freestream conditions
Configuration  Mach  Kn  λ definition  Working gas  \( \overline{L} \)(m)  ω  T_{∞}(K)  T_{w}(K) 

Wedge  10  0.05  HS  Argon  0.2  0.81  200  300 
Cylinder  1.96  0.0162, 0.162  VHS  Nitrogen  radius  0.74  124.94  259.87 
Cylinder  5  0.01, 0.1, 1  VHS  Argon  radius  0.81  273  273 
Cylinder  10  3.03e3, 7.58e2  VHS  Argon  radius  0.81  200  500 
Cylinder  25  3.69e3, 1.84e2 9.22e2, 0.461  VHS  Argon  radius  0.734  200  1500 
Cone  8.1  9.75e3, 3.38e1  VHS  Argon  0.02  0.81  247, 189  273 
Sphere  4.25  0.031~0.672  VHS  Nitrogen  0.002  0.74  65  302 
Sphere  5.45  0.256~1.96  VHS  Nitrogen  0.002  0.74  43  315 
X38like  4  8.41e5~8.41e2  VHS  Argon  0.28  0.81  56  300 
X38like  6  1.26e4~1.26e1  VHS  Argon  0.28  0.81  56  300 
X38like  8  1.68e4~1.68e1  VHS  Argon  0.28  0.81  56  300 
4.1 Hypersonic flow over a 40^{0} wedge
4.2 Super and hypersonic flows over a 2D cylinder
This is a quite comprehensive test case covering supersonic and hypersonic flows in all regimes. We also use this case for validating the CDOM and implicit techniques described in section 2.
Comparison of the explicit and implicit methods in convergence rate
Ma  Kn  Nc.E  Nc.I  Rs = Nc.E/ Nc.I/1.02 

5  0.01  502,450  5800  84.93 
5  0.1  463,500  3500  129.83 
10  0.01  505,900  4645  106.78 
Comparisons of cylinder drag
M_{∞}  Kn_{∞}  UGKS  DS2V/MONACO  Relative error (%) 

1.96  0.0162  1.597  1.582  0.92 
1.96  0.162  1.862  1.863  −0.06 
5  0.01  1.320  1.316  0.31 
5  0.1  1.527  1.523  0.28 
5  1  1.929  1.917  0.62 
5.43  0.303  1.774  1.775  −0.05 
5.43  1.52  2.277  2.304  −1.19 
10  0.00303  1.258  1.252  0.56 
10  0.0758  1.500  1.496  0.27 
25  0.00369  1.256  NA/1.259  −0.21 
25  0.0184  1.348  1.349/1.347  0.07 
25  0.0922  1.531  1.521/1.528  0.22 
25  0.461  1.807  1.792/1.771  2.03 
4.3 Hypersonic flow over a 2D cone
4.4 Supersonic and hypersonic flows over a sphere
The flow past a sphere is simulated with Rykov model to compare with the experimental drag coefficients [33]. The space mesh contains 21,840 cells while a velocity mesh of 41 × 41 × 41 is used.
Comparisons of sphere drag
M_{∞}  Re  Kn_{∞}  UGKS (Nitrogen)  Exp (Air)  Relative error (%) 

4.25  9.55  0.672  2.356  2.42  2.64% 
4.25  19.0  0.338  2.101  2.12  0.87% 
4.25  53.0  0.121  1.694  1.69  −0.27% 
4.25  80.5  0.080  1.558  1.53  −1.80% 
4.25  150.0  0.043  1.410  1.37  −2.91% 
4.25  210.0  0.031  1.355  1.35  −0.39% 
5.45  4.2  1.960  2.595  2.60  0.18% 
5.45  8.6  0.957  2.449  2.44  −0.36% 
5.45  16.8  0.490  2.248  2.28  1.41% 
5.45  32.1  0.256  2.005  2.04  1.71% 
4.5 Supersonic and hypersonic flows over a X38like vehicle
The angle of attack is 20 degrees in this case. The space mesh contains 334,434 cells while a velocity mesh of 33 × 33 × 33 is used. The total sixdimensional mesh reaches 1.2 × 10^{10}. The reference area for the aerodynamic coefficient is 2.41 × 10^{− 2} m^{2}.
Comparisons of X38like coefficients with Mach number 8
Coefficients  Kn  UGKS  RariHV  Relative error(%) 

Lift  1.68e1  1.98E01  1.96E01  0.98 
1.68e2  1.94E01  1.92E01  1.15  
1.68e3  1.93E01  1.94E01  −0.69  
Drag  1.68e1  1.00E+ 00  9.79E01  2.27 
1.68e2  5.56E01  5.46E01  1.85  
1.68e3  2.90E01  2.95E01  −1.71 
5 Conclusions
Our UGKS solver package is introduced including the main numerical techniques for improving the efficiency and accuracy, such as implicit method and local mesh refinement technique in the velocity space. It is devised for simulating flow fields around complex configurations for all flow regimes.
Several validations are conducted by comprehensive comparisons with industrystandard DSMC code and experimental results including the pressure, heat flux, shear stress and aerodynamic coefficients for supersonic and hypersonic flows at almost all regimes. The agreements are satisfactory in all cases.
Future work include more application to 3D complex configurations and complex flow, improvement on physical models to consider vibrational degree, implementation of models for gas mixtures, and increases in computational efficiency and accuracy.
Notes
Acknowledgements
The authors would like to thank professor Kun Xu for his help in the code development in the past 8 years and his advice on preparing the manuscript.
Funding
This work was supported by the National Natural Science Foundation of China (11402287 and 11372342).
Availability of data and materials
All data generated or analysed during this study are included in this published article.
Authors’ contributions
DWJ programmed the whole code and conducted the UGKS simulations, and was a major contributor in writing the manuscript. MLM devised the code framework and guided the programming. JL conducted the DSMC simulations. XGD suggested some validation cases and analyzed the related results. All authors read and approved the final manuscript.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
 1.Ivanov MS, Gimelshein SF (1998) Computational hypersonic rarefied flows. Annu Rev Fluid Mech 30:469–505MathSciNetCrossRefGoogle Scholar
 2.Bird GA (1963) Approach to translational equilibrium in a rigid sphere gas. Phys Fluids 6(10):1518–1519CrossRefGoogle Scholar
 3.Bird GA (1990) Application of the direct simulation Monte Carlo method to the full shuttle geometry. AIAA Paper 90–1692Google Scholar
 4.PhamVanDiep G, Erwin D, Muntz EP (1989) Nonequilibrium molecular motion in a hypersonic shock wave. Science 245:624–626CrossRefGoogle Scholar
 5.Bird GA (2005) The DS2V/3V program suite for DSMC calculations. Rarefied Gas Dynamics. Amer Inst Physics, Melville, 541–546Google Scholar
 6.LeBeau GJ (1999) A parallel implementation of the direct simulation Monte Carlo method. Comput Methods Appl Mech Eng 174:319–337CrossRefGoogle Scholar
 7.Ivanov MS, Markelov GN, Gimelshein SF (1998) Statistical simulation of reactive rarefied flows: numerical approach and applications. AIAA Paper 98–2669Google Scholar
 8.Dietrich S, Boyd ID (1996) Scalar and parallel optimized implementation of the direct simulation Monte Carlo method. J Comput Phys 126(2):328–342CrossRefGoogle Scholar
 9.Scanlon TJ et al (2010) An open source, parallel DSMC code for rarefied gas flows in arbitrary geometries. Comput Fluids 39(10):2078–2089CrossRefGoogle Scholar
 10.Bhatnagar PL, Gross EP, Krook M (1954) A model for collision processes in gases I: small amplitude processes in charged and neutral onecomponent systems. Phys Rev 94(3):511–525CrossRefGoogle Scholar
 11.Shakhov E (1968) Generalization of the Krook kinetic equation. Fluid Dynamics 3(5):95–96CrossRefGoogle Scholar
 12.Rykov VA (1975) A model kinetic equation for a gas with rotational degrees of freedom. Fluid Dynamics 10(6):959–966CrossRefGoogle Scholar
 13.Titarev V, Dumbser M, Utyuzhnikov S (2014) Construction and comparison of parallel implicit kinetic solvers in three spatial dimensions. J Comput Phys 256:17–33MathSciNetCrossRefGoogle Scholar
 14.Wadsworth DC et al (2009) Assessment of Translational Anisotropy in Rarefied Flows Using Kinetic Approaches. Rarefied Gas Dynamics. Amer Inst Physics, Melville, 206–211Google Scholar
 15.Mieussens L (2000) Discretevelocity models and numerical schemes for the BoltzmannBGK equation in plane and axisymmetric geometries. J Comput Phys 162(2):429–466MathSciNetCrossRefGoogle Scholar
 16.Baranger C et al (2014) Locally refined discrete velocity grids for stationary rarefied flow simulations. J Comput Phys 257:572–593MathSciNetCrossRefGoogle Scholar
 17.Li ZH, Zhang HX (2009) Gaskinetic numerical studies of threedimensional complex flows on spacecraft reentry. J Comput Phys 228(4):1116–1138CrossRefGoogle Scholar
 18.Peng AP et al (2016) Implicit gaskinetic unified algorithm based on multiblock docking grid for multibody reentry flows covering all flow regimes. J Comput Phys 327:919–942MathSciNetCrossRefGoogle Scholar
 19.Li Z et al (2015) A massively parallel algorithm for hypersonic covering various flow regimes to solve Boltzmann model equation. Acta Aeronautica et Astronautica Sinica 36(1):201–212Google Scholar
 20.Xu K, Huang JC (2010) A unified gaskinetic scheme for continuum and rarefied flows. J Comput Phys 229(20):7747–7764MathSciNetCrossRefGoogle Scholar
 21.Xu K (2015) Direct modeling for computational Fluid dynamics. World Scientific, SingaporeCrossRefGoogle Scholar
 22.Huang JC, Xu K, Yu P (2012) A Unified GasKinetic Scheme for Continuum and Rarefied Flows II: MultiDimensional Cases. Communications in Computational Physics 12(3): 662690.Google Scholar
 23.Jiang D et al (2015) Study on the numerical error introduced by dissatisfying the conservation constraint in UGKS and its effects. Chinese J Theor Appl Mech 47(1):163–168Google Scholar
 24.Titarev VA (2007) Conservative numerical methods for model kinetic equations. Comput Fluids 36(9):1446–1459MathSciNetCrossRefGoogle Scholar
 25.Mao M et al (2015) Study on implicit implementation of the unified gas kinetic scheme. Chinese J Theor Appl Mech 47(5):822–829Google Scholar
 26.Zhu Y, Zhong C, Xu K (2016) Implicit unified gaskinetic scheme for steady state solutions in all flow regimes. J Comput Phys 315:16–38Google Scholar
 27.Yu P (2013) A Unied Gas Kinetic Scheme For All Knudsen Number Flows. The Hong Kong University of Science and Technology, Hong KongGoogle Scholar
 28.Chen SZ et al (2012) A unified gas kinetic scheme with moving mesh and velocity space adaptation. J Comput Phys 231(20):6643–6664MathSciNetCrossRefGoogle Scholar
 29.Mao M (2006) Study of Practical Algorithm for numerical Simulation of Complicated hypersonic Flow. Dissertation, China Aerodynamics Research and Development Center.Google Scholar
 30.Bird GA (1994) Molecular gas dynamics and the direct simulation of gas flows. Oxford Univ. Press,Inc, New YorkGoogle Scholar
 31.Lofthouse AJ (2008) Nonequilibrium hypersonic aerothermodynamics using the Direct Simulation Monte Carlo and NavierStokes models. Dissertation, University of MichiganGoogle Scholar
 32.Li J et al (2018) Novel hybrid hard sphere model for direct simulation Monte Carlo computations. J Thermophys Heat Transf 32(1):156–160CrossRefGoogle Scholar
 33.Wendt JF (1971) Drag Coefficients of Spheres in Hypersonic NonContinuum Flow. von Karman Institute for Fluid Dynamics, BelgiumGoogle Scholar
 34.Boyd ID, Chen G, Candler GV (1995) Predicting failure of the continuum FLUID equations in transitional hypersonic flows. Phys Fluids 7(1):210–219CrossRefGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.