Sensitivity and calibration of turbulence model in the presence of epistemic uncertainties

Abstract

The solution of Reynolds-averaged Navier–Stokes equations employs an appropriate set of equations for the turbulence modelling. The closure coefficients of the turbulence model were calibrated using empiricism and arguments of dimensional analysis. These coefficients are considered universal, but there is no guarantee this property applies to test cases other than those used in the calibration process. This work aims at revisiting the calibration of the closure coefficients of the original Spalart–Allmaras turbulence model using machine learning, adaptive design of experiments and accessing a high-performance computing facility. The automated calibration procedure is carried out once for a transonic, wall-bounded flow around the RAE 2822 aerofoil. It was found that: (a) an optimal set of closure coefficients exists that minimises numerical deviations from experimental data; (b) the improved prediction accuracy of the calibrated turbulence model is consistent across different flow solvers; and (c) the calibrated turbulence model outperforms slightly the standard model in analysing complex flow features around additional test cases (ONERA M6 wing, axisymmetric transonic bump, forced sinusoidal motion of NACA 0012 aerofoil). A by-product of this study is a fully calibrated turbulence model that leverages on current state-of-the-art computational techniques, overcoming inherent limitations of the manual fine-tuning process.

1 Introduction

A deterministic computational fluid dynamics (CFD) analysis gives a single solution for a certain set of input parameters, e.g. geometry, free-stream flow conditions, etc. In practice, these parameters may be uncertain and the associated variability may have a significant impact on the final results. For this reason, stochastic CFD analyses are needed to assess the uncertainty in the solution and to achieve a certain level of robustness or reliability in the final aerodynamic design [24]. The point-collocation nonintrusive polynomial chaos technique is the method of choice to propagate the uncertainty in CFD analyses, as exemplified in [8] (and references therein). This technique requires less deterministic CFD analyses than Monte Carlo techniques by assuming a polynomial chaos expansion of low degree for the uncertain output variables.

Today, it is apparent that CFD workflows contain considerable uncertainty, often not quantified [18]. Numerical uncertainties in the results come from: (a) physical modelling errors and uncertainties, for example, in accurate predictions of turbulent flows; (b) numerical errors arising from mesh and discretisation inadequacies; and (c) aleatory uncertainties derived from natural variability, and epistemic uncertainties due to the lack of knowledge in the parameters of a specific fluid problem. The work presented in the current paper addresses the last type of uncertainty above-mentioned, which calls for turbulence modelling uncertainty quantification, sensitivity analysis and parameter calibration.

Uncertainty in the closure coefficients of a turbulence model is an important source of error in Reynolds-averaged Navier–Stokes (RANS) analyses, but no reliable estimator for this error component exists. It is important to stress that RANS equations rely on several assumptions, e.g. the Boussinesq approximation, which assumes that the turbulent shear stress depends linearly on the mean rate of strain rate. Due to this, it is very unlikely that truly universal coefficients exist. There is no consensus on the best values of these coefficients, as suggested by the wide range of values proposed in the open literature [1]. Current efforts to address these concerns use Bayesian approaches. For example, Ref. [6] described a stochastic error estimate of turbulence models based on variability in the model coefficients. In a sensitivity analysis, it was found that Von Kármán constant, k, has the largest impact on uncertainty in $u^+$ in the log layer of a flat plate boundary layer. This conclusion was suggested analysing results from several turbulence models, including Spalart–Allmaras [21] and Wilcox $k-\omega$ models. In [12], a Bayesian inference framework was used to quantify the uncertainty in Spalart–Allmaras model due to the uncertainty in the closure coefficients. For a flat plate and a backward-facing step problem, the coefficients k, $c_{v1}$, and $c_{b1}$ were found to contribute most to the uncertainty in Spalart–Allmaras model for the chosen output quantities of interest.

References [10, 11] proposed methodologies to estimate the closure coefficients of the Wilcox $k-\omega$ turbulent model for steady and unsteady flow scenarios, respectively. In [11], similar in scope to the one proposed herein, a gradient-based optimization strategy was employed to find the values of the model coefficients providing the best fit between experimental observations and simulated data. Differently from [11], the methodology developed in this work exploits machine learning techniques to build accurate surrogate models that: (a) are used to perform a global sensitivity analysis, providing interesting insights about the problem under study; and (b) are coupled with a global optimization strategy, mitigating the impact of the choice of the initial point on the optimizer’s behaviour and the risk to find sub-optimal solutions due to local minima.

Reference [14] quantified the uncertainty and sensitivity of three turbulence models (Spalart–Allmaras, Wilcox $k-\omega$, and Menter shear–stress transport models) due to uncertainty in the values of closure coefficients for transonic, wall-bounded flows. The analysis was carried out using point-collocation nonintrusive polynomial chaos technique. The test cases were for the flow around an asymmetric bump at 0$^{\circ }$ angle of attack and for Case 6 of the RAE 2822 aerofoil at a prescribed normal force coefficient [2]. For the aerofoil case, the angle of attack was adjusted for each (baseline) turbulence model to match the target normal force coefficient. The same angle of attack was then used in all subsequent simulations where the closure coefficients were modified for uncertainty quantification. Observe that this approach fails to meet the prescribed normal force coefficient for any variation of the closure coefficients from their baseline values. Furthermore, no indications were given on the best values of the closure coefficients for each turbulence model that improved the agreement with experimental data.

The aim of this study is to revisit the calibration of the standard values of the closure coefficients commonly employed in Spalart–Allmaras turbulence model. The work is structured around three technical objectives. The first objective is to exploit current state-of-the-art machine-learning techniques to assess the sensitivity of the output quantities of interest on the uncertainty in turbulence model closure coefficients. The second objective is to calibrate the closure coefficients of Spalart-Allmaras turbulence model by minimising the deviation of numerical results from available experimental data for transonic flows around an aerofoil (Case 6 of RAE 2822). The third objective evaluates the generality of the calibrated Spalart–Allmaras turbulence model on different flow solvers and the expected improvements in prediction accuracy for a variety of transonic flows (Case 9 of RAE 2822, ONERA M6, axisymmetric transonic bump, forced sinusoidal motion of NACA 0012 aerofoil).

The need for an automated calibration, which overcomes the limitations imposed by a manual tuning, is not a conjecture, but an intrinsic requirement to deliver a complete and usable turbulence model. As an example, ANSYS Fluent informs the user that:

The $\gamma$ transition model has only been calibrated for classical boundary layer flows. Application to other types of wall-bounded flows is possible, but might require a modification of the underlying correlations.¹

The uncertainty quantification, the sensitivity analyses, and the calibration of the turbulence model closure coefficients suffer from the curse of dimensionality [3]. In this respect, the reader is invited to reflect upon the work by Sørensen [20]:

Determining the empirical correlations by numerical optimization, along with debugging the model, demands a very large amount of computations, and it is the hope that other researchers can confirm the present expressions by implementation in other flow solvers.

To overcome the large amount of computations, a strategy based on surrogate models is employed. This requires setting up and running a design of experiments (DOE) plan to acquire the relevant information on the system behaviour. A surrogate model that mimics the dependence between the turbulence model closure coefficients and the output quantities of interest is then built and employed to perform the sensitivity analysis and the model calibration. The key aspect of this strategy is to minimise the number of deterministic CFD simulations while maintaining an accurate representation of the system behaviour. In this study, these aspects are encapsulated in an adaptive DOE (ADOE) algorithm that: (a) identifies the regions of the design space that are more difficult to model due to strong non-linearities or scarcity of data, for example; (b) distributes iteratively the design points in those areas of the design space; and (c) selects automatically the surrogate model that best fits the results obtained from the DOE plan. All these features are supported by the machine-learning framework described in [4], where the robustness, efficiency and accuracy of the proposed ADOE algorithm were found to be superior to traditional DOE techniques.

The paper continues in Sect. 2 with a description of the flow solver and the machine-learning approach used to calibrate the closure coefficients of Spalart–Allmaras turbulence model. Then, the sensitivity and calibration of Spalart–Allmaras turbulence model is presented in Sect. 3. The application of the calibrated turbulence model to cases different from the calibration case is discussed in Sect. 4. Finally, conclusions and future recommendations are given in Sect.on 5.

2 Methodology

The computational framework consists of two software tools. The flow solver in Sect. 2.1 was used for the flow predictions. The uncertainty quantification, the sensitivity analysis and the optimisation of the closure coefficients were carried out with the software described in Sect. 2.2.

2.1 Flow solver

The flow solver employed in this study is DLR-Tau [17], a finite volume based CFD flow solver used by several aerospace industries across Europe. The DLR-Tau solver uses an edge-based vertex-centred scheme, where the convective terms are computed via several first and second-order schemes, including central and upwind types. The viscous terms are computed with a second-order central scheme. Time integration is performed either with various explicit Runge–Kutta schemes or the Lower-Upper Symmetric Gauss-Seidel (LU-SGS) implicit approximate factorisation scheme. For time accurate computations, the dual time stepping approach of Jameson [9] is employed. Convergence is improved with a multi-grid acceleration technique based on agglomerated coarse grids. Several models for turbulence closure are available.

For the Spalart–Allmaras model [21], the transport equation is

$$\begin{aligned} \begin{aligned} \frac{\partial \tilde{\nu }}{\partial t}\,+\,u_j\,\frac{\partial \tilde{\nu }}{\partial x_j}\, =&\,c_{b1}\,\left( 1\,-\,f_{t2}\right) \,\tilde{S}\,\tilde{\nu }\,-\,\left[ c_{w1}\,f_{w}\,-\,\frac{c_{b1}}{k^2}\,f_{t2}\right] \,\left( \frac{\tilde{\nu }}{d}\right) ^2\,\\&+\,\cdots \quad \frac{1}{\sigma }\left[ \frac{\partial }{\partial x_j}\left( \left( \nu \,+\,\tilde{\nu }\right) \,\frac{\partial \tilde{\nu }}{\partial x_j}\right) \,+\,c_{b2}\,\frac{\partial \tilde{\nu }}{\partial x_i}\,\frac{\partial \tilde{\nu }}{\partial x_i}\right] \end{aligned} \end{aligned}$$

(1)

The turbulent eddy viscosity is calculated by

$$\begin{aligned} \mu _t\,=\,\rho \,\tilde{\nu }\,f_{v1} \end{aligned}$$

(2)

where

$$\begin{aligned} f_{v1}\,=\,\frac{\chi ^3}{\chi ^3\,+\,c_{v1}^3}, \quad \,\chi \,=\,\frac{\tilde{\nu }}{\nu },\quad \,\text {and} \quad \,\nu \,=\,\frac{\mu }{\rho } \end{aligned}$$

(3)

Furthermore, one has that

$$\begin{aligned}&\tilde{S}\,=\,\varOmega \,+\,\frac{\tilde{\nu }}{k^2\,d^2}\,f_{v2}&\varOmega \,=\,\sqrt{2\,W_{ij}\,W_{ij}}&\quad f_{v2}\,=\,1\,-\,\frac{\chi }{1\,+\,\chi \,f_{v1}} \end{aligned}$$

(4)

$$\begin{aligned}&f_{w}\,=\,g\,\left( \frac{1+c_{w3}^{6}}{g^6+c_{w3}^{6}}\right) ^{\frac{1}{6}}&g\,=\,r+c_{w2}\,\left( r^6-r\right)&\quad r=\,\min \left[ \frac{\tilde{\nu }}{\tilde{S}\,k^2\,d^2},\,10\right] \end{aligned}$$

(5)

$$\begin{aligned}&f_{t2}\,=\,c_{t3}\,e^{\left( -c_{t4}\,\chi ^2\right) }&W_{ij}\,=\,\frac{1}{2}\left( \frac{\partial u_{i}}{\partial x_{j}}-\frac{\partial u_{j}}{\partial x_{i}}\right)&\quad c_{w1}\,=\,\frac{c_{b1}}{k^2}\,+\,\frac{1\,+\,c_{b2}}{\sigma } \end{aligned}$$

(6)

In its original formulation [21], Spalart–Allmaras model includes 11 closure coefficients. In DLR-Tau, the model implementation neglects the trip terms, $c_{t1}$ and $c_{t2}$, which are also passive for the transonic, wall-bounded flows of this study. Herein, nine closure coefficients (after removing $c_{t1}$ and $c_{t2}$) were varied for the uncertainty quantification and sensitivity analysis. A summary of Spalart–Allmaras closure coefficients to be varied and their associated epistemic intervals are reported in Table 1. The choice of the epistemic intervals, some of which differ slightly from [14], lies on empirical suggestions, physical constraints, and experimental evidence [1, 14, 21].

Table 1 Spalart–Allmaras closure coefficients and epistemic intervals

The choice for using DLR-Tau was made to demonstrate that uncertainty in closure coefficients has been overlooked in the past, even for an industrial-grade software tool. This situation may have arisen for convenience, by removing additional difficulties from the multifaced complexities of CFD algorithmic implementation, or negligence, by treating CFD as an established technique. The present work carries out an investigation into an intrinsic weakness of turbulence modelling, and creates a preliminary background knowledge for a robust engineering design.

2.2 Machine-learning Framework

The machine-learning framework is provided by the software platform Noesis Optimus.² The framework consists of an iterative ADOE technique that analyses available data, generally produced by previous iterations or previous DOE runs, to distribute the design points of the next iteration in areas of the parameter space considered of interest. The choice of the location of new sample points is driven by two competing factors. The first factor, denoted space-learning, tends to cover uniformly the design space. No information about the response of the model is, therefore, needed. The second factor, denoted feature-learning, aims at improving the accuracy of the surrogates by identifying critical areas of the design space, such as non-linearities and discontinuities. The reader is referred to [4] for more details on our implementation of space-learning and feature-learning factors, the associated algorithms, and the relevant benchmark cases (analytical and industrially-relevant).

A key aspect of the machine-learning framework, which represents the backbone of the ADOE algorithm, is the capability to identify automatically the best surrogate models for a given set of design points. In the current implementation, Kriging interpolating models together with linear, cubic and thin-plate radial basis functions are considered, without applying any regularization technique. The selection of the best model is accomplished through the following steps:

• 1. for each type of model, a leave-1-out cross-validation is applied;

• 2. the corresponding values of the coefficient of determination (denoted as $R_{\text {PRESS}}^2$) is calculated based on the cross-validation residuals;

• 3. the best model is identified as the one characterized by the smallest value of the $R_{\text {PRESS}}^2$ metric.

The advantages of the ADOE algorithm consist, therefore, on the possibility to perform in a completely unsupervised fashion: (a) the iterative selection of the design point locations considered in the DOE campaign; and (b) the choice of the response surface model type. These features, embedded in Noesis Optimus, are exploited in the current study to assess the sensitivity of the output quantities of interest (flow solution and aerodynamic coefficients) on the uncertainty in the closure coefficients of Spalart–Allmaras turbulence model, and to calibrate automatically the values of these coefficients based on available experimental data.

3 Transonic wall-bounded test case

Uncertainty quantification and sensitivity analysis for the aerofoil test case are reported in this section. The section continues with the calibration of the closure coefficients, discusses the implementation across different flow solvers, and investigates the influence of the spatial discretisation on the results.

3.1 Description

Navier–Stokes calculations for Case 6 of the RAE 2822 aerofoil [2] were performed with Spalart–Allmaras turbulence model. The experimental data for this case are for $M\,=\,0.729$ and $Re\,=\,6.5\times 10^6$ at a prescribed normal force coefficient $C_{\text {N}}\,=\,0.743$. For all calculations, the angle of attack, $\alpha$, was adjusted to match this value of $C_{\text {N}}$. This is a countermeasure taken to minimise the possibility of an inaccurate measurement of the angle of attack, favouring the use of the numerical integration of the measured pressure coefficient data as a more reliable reference condition. Ensuring the accuracy of experimental data is at the centre of much research, and the interested reader is referred to the concluding remarks for recent initiatives in this direction.

The computational grid adopted for the flow simulations, shown in Fig. 1a, is available from the NPARC Alliance Validation Archive web site.³ The C-grid, denoted hereafter the coarse grid, consists of a single-block with $369\times 65$ points. The far-field boundary is placed at about 20 chords from the aerofoil, and the distance of the first grid points off the aerofoil surface is about ${10}^{-5}$ chord.

For all steady calculations, the explicit time stepping and the fourth order Runge–Kutta scheme were used. To accelerate the convergence to a steady state, a local time-stepping, implicit residual smoothing and a full multigrid method were used. The discretisation of the convective and diffusive fluxes of both RANS and Spalart–Allmaras equations is based on the second order Roe’s flux difference splitting scheme. Venkatakrishnan’s flux limiter was used for all simulations reported in this paper. A no-slip boundary condition was applied on the aerofoil surface, and far-field boundary conditions were applied to the far-field boundaries. The ratio of eddy viscosity to molecular viscosity 0.5 is prescribed at the far-field boundaries, whereas at smooth walls, zero turbulence condition is enforced. The CFL number was set to 1.2 and the number of multigrid (MG) levels to 3. Simulations were run for 4000 MG cycles to compute the steady state solution. With this setup, the overall residuals of Navier–Stokes and Spalart–Allmaras equations decreased by about five orders of magnitude, and all force and moment components converged within 2000 MG cycles.

For Case 6 ($M\,=\,0.729$, $Re\,=\,6.5\times 10^6$, and prescribed $C_{\text {N}}\,=\,0.743$), the angle of attack with the standard Spalart–Allmaras model was found to be $\alpha \,=\,2.51^{\circ }$, the drag coefficient $C_{\text {D}}\,=\,0.0150$, and the pitch moment coefficient about quarter chord $C_{\text {m}}\,=\,-\,0.0909$, compared, respectively, with 2.92$^{\circ }$, 0.0127, and − 0.095 in the experiment. The comparison of the pressure coefficient distribution with experimental data is shown in Fig. 1b. The overall agreement is good, but differences are visible near the leading edge and at the shock front ($x/c\,=\,0.15$ and 0.55, respectively).

Fig. 1

Validation of RAE 2822 aerofoil; in a, C-type structured grid; in b pressure coefficient distribution for Case 6 ($M\,=\,0.729$, $Re\,=\,6.5\times 10^6$, and $C_{\text {N}}\,=\,0.743$)

3.2 Generation of the response surface model

The computational framework described in Sect. 2 was used to generate a response surface model between the input parameters and the system outputs. The input parameters include nine uncertain closure coefficients of Spalart–Allmaras turbulence model, which are described by the epistemic intervals in Table 1, and the angle of attack for matching the prescribed normal force coefficient. The system outputs that were monitored consist of three quantities: the lift and drag coefficients, and the sum of squared errors (SSE) between the pressure coefficient distribution from experimental data and that from numerical results. For uncertainty quantification and sensitivity analysis, and for the calibration of the closure coefficients, SSE is used as the output quantity of interest.

The generation of the response surface model followed the procedure:

• 1. the ADOE algorithm was initialised with 1025 sample points selected by a two-level full factorial approach including the central point;

• 2. these results were then used to initialize the ADOE strategy and to compute sequentially (in batches of 25 analyses) an additional set of 1500 CFD simulations. At the end of this step, the machine-learning framework identified automatically the best surrogate models to link the ten input variables to the system outputs based on a table containing 2525 design points;

• 3. for each set of values of the closure coefficients included in this table, the surrogate models were interrogated to find the angle of attack that matched the target $C_{\text {N}}\,=\,0.743$. By removing the dependence on the angle of attack, the 1024 sample points of the full factorial plan were reduced to 512. Hence, this step reduced the size of the table to 2013 sample points by adjusting the angle of attack to match the target $C_{\text {N}}$;

• 4. finally, results at the previous step were validated by running a set of deterministic CFD simulations. Due to problems occurred on the local network connectivity, only 1980 experiments out of 2013 were successful and used to build the final surrogate models employed in the sensitivity analysis and calibration of the turbulence closure coefficients.

A total of 4538 deterministic CFD simulations were run on the high-performance computing (HPC) facility of the University of Southampton (Iridis4) in just over 1000 CPU hours. As an example, the Optimus simulation workflow is shown in Fig. 2. This workflow integrates and automates the following main tasks that are performed during each iteration of the DOE/optimisation analyses:

• 1. map the value assigned to the turbulence coefficients within the input file required to run the CFD analysis;

• 2. submit the job with the instructions to launch the DLR-Tau flow solver to the resource manager of the HPC facility Iridis4, monitor its status and retrieve the output files upon job completion;

• 3. parse the output files to extract the values of the drag, lift, pitch moment, and pressure coefficients;

• 4. reorder the list of the calculated pressure coefficients to match those available from the experimental dataset;

• 5. calculate the mismatch between simulated and measured pressure coefficients in terms of the SSE metric.

Fig. 2

Optimus workflow for the job submissions on the HPC facility and for running the ADOE and optimisation analyses

Similar to the approach in [13], we employ the generated response surface models to assess how the uncertainty characterizing the turbulence model coefficients impacts the values of the SSE metric. The latter provides a global quantification of the deviation occurring between experimental and numerical pressure coefficients along the section of the airfoil. This uncertainty quantification study is performed by evaluating the SSE values for 10,000 random perturbations of the turbulence model coefficients. A uniform distribution within the design space identified by the low and high boundaries defined in Table 1 was adopted. As depicted in Fig. 3, the uncertainty in the turbulence model coefficients has a significant impact on the SSE metric, with a variation ranging between 0.14 and 0.24, corresponding to values of the root mean square error equal to 0.037 and 0.048, respectively. The observed degree of uncertainty highlights the need to perform a sensitivity analysis study to identify the set of turbulence model coefficients that have the greatest impact on the uncertainty of SSE.

Fig. 3

Histogram of SSE values obtained from 10,000 random

3.3 Global sensitivity analysis

The global sensitivity analysis of the SSE between measured and simulated pressure coefficient on the uncertainty of the turbulence closure coefficients was analysed using the surrogate model generated with 1980 sample points. The relative contribution of the input parameters to the total variability of SSE was quantified by relying on the variance-based Sobol indices [19]. This information is a useful measure as it allows identifying the closure coefficients with the largest influence on the output variability. This, in turn, informs the calibration process by retaining only those coefficients with a Sobol index larger than a user predefined threshold.

Table 2 reports the values of the first order Sobol indices estimated via Monte Carlo integration performed with 10,000 random points evaluated on the surrogate model. In agreement with [6], the Von Kármán constant, k, is the parameter that has the greatest impact on the variability of the system output. The mismatch between experimental data and numerical simulations is also influenced by the value of $c_{v1}$ and, to a lesser extent, by $c_{b1}$ and $\sigma$. The sensitivities with respect to the remaining five parameters ($c_{w2}$, $c_{b2}$, $c_{t3}$, $c_{w3}$, $c_{t4}$) are virtually null. Being the value of the residual approximately equal to 0.01 (not reported in Table 2), no significant sensitivity can be attributed to the mutual interaction between the different input parameters.

Results in Table 2 are not unexpected because Spalart–Allmaras model consists of four nested versions, from the simplest which is applicable to free shear flows to the most complete, applicable to viscous flows past solid bodies and with laminar regions. The terms of each version are passive in all the lower versions of the model. The test case is for transonic wall-bounded flow, and only few terms of Spalart–Allmaras model (k, $c_{v1}$, $c_{b1}$, $\sigma$) are therefore active.

Table 2 First order Sobol indices of Spalart–Allmaras closure coefficients with respect to SSE

3.4 Calibration of turbulence model closure coefficients

A two-steps approach was adopted to calibrate the closure coefficients of Spalart–Allmaras turbulence model.

The first step entails the identification of a global optimum by a differential evolution algorithm [22]. The differential evolution is a genetic algorithm that is well-suited to find the global minimum of continuous functions, but at the cost of many expensive evaluation calls to reach convergence. To overcome this problem, the function evaluations were performed on the surrogate model, without the need to run the (expensive) CFD analyses. Because the differential evolution algorithm is characterised by a non-deterministic behaviour, three separate runs initialised with different values of the random number generator seed were performed. This action was taken to mitigate the impact of the inherent randomness of the optimisation scheme on the obtained solution. Table 3 lists the optimum values of the turbulence model closure coefficients for the three optimisation runs. The five parameters with the lowest Sobol indices ($c_{w2}$ through $c_{t4}$) were kept at their nominal values. As expected, the optimum solutions found by the three test runs yield consistent results for the input parameters with the highest Sobol indices (k, $c_{v1}$, $\sigma$, $c_{b1}$). These findings agree well with the set of optimised closure coefficients obtained through one of the deterministic CFD analyses executed during the DOE plan. Discrepancies on the remaining five parameters ($c_{w2}$ through $c_{t4})$ are non-influential, being their Sobol indices virtually null.

Table 3 Optimal values of closure coefficients and corresponding SSE based on genetic algorithm evaluated on surrogate model and best experiment found during the DOE plan

The second step of the calibration procedure uses a gradient-based approach that launches additional deterministic CFD analyses. The non-linear programming quadratic line (NLPQL) optimisation scheme [15] was the method of choice. To mitigate the possibility of being entrapped in local minima, the algorithm is initialised from the optimal point found during the second global optimisation run and reported in Table 3. The optimal values of the closure coefficients are summarised in Table 4 after only one iteration that entailed five additional CFD analyses. It was found that the results of the gradient-based optimisation are virtually unchanged compared with the optimal values reported in Table 3. This reflects the good quality of the surrogate model generated by the machine-learning framework and provides also a validation of the overall optimisation approach, according to which: (a) a large number of model evaluations performed on the surrogate model is first employed to find a global optimum; and (b) a smaller number of CFD analyses is then used to refine the output of the calibration.

Table 4 Spalart–Allmaras closure coefficients for the standard and calibrated versions

3.5 Improved prediction accuracy

This section discusses two aspects related to the optimal values of the closure coefficients in Table 4. The first aspect is concerned with the improved prediction accuracy of the flow solution for Case 6. Figure 4a shows a comparison of the pressure coefficient obtained with the standard and calibrated Spalart–Allmaras models. Qualitatively, the solution with the optimal values of the closure coefficients improves the agreement with the experimental data near the leading edge and at the shock front. Quantitatively, the SSE is reduced from 0.206 for the standard Spalart–Allmaras model, obtained for $\alpha \,=\,2.51^{\circ }$, to 0.137 for the calibrated model, for $\alpha \,=\,2.37^{\circ }$. The second aspect is about consistency and generality of the above conclusions across different flow solvers. For this purpose, two runs were performed with the SU2 flow solver [5] using the standard and calibrated Spalart–Allmaras models, on the same grid employed in DLR-Tau. Numerical settings were similar to those of DLR-Tau simulations. Figure 4 shows the pressure coefficient obtained with the standard and calibrated turbulence models in SU2. From the comparison, the improvement in prediction accuracy when using the calibrated model in SU2 is identical to that observed in DLR-Tau. This confirms the above conclusions are consistent on different flow solvers, and so the advantages of using the calibrated turbulence model are solver-independent.

Fig. 4

Case 6 of RAE 2822 aerofoil, coarse grid: pressure coefficient with standard and calibrated turbulence models

3.6 Concluding remarks

A finer grid was generated to investigate the influence of the spatial discretisation on the prediction capability for Case 6. The grid of Fig. 5a, denoted hereafter the medium grid, consists of a single-block with $865\times 161$ points. The far-field boundary is placed at 50 chords from the aerofoil, and the distance of the first grid points off the aerofoil surface is about ${5\times 10}^{-6}$ chord.

As similarly observed for the pressure coefficient distribution on the coarse grid, the calibrated turbulence model achieves a better agreement with experimental data than the standard version. The comparison is reported in Fig. 5b. A finer grid resolution improves significantly the pressure coefficient on the upper surface close to the leading edge and the shock position.

Fig. 5

Case 6 of RAE 2822 aerofoil, medium grid: pressure coefficient with standard and calibrated turbulence models

A quantitative analysis of the influence of the spatial discretisation (coarse and medium grids) and of the turbulence model (standard and calibrated versions) on aerodynamic coefficients is summarised in Table 5. In all cases, the reported angle of attack is such that $C_{\text {N}}\,=\,0.743\pm 0.0005$. On the coarse grid, the solution of the calibrated Spalart–Allmaras model leads to good predictions of the drag and pitch moment coefficients. Taking as reference the experimental values, the percentage error in $C_{\text {D}}$ is reduced from 18.1 to 7.9%, and the error in $C_{\text {m}}$ from 4.2 to 1.1% when switching from the standard to the calibrated turbulence model. Furthermore, results converge towards the experimental values as the grid is refined, for both versions of the turbulence model. On the medium grid, the solution of the calibrated turbulence model achieves an error of 2.3% in $C_{\text {D}}$, compared to 7.9% on the coarse grid. The pitch moment coefficient is unaffected by the spatial discretisation, with an error as small as 1.1%.

Table 5 Case 6 of RAE 2822 aerofoil: impact of spatial discretisation on aerodynamic coefficients for standard and calibrated Spalart–Allmaras turbulence models

With the above paragraphs as background, one can identify different scenarios for the applicability of the current work. In a research scenario, the work can be used to determine the robustness of a turbulence model, and whether efforts should be addressed to improve the model in some way. In a design scenario, there is an interest to reduce the uncertainty arising from the turbulence modelling, or else to design a factor of safety around it. In a commercial scenario, the work could be used in a cost analysis, whereby uncertainty in drag coefficient will affect projected fuel costs.

4 Additional test cases

The prediction capability of the calibrated turbulence model was assessed on additional test cases which were not used in the calibration process. These four cases include three steady-state problems, and one unsteady problem for forced sinusoidal oscillations in pitch.

4.1 RAE 2822 Aerofoil, Case 9

The first additional test case involves Case 9 of the RAE 2822 aerofoil. The experimental measurements suffered from wind tunnel interference effects, which cannot be estimated a posteriori. Hence, the analyses presented herein are performed for a fixed normal force coefficient, as done for Case 6 in Sect. 3. Case 9 is run at $M\,=\,0.73$, $Re\,=\,6.5\times 10^6$, and $C_{\text {N}}\,=\,0.803$, and this results in little separation downstream of the shock position. Both the coarse and medium grids were used with the standard and calibrated Spalart–Allmaras turbulence models.

Computed results of the pressure coefficient are compared to experimental data in Fig. 6. The suction peak on the upper surface and the shock location are predicted reasonably well by the standard Spalart–Allmaras turbulence model. The improvement revealed by the calibrated turbulence model is, however, visible at these two locations. As reference, the pitch moment coefficient from experiments is − 0.099, compared to − 0.094 for the standard model computed at $\alpha \,=\,2.7^{\circ }$, and − 0.098 for the calibrated version at $\alpha \,=\,2.6^{\circ }$.

Fig. 6

Case 9 of RAE 2822 aerofoil, medium grid: pressure coefficient with standard and calibrated Spalart–Allmaras turbulence model ($M\,=\,0.73$, $Re\,=\,6.5\times 10^6$, $C_{\text {N}}\,=\,0.803$)

4.2 ONERA M6 wing

The ONERA M6 wing is a swept wing with no twist, built with the symmetric ONERA D aerofoil. The computational grid is available from the validation web site of the NASA CFD code CFL3D.⁴ The grid with $288\times 64\times 48$ cells consists of one zone wrapped as a C-grid about the wing leading edge. Symmetry boundary condition was used on one side of the domain. The wing span features 256 cells in the chord-wise direction and 48 cells in the span-wise direction. The minimum wall distance of the first grid points off the wing surface is about $2.5\times {10}^{-6}$ chords at the leading edge and about $5\times {10}^{-6}$ chords at the trailing edge. The grid is non-dimensionalised by the span, and the mean aerodynamic chord is $c\,=\,0.54\,b$.

Calculations were performed with Spalart–Allmaras turbulence model. Experimental data are available for $M\,=\,0.84$ and Reynolds number, based on the mean aerodynamic chord, $Re\,=\,12.7\times 10^6$ at a prescribed angle of attack $\alpha \,=\,3.06^{\circ }$ [16].

As for the aerofoil case, an explicit time stepping and the fourth order Runge–Kutta scheme were used. To accelerate the convergence to a steady state, a local time-stepping and implicit residual smoothing were used. The discretisation of the convective and diffusive fluxes of the RANS equations was based on the second order Roe’s flux difference splitting scheme, and the first order accurate scheme was used for Spalart–Allmaras fluxes. Venkatakrishnan’s flux limiter was used. A no-slip boundary condition was set at the wing surface, and far-field boundary conditions were applied to the far-field boundaries. The CFL number was set to 1.2 and simulations were run for 20,000 iterations to compute the steady state solution. With this setup, the overall residuals of Navier-Stokes and Spalart–Allmaras equations decreased by about five orders of magnitude, and all force and moment components converged within 15,000 iterations.

The pressure coefficients at six span-wise locations of the ONERA M6 wing are shown in Fig. 7. The agreement with experimental data reveals the difficulty of a RANS solution to capture the double shock at 80% span-wise location and to predict the pressure coefficient in the cove region at 99% span-wise location. These deficiencies, commonly documented in the open literature, are attributed to physical modelling errors. The calibrated turbulence model has no effect in these two areas, suggesting that the deficiency is intrinsic to turbulence modelling and requires higher modelling fidelity in flow physics. By close inspection, the shock position and intensity of the calibrated turbulence model achieves a favourable agreement with experimental data, particularly, at locations 90 and 95% of the span. Although of limited extent, the solution of the calibrated Spalart–Allmaras model moves towards the reference data.

Fig. 7

ONERA M6 wing: pressure coefficient with standard and calibrated Spalart–Allmaras turbulence model ($M\,=\,0.84$, $Re\,=\,12.7\times 10^6$, $\alpha \,=\,3.06^{\circ }$)

4.3 Axisymmetric transonic bump

The axisymmetric transonic bump is one of the turbulence validation test cases available from the validation web site of the NASA CFD code CFL3D.⁵ The experiment, which used a circular-arc bump, was performed in the NASA-Ames wind tunnel at Mach 0.875 and Reynolds number 2.8 million. The grid with dimensions of $360\times 160$ cells (available from the NASA website) was used in this study, see Fig. 8.

Fig. 8

Axisymmetric transonic bump

A preliminary study was carried out to compare DLR-Tau results with CFL3D results. The agreement between codes, using the standard Spalart–Allmaras turbulence model, was good. We then proceeded to assess the influence of the turbulence closure coefficients using DLR-Tau. Computed results of pressure coefficient are shown in Fig. 9a. The standard and calibrated turbulence models provide a similar overall trend, but the calibrated model provides a sharper pressure jump at the location of separation which is caused by a combination of shock and trailing-edge adverse gradient, and a slight under-prediction at or around the flow reattachment. Figure 9b shows the skin friction coefficient. The computed flow separation point is at about $x/c\,=\,0.7$ and the reattachment point is observed between $x/c\,=\,1.1$ and 1.2, in agreement with the locations extracted from oil-flow visualizations. The sensitivity of the skin friction coefficient on the turbulence closure coefficients is visible in the attached ($x/c\,<\,0.7$), separated, and reattached ($x/c\,>\,1.1$) flow regions, but the locations of separation and reattachment points are virtually unaffected.

Fig. 9

Axisymmetric transonic bump: standard and calibrated Spalart–Allmaras turbulence model ($M\,=\,0.875$ and $Re\,=\,2.8\times 10^6$)

4.4 NACA 0012 aerofoil forced sinusoidal motion

This test case concerns transonic flow predictions for the NACA 0012 aerofoil undergoing a forced sinusoidal motion in pitch around one-quarter of the chord. The flow conditions for AGARD CT5 are Mach number 0.755 and Reynolds number 5.5 million. The motion is characterised by the reduced frequency $k\,=\,0.0814$, mean angle of attack $\alpha _0\,=\,0.016^{\circ }$, and amplitude $\alpha _A\,=\,2.51^{\circ }$.

The unstructured grid, which was also used in a previous work [7], consists of about 15.3 thousand mesh elements, see Fig. 10a. The first grid layer on the wall was placed at $5\times 10^{-6}$ (for a chord of one) to ensure that $y^+$ was smaller than 1. An implicit dual-time stepping scheme was used, with a target residual drop of three orders at each physical time step.

The flow field presents the formation of a strong and highly dynamic shock wave experiencing Tijdeman and Seebass’s [23] type-B shock motion. The steady solution includes a virtually symmetric shock wave which periodically appears and disappears on the upper and lower surfaces as consequence of the harmonic motion. Figure 10 reports the hysteresis in the normal force coefficient. Deviations between experiments and analyses are standard and have already been reported in the literature. The calibrated Spalart–Allmaras turbulence model has a marginal improvement, mostly confined to the down-stroke for positive angles of attack and to the up-stroke for negative values. Corresponding to these two regions, the computed instantaneous pressure coefficient distributions are presented in Fig. 11. In both cases, the calibrated model reveals a sharper pressure jump across the shock compared to that of the standard model, and the shock location moves closer to the experimentally measured data.

Fig. 10

NACA 0012 aerofoil; in a unstructured grid; in b normal force coefficient hysteresis loop for AGARD CT5 ($M\,=\,0.755$, $k\,=\,0.0814$, $\alpha _0\,=\,0.016^{\circ }$, $\alpha _A\,=\,2.51^{\circ }$)

Fig. 11

NACA 0012 aerofoil: instantaneous pressure coefficient for AGARD CT5 ($M\,=\,0.755$, $k\,=\,0.0814$, $\alpha _0\,=\,0.016^{\circ }$, $\alpha _A\,=\,2.51^{\circ }$)

5 Conclusions

Uncertainty in the closure coefficients of a turbulence model is an important source of error in Reynolds-averaged Navier–Stokes simulations. This requires turbulence modelling uncertainty quantification, sensitivity analysis and parameter calibration. The work detailed in this study addressed these aspects using state-of-the-art computational techniques, including a machine-learning software platform with an adaptive design of experiments algorithm, a modern flow solver, and a high-performance computing facility. The original Spalart–Allmaras turbulence model was analysed. The key elements of this work are: (a) uncertainty quantification and sensitivity analysis required only about 1,000 CPU hours to explore a ten-dimensional design space; (b) only a selected number of closure coefficients have a large impact on the uncertainty of the output quantities of interest; this is not unexpected because Spalart–Allmaras turbulence model has a nested structure, with the outer versions not altering the lower ones; (c) the optimal values of the closure coefficients, which were implemented in a calibrated version of Spalart–Allmaras model, were chosen to minimise the sum of square error of the pressure coefficient between experimental data and numerical results; (d) the closure coefficients with the largest Sobol indices are determined accurately as they have the largest impact on the output quantities of interest. It was found that the calibrated Spalart–Allmaras turbulence model slightly outperforms the standard version for transonic, wall-bounded flows around the RAE 2822 aerofoil. The expected prediction accuracy holds for a variety of other test cases (Case 9 of RAE 2822, ONERA M6, axisymmetric transonic bump, forced sinusoidal motion of NACA 0012 aerofoil) other than that used in the calibration process, as well as across different flow solvers.

Part of the problem with improving the simulation accuracy of the various effects that occur at challenging flow conditions has been the inability to access unbiased, data-rich experimental data-banks. There certainly are alternative computational approaches, e.g. Direct Numerical Simulation, that are being developed and play an important role in turbulence modelling development and validation. However, these approaches are limited in the types of geometries and flow conditions to be simulated. Recently, the European Commission through the Horizon 2020 programme has recognised these challenging issues, and supported the “Holistic Optical Metrology for Aero-elastic Research” (HOMER, project ID: 769237) project. Among the basic science research directions, the project aims at providing space and time-accurate experimental measurements of fluid and structure for a range of aeronautical flows, and at supporting the development and enhancement of numerical methods following a similar approach to that presented herein. Considering a variety of flow conditions and geometries is necessary to extend the approach here discussed to a large subset of the turbulence model closure coefficients.