Sequential improvement for robust optimization using an uncertainty measure for radial basis functions

Engineering can be seen as a sequence of making choices, from fundamental design choices to defining design details. In the last century, many empirical and theoretical models have been developed to help with these decisions. Nowadays engineers frequently use computational models to assess the effects of design choices and determine optimal designs.

In many engineering problems, it is infeasible to find an optimal design through trial and error due to a large number of interacting parameters. Therefore, many optimization methods have been developed to automate this process, each method being suitable to solve a specific type of problem in an efficient way. With these methods an optimal configuration of the design may be found. However, such a design may malfunction in real-life due to variability of the model parameters. A specific branch of optimization research focuses on these problems: within robust optimization the goal is to find a design which fulfills the requirements even under the influence of parameter variations.

When dealing with expensive computational models, it is desired to acquire insight into the problem at hand without performing an excessive amount of model evaluations. One approach is to replace the model with an easy to evaluate metamodel, also known as surrogate or response surface model. The metamodel is built based on a moderate number of evaluations of the computational model, and thereafter used for optimization. Some researchers developed methods to iteratively enrich the metamodels with new evaluations of the computational model. This is known as sequential optimization or sequential improvement.

Within this research the computational efficiency of sequential metamodel based robust optimization with expensive computational models is studied. The optimization flowchart is shown in Fig. 1. Robust optimization, metamodeling and sequential improvement have been combined in several studies, such as for the designs of an ordering policy of an inventory system (Huang et al. 2006), a loaded framework (Jurecka et al. 2007) and an industrial V-bending process (Wiebenga et al. 2012), as well as for the optimization of the crashworthiness of a foam-filled thin-walled structure (Sun et al. 2014), the blankholder force trajectory for springback reduction of an U-shaped high strength steel strip (Kitayama and Yamazaki 2014) and many mathematical benchmark problems (Huang et al. 2006; Wiebenga et al. 2012; Marzat et al. 2013; Janusevskis and Le Riche 2013; Ur Rehman et al. 2014).

Open image in new window

The robust optimization procedure consists of several steps (Fig. 1). For some steps a choice has to be made between different methods to perform a prescribed task. For example, latin hypercube sampling, fractional factorial sampling, random sampling or a combination of these may be chosen as sampling strategy for the initial Design Of Experiments (DOE). To fit a set of data points, many methods have been developed, such as polynomial regression, neural networks, kriging, Radial Basis Functions (RBF), moving least squares and many others. Even within these methods many choices can be made on the exact configuration of the method. Hence, these methods can be seen as building blocks of the optimization algorithm. Every building block may be replaced with any other building block which is capable of performing the prescribed task. Clearly, the choice for a certain building block may severely affect the efficiency of the optimization algorithm.

In the present work, the efficiency of the robust optimization algorithm is studied. More specifically, the effects of the metamodeling method and the sequential improvement criterion on the optimization are examined. A widely used metamodeling method is kriging, as observed by several researchers (Wang and Shan 2007; Kleijnen 2009; Forrester and Keane 2009). One advantage of kriging is that it includes a prediction of the metamodel uncertainty. However, clustering of DOE points adversely affects the predictive capability of kriging methods (Zimmerman et al. 1999; Havinga et al. 2013). This is of interest because point clusters are formed during optimization due to the sequential improvement procedure. An interpolating metamodeling method which may be adapted to account for clustering of DOE points is RBF interpolation. Good fitting capabilities for nonlinear functions have been reported for RBF interpolation in several studies (Franke 1982; Jin et al. 2001). However, the uncertainty measures which have been developed for RBF are not suitable for sequential optimization. Therefore, an improved configuration of the RBF metamodel is proposed, taking into account the local DOE density and including a prediction of the metamodel uncertainty based on the Leave-One-Out Cross-Validation (LOOCV) measure.

Sequential improvement in robust optimization has been scarcely investigated up to now. For deterministic optimization, the Expected Improvement (EI) criterion was proposed by Jones et al. (1998). This criterion can be used for robust optimization, but does not take into account the uncertainty of the objective function value at the sampled design points. Jurecka (2007) adapted the expected improvement criterion of Jones for robust optimization. However, this criterion has some disadvantages which will be shown in this work. Therefore, a new expected improvement criterion for robust optimization is proposed.

The sequential robust optimization procedure is generic and, therefore, not limited to specific research fields. However, our interest goes to robust optimization of metal forming processes. The aim is to design metal forming processes with a low sensitivity to variations of material properties and process parameters such as friction and tool wear. Our background may be noticed in the examples given throughout this work, without violating the genericity of the approach.

This paper is structured as follows: in Section 2 a selective overview of the state of the art of robust optimization is given: the metamodeling methods including error prediction (Section 2.1) and the sequential improvement criterion (Section 2.2) are discussed. The RBF metamodel with uncertainty measure is proposed in Section 3. The novel robust expected improvement formulation is introduced in Section 4. The efficiency of the expected improvement criteria and metamodeling methods are compared in Section 5 based on mathematical functions. In Section 6 a demonstration problem is presented: a V-bending process with two noise variables, five design variables and two nonlinear constraints. The robust optimization results are given in Section 7 and the conclusion is given in Section 8.

The famous paper of Jones et al. (1998) in which the expected improvement criterion was introduced, is titled Efficient Global Optimization of Expensive Black-Box Functions. The term black-box function denotes that it is not of interest how the function works. The only requirement is that the function is able to process a set of input variables and return one or multiple output values. Therefore, the optimization procedure is suitable for optimization with any kind of numerical model. Unlike experiments, trials with the same input variable values always give the same output value. Therefore, there is no need to perform multiple tests of the same variable set. The adjective expensive emphasizes the computational cost of the function and stresses the need to keep the number of function evaluations as low as possible.

In the case of deterministic optimization, both the black-box function as well as the optimization problem are defined in the design variable space x. In contrary, in robust optimization the black-box function is defined in the combined space of the design variable x and the noise variable z, while the optimization problem is stated in the design variable space x. At a certain point in the design space x ^′, the objective function is defined by the behavior of the black-box function throughout subspace (x ^′, z). Given an assumption on the probability distribution of the noise variable z, the probability distribution for the output of the black-box function may be determined at x ^′, and subsequently this probability is used in the objective function.

The objective function Obj(Y(x)) determines which property of the probability distribution Y(x) should be minimized. The robustness of a production process can be defined in several ways, such as the average error of the production process or the percentage of the products that meet the product requirements. The most straightforward way to quantify these measures is to determine the statistical parameters of the process output Y(x), such as the mean μ _Y , standard deviation σ _Y or median. Two different ways to improve the accuracy of the process are by shifting the mean μ _Y or decreasing the standard deviation σ _Y (Koch et al. 2004). In some cases, these may be conflicting objectives (Beyer and Sendhoff 2007). Some researchers choose to combine these measures in a single objective, for example by minimizing μ _Y + kσ _Y , as is done in this study. Many other robust optimization objectives can be found in literature, such as Pareto front determination (Coelho and Bouillard 2011; Nishida et al. 2013), quantile optimization (Rhein et al. 2014), minimization of the worst case scenario (Marzat et al. 2013) and minimization of the signal-to-noise ratio (Taguchi and Phadke 1984; Leon et al. 1987).

The lower bound and upper bound of the design variables x are defined by L B and U B. The constraint functions C _i (G _i (x)) and \(\text {C}_{i}^{\text {eq}} (H_{i} (\mathbf {x} ) )\) determine the constraints for the distributions G _i (x) and H _i (x).

Usually, every simulation yields the values of the functions f, g _i and h _i for one variable set (x, z). However, the whole subspace (x, Z) must be known to exactly evaluate the objective function Obj(Y(x)) and the constraint functions C _i (G _i (x)) and \(\text {C}_{i}^{\text {eq}} (H_{i} (\mathbf {x} ) )\) for one value of the design variable x. Clearly, it is not desired to spend too many simulations for just one point in the design variable space. Therefore, the optimization algorithm should be able to handle uncertainty in the objective and constraint function values, and efficiently select whether to sample multiple points in the noise space or explore other regions of the design space.

When defining a robust optimization problem, one of the first problems is to determine the distribution p _Z (z) of the noise variable z. In the optimization of a production process, these may be properties such as material variation, friction variation or sheet thickness variation. These variations can be characterized by min-max values or by statistical properties such as mean and standard deviation. In some cases, knowledge of the correlations between the different noise variables is required for a correct estimation of the robustness of the process (Wiebenga et al. 2014).

In the following sections the different components used to solve the optimization problem of (6) are discussed (see Fig. 1).

2.1 Metamodeling methods

A major component of the optimization algorithm is the metamodeling method which is used to fit the objective function f and the constraint functions g _i and h _i using a set of observations D = {(x ⁽ⁱ⁾, y ⁽ⁱ⁾)|i = 1…n}. The sequential improvement strategies discussed in this study (Section 2.2) do not only need the estimate of the functions \(\hat {f}\), \(\hat {g}_{i}\) and \(\hat {h}_{i}\) at untried points x but also require an estimate of the prediction uncertainty at these points. Therefore, only methods that provide this estimate will be discussed.

As a basis for interpolating simulation data, it is good practice to normalize the inputs and to use a regression model as underlying model. Often, a simple polynomial model is used to capture the major trends of the observations y. Hence, the metamodeling methods are used to fit the residue \(\mathbf {y} - \hat {f}_{\text {regr}}(\mathbf {x})\) of the polynomial fit. In the following text, fitting of the residue is assumed. For clarity, y is used in the equations instead of \(\mathbf {y} - \hat {f}_{\text {regr}}(\mathbf {x})\).

2.1.1 Kriging

A popular method for metamodeling with simulation data is kriging. It has been developed by Krige (1951) and became popular in engineering design after the work of Sacks et al. (1989). The data is assumed to be a realization of a Gaussian random field. The spatial correlation between two points x ⁽ⁱ⁾ and x ^(j) is assumed to be:

$$ \phi_{i,j} = \exp \left( - \sum\limits_{d = 1}^{k} \theta_{d} \left| \mathbf{x}^{(i)}_{d} - \mathbf{x}^{(j)}_{d} \right|^{p_{d}} \right) $$

(7)

The number of dimensions is k. For any correlation function, the prediction \(\hat {f}\) and Mean Squared Error (MSE) estimate \(\hat {s}^{2}\) at any point x can be derived (Forrester et al. 2008):

$$\begin{array}{@{}rcl@{}} && \hat{f}(\mathbf{x}) = \boldsymbol{\phi}^{T}\boldsymbol{\Phi}^{-1}\mathbf{y} \end{array} $$

(8)

$$\begin{array}{@{}rcl@{}} && \hat{s}^{2}(\mathbf{x}) = \hat{\sigma}^{2} \left[ 1 - \boldsymbol{\phi}^{T}\boldsymbol{\Phi}^{-1}\boldsymbol{\phi} \right] \end{array} $$

(9)

$$\begin{array}{@{}rcl@{}} && \hat{\sigma}^{2}=\frac{\mathbf{y}^{T}\boldsymbol{\Phi}^{-1}\mathbf{y}}{n} \end{array} $$

(10)

In these equations ϕ is a vector with the correlation between point x and all observations x ⁽ⁱ⁾. The n×n matrix Φ contains the correlations between all observation points x ⁽ⁱ⁾. The model parameters 𝜃 and p are found through maximization of the likelihood function L(𝜃, p) (Forrester and Keane 2009):

$$ L(\boldsymbol{\theta},\mathbf{p}) = -\frac{n}{2}\ln(\hat{\sigma}^{2})-\frac{1}{2}\ln(\det(\boldsymbol{\Phi}))) $$

(11)

The exponents p are related to the smoothness of the function (Jones et al. 1998). Under assumption that the underlying function f is smooth, the values of p can be set to 2 and only the 𝜃 parameters have to be optimized. The ability to fit complex nonlinear models and to estimate the prediction uncertainty attracted many researchers in the field of sequential optimization to the use of kriging interpolation.

2.1.2 Radial basis functions

In RBF interpolation a set of n axisymmetrical basis functions ψ are placed at the location of each observation point x ⁽ⁱ⁾. The function ψ ⁽ⁱ⁾ is a function of the Euclidean distance between the points x ⁽ⁱ⁾ and x and may have additional parameters.

$$ \hat{f}(\mathbf{x}) = \sum\limits_{i=1}^{n} w_{i} \psi^{(i)} \left( \| \mathbf{x}^{(i)} - \mathbf{x} \| \right) $$

(12)

The n weight factors w _i can be solved with the n interpolation equations \(\hat {f}(\mathbf {x}^{(i)}) = y^{(i)}\), leading to w = Ψ ⁻¹ y. Hence, the prediction becomes:

$$ \hat{f}(\mathbf{x}) = \boldsymbol{\psi}^{T}\boldsymbol{\Psi}^{-1}\mathbf{y} $$

(13)

The resemblance to the kriging prediction (10) is clear. The procedure for construction of ψ and Ψ is identical to the procedure for ϕ and Φ. Choosing the Gaussian basis function (7) with the same model parameters obviously yields the same prediction. However, kriging is constrained to functions which are decaying from the center point, whereas any function may be used in the more general RBF formulation. Several basis functions have been proposed in literature, such as Gaussian, thin plate spline, multiquadric, inverse multiquadric and others (Fornberg and Zuev 2007). The Gaussian basis function is most frequently used, but the multiquadric basis function (Hardy 1971) has proven to have good predictive capability in several comparative studies (Franke 1982; Jin et al. 2001):

$$ \psi^{(i)}(x) = \sqrt{{c_{i}^{2}} + x^{2}} $$

(14)

The shape of the Gaussian and the multiquadric basis function with varying model parameters is shown in Fig. 2. A major advantage of RBF over kriging is the possibility to use different shape parameter values c _i at each observation point x ⁽ⁱ⁾. This is especially desired in the case of unevenly distributed DOE’s, such as the ones obtained by sequential optimization (Fornberg and Zuev 2007).

Open image in new window

As stated before, an estimate of the prediction uncertainty is required for sequential sampling. Some researchers derived formulations for the prediction uncertainty for RBF. Li et al. (2010) fit a RBF with Gaussian basis functions with a single parameter for the kernel width using LOOCV. They assume the prediction \(\hat {f}(\mathbf {x})\) to be a realization of a stochastic process and assume the variance of the stochastic process to be ψ(0)−ψ ^T Ψ ⁻¹ ψ (Gibbs 1997; Sóbester et al. 2004; Ji and Kim 2013). This uncertainty measure is applicable for basis functions which are decaying from the center point. A more general uncertainty measure is given by Yao et al. (2014), which derive the relation between the unknown weight factor w at an untried point x with the error of the prediction \(f(\mathbf {x}) - \hat {f}(\mathbf {x})\). They predict the weight factor \(\hat {w}(\mathbf {x})\) by interpolating the weight factors with a RBF model with the same model parameters as the underlying model for the prediction \(\hat {f}(\mathbf {x})\). The disadvantage of this measure is that the prediction uncertainty reduces to zero for points which are far away from all DOE points. Nikitin et al. (2012) propose to directly interpolate the LOOCV values at all observation points. However, it is not clear how this uncertainty measure should be used for sequential optimization, since the uncertainty value at the DOE points is not equal to zero. Therefore, we propose a new uncertainty measure in Section 3 with two parameters which is determined with the LOOCV values and a likelihood function.

2.2 Sequential improvement

Due to the computational cost of the simulations, efficient exploration of the design space is required. This can be achieved with sequential improvement strategies. The concepts of sequential improvement will be first explained for the deterministic case with the objective to minimize f and thereafter the robust optimization case will be discussed.

An important question is whether to search in the neighbourhood of the optimum of the predictor \(\hat {f}\) or to search in a region with high prediction uncertainty. The former is denoted as local search and can be achieved by sampling \(\min _{\mathbf {x}} \hat {f}(\mathbf {x})\) whereas the latter is denoted as global search and can be achieved by sampling \(\max _{\mathbf {x}} \hat {s}^{2}(\mathbf {x})\).

Jones et al. (1998) proposed a sequential improvement criterion that combines local and global search in a single measure. The lowest value of y in the current dataset D is y _min = min(y). An improvement function is defined as max{y _min−y,0}. The predicted value at an unknown point x has the probability \(p_{y} \sim \mathcal {N}(\hat {f}(\mathbf {x}),\hat {s}^{2}(\mathbf {x}))\). Hence, the expected value of the improvement at x is:

$$\begin{array}{@{}rcl@{}} EI = {\int}_{-\infty}^{y_{\min}}\!(y_{\min} - y) p_{y}\,\mathrm{d}y = \\ (y_{\min} - \hat{f} ) {\Phi} \left( \frac{y_{\min} - \hat{f}}{\hat{s}} \right) + \hat{s} \phi \left( \frac{y_{\min} - \hat{f}}{\hat{s}} \right) \end{array} $$

(15)

with ϕ and Φ the Probability Density Function (PDF) and the Cumulative Distribution Function (CDF) of a standard normal distribution. The term with Φ relates to local search and the term with ϕ relates to global search. Therefore, Sóbester et al. (2004) proposed to include a weight factor w∈[0,1] to be able to adapt the inclination towards local or global search:

$$ EI = w (y_{\min} - \hat{f} ) {\Phi} \left( \frac{y_{\min} - \hat{f}}{\hat{s}} \right) + (1-w) \hat{s} \phi \left( \frac{y_{\min} - \hat{f}}{\hat{s}} \right) $$

(16)

When constraints are included in the optimization problem, it is unwise to sample new points at locations with high probability of not fulfilling the constraints. Therefore, it is proposed to multiply the expected improvement with the probability of fulfilling the constraints (Lehman et al. 2004; Forrester and Keane 2009):

$$ CEI \left( \mathbf{x} \right) = EI \left( \mathbf{x} \right) P \left[ \text{C} (\mathbf{\mathbf{x}} ) \leq 0 \right] $$

(17)

In the case of robust optimization a clear distinction should be made between uncertainty over the predictor \(\hat {f} (\mathbf {x},\mathbf {z})\) which is defined in (x, z) and uncertainty over the objective function value \(\text {Obj} (\hat {Y} (\mathbf {x}) )\) which is defined in x. Estimating the latter enables the use of an improvement criterion similar to (15). In the case of using μ _Y + kσ _Y as objective function the mean MSE can be used as an estimate of the objective function MSE (Jurecka et al. 2007; Wiebenga 2014):

$$ \hat{s}_{\text{Obj}}^{2}(\mathbf{x}) = {\int}_{\mathbf{z}}\hat{s}^{2}(\mathbf{x},\mathbf{z}) p_{\mathbf{Z}}(\mathbf{z})d\mathbf{z} $$

(18)

The sequential improvement procedure for robust optimization is split in two steps: first a location in the design space x is selected based on expected improvement, thereafter the location in the noise space z is selected. At any point in the design space x an estimate of the objective function value \(\text {Obj} (\hat {Y} (\mathbf {x}) )\) and an estimate of its uncertainty \(\hat {s}_{\text {Obj}} (\mathbf {x}) \) can be determined. In contrast to the deterministic optimization case, there is not a clearly defined best point y _min in the design space, since the prediction error \(\hat {s}_{\text {Obj}} (\mathbf {x})\) is larger than zero for any x. Some authors propose to define the best point x _b as the point {x _b ∈x ⁽ⁱ⁾:i = 1…n} with the lowest value for the objective function prediction (Jurecka 2007; Wiebenga et al. 2012). In this work a different criterion is used. The criterion and the reasoning behind it are discussed in Section 6.2.

Now a criterion is needed to quantify whether a candidate infill point x can potentially yield an improvement over the current best solution x _b , given \(\text {Obj}(\hat {Y}(\mathbf {x}))\), \(\hat {s}_{\text {Obj}}(\mathbf {x})\), \(\text {Obj}(\hat {Y}(\mathbf {x}_{b}))\) and \(\hat {s}_{\text {Obj}}(\mathbf {x}_{b})\). One approach is to ignore the prediction error at the best point \(\hat {s}_{\text {Obj}}(\mathbf {x}_{b})\) and use the criterion of (15) (Jurecka et al. 2007). Note that selecting the current best point as candidate infill point x = x _b yields a nonzero value of the expected improvement. Clearly it is desired to sample more points at x _b to decrease the prediction error of the current best estimate.

To our knowledge there has been only one criterion proposed which includes the prediction uncertainty \(\hat {s}_{\text {Obj}}(\mathbf {x}_{b})\) and is suitable for metamodel based sequential robust optimization. Jurecka (2007) proposed to replace the probability p _y in (15) with \(\max \{p_{\text {Obj}(\mathbf {x})}-p_{\text {Obj}(\mathbf {x}_{b})},0\}\). A graphical representation of the new probability is shown in Fig. 3. Note that \(\int p_{y} \leq 1\). The reasoning behind this approach is that the probability for a certain objective function value at x should only be taken into account if it is higher than the probability of the same outcome at x _b . However, a new DOE point will never be picked at x _b because the expected improvement value at x = x _b is zero.

Open image in new window

The last step of the sequential improvement procedure is to select a location in the noise space z at the selected location in the design space x ^′. The purpose of the new simulation is to improve the prediction of \(\text {Obj}(\hat {Y}(\mathbf {x}^{\prime }))\). Hence criteria such as \(\arg \!\max _{\mathbf {z}} \hat {s}^{2} (\mathbf {x}^{\prime },\mathbf {z})\) (Jurecka et al. 2007) or \(\arg \!\max _{\mathbf {z}} (\hat {s}^{2} (\mathbf {x}^{\prime },\mathbf {z}) p_{\mathbf {Z}}(\mathbf {z}))\) (Jurecka 2007; Wiebenga et al. 2012) can be used for this purpose.

The abovementioned sequential improvement criteria can be regarded as global improvement criteria, in the sense that a criterion which is defined in the complete function domain is to be minimized to obtain a new infill point. A different class of sequential improvement methods is given by trust region strategies. Trust region strategies work with metamodels (also referred to as approximation models) which can be trusted within a certain trust region. The size and position of the trust region and the metamodel itself are updated based on evaluations of a high-fidelity model in a local search procedure. Hence, the choice of a new infill point is determined by the search path of the algorithm. The algorithm has been extended from local quadratic approximation models to any kind of approximation models by Alexandrov et al. (1998). One specific application of trust region strategies is to solve optimization problems using variable fidelity models which are coupled through mapping functions. Gano et al. (2006) used kriging interpolation functions to map variable fidelity models in a trust region optimization procedure. For further reading we refer to the application of trust region strategies to different engineering problems, such as the design of an autonomous hovercraft (Rodríguez et al. 2001) and the weight minimization of a fiber-reinforced cantilever beam (Zadeh et al. 2009).

This concludes the overview of the building blocks of the robust sequential optimization algorithm. In Section 3 an implementation of a RBF model is proposed, including estimation of the prediction error. In Section 4 a novel expected improvement criterion for robust optimization is proposed, including a discussion on the several criteria available for robust optimization. The effectiveness of these building blocks will be compared with other methods in Section 7.

As discussed in Section 2.1.2, there is a lot of freedom for the user on the exact implementation of the RBF method. Therefore, a proposal is made for an implementation for efficient sequential optimization. The procedure is elaborated in this section and the implementation choices are discussed.

First, the scaled distance to the nearest neighbor d _i is determined with (21). Thereafter, the distance to the nearest neighbor is normalized with (22), such that c _i ≤1∀c _i holds. The latter operation is needed, to ensure that each value for the scaling parameter 𝜃 results in a unique metamodel. Note that the scaling parameter 𝜃 has not yet been selected and can be freely chosen. When the scaling parameter 𝜃 is multiplied with a constant C, the scaled distance to the nearest neighbors d _i changes accordingly (d _i (C 𝜃) = Cd _i (𝜃)). If the local shape parameters are set to the distance to the nearest neighbor (c _i = d _i ), the relative effect of basis function ψ ⁽ⁱ⁾ throughout the scaled model space (𝜃∘x) will remain the same when multiplying the scaling parameter 𝜃 with a constant. Hence, all models with parameter sets C 𝜃 ∀C > 0 will be equal. To avoid this effect, the normalization from (22) is used.

In Fig. 4 the effect of this approach is illustrated. With constant shape parameters either a bad prediction in sparse sampled areas or severe oscillations in dense sampled areas and a badly conditioned matrix Ψ are obtained. However, scaling the shape factors c _i based on the local point density yields a good fit in all regions the model. Note that changing the shape parameters c _i changes the width of the basis function and the effect of the scaling parameter 𝜃.

Open image in new window

The optimal scaling parameters 𝜃 are determined by minimization of the L ₂ norm of the error ||𝜖||₂. The number of components of 𝜃 equals the number of dimensions of the metamodel.

The full derivation can be found in the Appendix. With zero uncertainty \(\hat {s}_{b}\) of the objective at x _b the equation reduces to (15).

In Sections 5 and 7 the efficiency of the novel criterion for sequential improvement is compared with the efficiency of the criteria of Jones et al. (1998) and Jurecka (2007). Some observations on the differences between these criteria can be made when investigating the functions. At first, the three criteria are identical when the objective uncertainty at the best point is zero. In Fig. 5 the expected improvement functions are shown for the case that the uncertainty at the best point x _b is found to be \(\hat {s}_{b} = 0.4\). Obviously, point x _b is a candidate infill point because it can be sampled at a new location in the noise space z. Therefore, a candidate point x should have an expected value of at least EI(x _b ) to be selected as the new infill point. In the figures, the value of EI(x _b ) is marked with a circle ∘. Looking at the difference between the Jones citerion (Fig. 5a) and the novel criterion (Fig. 5c), it is clear that the differences in the region with EI(x)<EI(x _b ) are unimportant since the new infill point will never be selected from this region. In the region with EI(x)>EI(x _b ) the differences between both criteria are small. In Section 7 it will be shown whether these differences affect the efficiency of the criteria.

Open image in new window

When observing the criterion of Jurecka (2007) (Fig. 5b) some remarkable characteristics are seen. First observation is that the expected improvement value at x _b equals zero. Therefore, the current best point will never be selected as the new infill point. Also a large region has an EI value close to zero, which makes it difficult for the algorithm to distinct between regions with high and low probability of improvement. Furthermore it is seen that in some regions the EI value decreases with increasing uncertainty \(\hat {s}\), such as around \((\hat {\mu }_{b}-\hat {\mu } = 0.2, \hat {s}=0)\). However if two points would have the same expected value \(\hat {\mu }\), the point with the highest uncertainty \(\hat {s}\) should be preferred. In the Sections 5 and 7 it is examined how these properties of the Jurecka criterion influence its efficiency.

The efficiency of the sequential improvement criteria and metamodeling methods for robust optimization have been compared based on a robust optimization test with a large set of Gaussian random fields. The approach and results are presented in Section 5.1 for the sequential improvement criterion and in Section 5.2 for the metamodeling methods.

5.1 Sequential improvement criterion

The efficiency of the three sequential improvement criteria for robust optimization have been compared based on an optimization test with mathematical functions. To eliminate the influence of the metamodeling technique a set of 15000 Gaussian random fields f(x, z) with predefined statistical parameters has been generated. Therefore, the kriging predictor can be used to obtain the best prediction and uncertainty measure given any set of observations D = {((x,z)⁽ⁱ⁾, f(x,z)⁽ⁱ⁾)|i = 1…n}. The random fields have one design parameter x and one noise parameter z in the range [0,1]. The correlation function of the random fields is Gaussian (7) with model parameters p _x = 2 and p _z = 2. The model parameters 𝜃 _x and 𝜃 _z are selected randomly from the respective sets {30,40,50…300} and {30,40,50…100}. The probability distribution of the random field f(x,z) is \(p_{f} \sim \mathcal {N}(0,1^{2})\). The probability of the noise parameter z is given by a truncated normal distribution with mean 0.5, standard deviation 0.1, truncated outside the range [0,1]. During the optimization the selection of new infill points is restricted to an equidistant grid of 25 points in x-direction and 21 points in z-direction. As an initial DOE the center point of the field (x,z)=(0.5,0.5) is taken and a total of 50 sequential improvement iterations are performed.

The sequential optimization procedure has been performed for all 15000 random fields with the three optimization algorithms. Selection of z is done with the criterion \(\arg \!\max _{z} (\hat {s}^{2} (x^{\prime },z) p_{z}(z))\). The results are shown in Fig. 6. The correct robust optimum was found after 50 iterations in 92.1 %, 61.3 % and 93.1 % of the cases for the Jones, Jurecka and novel criterion respectively. The novel criterion (Fig. 6c) is slightly more efficient than the Jones criterion (Fig. 6a). It should be noted that the extra computational cost of the novel criterion is negligible since the only difference is that the objective uncertainty \(\hat {s}_{\text {Obj}}\) at the current best point x _b has to be included. The statistical significance of the conclusion that the novel criterion is more efficient than Jones’ criterion can be tested with McNemar’s test for paired nominal data (McNemar 1947). Table 1 shows the number of correct solutions after 50 iterations for both criteria. The null hypothesis states that the probability that Jones’ criterion and the novel criterion find the correct solution is the same. The test statistic is χ ² = (513−367)²/(513+367). The statistic has a chi-squared distribution and the null hypothesis is rejected with a p-value of 8.6e-7. Hence, a statistically significant improvement with the new criterion is found in the test with 2-D Gaussian random fields.

Open image in new window

Table 1

Contingency table for results of Jones’ criterion and the novel criterion, stating whether the correct or wrong solution has been found after 50 iterations

		Novel criterion
		correct	wrong
Jones	correct	13450	367
	wrong	513	670

The convergence of the Jurecka criterion is shown in Fig. 6b. During the first 10 iterations the average convergence is similar to the average convergence of the other methods but thereafter the convergence rate drops. Furthermore it is observed that the Jurecka criterion fails to select the correct robust optimum after 50 iterations in 38.7 % of the runs. One cause for the poor convergence is that the current best optimum x _b is never selected as an infill point since the expected improvement value at x _b equals zero (Fig. 5b). Therefore, it may happen that an underestimated value of the current best optimum x _b is not verified by additional samples, leading to erroneous results.

The mathematical test reveals performance differences between the three expected improvement criteria. However limited conclusions should be drawn since only one type of function (2D Gaussian random field) with one metamodeling method (kriging) have been studied. Therefore, all expected improvement criteria are included in the study on the efficiency of the robust optimization procedure of a V-bending process with a Finite Element (FE) model with seven variables. The study is presented in Section 6.

5.2 Metamodeling method

As reported in the comparative studies of Franke (1982) and Jin et al. (2001), different problems may require different types of metamodels. To reflect this, four different sets of 2000 Gaussian random fields have been generated for a study on the effect of metamodeling methods on the efficiency of the optimization procedure. Each set has fixed statistical parameters. Two of these sets have additional normal distributed white noise with standard deviation σ _n. The parameters of the different sets are listen in Table 2.

Table 2

Overview of Gaussian random field parameters for the metamodel test

set nr.	1	2	3	4
\(\theta _{\text {x}}\)	40	40	40	40
\(\theta _{\text {z}}\)	40	100	40	100
\(\sigma _{\text {n}}\)	0	0	0.2	0.2

For each Gaussian random field, the same optimization procedure has been followed as in Section 5.1, except for the details mentioned in this paragraph. The novel criterion for sequential improvement has been used. A small initial DOE is used, with a full factorial design (four points) and six additional latin hypercube design points. Thereafter, 50 sequential optimization iterations have been performed. During the optimization the selection of new infill points is restricted to an equidistant grid of 101 points in x-direction and 21 points in z-direction. As metamodeling methods, kriging, RBF with Gaussian basis functions (RBFG) and RBF with multiquadric basis functions (RBFMQ) have been used. A constant value has been used to detrend the data for all metamodeling methods. Note that fitting of the kriging model requires determination of the statistical parameters (11), whereas the kriging model in Section 5.1 was fit with known statistical parameters.

It does not make sense to assess the quality of the metamodels with the estimate of the metamodel uncertainty at the real robust optimum, as done in Section 5.1, because the uncertainty estimates of the metamodels are not exact, whereas the best possible metamodel (wich exact uncertainty estimate) was used in the previous section. Therefore, another measure is used to assess the quality of the metamodels. In each run, the optimum is determined after each iteration and the difference between the real objective function value at this predicted optimum and the real objective function value at the real optimum is defined as the potential improvement PI:

$$ \text{PI} = \text{Obj}(\hat{\mathbf{x}}_{\text{opt}}) - \text{Obj}(\mathbf{x}_{\text{opt}}) $$

(30)

The average potential improvement over all 2000 runs in each set is shown in Fig. 7. With increasing number of iterations, the potential improvement converges to zero. For the runs without noise (σ _n = 0, Fig. 7a and b), the kriging metamodel performs best. This is expected, because the studied functions are Gaussian random fields, which are built with the same assumptions as used for kriging interpolation. However, kriging performs worst for the case with additional noise (σ _n = 0.2, Fig. 7c and d). The RBF metamodels are less sensitive to additional noise because the distance between DOE points is taken into account in the implementation (22). Therefore, large differences in function value for nearby points due to noise do not have a large effect on the metamodel quality. This example is relevant for metamodel based optimization with numerical models, because numerical models may suffer from numerical noise (Wiebenga and van den Boogaard 2014).

Open image in new window

A quantitative comparison between the effect of different metamodels on the robust optimization procedure is given in Table 3. For each run, the real objective function value is determined at the predicted optimum. In a comparison between two types of metamodels, it is listed how frequently each metamodel yields a better solution after 50 iterations. Based on McNemar’s statistical test for paired data, the statistical relevance of each comparison is listed (the procedure for determining the p-value with McNemar’s test has been discussed in Section 5.1). The results with a p-value lower than 0.001 are regarded as statistically significant. It can be concluded for this test that kriging has a higher chance of finding a lower objective function value in the case without noise, whereas RBFMQ has a higher chance than kriging of finding a better optimum when noise is present.

Table 3

Comparison between metamodeling methods based on the real objective function values at the predicted optima after 50 iterations. Statistical significant results (p-value smaller than 0.001) have a gray background

Open image in new window

The comparison of the effect of metamodeling methods on the efficiency of the robust optimization procedure shows that the choice for the best metamodel depends on the problem at hand. To assess the effect of the discussed metamodeling methods in a real engineering problem, the robust optimization problem of a V-bending metal forming process is presented in the following section.

At first, it can be seen by the characteristics of different optima that the process is strongly nonlinear (Fig. 11e–h). The influence of the thickness and the yield stress on the main angle 𝜃 _M is strongly dependent on the design variables. A striking example of the nonlinearity is the local minimum of the angle for the Jones criterion - RBFG0 run at a thickness of 0.52 mm (Fig. 11e), whereas the Jurecka criterion - RBFMQ0 optimum has a local maximum at the same thickness (Fig. 11g). Hence, is should be clear that the strip bending optimization problem has strong interactions between noise variables and design variables. Therefore, it is not surprising that the metamodel prediction of the optimum is of fluctuating quality (Fig. 11a–d).

Open image in new window

Obviously, it is not possible to know the global optimum for sure without performing an excessive amount of FE model runs. Therefore, it is not feasible to compare the convergence of the global optimum for all runs. However, the goal of a robust optimization run is to find a good solution of the optimization problem with moderate computational effort. Looking at the results in Table 5, it can be seen that the found optima have a checked objective function value in between 0.14° (novel criterion - RBFG0) and 0.72° (Jurecka - RBFG0). The objective function value is underestimated for 15 out of 18 runs. Only one run has constraint violations. Furthermore the RMSE (\(\hat {s}_{\text {Obj}}\)) is underestimated in 17 out of 18 runs for both the 𝜃 _M metamodel as well as for the 𝜃 _T metamodel. This is expected since the optimum is selected based on \(\text {Obj} (\hat {Y} (\mathbf {x}) ) + 6 \hat {s}_{\text {Obj}} (\mathbf {x})\), which includes both objective function value as well as RMSE value. Therefore, it is more probable for the optimum to be selected from regions where these values are underestimated. In general it can be said that the robust sequential optimization algorithm yields good results for the V-bending problem. Nearly all results fulfill the constraints and satisfying objective function values have been found. The reference for the objective function value is the best result found with the previous study performed with the same model, where an objective function value of 0.85° was found using four design variables (Wiebenga et al. 2012).

The number of sequential optimization runs is not sufficient to assess the statistical relevance of the results. Hence, it cannot be stated which metamodeling method or sequential improvement criterion is most efficient for robust optimization of the V-bending problem. For statistical relevant results, we refer to the test with mathematical functions in Section 5. However, the V-bending problem results give one example for the performance of the building blocks of the procedure. More general results may be obtained through new studies on the efficiency of the building blocks for other engineering problems. Furthermore, several other conclusions can be drawn based on detailed observation of the results.

Regarding the sequential improvement criteria, mean objective function values of 0.37°, 0.42° and 0.24° are obtained for the Jones, Jurecka and novel criterion runs respectively (Fig. 12). It is worth mentioning the run with Jurecka criterion and RBFMQ1 metamodel. During the first 190 iterations, 183 iterations had an expected improvement value smaller than 10⁻¹⁰ degrees. Hence, the algorithm was not able to distinguish between regions with high and low probability of improvement. During these iterations new DOE points were added throughout the whole domain, until the uncertainty in the best point \(\hat {s}_{\text {Obj}}(\mathbf {x}_{b})\) decreased, enabling the algorithm to find regions with a higher expected improvement value. This artifact clearly influences the performance of the criterion, as discussed in Section 4. However, it is noticeable that finally a good objective function value of 0.18° was found with this run.

Open image in new window

Looking at the metamodeling techniques, the RBF runs outperformed the kriging runs with a small margin, with final mean objective function values of 0.40°, 0.33° and 0.30° for the kriging, Gaussian RBF and multiquadric RBF runs respectively (Fig. 13). Another measure for the quality of the metamodels is their R ² value. This has been estimated for all iterations based on 5000 simulations, sampled with a latin hypercube design (Fig. 14). The R ² value is a measure of the predictive quality of the metamodels throughout the full domain, although it is only of interest how the regions with low objective function value are predicted. However, Fig. 14 is of interest because it gives an impression about the stability of the metamodeling methods. During some iterations a strong drop in the R ² value is observed. Recall that the fitting procedure for kriging and RBF includes the optimization of model parameters based on a likelihood measure. This measure can be strongly affected by the addition of an extra DOE point, yielding to a different set of model parameters and a sudden drop in the R ² value of the metamodel. It may be expected that these fluctuations in the predictive capability of the metamodel negatively affect the efficiency of the optimization. From Fig. 14 it can be seen that the RBF metamodels suffer less from these strong variations in global fitting accuracy.

Open image in new window

Open image in new window

In Fig. 15 the estimated and checked objective function values after every 50 iterations are shown. It would be expected that the objective function value and the prediction uncertainty decrease with increasing number of iterations. However, this is not always the case. Refitting the metamodels at every iteration leads to changing model parameters and subsequently to a changing prediction of the optimum. On average over all runs the predicted and checked objective function values and the prediction uncertainty decrease with increasing number of iterations.

Open image in new window

In this work the effect of the metamodeling method and the sequential improvement criterion on the efficiency of the robust optimization procedure is investigated.

The only available criterion for sequential improvement in robust optimization was the Jurecka criterion. However, this criterion has some artifacts, such as that, when two points have the same prediction for the objective function value, the point with the largest prediction uncertainty is not always preferred as new sampling point. Furthermore, the current estimate of the best design point has an expected improvement value of zero by definition. Therefore, an erroneously underpredicted objective function value will not be verified with additional simulations, even when the prediction uncertainty is large. Therefore, we propose a new criterion for sequential improvement in robust optimization, which is derived from the deterministic expected improvement criterion of Jones et al. (1998). This criterion shows better performance than the Jurecka criterion and than the deterministic Jones criterion, both in a test with mathematical test functions (Section 5) as well as in a V-bending optimization problem (Section 7). Whereas the validity of this conclusion is shown to be statistically significant for the test with mathematical test functions, no statistical test could be performed based on the small number of optimization runs in the V-bending test. It is recommended to verify the presented observations with new studies on robust optimization in engineering.

Regarding the metamodeling method, kriging has been used by many researchers for sequential optimization. We propose an improved implementation of RBF suitable for sequential optimization, including an estimation of the metamodel uncertainty. In a robust optimization test with mathematical test functions it was shown with statistical significance that kriging metamodels perform best when no noise is present on the black-box function, whereas the proposed implementation of the RBF interpolation function with multiquadric basis functions performs best when noise is present. Hence, it is shown that the choice for the best metamodeling method for robust optimization should depend on the investigated optimization problem. In the test with the optimization of a strongly nonlinear V-bending process the RBF metamodels outperformed the kriging metamodels with a small margin. Furthermore the RBF metamodels proved to suffer less from fluctuations in their global predictive capability throughout the optimization procedure.

Given these observations, we believe that the novel robust sequential improvement criterion as well as the proposed implementation of RBF may be successfully used for other applications in the field of robust optimization and metamodel based optimization respectively. Obviously, the effect on the optimization efficiency may be strongly dependent on the optimization problem at hand. Therefore, we believe that there is no general solution for an optimal sequential optimization algorithm. However, knowing the pros and cons of the available building blocks may significantly improve the chances of success in robust optimization.