Ensemble of metamodels: extensions of the least squares approach to efficient global optimization

Abstract

In this work we present LSEGO, an approach to drive efficient global optimization (EGO), based on LS (least squares) ensemble of metamodels. By means of LS ensemble of metamodels it is possible to estimate the uncertainty of the prediction with any kind of model (not only kriging) and provide an estimate for the expected improvement function. For the problems studied, the proposed LSEGO algorithm has shown to be able to find the global optimum with less number of optimization cycles than required by the classical EGO approach. As more infill points are added per cycle, the faster is the convergence to the global optimum (exploitation) and also the quality improvement of the metamodel in the design space (exploration), specially as the number of variables increases, when the standard single point EGO can be quite slow to reach the optimum. LSEGO has shown to be a feasible option to drive EGO with ensemble of metamodels as well as for constrained problems, and it is not restricted to kriging and to a single infill point per optimization cycle.

Appendices

Appendix: A: The kriging metamodel

The Kriging model, originally proposed by Krige (1951), is an interpolating metamodel in which the basis functions, as stated in (1), are of the form

$$ \psi^{(i)} = \psi \left( \left\|\mathbf{x}^{(i)} - \mathbf{x}\right\|\right) = \exp\left( -\sum\limits_{j=1}^{k} {\theta_{j} \left| x^{(i)} -x_{j} \right|^{p_{j}}} \right), $$

(12)

with tuning parameters 𝜃 _j and p _j normally determined by maximum likelihood estimates.

With the parameters estimated, the final kriging predictor is of the form

$$ \hat{f}(\mathbf{x}) = \hat{\mu} + \boldsymbol{\psi}^{T}\boldsymbol{\Psi}^{-1}\left( \mathbf{y} -\mathbf{1}\hat{\mu}\right), $$

(13)

where $\mathbf {y} = \left [y^{(1)}{\ldots } y^{(N)}\right ]^{T}$, 1 is a vector of ones, Ψ = ψ ^(r)(s) is the so called N × N matrix of correlations between the sample data, calculated by means of (12) as

$$ \boldsymbol{\Psi} =\psi \left( \left\|\mathbf{x}^{(r)} - \mathbf{x}^{(s)} \right\|\right) $$

(14)

and $\hat {\mu }$ is given by

$$ \hat{\mu} = \frac{\mathbf{1}^{T}\boldsymbol{\Psi}^{-1}\mathbf{y}}{\mathbf{1}^{T}\boldsymbol{\Psi}^{-1}\mathbf{1}}. $$

(15)

One of the key benefits of kriging models is the provision of uncertainty estimate for the prediction (mean squared error, MSE) at each point x, given by

$$ \hat{s}^{2}(\mathbf{x}) = \hat{\sigma}^{2}\left[1 - \boldsymbol{\psi}^{T}\boldsymbol{\Psi}^{-1}\boldsymbol{\psi} + \frac{1-\mathbf{1}^{T}\boldsymbol{\Psi}^{-1}\mathbf{y}}{\mathbf{1}^{T}\boldsymbol{\Psi}^{-1}\mathbf{1}}\right], $$

(16)

with variance estimated by

$$ \hat{\sigma}^{2} = \frac{\left( \mathbf{y} -\mathbf{1}\hat{\mu}\right)^{T}\boldsymbol{\Psi}^{-1}\left( \mathbf{y} -\mathbf{1}\hat{\mu}\right)}{N}. $$

(17)

Refer to Forrester et al. (2008) or Fang et al. (2006) for further details on metamodel formulation.

Appendix: B: Analytical benchmark functions

These functions were chosen since they are widely used to validate both metamodeling and optimization methods, as for example in and Jones et al. (1998) and Viana et al. (2013).

Branin-Hoo

$$\begin{array}{@{}rcl@{}} y\left( \mathbf{x}\right) & = &\left( x_{2} + \frac{5.1{x_{1}^{2}}}{4\pi^{2}} + \frac{5x_{1}}{\pi} -6\right)^{2}\\ & &+ 10\left( 1-\frac{1}{8\pi}\right)\cos\left( x_{1}\right) + 10, \end{array} $$

(18)

for the region − 5 ≤ x ₁ ≤ 10 and 0 ≤ x ₂ ≤ 15. There are 3 minima in this region, i.e., $\mathbf {x}^{\ast } \approx \left (-\pi , 12.275 \right ), \left (\pi , 2.275 \right ), \left (3\pi , 2.475 \right )$ with $f\left (\mathbf {x}^{\ast }\right ) = \frac {5}{4\pi }$.

Hartman

$$ y(\mathbf{x})= -\sum\limits_{i=1}^{4}{c_{i}\exp\left[ -\sum\limits_{j=1}^{n_{v}}{a_{ij}\left( x_{j} - p_{ij} \right)^{2}} \right]} , $$

(19)

where $x_{i} \in \left [0, 1\right ]^{n_{v}}$, with constants c _i , a _{i j} and p _{i j} given in Table 3, for the case n _v = 3 (Hartman-3); and in Tables 4 and 5, for the case n _v = 6 (Hartman-6).

Table 3 Data for Hartman-3 function

Table 4 Data for Hartman-6 function, c _i and a _{i j}

Table 5 Data for Hartman-6 function, p _{i j}

In case of Hartman-3, there are four local minima,

$$\mathbf{x}_{local} \approx \left( p_{i1},p_{i2},p_{i3}\right),$$

with f _{l o c a l} ≈−c _i and the global minimum is located at

$$\mathbf{x}^{\ast} \approx \left( 0.114614, 0.555649, 0.852547 \right),$$

with $f\left (\mathbf {x}^{\ast } \right ) \approx -3.862782$.

In case of Hartman-6, there are four local minima,

$$\mathbf{x}_{local} \approx \left( p_{i1},p_{i2},p_{i3},p_{i4},p_{i5},p_{i6}\right), $$

with f _{l o c a l} ≈−c _i and the global minimum is located at

$$\mathbf{x}^{\ast} \approx (0.201690, 0.150011,0.476874, $$

$$\ \ \ \ \ \ \ \ \ \ \ 0.275332, 0.3111652, 0.657301), $$

with $f\left (\mathbf {x}^{\ast } \right ) \approx -3.322368$.

Giunta-Watson

This is the “noise-free” version of the function used by Giunta and Watson (1998)

$$ y(\mathbf{x})=\sum\limits_{i=1}^{n_{v}}\left[ \frac{3}{10}+\sin \left( \frac{16}{15}x_{i}-1\right) +\sin^{2}\left( \frac{16}{15}x_{i}-1\right) \right] , $$

(20)

where $\mathbf {x} \in \left [-2, 4\right ]^{n_{v}}$.

Appendix: C: SURROGATES Toolbox

The SURROGATES Toolbox (ref. Viana 2009) is a Matlab based toolbox that aggregates and extends several open-source tools previously developed in the literature for design and analysis of computer experiments, i.e., metamodeling and optimization. We used the version v2.0, but v3.0 already includes EGO variants.⁵

The SURROGATES Toolbox uses the following collection of third party software published: SVM by Gunn (1997), DACE by Lophaven et al. (2002), GPML by Rasmussen and Williams (2006), RBF by Jekabsons (2009), and SHEPPACK by Thacker et al. (2010). The compilation in a single framework has been implemented and applied in previous research by Viana and co-workers, as for example Viana et al. (2009) and Viana (2011).

Appendix: D: A note on sequential sampling vs. one-stage approach

In Ferreira (2016) we investigated some examples with analytical engineering functions. We repeated the one-stage optimization ten times, with different initial DOE, at a very large rate of number of sampling points in terms of number of variables,⁶ i.e., for f ₁(x) of Three-Bar Truss N = 120 (60n _v ); for Cantilever Beam N = 120 (60n _v ); for Helical Spring N = 360 (120n _v ) and for Pressure Vessel N = 460 (120n _v ).

The results for this experiment are presented in Fig. 18. For the cases investigated, the results showed that there is no guarantee to achieve the exact optimum with a one-stage approach, even starting the optimization with a high density of sampling points in the design space. Probably, the majority of these points are working only for improving the overall quality of the metamodels (exploration) and these points are not being effective to help finding the exact minimum (exploitation), what is clearly a waste of resources for optimization objectives in mind.

Fig. 18

Boxplots with the converged results for analytical engineering functions with one-stage optimization. Each problem was repeated 10 times with different initial DOE, i.e., for f ₁(x) of Three-Bar Truss N = 120 (60n _v ); for Cantilever Beam N = 120 (60n _v ); for Helical Spring N = 360 (120n _v ) and for Pressure Vessel N = 460 (120n _v ). Even with very dense number of initial sampling points, there is no guarantee of achieving the exact optimum. For details refer to Ferreira (2016)

These results confirm our beliefs that it is worthwhile to apply sequential sampling approaches like EGO-type algorithms, or some hybrid approach (allied to clustering, for instance), in order to increase the number of points slowly and “surgically” at regions of the design space, with real chance or expectation of improvement in the objective and constraint responses.

In this sense, we reinforce the comments of Forrester and Keane (2009), that the metamodel based optimization must always include some form of iterative search and repetitive infill process to ensure the accuracy in the areas of interest in the design space. In this direction, we agree on the recommendations that a reasonable number of points for starting the sequential sampling metamodel based optimization is about one third (33%) of the available budget in terms of true function/model evaluations (or processing time) to be spent in the whole optimization cycle.