An adaptive-topology ensemble algorithm for engineering optimization problems

Abstract

Modern engineering design optimization often relies on computer simulations to evaluate candidate designs, a scenario which formulates the problem of optimizing a computationally expensive black-box functions. In such problems, there will often exist candidate designs which cause the simulation to fail, and this can degrade the optimization effectiveness. To address this issue, this paper proposes a new optimization algorithm which incorporates classifiers into the optimization search. The classifiers predict which candidate design are expected to cause the simulation to fail, and their prediction is used to bias the search towards valid designs, namely, for which the simulation is expected to succeed. However, the effectiveness of this approach strongly depends on the type of metamodels and classifiers being used, but due to the high cost of evaluating the simulation-based objective function it may be impractical to identify by numerical experiments the most suitable types of each. Leveraging on these issues, the proposed algorithm offers two main contributions: (a) it uses ensembles of both metamodels and classifiers to benefit from a diversity of predictions of different metamodels and classifiers, and (b) to improve the search effectiveness, it continuously adapts the ensembles’ topology during the search. The performance of the proposed algorithm was evaluated using an engineering problem of airfoil shape optimization. Performance analysis of the proposed algorithm using an engineering problem of airfoil shape optimization shows that: (a) incorporating classifiers into the search was an effective approach to handle simulation failures (b) using ensembles of metamodels and classifiers, and updating their topology during the search, improved the search effectiveness in comparison to using a single metamodel and classifier, and (c) it is beneficial to update the topology of the metamodel ensemble in all problem types, and it is beneficial to update the classifier ensemble topology in problems where simulation failures are prevalent.

Appendices

Appendix 1: Ensemble metamodels

Metamodels serve as computationally-cheaper mathematical approximations of the true expensive objective function, and are typically interpolants which have been trained by using previously evaluated vectors. In this study, the proposed algorithm used a metamodel ensemble comprising of two metamodel types (Simpson et al. 2001b):

• Radial Basis Functions (RBF) The metamodel approximates the objective function by a superposition of basis functions of the form

  $$\begin{aligned} \phi _{ i } ( {\varvec{x}}) = \phi ( ||{\varvec{x}}- {\varvec{x}}_{ i } ||_2 ) \,, \end{aligned}$$

  (18)

  where ${\varvec{x}}_i$ is an interpolation point. Given the training data ${\varvec{x}}_i \in \mathbb { R }^d,$ $i = 1 \ldots n,$ and corresponding objective values ${f}( {\varvec{x}}_i),$ the RBF metamodel is then given by

  $$\begin{aligned} {m}( {\varvec{x}}) = \sum _{i = 1}^{n} \alpha _i \phi _i( {\varvec{x}}) + c \,, \end{aligned}$$

  (19)

  where α _i and c are coefficients determined from the interpolation conditions

  $$\begin{aligned} {m}( {\varvec{x}}_i )&= {f}( {\varvec{x}}_i ) , \, i = 1 \ldots n\,,\end{aligned}$$

  (20a)
  $$\begin{aligned}&\sum _{i = 1 }^{ n } \alpha _i = 0 \, . \end{aligned}$$

  (20b)

  A common choice is the Gaussian basis function (Koehler and Owen 1996), defined as

  $$\begin{aligned} \phi _i( {\varvec{x}}) = \exp \left( -\frac{ {\varvec{x}}- {\varvec{x}}_i }{ \tau } \right) \,, \end{aligned}$$

  (21)

  where τ controls the width of the Gaussians and is determined by CV (Golberg et al. 1996). The set of interpolation equations (20) is often ill-conditioned, and so the proposed algorithm solves them using the singular value decomposition (SVD) procedure (Simpson et al. 2001b).

• Radial Basis Functions Network (RBFN) The metamodel aims to mimic the operation of a biological neural network. The metamodel uses RBFs as its ‘neurons’ or ‘processing units’, but in contrast to the RBF metamodel, it typically uses less RBFs than the number of sample points. This is done to improve the metamodel’s generalization ability, and therefore its accuracy of prediction. Given a set of sample points, ${\varvec{x}}_i,$ $i = 1 \ldots n,$ and their respective objective values, ${f}( {\varvec{x}}_i ),$ the RBFN metamodel is given by

  $$\begin{aligned} {m}( {\varvec{x}}) = \sum _{i = 1}^{p} \alpha _i \phi _i( {\varvec{x}}) \,, \end{aligned}$$

  (22)

  where the processing units $\phi _i( {\varvec{x}})$ are

  $$\begin{aligned} \phi _i( {\varvec{x}}) = \exp \left( -\frac{ {\varvec{x}}- \hat{ {\varvec{x}}}_i }{ \tau } \right) \,, \end{aligned}$$

  (23)

  and $p \le n.$ The neurons’ centres, $\hat{ {\varvec{x}}}_i,$ $i = 1 \ldots p,$ need to be determined such that they provide a good representation of the sample points, and an established approach to achieve this is to determine them by clustering (Bishop 1995). Accordingly, the proposed algorithm uses the k-means clustering algorithm to determine the centres. The parameter τ, which determines the width of the Gaussian functions of the neurons, is selected by CV (Golberg et al. 1996). The coefficients α are determined as the least-squares solution of the underdetermined linear set of equations

  $$\begin{aligned} \varPhi ^T \varPhi \varvec{ \alpha }&= \varPhi ^T {\varvec{ f }} \end{aligned}$$

  (24)

  where

  $$\begin{aligned} \varPhi _{ i, j }&= \phi _j( {\varvec{x}}_i ) \end{aligned}$$

  (25)

  where ${\varvec{x}}_i,$ $i = 1 \ldots n$ are the sample points, and f is the vector of corresponding objective function values. As mentioned in Sect. 3, the proposed algorithm used two RBFNs, one with $p = 0.8n$ and another with $p = 0.6n,$ which correspond to two different RBFN layouts, and therefore, two different approximations of the objective function.

Appendix 2: Ensemble classifiers

Classifiers originated in the domain on machine learning, and their goal is predict the class of new vectors, that is, given a set of vectors where each is associated with a group, a classifier associates a new input vector with one of these groups. Mathematically, given a set of vectors ${\varvec{x}}_i \in \mathbb { R }^d,$ $i = 1 \ldots n,$ which are grouped into several classes such that each vector has a corresponding class label ${F}( {\varvec{x}}_i ) \in \mathbb{I} ,$ for example, $\mathbb{I} = \lbrace -1, +1 \rbrace$  in a two-class problem, a classifier performs the mapping

$$\begin{aligned} {c}( {\varvec{x}}): \mathbb { R }^d \rightarrow \mathbb{I} \, , \end{aligned}$$

(26)

where ${c}( {\varvec{x}})$ is the class assigned by the classifier.

In this study, the proposed algorithm employs a classifier ensemble comprising of three established variants (Wu et al. 2008):

• k Nearest Neighbours (KNN) The classifier assigns the new vector the same class as its closest training vector, namely:

  $$\begin{aligned} {c}( {\varvec{x}}) = {F}( {\varvec{x}}_{\mathrm { NN} } ):d( {\varvec{x}}, {\varvec{x}}_{\mathrm { NN }} ) = \min \, d \bigl ( {\varvec{x}}, {\varvec{x}}_i \bigr ) , \; i = 1 \ldots n \, , \end{aligned}$$

  (27)

  where $d({\varvec{x}}, {\varvec{y}} )$ is distance measure such as the $l_2$ norm. An extension of this technique is to assign the class most frequent among the $k>1$ nearest neighbours (KNN). In this study the classifier used $k=3.$

• Linear Discriminant Analysis (LDA) In a two-class problem, where the class labels are $F ( {\varvec{x}}_i ) \in \mathbb{I} = \lbrace -1, +1 \rbrace,$ the classifier attempts to model the conditional probability density functions of a vector belonging to each class, where the latter functions are assumed to be normally distributed. The classifier considers the separation between classes as the ratio of: (a) the variance between classes, and (b) the variance within the classes, and obtains a vector w which maximizes this ratio. The vector w is such that it is orthogonal to the hyperplane separating the two classes. A new vector x is classified based on its projection with respect to the separating hyperplane, that is,

  $$\begin{aligned} {c}( {\varvec{x}}) = {{\mathrm{sign}}}{ \left( {\varvec{w}} \cdot {\varvec{x}} \right) } \, . \end{aligned}$$

  (28)

• Support Vector Machines (SVM) The classifier projects the data into a high-dimensional space where it can be more easily separated into disjoint classes. In a two-class problem, and assuming class labels $F ( {\varvec{x}}_i ) \in \mathbb{I} = \lbrace -1, +1 \rbrace,$ an SVM classifier tries to find the best classification function for the training data. For a linearly separable training set, a linear classification function is the separating hyperplane passing through the middle of the two classes. Once this hyperplane has been fixed, new vectors are classified based on their relative position to this hyperplane, that is, whether they are “above” or “below” it. Since there are many possible separating hyperplanes, an SVM classifier adds the condition that the hyperplane should maximize its distance to the its nearest vectors from each class. This is accomplished by maximizing the Lagrangian

  $$\begin{aligned} L_P = \frac{1}{2} ||{{\varvec{w}} } ||- \sum _{ i = 1 }^{ n } \alpha _i {F}( {\varvec{x}}_i ) ( {{\varvec{w}} } \cdot {{\varvec{x}} }_i + b ) + \sum _{ i = 1 }^{ n } \alpha _i \, , \end{aligned}$$

  (29)

  where n is the number of samples (training vectors), ${F}( {\varvec{x}}_i )$ is the class of the ith training vector, and $\alpha _i \geqslant 0,$ $i = 1 \ldots n,$ are the Lagrange multipliers, such that the derivatives of $L_P$ with respect to α _i are zero. The vector ${{\varvec{w}} }$ and scalar b define the hyperplane.