Delaunay-based derivative-free optimization via global surrogates, part I: linear constraints

Abstract

A new derivative-free optimization algorithm is introduced for nonconvex functions within a feasible domain bounded by linear constraints. Global convergence is guaranteed for twice differentiable functions with bounded Hessian, and is found to be remarkably efficient even for many functions which are not differentiable. Like other Response Surface Methods, at each optimization step, the algorithm minimizes a metric combining an interpolation of existing function evaluations and a model of the uncertainty of this interpolation. By adjusting the respective weighting of these two terms, the algorithm incorporates a tunable balance between global exploration and local refinement; a rule to adjust this balance automatically is also presented. Unlike other methods, any well-behaved interpolation strategy may be used. The uncertainty model is built upon the framework of a Delaunay triangulation of existing datapoints in parameter space. A quadratic function which goes to zero at each datapoint is formed within each simplex of this triangulation; the union of each of these quadratics forms the desired uncertainty model. Care is taken to ensure that function evaluations are performed at points that are well situated in parameter space; that is, such that the simplices of the resulting triangulation have circumradii with a known bound. This facilitates well-behaved local refinement as additional function evaluations are performed.

Appendix: Polyharmonic splines

The algorithms described above require the gradient and Hessian of the interpolant being used to facilitate Newton-based minimizations of the search function. Since our numerical tests all implement the polyharmonic spline interpolation formula, we now derive analytical expressions of the gradient and Hessian in this case.

The polyharmonic spline interpolation p(x) of a function f(x) in $\mathbb {R}^n$ is defined as a weighted sum of a set of radial basis functions $\varphi (r)$ built around the location of each evaluation point, plus a linear function of x:

$$\begin{aligned}&p(x) = \sum _{i = 1}^N w_i\,\varphi (r) + v^T \begin{bmatrix} 1 \\ x \end{bmatrix}, \nonumber \\&\quad \text {where} \quad \varphi (r) = r^3 \quad \text {and} \quad r = {||}x - x_i{||}. \end{aligned}$$

(83)

The weights $w_i$ and $v_i$ represent N and $n+1$ unknowns, respectively, to be determined through appropriate conditions. First, we match the interpolant p(x) to the known values of f(x) at each evaluation point $x_i$, i.e. $p(x_i) = f(x_i)$; this gives N conditions. Then, we impose the orthogonality conditions $\sum _i w_i = 0$ and $\sum _i w_i x_{ij} = 0, \quad j = 1,\,2,\,\ldots ,\,n$. This gives $n+1$ additional conditions. Thus,

$$\begin{aligned} \begin{bmatrix} F&V^T \\ V&0 \end{bmatrix} \begin{bmatrix} w \\ v \end{bmatrix}= & {} \begin{bmatrix} f(x_i) \\ 0 \end{bmatrix} \quad \text {where} \nonumber \\ F_{ij}= & {} \varphi ({||}x_i-x_j{||}) \quad \text {and} \nonumber \\ V= & {} \begin{bmatrix} 1&1&\ldots&1 \\ x_1&x_2&\ldots&x_N \end{bmatrix}. \end{aligned}$$

(84)

The gradient and Hessian of p(x) may now be written as follows:

$$\begin{aligned} \begin{aligned} \nabla p (x)&= \nabla \left( \sum _{i = 1}^N w_i {||}x - x_i{||}^3 + v^T\, \begin{bmatrix} 1 \\ x \end{bmatrix} \right) \\&= 3 \sum _{i = 1}^N w_i{||}x - x_i{||} (x-x_i) + \bar{v}, \end{aligned} \end{aligned}$$

where $\bar{v}=[v_2,v_3,\ldots ,v_{n+1}]^T$, and

$$\begin{aligned} \begin{aligned}&\nabla ^2 p(x) = \nabla ^2 \left( \sum _{i = 1}^N w_i {||}x - x_i{||}^3 + v^T\, \begin{bmatrix} 1 \\ x \end{bmatrix} \right) \\&= 3\sum _{i = 0}^N w_i \left( \frac{(x - x_i)(x - x_i)^T}{{||}x - x_i{||}} + {||}x - x_i{||} I_{n \times n} \right) . \end{aligned} \end{aligned}$$

Note that the calculation of the weights of a polyharmonic spline interpolant requires the solution of a $(N+n+1)\times (N+n+1)$ linear system. This system is not diagonally dominant, and does not show an easily-exploitable sparsity pattern facilitating fast factorization techniques. Nevertheless, since our algorithm adds only one point to the set of N evaluation points at each iteration, we can avoid the solution of the new linear system from scratch, and instead implement a rank-one update at each iteration as follows. First, for the set of initial points, we calculate the inverse $A = \bigl [ {\begin{matrix} F &{} V^T \\ V &{} 0 \end{matrix}} \bigr ]$. This step is somewhat time consuming, but reduces the computations required in subsequent steps. Using Matrix Inversion Lemma, we then update the inverse of A with the new information given at each step as follows:

$$\begin{aligned} A_{N+1}^{-1} = \begin{bmatrix} A_N&b^T \\ b&0 \end{bmatrix} ^{-1} = \begin{bmatrix} A_N^{-1} + A_N^{-1} b^T b A_N^{-1} / c&-A_N^{-1} b^T / c \\ - b A_N^{-1} / c&1 / c \end{bmatrix}, \end{aligned}$$

(85)

where b is a vector of length $n+1$ defined as $b = \bigl [ {\begin{matrix} 1&x_{N+1} \end{matrix}} \bigr ]^T$, and $c = - b A_N^{-1} b^T$ is a scalar. Multiplication of $A_{N+1}^{-1}$ in (85) with the vector $\bigl [ {\begin{matrix} f(x_i)&0&f(x_{N+1}) \end{matrix}} \bigr ]^T$ gives the vector of weights in an unordered fashion, i.e. $\bigl [ {\begin{matrix} w_i&v_i&w_{N+1}) \end{matrix}} \bigr ]^T$. Therefore, before adding the new function evaluation in the following iteration and performing the next rank-one update, it is necessary to permute the matrix $A_{N+1}^{-1}$, given by

$$\begin{aligned}{A_{N+1}^{-1} = \begin{bmatrix} F&V^T&\varphi ({||}x_{N+1} - x_i{||})^T \\ V&0&\begin{bmatrix} 1&x_{N+1} \end{bmatrix} \\ \varphi ({||}x_{N+1} - x_i{||})&\begin{bmatrix} 1 \\ x_{N+1} \end{bmatrix}&0 \end{bmatrix}^{-1}}, \end{aligned}$$

such that the desired $2 \times 2$ block form at the next iteration is recovered:

$$\begin{aligned} \begin{aligned}&P A_{N+1}^{-1} P^T = \begin{bmatrix} F_+&V_+^T \\ V_+&0 \end{bmatrix} ^{-1} \\&=\begin{bmatrix} F&\varphi ({||}x_{N+1} - x_i{||})^T&V^T \\ \varphi ({||}x_{N+1} - x_i{||})&0&\begin{bmatrix} 1&x_{N+1} \end{bmatrix} \\ V&\begin{bmatrix} 1 \\ x_{N+1} \end{bmatrix}&0 \end{bmatrix} ^{-1}. \end{aligned} \end{aligned}$$

After this permutation, it is possible to apply the Matrix Inversion Lemma (85) at the following step.

Remark 13

Another fast method to find the coefficients of radial basis functions is described in [35]. Since the present algorithms build the dataset incrementally, the method described above is less expensive in the present case.

Fig. 20

The difference between the actual function, $f(r)=r* \sin {1/r}$, where $r^2=x^2+y^2$, and its interpolant for two different interpolation strategies when 1000 function evaluations are clustered near the center of a square domain. a The error of the polyharmonic spline interpolation interpolant (83). b The error of the Kriging interpolant with a Gaussian model for the corrrelation, computed using DACE

As mentioned earlier, variations of Kriging interpolation are often used in Response Surface Methods, such as the Surrogate Management Framework, for derivative-free optimization. DACE (see [8]) is one of the standard packages used for numerically computing the Kriging interpolant. Figure 20a and b compare of the polyharmonic spline interpolation method described above and the Kriging interpolation method computed using DACE, as applied to the test function $f(r)=r* \sin {1/r}$, where $r^2=x^2+y^2$ with $N=1004$ data points. The data points used in this example are the 4 corners of a square domain, and 1000 random-chosen points clustered within a small neighborhood of the center of the square, which highlights the numerical challenge of performing interpolation when grid points begin to cluster in a particular region, which is common when a response surface method for derivative-free optimization approaches convergence. Figure 20a and b plot the difference between the real value of f and the value of the corresponding interpolants.

An observation which motivated the present study is that, in such problems, the Kriging interpolant is often spurious in comparison with other interpolation methods, such as polyharmonic splines. Note that various methods have been proposed to regularize such spurious interpolations in the presence of clustered datapoints, such as combining interpolants which summarize global trends with interpolants which account for local fluctuations. Our desire in the present effort was to develop a robust response surface method that can implement any good interpolation strategy, the selection of which is expected to be somewhat problem dependent.