Optimization via Chebyshev polynomials

Abstract

This paper presents for the first time a robust exact line-search method based on a full pseudospectral (PS) numerical scheme employing orthogonal polynomials. The proposed method takes on an adaptive search procedure and combines the superior accuracy of Chebyshev PS approximations with the high-order approximations obtained through Chebyshev PS differentiation matrices. In addition, the method exhibits quadratic convergence rate by enforcing an adaptive Newton search iterative scheme. A rigorous error analysis of the proposed method is presented along with a detailed set of pseudocodes for the established computational algorithms. Several numerical experiments are conducted on one- and multi-dimensional optimization test problems to illustrate the advantages of the proposed strategy.

Appendices

Appendix 1: Preliminaries

In this section, we present some useful results from approximation theory. The first kind Chebyshev polynomial (or simply the Chebyshev polynomial) of degree n, $T_n(x)$, is given by the explicit formula

$$\begin{aligned} {T_n}(x) = \cos \left( {n\,{{\cos }^{ - 1}}(x)} \right) \;\forall x \in [ - 1,1], \end{aligned}$$

(8.1)

using trigonometric functions, or in the following explicit polynomial form [24]:

$$\begin{aligned} {T_n}(x) = \frac{1}{2}\sum \limits _{k = 0}^{\left\lfloor {n/2} \right\rfloor } {T_{n - 2k}^k\,{x^{n - 2k}}} , \end{aligned}$$

(8.2)

where

$$\begin{aligned} T_m^k = {2^m}{( - 1)^k}\left\{ {\frac{{m + 2\,k}}{{m + k}}} \right\} \left( \begin{array}{c} m + k\\ k \end{array} \right) ,\quad m,k \ge 0. \end{aligned}$$

(8.3)

The Chebyshev polynomials can be generated by the three-term recurrence relation

$$\begin{aligned} T_{n + 1} (x) = 2xT_n (x) - T_{n - 1} (x), \quad n \ge 1, \end{aligned}$$

(8.4)

starting with ${T_0}(x) = 1$ and ${T_1}(x) = x$. They are orthogonal in the interval $[-1, 1]$ with respect to the weight function $w(x) = \left( 1 - x^2\right) ^{-1/2}$, and their orthogonality relation is given by

$$\begin{aligned} \left\langle {{T_n},{T_m}} \right\rangle _w = \int _{ - 1}^1 {{T_n}(x)\,{T_m}(x)\,{{\left( 1 - {x^2}\right) }^{ - \frac{1}{2}}}dx = \frac{\pi }{2}{c_n}{\delta _{nm}}} , \end{aligned}$$

(8.5)

where $c_0 = 2, c_n = 1, n \ge 1$, and $\delta _{n m}$ is the Kronecker delta function defined by

$$\begin{aligned} {\delta _{nm}} = \left\{ {\begin{array}{ll} 1,&{}\quad n = m,\\ 0,&{}\quad n \ne m. \end{array}} \right. \end{aligned}$$

The roots (aka Chebyshev–Gauss points) of $T_n(x)$ are given by

$$\begin{aligned} {x_k} = \cos \left( {\frac{{2k - 1}}{{2n}}\pi } \right) ,\quad k = 1, \ldots ,n, \end{aligned}$$

(8.6)

and the extrema (aka CGL points) are defined by

$$\begin{aligned} {x_k} = \cos \left( {\frac{{k \pi }}{n}} \right) ,\quad k = 0,1, \ldots ,n. \end{aligned}$$

(8.7)

The derivative of $T_n(x)$ can be obtained in terms of Chebyshev polynomials as follows [18]:

$$\begin{aligned} \frac{d}{{dx}}{T_n}(x) = \frac{n}{2}\frac{{{T_{n - 1}}(x) - {T_{n + 1}}(x)}}{{1 - {x^2}}},\quad \left| x \right| \ne 1. \end{aligned}$$

(8.8)

Clenshaw and Curtis [5] showed that a continuous function f(x) with bounded variation on $[-1, 1]$ can be approximated by the truncated series

$$\begin{aligned} ({P_n}f)(x) = \sum \limits _{k = 0}^n {^{''}}{{a_k}\,{T_k}(x)} , \end{aligned}$$

(8.9)

where

$$\begin{aligned} {a_k} = \frac{2}{n}\sum \limits _{j = 0}^n {^{''}}{{f_j}\,{T_k}({x_j})} ,\quad n > 0, \end{aligned}$$

(8.10)

$x_j, j = 0, \ldots , n$, are the CGL points defined by Eq. (8.7), $f_j = f(x_j)\, \forall j$, and the summation symbol with double primes denotes a sum with both the first and last terms halved. For a smooth function f, the Chebyshev series (8.9) exhibits exponential convergence faster than any finite power of 1 / n [15].

Appendix 2: Pseudocodes of developed computational algorithms