A novel model modification method for support vector regression based on radial basis functions

Abstract

There are some inherent limitations to the performance of support vector regression (SVR), such as (i) the loss function, penalty parameter, and kernel function parameter usually cannot be determined accurately; (ii) the training data sometimes cannot be fully utilized; and (iii) the local accuracy in the vicinity of training points still need to be improved. To further enhance the performance of SVR, this paper proposes a novel model modification method for SVR with the help of radial basis functions. The core idea of the method is to start with an initial SVR and modify it in a subsequent stage to extract as much information as possible from the existing training data; the second stage does not require new points. Four types of modified support vector regression (MSVR), including MSVg, MSVm, MSVi, and MSVc, are constructed by using four different forms of basis functions. This paper evaluates the performances of SVR, MSVg, MSVm, MSVi, and MSVc by using six popular benchmark problems and a practical engineering problem, which is designing a typical turbine disk for an air-breathing engine. The results show that all the four types of MSVR perform better than SVR. Notably, MSVc has the best performance.

Appendices

Appendix 1. Support vector regression

The general form of SVR can be written as

$$ \hat{y}(\textbf{x})=\boldsymbol{\omega}^{\mathrm{T}}\boldsymbol{\psi}(\textbf{x})+b $$

(14)

where x denotes the vector of input variables, $\hat {y}(\textbf {x})$ denotes the approximation response, ψ(x) denotes the non-linear feature mapping function, ω denotes the weight vector, and b denotes the bias.

To construct the model, a following optimization problem should be solved.

$$ \begin{array}{llll} &\min& &\frac{1}{2}||\boldsymbol{\omega}||^{2}+C \sum\limits_{i = 1}^{m}(\xi^{+}_{i}+\xi^{-}_{i})\\ &\text{s.t.} & &\left\{ \begin{array}{llll} &\boldsymbol{\omega}^{\mathrm{T}}\boldsymbol{\psi}(\textbf{x}_{i})+b-y_{i} \leq \epsilon+\xi^{+}_{i}\\ &y_{i}-\boldsymbol{\omega}^{\mathrm{T}}\boldsymbol{\psi}(\textbf{x}_{i})-b \leq \epsilon+\xi^{-}_{i}\\ &\xi^{+}_{i},\xi^{-}_{i}\geq 0\\ &i = 1,\dots,m \end{array} \right. \end{array} $$

(15)

where (x_i,y_i) (i = 1,…,m) denotes the training data subset (or the entire training dataset), 𝜖 denotes the loss function, C denotes the penalty parameter, $\xi ^{+}_{i}$ and $\xi ^{-}_{i}$ denotes the slack variables.

The Lagrange dual form of the above model is expressed as

$$ \begin{array}{llll} &\max& &\left\{ \begin{array}{llll} &-\frac{1}{2} \sum\limits_{i,j = 1}^{m}(\alpha^{+}_{i}-\alpha^{-}_{i})(\alpha^{+}_{j}-\alpha^{-}_{j})\\ &k\left\langle\textbf{x}_{i},\textbf{x}_{j}\right\rangle+\sum\limits_{i = 1}^{m}(\alpha^{+}_{i}-\alpha^{-}_{i})y_{i} \\ &-\sum\limits_{i = 1}^{m}(\alpha^{+}_{i}+\alpha^{-}_{i})\epsilon \end{array} \right.\\ &\text{s.t.} & &\left\{ \begin{array}{lllll} &\sum\limits_{i = 1}^{m}(\alpha^{+}_{i}-\alpha^{-}_{i})= 0\\ &0\leq\alpha^{+}_{i},\alpha^{-}_{i}\leq C\\ &i = 1,\dots,m \end{array} \right. \end{array} $$

(16)

where $\alpha ^{+}_{i}$ and $\alpha ^{-}_{i}$ denote the Lagrange multipliers. k 〈x_i,x_j〉 = ψ(x_i)^Tψ(x_j) is a kernel function, which has to be continuous, symmetric, and positive definite. One of the most popular kernel function that is Gaussian kernel is selected in this paper and expressed as

$$ k\left\langle\textbf{x},\textbf{x}_{i}\right\rangle=\exp (-\gamma||\textbf{x}-\textbf{x}_{i}||^{2}) $$

(17)

According to KKT conditions, SVR can be finally obtained through

$$ \hat{y}(\textbf{x})=\sum\limits_{i = 1}^{m}(\alpha^{+}_{i}-\alpha^{-}_{i})k\left\langle\textbf{x},\textbf{x}_{i}\right\rangle+b $$

(18)

The efficient package LIBSVM developed by Chang and Lin (2011) is used to construct the SVR model in this paper.

Appendix 2. Analytical functions

• (1) 1-variable Forrester function

  $$ f(x)=(6x-2)^{2}\sin (12x-4) $$

  (19)

  where x ∈ [0, 1].

• (2) 2-variable Goldstein Price function

  $$ \begin{array}{llll} f(\textbf{x})=\left[1+(x_{1}+x_{2}+ 1)^{2}\times\right.\\ \left.(19-14x_{1}+ 3{x_{1}^{2}}-14x_{2}+ 6x_{1}x_{2}+ 3{x_{2}^{2}})\right]\times\\ \left[30+(2x_{1}-3x_{2})^{2}\times\right.\\ \left.(18-32x_{1}+ 12{x_{1}^{2}}+ 48x_{2}-36x_{1}x_{2}+ 27{x_{2}^{2}})\right] \end{array} $$

  (20)

  where x₁ ∈ [− 2, 2], and x₂ ∈ [− 2, 2].

• (3) 3-variable Perm function

  $$ f(\textbf{x})=\sum\limits_{i = 1}^{3} \left( {\sum}_{j = 1}^{3} \left( j + 2 \right)\left( {x^{i}_{j}}-\frac{1}{j^{i}}\right)\right)^{2} $$

  (21)

  where x_j ∈ [0, 1] for all j = 1, 2, 3.

• (4) 4-variable Hartmann function

  $$ f(\textbf{x})=-\sum\limits_{i = 1}^{4} c_{i} \exp \left[-\sum\limits_{j = 1}^{4} a_{ij}(x_{j}-p_{ij})^{2} \right] $$

  (22)

  where x_j ∈ [0, 1] for all j = 1, 2, 3, 4, $\textbf {c}=\left [\begin {array}{lllll} 1.0 &1.2& 3.0& 3.2 \end {array}\right ]^{\mathrm {T}}$, A and P are expressed as follows.

  $$\begin{array}{ll} &\textbf{A}=\left[\begin{array}{llllllll} &10&3.0&17&3.5\\ &0.05&10&17&0.1\\ &3.0&3.5&1.7&10\\ &17&8.0&0.05&10 \end{array}\right]\\ &\textbf{P}=\left[\begin{array}{lllll} &0.1312&0.1696&0.5569&0.124\\ &0.2329&0.4135&0.8307&0.3736\\ &0.2348&0.1451&0.3522&0.2883\\ &0.4047&0.8828&0.8732&0.5743 \end{array}\right] \end{array} $$

• (5) 5-variable Zakharov function

  $$ f(\textbf{x})=\sum\limits_{i = 1}^{5} {{x_{i}^{2}}}+\left( \sum\limits_{i = 1}^{d} {0.5ix_{i}} \right)^{2}+\left( \sum\limits_{i = 1}^{d} {0.5ix_{i}}\right)^{4} $$

  (23)

  where x_i ∈ [− 5, 10], for all i = 1,…, 5.

• (6) 6-variable Power Sum function

  $$ f(\textbf{x})=\sum\limits_{i = 1}^{6} \left[ \left( -\sum\limits_{j = 1}^{6} {x_{j}^{i}} \right)-13\right]^{2} $$

  (24)

  where x_j ∈ [0, 4], for all j = 1,…, 6.