High dimensional model representation constructed by support vector regression. I. Independent variables with known probability distributions

Abstract

High dimensional model representation (HDMR) is a general set of quantitative model assessment and analysis tools for capturing high dimensional input-output system behavior. In practice, the HDMR component functions are each approximated by an appropriate basis function expansion. This procedure often requires many input-output samples which can restrict the treatment of high dimensional systems. In order to address this problem we introduce svr-based HDMR to efficiently and effectively construct the HDMR expansion by support vector regression (SVR) for a function $f(\mathbf{x})$. In this paper the results for independent variables sampled over known probability distributions are reported. The theoretical foundation of the new approach relies on the kernel used in SVR itself being an HDMR expansion (referred to as the HDMR kernel ), i.e., an ANOVA kernel whose component kernels are mutually orthogonal and all non-constant component kernels have zero expectation. Several HDMR kernels are constructed as illustrations. While preserving the characteristic properties of HDMR, the svr-based HDMR method enables efficient construction of high dimensional models with satisfactory prediction accuracy from a modest number of samples, which also permits accurate computation of the sensitivity indices. A genetic algorithm is employed to optimally determine all the parameters of the component HDMR kernels and in SVR. The svr-based HDMR introduces a new route to advance HDMR algorithms. Two examples are used to illustrate the capability of the method.

Appendices

Appendices

The material below will summarize the features of the HDMR kernels employed in this paper.

Appendix 1: $K_0(x_{i_s},x_i)$ is a kernel

The commonly used kernels $K(x_{i_s},x_i)$ in SVR, like Gaussian rbf, polynomial kernels, etc. do not satisfy the zero expectation condition with respect to the measure $\mu$ required for defining HDMR component functions:

$$\begin{aligned} \int K(s,x_i){\mathrm d}\mu (s) =\int K(x_{i_s},t){\mathrm d}\mu (t) =0, \end{aligned}$$

(59)

or using ${\mathrm d}\mu (s)=p_i(s){\mathrm d}s$ and ${\mathrm d}\mu (t)=p_i(t){\mathrm d}t$

$$\begin{aligned} \int K(s,x_i)p_i(s){\mathrm d}s =\int K(x_{i_s},t)p_i(t){\mathrm d}t =0, \end{aligned}$$

(60)

where $p_i(x_i)$ is the marginal pdf of $x_i$. To solve this problem, define

$$\begin{aligned} K_0(x_{i_s},x_i)= & {} K(x_{i_s},x_i) - \frac{\int K(x_{i_s},t){\mathrm d}\mu (t) \int K(s,x_i){\mathrm d}\mu (s)}{\int \int K(s,t){\mathrm d}\mu (s){\mathrm d}\mu (t)}\nonumber \\= & {} K(x_{i_s},x_i) - \frac{\int K(x_{i_s},t)p_i(t){\mathrm d}t \int K(s,x_i)p_i(s){\mathrm d}s}{\int \int K(s,t)p_i(s)p_i(t){\mathrm d}s{\mathrm d}t}. \end{aligned}$$

(61)

For a uniform distribution of $x_i$, Eq. 61 reduces to

$$\begin{aligned} K_0(x_{i_s},x_i)=K(x_{i_s},x_i) - \frac{\int K(x_{i_s},t){\mathrm d}t \int K(s,x_i){\mathrm d}s}{\int \int K(s,t){\mathrm d}s{\mathrm d}t}. \end{aligned}$$

(62)

From Eq. 61, it is easy to prove that $K_0(x_{i_s},x_i)$ satisfies Eq. 60 by direct integration.

If $K(x_i,x'_i)$ is a kernel, it can be proved that $K_0(x_i,x'_i)$ is also a kernel satisfying Mercer’s condition [17]

$$\begin{aligned} \int \int K_0(x_i,x_i')f(x_i)f(x_i') {\mathrm d}\mu (x_i){\mathrm d}\mu (x'_i)\ge 0. \end{aligned}$$

(63)

The proof is the following:

$$\begin{aligned}&\int \int K_0(x_i,x'_i)f(x_i)f(x'_i){\mathrm d}\mu (x_i){\mathrm d}\mu (x'_i) \nonumber \\&\quad =\int \int K(x_i,x'_i)f(x_i)f(x'_i){\mathrm d}\mu (x_i){\mathrm d}\mu (x'_i) \nonumber \\&\qquad -\frac{1}{\int \int K(s,t){\mathrm d}\mu (s){\mathrm d}\mu (t)} \left[ \int \int K(x_i,t)f(x_i){\mathrm d}\mu (x_i){\mathrm d}\mu (t) \right. \nonumber \\&\qquad \left. \times \int \int K(s,x'_i)f(x'_i){\mathrm d}\mu (s){\mathrm d}\mu (x'_i) \right] \nonumber \\&\quad =\langle f(x_i), f(x'_i)\rangle - \frac{\langle f(x_i),1 \rangle \langle 1,f(x'_i) \rangle }{ \langle 1,1\rangle }\ge 0. \end{aligned}$$

(64)

Here, the inner product $\langle *,* \rangle$ is defined with respect to the kernel $K(x_i,x'_i)$

$$\begin{aligned} \langle f(x_i), g(x'_i)\rangle = \int \int K(x_i,x'_i)f(x_i)g(x'_i){\mathrm d}\mu (x_i){\mathrm d}\mu (x'_i), \end{aligned}$$

(65)

and the symmetric property

$$\begin{aligned} \langle 1,f(x'_i) \rangle= & {} \langle f(x'_i),1 \rangle = \langle f(x_i),1 \rangle , \end{aligned}$$

(66)

$$\begin{aligned} \langle f(x_i), f(x'_i)\rangle= & {} \langle f(x_i), f(x _i)\rangle \end{aligned}$$

(67)

along with the Cauchy-Schwarz inequality

$$\begin{aligned} \langle x,y\rangle \langle x,y\rangle \le \langle x,x \rangle \langle y,y \rangle \end{aligned}$$

(68)

were used.

Appendix 2: Formulas of some commonly used HDMR kernels

Using Eq. 61 some commonly used HDMR kernels are given below. To save the space, the detailed derivations of these formulas are omitted.

• 1a. Gaussian rbf HDMR kernel with a uniform distribution in [−1,1]

  $$\begin{aligned} K_0(x_i,x'_i)= & {} \exp {\left[ \frac{(x_i-x'_i)^2}{2l_i^2}\right] } \nonumber \\&\quad -\frac{\left[ \mathrm{erf}\left( \frac{1+x_i}{\sqrt{2}l_i}\right) +\mathrm{erf}\left( \frac{1-x_i}{\sqrt{2}l_i}\right) \right] \left[ \mathrm{erf}\left( \frac{1+x'_i}{\sqrt{2}l_i}\right) +\mathrm{erf}\left( \frac{1-x'_i}{\sqrt{2}l_i}\right) \right] }{\frac{4}{\sqrt{\pi }}\frac{\sqrt{2}}{l_i}\, \mathrm{erf}\left( \frac{\sqrt{2}}{l_i}\right) +\frac{4}{\pi }\left( e^{-\frac{2}{l_i^2}}-1\right) }.\nonumber \\ \end{aligned}$$

  (69)

• 1b. Gaussian rbf HDMR kernel with a standard normal distribution in $[-\infty ,\infty ]$

  $$\begin{aligned} K_0(x_i,x'_i) =\exp {\left[ -\frac{(x_i-x'_i)^2}{2l_i^2}\right] } -\frac{l_i\sqrt{l_i^2+2}}{l_i^2+1} \exp {\left[ -\frac{x_i^2+(x'_i)^2}{2(l_i^2+1)}\right] }. \end{aligned}$$

  (70)

• 2a. Exponential HDMR kernel with a uniform distribution in [−1,1]

  $$\begin{aligned} K_0(x_i,x'_i)= & {} \exp {\left[ -\frac{|x_i-x'_i|}{2l_i^2}\right] } \nonumber \\&\quad -\frac{\left( 2-\exp {\left[ -\frac{(1+x_i)(1-x_i)}{4l_i^4}\right] }\right) \left( 2-\exp {\left[ -\frac{(1+x'_i)(1-x'_i)}{4l_i^4}\right] }\right) }{2\left[ \frac{1}{l_i^2}-\left( 1-e^{-\frac{1}{l_i^2}}\right) \right] }.\nonumber \\ \end{aligned}$$

  (71)

• 2b. Exponential HDMR kernel with a standard normal distribution in $[-\infty ,\infty ]$

  $$\begin{aligned}&K_0(x_i,x'_i)=\exp {\left[ -\frac{|x_i-x'_i|}{2l_i^2}\right] } -\frac{1}{4\,\mathrm{erfc}(\frac{1}{2l_i^2})} \nonumber \\&\quad \times \left[ \exp \left( -\frac{x_i}{2l_i^2}\right) \mathrm{erfc} \left( \frac{1}{2\sqrt{2}l_i^2}-\frac{x_i}{\sqrt{2}}\right) +\exp \left( \frac{x_i}{2l_i^2}\right) \mathrm{erfc} \left( \frac{1}{2\sqrt{2}l_i^2}+\frac{x_i}{\sqrt{2}}\right) \right] \nonumber \\&\quad \times \left[ \exp \left( -\frac{x'_i}{2l_i^2}\right) \mathrm{erfc} \left( \frac{1}{2\sqrt{2}l_i^2}-\frac{x'_i}{\sqrt{2}}\right) +\exp \left( \frac{x'_i}{2l_i^2}\right) \mathrm{erfc} \left( \frac{1}{2\sqrt{2}l_i^2}+\frac{x'_i}{\sqrt{2}}\right) \right] . \nonumber \\ \end{aligned}$$

  (72)

• 3a. Polynomial HDMR kernel with a uniform distribution in [0,1]

  $$\begin{aligned} K_0(x_i,x'_i) = (x_ix'_i+1)^k - \frac{\left[ \displaystyle {\sum _{j=1}^{k+1}} C_{k+1}^j x_i^{j-1}\right] \left[ \displaystyle {\sum _{j=1}^{k+1}}C_{k+1}^j (x_i')^{j-1}\right] }{(k+1)\displaystyle {\sum _{j=1}^{k+1}}\frac{2^j-1}{j}}, \end{aligned}$$

  (73)

  where

  $$\begin{aligned} C_{k+1}^j=\frac{(k+1)!}{j!(k+1-j)!} \end{aligned}$$

  (74)

  denotes the number of all combinations of $k+1$ things taken j at a time.

• 3b. Polynomial HDMR kernel with a uniform distribution in [−1,1]

  $$\begin{aligned} K_0(x_i,x'_i)= & {} \,(x_ix'_i+1)^k - \frac{1}{(k+1)\displaystyle {\sum _{j=1}^{\mathrm{Int}((k+1)/2)}} C_{k+1}^{2j-1}/(2j-1)} \times \nonumber \\&\quad \times \left[ \sum _{j=1}^{\mathrm{Int}((k+1)/2)} C_{k+1}^{2j-1} x_i^{2(j-1)} \right] \cdot \left[ \sum _{j=1}^{\mathrm{Int}((k+1)/2)} C_{k+1}^{2j-1}(x'_i)^{2(j-1)} \right] .\nonumber \\ \end{aligned}$$

  (75)

• 4. Fourier HDMR kernel with a uniform distribution in $[-\pi ,\pi ]$

  $$\begin{aligned} K_0(x_i,x'_i)= & {} \frac{1-q^2}{2(1-2q\cos (x_i-x'_i)+q^2)} - \frac{1}{2}\nonumber \\= & {} \frac{q\cos (x_i-x'_i)-q^2}{1-2q\cos (x_i-x'_i)+q^2}. \end{aligned}$$

  (76)

Complex and hybrid HDMR kernels

If $K_{01}(x_i,x'_i)$ and $K_{02}(x_i,x'_i)$ are two HDMR kernels, then

$$\begin{aligned} K_{03}(x_i,x'_i)=\alpha K_{01}(x_i,x'_i) + \beta K_{02}(x_i,x'_i), \qquad \alpha , \beta > 0 \end{aligned}$$

(77)

is another HDMR kernel. Using Eq. 31, we know that $K_{03}(x_i,x'_i)$ is a kernel. Moreover, it can be also proved that $K_{03}(x_i,x'_i)$ has zero expectation

$$\begin{aligned} \int K_{03}(x_i,x'_i){\mathrm d}\mu (x_i) = \alpha \int K_{03}(x_i,x'_i){\mathrm d}\mu (x_i) + \beta \int K_{03}(x_i,x'_i){\mathrm d}\mu (x_i)=0. \end{aligned}$$

(78)

Therefore $K_{03}(x_i,x'_i)$ is an HDMR kernel. Using Eq. 77, various complex and hybrid HDMR kernels can be constructed.