Global resolution of the support vector machine regression parameters selection problem with LPCC

Abstract

Support vector machine regression is a robust data fitting method to minimize the sum of deducted residuals of regression, and thus is less sensitive to changes of data near the regression hyperplane. Two design parameters, the insensitive tube size ($\varepsilon _\mathrm{e}$) and the weight assigned to the regression error trading off the normed support vector ($C_\mathrm{e}$), are selected by user to gain better forecasts. The global training and validation parameter selection procedure for the support vector machine regression can be formulated as a bi-level optimization model, which is equivalently reformulated as linear program with linear complementarity constraints (LPCC). We propose a rectangle search global optimization algorithm to solve this LPCC. The algorithm exhausts the invariancy regions on the parameter plane ($(C_\mathrm{e},\varepsilon _\mathrm{e})$-plane) without explicitly identifying the edges of the regions. This algorithm is tested on synthetic and real-world support vector machine regression problems with up to hundreds of data points, and the efficiency are compared with several approaches. The obtained global optimal parameter is an important benchmark for every other selection of parameters.

Appendices

Appendix A: Semismooth method for SVR

Algorithm 12

Damped Newton method (Algorithm 2 in [17])

Step 0: Initialization: :

Let $\mathbf {a}^0 \in {\mathbb {R}}^n$, $\rho \ge 0$, $p \ge 2$, and $\sigma \in (0, \frac{1}{2})$ be given. Set $k = 0$. Set tol ¹².

Step 1: Termination: :

If $g(\mathbf {a}^k) := \frac{1}{2} \parallel \Phi (\mathbf {a}^k) \parallel _2^2 \le tol$, stop.

Step 2: Direction Generation: :

Otherwise, let $\mathbf {H}^k \in \partial _B \Phi (\mathbf {a}^k)$, and calculate $\mathbf {d}^k \in {\mathbb {R}}^n$ solving the Newton system:

$$\begin{aligned} \mathbf {H}^k\mathbf {d}^k = - \Phi (\mathbf {a}^k). \end{aligned}$$

(31)

If either (31) is unsolvable or the decent condition

$$\begin{aligned} \nabla g(\mathbf {a}^k)^T\mathbf {d}^k < - \rho \parallel \mathbf {d}^k \parallel _2^p \end{aligned}$$

(32)

is not satisfied, then set

$$\begin{aligned} \mathbf {d}^k = - \nabla g(\mathbf {a}^k). \end{aligned}$$

(33)

Step 3: Line Search: Choose $t^k = 2^{-i_k}$, where $i_k$ is the smallest integer such that

$$\begin{aligned} g\big ( \mathbf {a}^k + 2^{-i_k}\mathbf {d}^k \big ) \le g(\mathbf {a}^k) + \sigma 2^{-i_k} \nabla g (\mathbf {a}^k)^T \mathbf {d}^k. \end{aligned}$$

Step 4: Update: Let $\mathbf {a}^{k+1} := \mathbf {a}^k + t^k\mathbf {d}^k$ and $k := k + 1$. Go to Step 1. $\square$

The B-subdifferential used in step 3 of Algorithm 12 is obtained using the following theorem.

Theorem 13

(Theorem 5 in Ferris and Munson (2004)) Let $\mathbf {U}: {\mathbb {R}}^n \rightarrow {\mathbb {R}}^n$ be continuously differentiable. Then

$$\begin{aligned} \partial _B\Phi (\mathbf {a}) \subseteq \{ D_a + D_bU'(\mathbf {a})\}, \end{aligned}$$

where $D_a \in {\mathbb {R}}^{n \times n}$ and $D_b \in {\mathbb {R}}^{n\times n}$ are diagonal matrices with entries defined as follows:

• 1. For all $i \in \mathbb {I}$: If $\parallel (a_i, U_i(\mathbf {a})) \parallel \ne 0$, then

  $$\begin{aligned} (D_a)_{ii}= & {} 1 - \frac{a_i}{\parallel a_i, U_i(\mathbf {a}) \parallel }, \nonumber \\ (D_b)_{ii}= & {} 1 - \frac{U_i(\mathbf {a})}{\parallel a_i, U_i(\mathbf {a}) \parallel }; \end{aligned}$$

  (34)

  otherwise

  $$\begin{aligned} ((D_a)_{ii}, (D_b)_{ii} ) \in \{ (1- \kappa , 1-\gamma ) \in {\mathbb {R}}^2 \mid \parallel (\eta , \gamma ) \parallel \le 1\}. \end{aligned}$$

  (35)

• 2. For all $i\in \mathbb {E}$:

  $$\begin{aligned} (D_a)_{ii}= & {} 0,\\(D_b)_{ii}= & {} 1. \end{aligned}$$

                                                                                                                                   $\square$

If $\parallel (a_i, U_i(\mathbf {a})) \parallel \ne 0$, $\Phi (\mathbf {a})$ is differentiable at $\mathbf {a}$ and the formula (34) computes the exact Jacobian. On the other hand, if $\parallel (a_i, U_i(\mathbf {a})) \parallel = 0$ occurs at the $i^{th}$ complementarity, $\kappa$ and $\gamma$ appeared in (35) are computed as suggested in Facchinei and Pang (2003):

$$\begin{aligned} \kappa = \frac{v_i}{\sqrt{v_i^2 + (U'v)_i^2}}, \quad \text{ and } \quad \gamma = \frac{(U'v)_i}{\sqrt{v_i^2 + (U'v)_i^2}}, \end{aligned}$$

(36)

where $\mathbf {v} \in {\mathbb {R}}^n$ is a vector of user’s choice whose ith element is nonzero.

To compute $\kappa$ and $\gamma$ in (36), we can choose $\mathbf {v} = \mathbf {1}$, and let

which doesn’t require update. Then the computation of $\kappa$ and $\gamma$ is simplified as

$$\begin{aligned} \kappa = \frac{1}{\sqrt{1 + (hv)_i^2}},\quad \text{ and } \quad \gamma = \frac{(hv)_i}{\sqrt{1 + (hv)_i^2}}. \end{aligned}$$

where the index i is the same as defined in (36).

In our experiments, we ignore steps (32) and (33) to save running time because the system (31) is always solvable, and the convergence is obtained in most cases with the initial $\{\mathbf {a}^0\} = \mathbf {0}$. For rare cases where the condition $g(\mathbf {a}^k) \le tol$ in Step 2 can not be fulfilled, we try a different initial point to restart.

Appendix B: Steps 2–4 of Algorithm 8

Step 2. Search on the vertical line $C_e = \underline{C}$.

Initialize the set ${\mathbb {G}}rouping{\mathbb {VS}}et^{left} = {\mathbb {G}}rouping{\mathbb {VS}}et^{lu}$.

• 2a: For every pieces corresponding to a member of ${\mathbb {G}}rouping{\mathbb {VS}}et^{left}$, solve (20) subject to $C_e = \underline{C}$ to obtain invariancy intervals. Let $\big [\varepsilon _{min}, \,\varepsilon _{max}\big ]$ be the largest intervals among others. Do Add-In when repeating. $countLeft+1$.

• 2b: Solve ${\mathbb {LCP}}_{SVR}^f$ at $(\underline{C},\varepsilon _{min})$ and obtain the grouping vectors set.

• 2c: Replace ${\mathbb {G}}rouping{\mathbb {VS}}et^{left}$ by the set of the grouping vectors obtained in 2b. If any members of ${\mathbb {G}}rouping{\mathbb {VS}}et^{left}$ are not in the set ${\mathbb {G}}rouping{\mathbb {VF}}ound$, add them to the later set.

• 2d: If the objective value of ${\mathbb {RLP}}$ is smaller than LeastUpperBound, update LeastUpperBound.

• 2e: If $\varepsilon _{min}$ is greater than $\underline{\varepsilon }$, let $newStarting = (\underline{C}\,,\, \varepsilon _{min}-deviation)$.

• 2f: Solve ${\mathbb {LCP}}_{SVR}^f$ at newStarting and obtain the grouping vectors set. Do as in 2c–2d.

• 2g: Repeat 2a–2d until $\varepsilon _{min} = \underline{\varepsilon }$.

Step 3. Search on the horizontal line $\varepsilon _e = \underline{\varepsilon }$.

Initialize the set ${\mathbb {G}}rouping{\mathbb {VS}}et^{bottom} = {\mathbb {G}}rouping{\mathbb {VS}}et^{ul}$.

• 3a: For every pieces corresponding to a member of $GroupingVSet^{bottom}$, solve (19) subject to $\varepsilon _e = \underline{\varepsilon }$ to obtain invariancy intervals. Let $\big [C_{min}, \,C_{max}\big ]$ be the largest intervals among others. Do Add-In when repeating. $countBottom+1$.

• 3b: Solve ${\mathbb {LCP}}_{SVR}^f$ at $(C_{min},\underline{\varepsilon })$ and obtain the grouping vectors set.

• 3c: Replace ${\mathbb {G}}rouping{\mathbb {VS}}et^{bottom}$ by the set of the grouping vectors obtained in 3b. If any members of ${\mathbb {G}}rouping{\mathbb {VS}}et^{bottom}$ are not in the set ${\mathbb {G}}rouping{\mathbb {VF}}ound$, add them to the later set.

• 3d: If the objective value of ${\mathbb {RLP}}$ is smaller than LeastUpperBound, update LeastUpperBound.

• 3e: If $C_{min}$ is greater than $\underline{C}$, let $newStarting = (C_{min}-deviation\,,\, \underline{\varepsilon })$.

• 3f: Solve ${\mathbb {LCP}}_{SVR}^f$ at newStarting and obtain the grouping vectors set. Do as in 3c-3d.

• 3g: Repeat 3a–3d until $C_{min} = \underline{C}$.

Step 4. Search on the vertical line $C_e = \bar{C}$.

Initialize the ${\mathbb {G}}rouping{\mathbb {VS}}et^{right} = {\mathbb {G}}rouping{\mathbb {VS}}et^{uu}$.

• 4a: For every pieces corresponding to a member of ${\mathbb {G}}rouping{\mathbb {VS}}et^{right}$, solve (20) subject to $C_e = \bar{C}$ to obtain invariancy intervals. Let $\big [\varepsilon _{min}, \,\varepsilon _{max}\big ]$ be the largest intervals among others. Do Add-In when repeating. $countRight+1$.

• 4b: Solve ${\mathbb {LCP}}_{SVR}^f$ at $(\bar{C},\varepsilon _{min})$ and obtain the grouping vectors set.

• 4c: Replace ${\mathbb {G}}rouping{\mathbb {VS}}et^{right}$ by the set of the grouping vectors obtained in 4b. If any members of ${\mathbb {G}}rouping{\mathbb {VS}}et^{right}$ are not in the set ${\mathbb {G}}rouping{\mathbb {VF}}ound$, add them to the later set.

• 4d: If the objective value of ${\mathbb {RLP}}$ is smaller than LeastUpperBound, update LeastUpperBound.

• 4e: If $\varepsilon _{min}$ is greater than $\underline{\varepsilon }$, let $newStarting = (\bar{C}, \varepsilon _{min}-deviation)$.

• 4f: Solve ${\mathbb {LCP}}_{SVR}^f$ at newStarting and obtain the grouping vectors set. Do as in 4c–4d.

• 4g: Repeat 4a–4d until $\varepsilon _{min} = \underline{\varepsilon }$.

Appendix C: Cases 2–6 of Algorithm 10