A new formulation for second-order cone programming support vector machine

Abstract

A new second-order cone programming (SOCP) formulation inspired by the soft-margin linear programming support vector machine (LP-SVM) formulation and cost-sensitive framework is proposed. Our proposed method maximizes the slack variables related to each class by appropriately relaxing the bounds on the VC dimension using the l$_{\infty }$-norm, and penalizes them using the corresponding regularization parametrization to control the trade-off between margin and slack variables. The proposed method has two main advantages: firstly, a flexible classifier is constructed that extends the advantages of the soft-margin LP-SVM problem to the second-order cone; secondly, due to the elimination of a conic restriction, only two SOCP problems containing second-order cone constraints need to be solved. Thus similar results to the SOCP-SVM problem are obtained with less calculational effort. Numerical experiments show that our method achieves the better classification performance than the conventional SOCP-SVM formulations and standard linear SVM formulations.

Appendix A dual problem for LP-SVM

We use $m_1$ and $m_2$ to indicate the number of samples for positive and negative classes, respectively, the data matrix $A \in {\Re ^{n \times {m_1}}}$ for positive classes, and the data matrix $B \in {\Re ^{n \times {m_2}}}$ for negative classes, $\mathbf{{e}} = (1,...,1)$ as a ones vector of appropriate dimensionality. The vector in ${\Re ^{{m_1}}}$ we denote by subscript 1, the vector in ${\Re ^{{m_2}}}$ we denote by subscript 2, the subscriptless in ${\Re ^{{n}}}$. Based on this, the Lagrange formulation for Eq. (4) is:

$$\begin{aligned} \begin{aligned} L(\mathbf{{w}},b,r,{\mathbf{{z}}_1},{\mathbf{{z}}_2},t,\mathbf{{\alpha }},\mathbf{{\beta }}) = - r - \left\langle {{A^{\textrm{T}}}{} \mathbf{{w}} + (b - r)\mathbf{{e}},{\mathbf{{z}}_1}} \right\rangle \\ - \left\langle { - {B^{\textrm{T}}}{} \mathbf{{w}} - (b + r)\mathbf{{e}},{\mathbf{{z}}_2}} \right\rangle \\ - rt - \left\langle {\mathbf{{e}} - \mathbf{{w}},\mathbf{{\alpha }}} \right\rangle - \left\langle {\mathbf{{e}} + \mathbf{{w}},\mathbf{{\beta }}} \right\rangle . \end{aligned} \end{aligned}$$

Then the dual equation for problem (4) is:

$$\begin{aligned} \begin{aligned} \max \;\;\;&L(\mathbf{{w}},b,r,{\mathbf{{z}}_1},{\mathbf{{z}}_2},t,\mathbf{{\alpha }},\mathbf{{\beta }})\\ s.t.\;\;\;&\frac{{\partial L}}{{\partial \mathbf{{w}}}} = - A{\mathbf{{z}}_1} + B{\mathbf{{z}}_2} + \mathbf{{\alpha }} - \mathbf{{\beta }} = 0,\\&\frac{{\partial L}}{{\partial b}} = - {\mathbf{{e}}^{\textrm{T}}}{\mathbf{{z}}_1} + {\mathbf{{e}}^{\textrm{T}}}{\mathbf{{z}}_2} = 0,\\&\frac{{\partial L}}{{\partial r}} = - 1 + {\mathbf{{e}}^{\textrm{T}}}{\mathbf{{z}}_1} + {\mathbf{{e}}^{\textrm{T}}}{\mathbf{{z}}_2} - t = 0,\\&{\mathbf{{z}}_1} \ge 0,{\mathbf{{z}}_2} \ge 0,t \ge 0,\mathbf{{\alpha }} \ge 0,\mathbf{{\beta }} \ge 0. \end{aligned} \end{aligned}$$

Bringing the above equation into the Lagrange formulation, we can get:

$$\begin{aligned} \begin{aligned} \mathop {\max }\limits _{\mathbf{{\alpha }},\mathbf{{\beta }},{\mathbf{{z}}_1},{\mathbf{{z}}_2}} \;\;\;&-{\mathbf{{e}}^{\textrm{T}}}(\mathbf{{\alpha }} + \mathbf{{\beta }})\\ s.t.\;\;\;\;\;&\mathbf{{\alpha }} - \mathbf{{\beta }} = A{\mathbf{{z}}_1} - B{\mathbf{{z}}_2},\\&{\mathbf{{e}}^{\textrm{T}}}{\mathbf{{z}}_1} = 1,{\mathbf{{e}}^{\textrm{T}}}{\mathbf{{z}}_2} = 1,\\&{\mathbf{{z}}_1} \ge 0,{\mathbf{{z}}_2} \ge 0,\mathbf{{\alpha }} \ge 0,\mathbf{{\beta }} \ge 0. \end{aligned} \end{aligned}$$

Due to the following relation $\mathbf{{z}} = \mathbf{{\alpha }} - \mathbf{{\beta }},{\left\| \mathbf{{z}} \right\| _1} = {\mathbf{{e}}^{\textrm{T}}}(\mathbf{{\alpha }} + \mathbf{{\beta }}),\mathbf{{\alpha }},\mathbf{{\beta }} \ge 0.$, we can obtain the final result:

$$\begin{aligned} \begin{aligned} \mathop {\max }\limits _{{\mathbf{{z}}_1},{\mathbf{{z}}_2}} \;\;\;&- {\left\| {A{\mathbf{{z}}_1} - B{\mathbf{{z}}_2}} \right\| _1}\\ s.t.\;\;\;&{\mathbf{{e}}^{\textrm{T}}}{\mathbf{{z}}_1} = 1,\;\;{\mathbf{{e}}^{\textrm{T}}}{\mathbf{{z}}_2} = 1,\\&{\mathbf{{z}}_1} \ge 0,\;\;{\mathbf{{z}}_2} \ge 0. \end{aligned} \end{aligned}$$