Support matrix machine with pinball loss for classification

Abstract

Support vector machine (SVM) is one of the highly efficient classification algorithms. Unfortunately, it is designed only for input samples expressed as vectors. In real life, most input samples are naturally in matrix form and include structural information, such as electroencephalogram (EEG) signals and gray images. Support matrix machine (SMM), which can capture the latent structure within input matrices by regularizing the regression matrix to be low rank, is more suitable for matrix-form data than the SVM. However, the SMM adopts hinge loss, which is easily sensitive to noise and unstable to re-sampling. In this paper, to tackle this issue, we propose a new SMM with pinball loss (Pin−SMM), which can simultaneously consider the intrinsic structural information of input matrices and noise insensitivity. Our Pin−SMM is defined as a spectral elastic net with pinball loss, penalizing the rightly classified points. The optimization problem of Pin−SMM is also convex, which motivates us to construct the fast alternating direction method of multipliers (Fast ADMM) to solve it. Comprehensive experiments on two popular image datasets and an EEG dataset with different noises are conducted, and the experimental results confirm the effectiveness of our presented algorithm.

Appendix A. Convexity of Pin−SMM

In order to prove that problem (9) is convex, we first define

$$\begin{aligned}&\varPhi _{1}({\mathbf {W}}) = \frac{1}{2}{\text {tr}}({\mathbf {W}}^{T}{\mathbf {W}}), \nonumber \\&\varPhi _{2}({\mathbf {W}}) = \Vert {\mathbf {W}}\Vert _{*}\nonumber , \\&\varPhi _{3}({\mathbf {W}}, b) = \sum _{i=1}^{n}L_{\tau } ({\mathbf {X}}_{i},y_{i},f({\mathbf {X}}_{i})). \nonumber \end{aligned}$$

It is obvious that the function $\varPhi _{1}({\mathbf {W}})$ is convex. The nuclear norm of a matrix is a convex alternative of the matrix rank [10], which has been proved in Appendix A of the literature [43]. Therefore, $\varPhi _{2}({\mathbf {W}})$ is also convex. Next, we demonstrate that the $\varPhi _{3}({\mathbf {W}}, b)$ is convex. The sum of the convex functions is a convex function. We only need to prove that the function $L_{\tau }({\mathbf {X}}_{i},y_{i},f({\mathbf {X}}_{i}))$ is convex. Let $y_{i}f({\mathbf {X}}_{i}) - 1 = \mu$, and the pinball loss can be rewritten as:

$$\begin{aligned} L_{\tau }(\mu )=\left\{ \begin{aligned}&-\mu ,&\mu < 0, \\&\tau \mu ,&\mu \ge 0. \end{aligned}\right. \end{aligned}$$

(30)

Then we have

$$\begin{aligned} L_{\tau }(\mu )= \tau \max (0, \mu ) + \max (0, -\mu ), \end{aligned}$$

(31)

where $\tau > 0$. Thus the function $L_{\tau }(\mu )$ is convex. Because $f({\mathbf {X}}_{i}) = {\text {tr}}({\mathbf {W}}^{T}{\mathbf {X}}_{i})+b$, we have $L_{\tau }({\mathbf {X}}_{i},y_{i},f({\mathbf {X}}_{i}))$ is convex. So $\varPhi _{3}({\mathbf {W}}, b)$ is convex with respect to ${\mathbf {W}}$ and b, respectively.

Moreover, parameters $\lambda$ and C are positive; therefore, $\varPhi _{1}({\mathbf {W}}) + \lambda \varPhi _{2}({\mathbf {W}}) + C\varPhi _{3}({\mathbf {W}}, b)$ is convex. That is, the Pin−SMM is convex.

This completes the proof. $\Box$.