Tackle balancing constraints in semi-supervised ordinal regression

Abstract

Semi-supervised ordinal regression (S²OR) has been recognized as a valuable technique to improve the performance of the ordinal regression (OR) model by leveraging available unlabeled samples. The balancing constraint is a useful approach for semi-supervised algorithms, as it can prevent the trivial solution of classifying a large number of unlabeled examples into a few classes. However, rapid training of the S²OR model with balancing constraints is still an open problem due to the difficulty in formulating and solving the corresponding optimization objective. To tackle this issue, we propose a novel form of balancing constraints and extend the traditional convex–concave procedure (CCCP) approach to solve our objective function. Additionally, we transform the convex inner loop (CIL) problem generated by the CCCP approach into a quadratic problem that resembles support vector machine, where multiple equality constraints are treated as virtual samples. As a result, we can utilize the existing fast solver to efficiently solve the CIL problem. Experimental results conducted on several benchmark and real-world datasets not only validate the effectiveness of our proposed algorithm but also demonstrate its superior performance compared to other supervised and semi-supervised algorithms

Appendices

Appendix A: Proof of Lemma 1

Firstly, we define some data subsets where $j \in \{1,\ldots ,r\}$:

$$\begin{aligned} \begin{aligned}&I_j^{low}(\theta )\overset{\text {def}}{=}\{i \in \{1,\ldots ,n\}:y_i=j,w^T\phi (x_i)-\theta \ge -1 \}, \\&I_j^{up}(\theta )\overset{\text {def}}{=}\{i \in \{1,\ldots ,n\}:y_i=j,w^T\phi (x_i)-\theta \le 1 \}, \\&I^{low}(\theta )\overset{\text {def}}{=}\{i \in \{n+1,\ldots ,n+u\}: -1 \le w^T\phi (x_i)-\theta \le 0 \}, \\&I^{up}(\theta )\overset{\text {def}}{=}\{i \in \{n+1,\ldots ,n+u\}: 0 < w^T\phi (x_i)- \theta \le 1 \}. \end{aligned} \end{aligned}$$

It is easy to see that $\theta _k$ is optimal if it minimizes the function:

$$\begin{aligned} e_k(\theta )&=C\sum _{j=1}^{k} \sum _{i \in I_j^{low}(\theta )}(w^T\phi (x_i)-\theta +1) \nonumber \\&\quad +C\sum _{j=k+1}^{r} \sum _{i \in I_j^{up}(\theta )}(-w^T\phi (x_i)+\theta +1) \nonumber \\&\quad +C^* \sum _{i \in I^{low}(\theta )}(w^T \phi (x_i)-\theta +1) \nonumber \\&\quad +C^*\sum _{i \in I^{up}(\theta )}(-w^T\phi (x_i)+\theta +1). \end{aligned}$$

(27)

Then, we obtain the derivative of $e_k(\theta )$ with respect to $\theta$:

$$\begin{aligned} \begin{aligned} \bigtriangledown e_k(\theta )&=-C\sum _{j=1}^{k} \vert I_j^{low}(\theta )\vert + C\sum _{j=k+1}^{r} \vert I_j^{up}(\theta ) \vert \\&\quad -C^*\vert I^{low}(\theta ) \vert + C^*\vert I^{up}(\theta ) \vert :=l_k(\theta ). \end{aligned} \end{aligned}$$

(28)

And through simple calculation, we have:

$$\begin{aligned} \begin{aligned} l_{k+1}(\theta )-l_k(\theta )=-C\vert I_{k+1}^{low}(\theta )\vert -C\vert I_{k+1}^{up}(\theta )\vert < 0. \end{aligned} \end{aligned}$$

(29)

In order to prove that the order of $\theta$ is no longer guaranteed in $\hbox {S}^2$OR problem, we construct a counterexample. Firstly, we find that the derivative $l_k(\theta )$ (28) is not a monotone function. What’s more, when $\theta$ tends to be positive infinity, we have $l_k(\theta )>0$ and when $\theta$ tends to be negative infinity, we have $l_k(\theta )<0$. In this case, we assume that $l_k(\theta )$ and $l_{k+1}(\theta )$ both own three zero points, and the first derivative of these points is not zero. Next, we set the zero points of $l_k(\theta )$ from left to right as $a_1$, $a_2$ and $a_3$, and the zero points of $l_{k+1}(\theta )$ as $b_1$, $b_2$ and $b_3$. According to (29), we assume one situation that $a_1<b_1<b_2<a_2<a_3<b_3$. Under such condition, $e_k(\theta )$ and $e_{k+1}(\theta )$ both own two local minimum points $a_1, a_3$ and $b_1, b_3$. Finally, we assume that point $a_3$ and point $b_1$ are global optimal solutions of $e_k(\theta )$ and $e_{k+1}(\theta )$. Then we have $\theta _k=a_3 > \theta _{k+1}=b_1$, and this situation conflicts with the order $\theta _k \le \theta _{k+1}$ we asked for. Therefore, we conclude that the order of $\theta$ is no longer guaranteed in $\hbox {S}^2$OR problem.

Appendix B: Proof of the DC formulation

In order to handle the non-convex objective function (5) better, we rewrite the loss of unlabeled samples:

$$\begin{aligned} H_1(\vert z \vert )=R_0(z)+R_0(-z)+ \text {const} , \end{aligned}$$

(30)

where $R_s(t)=\min \{ 1-s,\max \{ 0,1-t \} \}=H_1(t)-H_s(t)$ represents a ramp loss. Further more, we rewrite the loss term of unlabeled samples in the objective function (5) with the help of the ramp loss:

$$\begin{aligned}&\sum _{k=1}^{r-1} \sum _{i=n+1}^{n+u} H_1\left( \left| g\left( x_i^k\right) \right| \right) \nonumber \\&\quad = \sum _{k=1}^{r-1} \sum _{i=n+1}^{n+u} R_0\left( g\left( x_i^k\right) \right) +\sum _{k=1}^{r-1} \sum _{i=n+1}^{n+u} R_0\left( -g\left( x_i^k\right) \right) +\text {const} \nonumber \\&\quad =\sum _{k=1}^{r-1} \sum _{i=n+1}^{n+u} H_1\left( g\left( x_i^k\right) \right) +\sum _{k=1}^{r-1} \sum _{i=n+1}^{n+u} H_1\left( -g\left( x_i^k\right) \right) \nonumber \\&\qquad -\sum _{k=1}^{r-1} \sum _{i=n+1}^{n+u} H_0\left( g\left( x_i^k\right) \right) -\sum _{k=1}^{r-1} \sum _{i=n+1}^{n+u} H_0\left( -g\left( x_i^k\right) \right) +\text {const} . \end{aligned}$$

(31)

Then, according to the artificial labeled samples (14), the expression in (31) is in the new formulation:

$$\begin{aligned} \sum _{k=1}^{r-1} \sum _{i=n+1}^{n+u} H_1\left( \left| g\left( x_i^k\right) \right| \right)&= \sum _{k=1}^{r-1} \sum _{i=n+1}^{n+2u} H_1\left( y_i^k g\left( x_i^k\right) \right) \nonumber \\&\quad -\sum _{k=1}^{r-1} \sum _{i=n+1}^{n+2u} H_0\left( y_i^kg\left( x_i^k\right) \right) +\text {const}, \end{aligned}$$

(32)

where the constant does not affect the optimization problem obviously, and we can just ignore it.

Finally, original objective function is in the formulation of

$$\begin{aligned} \begin{aligned} J(\bar{w})&=\underbrace{\frac{1}{2}\Vert w \Vert ^2 +C \sum \nolimits _{k=1}^{r-1} \sum \nolimits _{i=1}^{n}H_1\left( y_i^kg\left( x_i^k\right) \right) +C^* \sum \nolimits _{k=1}^{r-1} \sum \nolimits _{i=n+1}^{n+2u} H_1\left( y_i^k g\left( x_i^k\right) \right) }_{o(\bar{w})} \\&\quad -\underbrace{C^* \sum _{k=1}^{r-1} \sum _{i=n+1}^{n+2u} H_0\left( y_i^k g\left( x_i^k\right) \right) }_{v(\bar{w})}. \end{aligned} \end{aligned}$$

(33)