A robust one-class transfer learning method with uncertain data

Abstract

One-class classification aims at constructing a distinctive classifier based on one class of examples. Most of the existing one-class classification methods are proposed based on the assumptions that: (1) there are a large number of training examples available for learning the classifier; (2) the training examples can be explicitly collected and hence do not contain any uncertain information. However, in real-world applications, these assumptions are not always satisfied. In this paper, we propose a novel approach called uncertain one-class transfer learning with support vector machine (UOCT-SVM), which is capable of constructing an accurate classifier on the target task by transferring knowledge from multiple source tasks whose data may contain uncertain information. In UOCT-SVM, the optimization function is formulated to deal with uncertain data and transfer learning based on one-class SVM. Then, an iterative framework is proposed to solve the optimization function. Extensive experiments have showed that UOCT-SVM can mitigate the effect of uncertain data on the decision boundary and transfer knowledge from source tasks to help build an accurate classifier on the target task, compared with state-of-the-art one-class classification methods.

Appendix

1.1 Proof for Theorem 1

Assume that $\alpha _{1i} \ge 0,\alpha _{2j} \ge 0,\beta _{1i} \ge 0$ and $\beta _{2j} \ge 0$ are Lagrange multipliers. The Lagrange function of problem (8) can be given as

$$\begin{aligned} L&= ||\mathbf w _0||^2 + C_1 ||\mathbf v _1||^2 + C_2 ||\mathbf v _2||^2 - \rho _1 - \rho _2 + C_1 \sum _{i=1}^{|S_1|} \xi _{1i} + C_2 \sum _{j=1}^{|S_2|} \xi _{2j} \nonumber \\&\quad +\,\sum _{i=1}^{|S_1|} \alpha _{1i} [ \rho _1 -\xi _{1i} - (\mathbf w _0+\mathbf v _1)^T \overline{\mathbf{x }}_{1i} ] +\sum _{j=1}^{|S_2|} \alpha _{2j} [ \rho _2 -\xi _{2j} - (\mathbf w _0+\mathbf v _2)^T \overline{\mathbf{x }}_{2j} ] \nonumber \\&\quad -\,\sum _{i=1}^{|S_1|} \beta _{1i} \xi _{1i} - \sum _{j=1}^{|S_2|} \beta _{2j} \xi _{2j} \end{aligned}$$

(37)

where it has $\overline{\mathbf{x }}_{1i} = \mathbf x _{1i} + \Delta \overline{\mathbf{x }}_{1i}$ and $\overline{\mathbf{x }}_{2j} = \mathbf x _{2j} + \Delta \overline{\mathbf{x }}_{2j}$.

By differentiating the Lagrange function (37) with $\mathbf w _0,\mathbf v _1,\mathbf v _2,\rho _1,\rho _2,\xi _{1i}$ and $\xi _{2j}$, respectively, the following equations can be obtained.

$$\begin{aligned} \frac{\partial L}{\partial \mathbf w _0}&= 2\mathbf w _0 - \sum _{i=1}^{|S_1|} \alpha _{i1} \overline{\mathbf{x }}_{i1} - \sum _{j=1}^{|S_2|} \alpha _{2j} \overline{\mathbf{x }}_{2j}=0,\end{aligned}$$

(38)

$$\begin{aligned} \frac{\partial L}{\partial \mathbf v _1}&= 2 C_1 \mathbf v _1 - \sum _{i=1}^{|S_1|} \alpha _{1i} \overline{\mathbf{x }}_{1i}=0, \end{aligned}$$

(39)

$$\begin{aligned} \frac{\partial L}{\partial \mathbf v _2}&= 2 C_2 \mathbf v _2- \sum _{j=1}^{|S_2|} \alpha _{2j} \overline{\mathbf{x }}_{2j}=0,\end{aligned}$$

(40)

$$\begin{aligned} \frac{\partial L}{\partial \rho _1}&= -1+\sum _{i=1}^{|S_1|} \alpha _{1i}=0, \end{aligned}$$

(41)

$$\begin{aligned} \frac{\partial L}{\partial \rho _2}&= -1+\sum _{j=1}^{|S_2|} \alpha _{2j}=0, \end{aligned}$$

(42)

$$\begin{aligned} \frac{\partial L}{\partial \xi _{1i}}&= C_1 - \alpha _{1i} - \beta _{1i} =0, \ \ i=1, \ldots , |S_1| \end{aligned}$$

(43)

$$\begin{aligned} \frac{\partial L}{\partial \xi _{2j}}&= C_2 - \alpha _{2j} - \beta _{2j} =0, \ \ j=1, \ldots , |S_2| \end{aligned}$$

(44)

From Eqs. (38)–(44), it is easy to deduce that

$$\begin{aligned}&\mathbf w _0 = \frac{1}{2} \left( \sum _{i=1}^{|S_1|}\alpha _{1i} \overline{\mathbf{x }}_{1i}+ \sum _{j=1}^{|S_2|} \alpha _{2j} \overline{\mathbf{x }}_{2j}\right) , \end{aligned}$$

(45)

$$\begin{aligned}&\mathbf v _1 = \frac{1}{ 2 C_1}\sum _{i=1}^{|S_1|} \alpha _{1i}\overline{\mathbf{x }}_{1i}, \end{aligned}$$

(46)

$$\begin{aligned}&\mathbf v _2 = \frac{1}{ 2 C_2}\sum _{j=1}^{|S_2|} \alpha _{2j}\overline{\mathbf{x }}_{2j}, \end{aligned}$$

(47)

$$\begin{aligned}&\sum _{i=1}^{|S_1|} \alpha _{1i} = 1, \end{aligned}$$

(48)

$$\begin{aligned}&\sum _{j=1}^{|S_2|} \alpha _{2j} = 1,\end{aligned}$$

(49)

$$\begin{aligned}&C_1 = \alpha _{1i} + \beta _{1i}, \ \ i=1, \ldots , |S_1|\end{aligned}$$

(50)

$$\begin{aligned}&C_2 =\alpha _{2j} + \beta _{2j}, \ \ \ j=1, \ldots , |S_2| \end{aligned}$$

(51)

Since it has $\beta _{1i} \ge 0$ and $\beta _{2j} \ge 0$, from (50) and (51), we can obtain

$$\begin{aligned}&0 \le \alpha _{1i} \le C_1, \ \ i=1, \ldots , |S_1| \end{aligned}$$

(52)

$$\begin{aligned}&0 \le \alpha _{2j} \le C_2, \ \ j=1, \ldots , |S_2| \end{aligned}$$

(53)

By substituting (38)–(53) into the Lagrange function (37), the dual form of problem (8) can be written as

$$\begin{aligned} \max&-\frac{1}{2} \sum _{i=1}^{|S_1|} \sum _{j=1}^{|S_2|} \alpha _{1i} \overline{\mathbf{x }}_{1i}^T \overline{\mathbf{x }}_{2j} \alpha _{2j} - \frac{C_1+1}{4C_1} \sum _{h=1}^{|S_1|} \sum _{g=1}^{|S_1|} \alpha _{1h} \overline{\mathbf{x }}_{1h}^T \overline{\mathbf{x }}_{1g} \alpha _{1g} \\&- \frac{C_2+1}{4C_2} \sum _{p=1}^{|S_2|} \sum _{k=1}^{|S_2|} \alpha _{2p} \overline{\mathbf{x }}_{2p}^T \overline{\mathbf{x }}_{2k} \alpha _{2k}\\ \mathrm{s.t.}&\sum _{i=1}^{|S_1|} \alpha _{1i} =1, \ \ \ 0 \le \alpha _{1i} \le C_1, \ \ i=1, \ldots , |S_1| \\&\sum _{j=1}^{|S_2|} \alpha _{2j} =1, \ \ \ 0 \le \alpha _{2j} \le C_2, \ \ j=1, \ldots , |S_2| \end{aligned}$$

$\square$

1.2 Proof for Theorem 2

In Theorem 2, we fix $\mathbf w _0,\mathbf v _1,\mathbf v _2,\rho _1$ and $\rho _2$ to be $\overline{\mathbf{w }}_0,\overline{\mathbf{v }}_1,\overline{\mathbf{v }}_2,\overline{\rho }_1$ and $\overline{\rho }_2$, respectively, and attempt to minimize the value of the objective function (7) by optimizing $\Delta \mathbf x _{1i}$ and $\Delta \mathbf x _{2j}$. From (7), the objective function’s value is determined by $\sum _{t=1}^{2} \sum _{i=1}^{|S_t|} \xi _{ti}$ since $\mathbf w _0,\mathbf v _1,\mathbf v _2,\rho _1$ and $\rho _2$ are fixed. Hence, we need to optimize $\Delta \mathbf x _{1i}$ and $\Delta \mathbf x _{2j}$ to minimize $\sum _{t=1}^{2} \sum _{ i=1}^{|S_t|} \xi _{ti}$.

Each training example $\mathbf x _{ti}$ ($i=1, \ldots , |S_t|, t=1, 2$) is associated with an error term $\xi _{ti}$ and the minimization of $\sum _{t=1}^{2} \sum _{ i=1}^{|S_t|} \xi _{ti}$ can be decomposed into subproblems of minimizing each error term $\xi _{ti}$:

$$\begin{aligned} \xi _{ti}&= \max \left\{ 0, \rho _t - (\overline{\mathbf{w }}_0 + \overline{\mathbf{v }}_t)^T (\mathbf x _{ti} + \Delta \mathbf x _{ti}) \right\} \nonumber \\&= \max \left\{ 0, \rho _t - (\overline{\mathbf{w }}_0 + \overline{\mathbf{v }}_t)^T\mathbf x _{ti} - (\overline{\mathbf{w }}_0 + \overline{\mathbf{v }}_t)^T \Delta \mathbf x _{ti} \right\} \end{aligned}$$

(54)

From Eq. (54), it is seen that we can minimize $\xi _{ti}$ by maximizing $(\overline{\mathbf{w }}_0 + \overline{\mathbf{v }}_t)^T \Delta \mathbf x _{ti}$. According to the Cauchy–Schwarz inequality [59], it has

$$\begin{aligned} - ||\overline{\mathbf{w }}_0 + \overline{\mathbf{v }}_t|| \cdot ||\mathbf x _{ti}|| \le (\overline{\mathbf{w }}_0 + \overline{\mathbf{v }}_t)^T \Delta \mathbf x _{ti} \le ||\overline{\mathbf{w }}_0 + \overline{\mathbf{v }}_t|| \cdot ||\mathbf x _{ti}|| \end{aligned}$$

(55)

In Eq. (55) becomes equation if and only if $\Delta \mathbf x _{ti} = c (\overline{\mathbf{w }}_0 + \overline{\mathbf{v }}_t)$, where $c$ is a constant number. Since $\Delta \mathbf x _{ti}$ is bounded by $\delta _{ti}$, the optimal value of $\Delta \mathbf x _{ti}$ is

$$\begin{aligned} \Delta \mathbf x _{ti}= \delta _{ti} \frac{\overline{\mathbf{w }}_0 + \overline{\mathbf{v }}_t}{||\overline{\mathbf{w }}_0 + \overline{\mathbf{v }}_t||}, \ \ i=1, \ldots , |S_t|, \ \ t=1, 2. \end{aligned}$$

(56)

$\square$

1.3 Proof for Theorem 3

We fix $\overline{\mathbf{w }}_0,\overline{\mathbf{v }}_1,\overline{\mathbf{v }}_2,\overline{\rho }_1$ and $\overline{\rho }_2$, and focus on minimizing each $\xi =\max \{ 0, \overline{\rho }_t - \frac{1}{2} \sum _{h=1}^{2} \sum _{j=1}^{|S_h|} \alpha _{hj} K(\mathbf x _{hj}+\Delta \overline{\mathbf{x }}_{hj}, \mathbf x + \Delta \mathbf x ) - \frac{1}{2 C_t}\sum _{j=1}^{|S_t|} \alpha _{tj} K(\mathbf x _{tj}+\Delta \overline{\mathbf{x }}_{tj}, \mathbf x + \Delta \mathbf x )$ ($\mathbf x \in S_t, t=1, 2$) over $\Delta \mathbf x$. According to the first order Taylor expansion of $K(\cdot )$ in Eq. (21), it is easy to deduce

$$\begin{aligned}&\frac{1}{2} \sum _{h=1}^{2} \sum _{j=1}^{|S_h|} \alpha _{hj} K(\mathbf x _{hj}+\Delta \overline{\mathbf{x }}_{hj}, \mathbf x + \Delta \mathbf x ) + \frac{1}{2 C_t}\sum _{j=1}^{|S_t|} \alpha _{tj} K(\mathbf x _{tj}+\Delta \overline{\mathbf{x }}_{tj}, \mathbf x + \Delta \mathbf x ) \nonumber \\&\quad = \frac{1}{2} \sum _{h=1}^{2} \sum _{j=1}^{|S_h|} \alpha _{hj} K(\mathbf x _{hj}+\Delta \overline{\mathbf{x }}_{hj}, \mathbf x ) + \Delta \mathbf x ^T \frac{1}{2} \sum _{h=1}^{2} \sum _{j=1}^{|S_h|} \alpha _{hj} K'(\mathbf x _{hj}+\Delta \overline{\mathbf{x }}_{hj}, \mathbf x ) \ \ \ \ \ \ \nonumber \\&\qquad + \frac{1}{2 C_t}\sum _{j=1}^{|S_t|} \alpha _{tj} K(\mathbf x _{tj}+\Delta \overline{\mathbf{x }}_{tj}, \mathbf x ) + \Delta \mathbf x ^T \frac{1}{2 C_t}\sum _{j=1}^{|S_t|} \alpha _{tj} K'(\mathbf x _{tj}+\Delta \overline{\mathbf{x }}_{tj}, \mathbf x ) \ \ \ \ \ \ \ \ \ \ \ \nonumber \\&\quad = \frac{1}{2} \sum _{h=1}^{2} \sum _{j=1}^{|S_h|} \alpha _{hj} K(\mathbf x _{hj}+\Delta \overline{\mathbf{x }}_{hj}, \mathbf x ) + \frac{1}{2 C_t}\sum _{j=1}^{|S_t|} \alpha _{tj} K(\mathbf x _{tj}+\Delta \overline{\mathbf{x }}_{tj}, \mathbf x ) \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \nonumber \\&\qquad + \Delta \mathbf x ^T \left[ \frac{1}{2} \sum _{h=1}^{2} \sum _{j=1}^{|S_h|} \alpha _{hj} K'(\mathbf x _{hj}+\Delta \overline{\mathbf{x }}_{hj}, \mathbf x ) + \frac{1}{2 C_t}\sum _{j=1}^{|S_t|} \alpha _{tj} K'(\mathbf x _{tj}+\Delta \overline{\mathbf{x }}_{tj}, \mathbf x ) \right] \qquad \end{aligned}$$

(57)

Similar to Sect. 8.2, by using the Cauchy–Schwarz inequality, the optimal value of $\Delta \mathbf x _{ti}$ is as follows

$$\begin{aligned} \Delta \mathbf{x}_{ti}= \delta _{ti } \frac{\mathbf{u}_{ti}}{||\mathbf{u}_{ti}||}, \ \ t=1, 2 \end{aligned}$$

where it has

$$\begin{aligned} \mathbf u _{ti}= \frac{1}{2} \sum _{h=1}^{2} \sum _{j=1}^{|S_h|} \alpha _{hj} K'(\mathbf x _{hj}+\Delta \overline{\mathbf{x }}_{hj}, \mathbf x ) + \frac{1}{2 C_t} \sum _{j=1}^{|S_t|} \alpha _{tj} K'(\mathbf x _{tj}+\Delta \overline{\mathbf{x }}_{tj}, \mathbf x ). \end{aligned}$$

$\square$

1.4 Proof for Theorem 4

Let $\alpha _{ti}\ge 0$ and $\beta _{ti} \ge 0$ be Lagrange multipliers. Based on the Lagrange multipliers, the Lagrange function of problem (29) can be given as

$$\begin{aligned} L&= ||\mathbf w ||^2 - {\mathbb {\rho }}^T \mathbf e + \sum _{t=1}^{K} C_t \sum _{j=1}^{|S_t|} \xi _{ti} \nonumber \\&\quad + \sum _{t=1}^{K} \sum _{i=1}^{|S_t|} \alpha _{ti} ({\mathbb {\rho }}^T \mathbf e _t - \xi _{ti} - \mathbf w ^T \overline{\mathbf{z }}(\mathbf x _{ti},t)) - \sum _{t=1}^{K} \sum _{i=1}^{|S_t|} \beta _{ti} \end{aligned}$$

(58)

Differentiating the Lagrange function (58) with $\mathbf w ,{\mathbb {\rho }},\xi _{ti}$ leads to

$$\begin{aligned}&\frac{\partial L}{\partial \mathbf w }= 2 \mathbf w - \sum _{t=1}^{K} \sum _{i=1}^{|S_t|} \alpha _{ti} \overline{\mathbf{z }}(\mathbf x _{ti},t) =0 \ \ \ \ \ \ \ \ \ \ \ \ \end{aligned}$$

(59)

$$\begin{aligned}&\frac{\partial L}{\partial {\mathbb {\rho }}}= - \mathbf e + \sum _{t=1}^{K} \sum _{i=1}^{|S_t|} \alpha _{ti} \mathbf e _t =0, \end{aligned}$$

(60)

$$\begin{aligned}&\frac{\partial L}{\partial \xi _{ti}}= C_t - \alpha _{ti} - \beta _{ti} =0. \end{aligned}$$

(61)

According to Eqs. (59)–(61), we can obtain

$$\begin{aligned}&\mathbf w = \frac{1}{2} \sum _{t=1}^{K} \sum _{i=1}^{|S_t|} \alpha _{ti} \overline{\mathbf{z }}(\mathbf x _{ti},t), \end{aligned}$$

(62)

$$\begin{aligned}&\sum _{t=1}^{K} \sum _{i=1}^{|S_t|} \alpha _{ti} \mathbf e _t =\mathbf e , \end{aligned}$$

(63)

$$\begin{aligned}&0 \le \alpha _{ti} \le C_{t}. \ \ \ \ \ \ \ \ \ \end{aligned}$$

(64)

By substituting (62)–(64) to problem (29), the dual form can be given as

$$\begin{aligned} \max&- \frac{1}{4} \sum _{t=1}^{K} \sum _{h=1}^{K} \sum _{i=1}^{|S_t|} \sum _{j=1}^{|S_h|} \alpha _{ti} \overline{\mathbf{z }}(\mathbf x _{ti},t)^T \overline{\mathbf{z }}(\mathbf x _{hj},h) \alpha _{hj}, \nonumber \\ \mathrm{s.t.}&\sum _{t=1}^{K} \sum _{j=1}^{|S_t|} \alpha _{ti} \mathbf e _t =\mathbf e , \nonumber \\&0 \le \alpha _{ti} \le C_{t}, \ \ i=1, \ldots , |S_t|, \ \ t=1, \ldots , K. \end{aligned}$$

(65)

$\square$

1.5 Proof for Theorem 6

We fix $\mathbf w ^{\phi }$ and $\rho$ to be $\overline{\mathbf{w }}^{\phi }$ and $\overline{\rho }$, respectively, and minimize each $\xi _{hj} =\max \{ 0, \overline{\rho }_h^T \mathbf e _h - (\mathbf w ^{\phi })^T \phi (\overline{\mathbf{z }}(\mathbf x _{hj}, h)) \}$ ($\mathbf x _{hj} \in S_h, h=1, \ldots , K$) over $\Delta \mathbf x _{hj}$. Since $\overline{\rho }_h^T \mathbf e _h$ is known, we minimize $\xi _{hj}$ by maximizing $(\mathbf w ^{\phi })^T \phi (\overline{\mathbf{z }}(\mathbf x _{hj}, h))$. Replacing $\overline{\mathbf{z }}(\mathbf x _{hj}, h)$ with $\phi (\overline{\mathbf{z }}(\mathbf x _{hj}, h))$ in Eq. (31) leads to

$$\begin{aligned} \mathbf w ^{\phi } = \frac{1}{2} \sum _{t=1}^{K} \sum _{i=1}^{|S_t|} \alpha _{ti} \phi (\overline{\mathbf{z }}(\mathbf x _{ti},t)) \end{aligned}$$

(66)

By employing the first order Taylor expansion of $K(\cdot )$ in Eq. (21) and substituting Eq. (66) into $(\mathbf w ^{\phi })^T \phi (\overline{\mathbf{z }}(\mathbf x _{hj}, h))$, it has

$$\begin{aligned}&(\mathbf w ^{\phi })^T \phi (\overline{\mathbf{z }}(\mathbf x _{hj}, h)) \nonumber \\&\quad = \frac{1}{2} \sum _{t=1}^{K} \sum _{i=1}^{|S_t|} \alpha _{ti} \phi (\overline{\mathbf{z }}(\mathbf x _{ti},t))^T \phi (\overline{\mathbf{z }}(\mathbf x _{hj}, h)) \nonumber \\&\quad = \frac{1}{2} \sum _{t=1}^{K} \sum _{i=1}^{|S_t|} \alpha _{ti} K(\mathbf z (\mathbf x _{ti},t)+ \Delta \overline{\mathbf{z }}(\mathbf x _{ti},t), \mathbf z (\mathbf x _{hj}, h)+\Delta \mathbf z (\mathbf x _{hj}, h)) \nonumber \\&\quad = \frac{1}{2} \sum _{t=1}^{K} \sum _{i=1}^{|S_t|} \alpha _{ti} K(\mathbf z (\mathbf x _{ti},t)+ \Delta \overline{\mathbf{z }}(\mathbf x _{ti},t), \mathbf z (\mathbf x _{hj},h)) \nonumber \\&\qquad + \frac{1}{2} \Delta \mathbf z (\mathbf x _{hj},h)^T \sum _{t=1}^{K} \sum _{i=1}^{|S_t|} \alpha _{ti} K'(\mathbf z (\mathbf x _{ti},t)+ \Delta \overline{\mathbf{z }}(\mathbf x _{ti},t), \mathbf z (\mathbf x _{hj},h)) \end{aligned}$$

(67)

By utilizing the Cauchy–Schwarz inequality, the optimal value of $\Delta \mathbf x _{hj}$ is

$$\begin{aligned} \Delta \mathbf z (\mathbf x _{hj},h) = \delta _{hj} \frac{\widetilde{\mathbf{u }}_{hj}}{||\widetilde{\mathbf{u }}_{hj}||}, \ \ j=1, \ldots , |S_h|, \ \ h=1, \ldots , K \end{aligned}$$

(68)

where it has

$$\begin{aligned} \widetilde{\mathbf{u }}_{hj}= \sum _{t=1}^{K} \sum _{i=1}^{|S_t|} \alpha _{ti} K'\left( \mathbf z (\mathbf x _{ti},t)+ \Delta \overline{\mathbf{z }}(\mathbf x _{ti},t), \mathbf z (\mathbf x _{hj},h)\right) . \end{aligned}$$

$\square$