Generalized Transfer Subspace Learning Through Low-Rank Constraint

Abstract

It is expensive to obtain labeled real-world visual data for use in training of supervised algorithms. Therefore, it is valuable to leverage existing databases of labeled data. However, the data in the source databases is often obtained under conditions that differ from those in the new task. Transfer learning provides techniques for transferring learned knowledge from a source domain to a target domain by finding a mapping between them. In this paper, we discuss a method for projecting both source and target data to a generalized subspace where each target sample can be represented by some combination of source samples. By employing a low-rank constraint during this transfer, the structure of source and target domains are preserved. This approach has three benefits. First, good alignment between the domains is ensured through the use of only relevant data in some subspace of the source domain in reconstructing the data in the target domain. Second, the discriminative power of the source domain is naturally passed on to the target domain. Third, noisy information will be filtered out during knowledge transfer. Extensive experiments on synthetic data, and important computer vision problems such as face recognition application and visual domain adaptation for object recognition demonstrate the superiority of the proposed approach over the existing, well-established methods.

Appendix

Proof of Lemma 1

Proof

$$\begin{aligned} \Vert AB\Vert _F^2&= \sum _j \Vert A [B]_{:,j}\Vert _2^2 \le \sum _j \Vert A\Vert _F^2\Vert [B]_{:,j} \Vert _2^2 \\&= \Vert A\Vert _F^2 \sum _j \Vert [B]_{:,j}\Vert _2^2 = \Vert A\Vert _F^2 \Vert B\Vert _F^2. \end{aligned}$$

$\square$

Next, due to space limitations, we only show a proof of Theorem 2 based on a specific learning method, Principal Component Analysis (PCA). Proof of others follows a similar scheme.

Proof of Theorem 2

Proof

Suppose $X_\mathrm{s}$ and $X_\mathrm{t}$ are strictly drawn from $\mathcal {S}_i$ and $\mathcal {T}_i$. We use $Y_\mathrm{s} = P^\mathrm{T} X_\mathrm{s}$ and $Y_\mathrm{t} = P^\mathrm{T} X_\mathrm{t}$ to denote their low-dimensional representations in the subspace $P$. Both $Y_\mathrm{s}$ and $Y_\mathrm{t}$ are of size $m \times n$. Therefore, the energy of PCA in the source domain $\mathcal {S}_i$ is:

$$\begin{aligned} F(P, \mathcal {S}_i) = \frac{1}{n-1} \left\| Y_\mathrm{s} - Y_\mathrm{s}*\frac{1}{n}\mathbf {e}\mathbf {e^\mathrm{T}}\right\| _F^2, \end{aligned}$$

(17)

where $\mathbf {e}$ is $\{\mathbf {e}_i = 1 | i = 1,2, \ldots , n\}$. For simplicity, we remove the constant term $\frac{1}{n-1}$ and replace $\frac{1}{n}\mathbf {e}\mathbf {e^\mathrm{T}}$ with the matrix $C$. Then the energy of PCA in $\mathcal {S}_i$ and $\mathcal {T}_i$ can be rewritten as:

$$\begin{aligned} F(P, \mathcal {S}_i)&= \left\| Y_\mathrm{s} - Y_\mathrm{s}*C\right\| _F^2.\end{aligned}$$

(18)

$$\begin{aligned} F(P, \mathcal {T}_i)&= \left\| Y_\mathrm{s} Z_i - Y_\mathrm{s}*C Z_i \right\| _F^2. \end{aligned}$$

(19)

Since $\widehat{Z}_i$ is non-singular, we have $\widehat{Z}_i\widehat{Z}_i^{-1} = \mathbf {I}$ and the above function can be rewritten as:

$$\begin{aligned} F(P, \mathcal {S}_i)&= \Vert Y_\mathrm{s} \widehat{Z}_i\widehat{Z}_i^{-1} - Y_\mathrm{s}*C \widehat{Z}_i\widehat{Z}_i^{-1} \Vert _F^2\nonumber \\&= \Vert \left( Y_\mathrm{s} \widehat{Z}_i - Y_\mathrm{s}*C \widehat{Z}_i \right) \widehat{Z}_i^{-1} \Vert _F^2\nonumber \\&\le \Vert Y_\mathrm{s} \widehat{Z}_i - Y_\mathrm{s}*C \widehat{Z}_i \Vert _F^2 \Vert \widehat{Z}_i^{-1} \Vert _F^2. \end{aligned}$$

(20)

Note that $\Vert Y_\mathrm{s} \widehat{Z}_i - Y_\mathrm{s}*C \widehat{Z}_i \Vert _F^2$ is the PCA energy of the data from the target domain that has been perturbed by $\widehat{Z}_i$. Therefore, $F(P, \widehat{\mathcal {T}}_i) = \Vert Y_\mathrm{s} \widehat{Z}_i - Y_\mathrm{s}*C \widehat{Z}_i \Vert _F^2$. Combining that with the inequality in (20) results in:

$$\begin{aligned} F(P, \mathcal {\widehat{T}}_i) \ge F(P, \mathcal {S}_i) \Vert \widehat{Z}_i^{-1}\Vert _F^{-2}. \end{aligned}$$

(21)

If we add $F(P, \mathcal {{T}}_i)$ to both sides of Inequality (21), then we derive Inequality (8). $\square$

There are three points to be made. First, the difference between $F(P, \mathcal {T}_i)$ and $F(P, \widehat{\mathcal {T}}_i)$ is that the first one is the PCA energy of $Y_\mathrm{s} Z_i$ while the second one is the PCA energy of $Y_\mathrm{s} \widehat{Z}_i$ where $\widehat{Z}_i = Z_i + \gamma \mathbf {I}$. Compared with $Y_\mathrm{s} Z_i$, $Y_\mathrm{s} \widehat{Z}_i$ adds a small term to each vector. However, this will not cause a significant change in $F(P, \mathcal {T}_i)$. Therefore, term $\xi$ in Theorem 2 will not be very large. Second, although we only compare the PCA energy of $\mathcal {S}_i$ and $\mathcal {T}_i$ that are drawn from one subspace, this theorem is easily extended to any other subspace included in the subspace union. Finally, other subspace learning methods can be similarly proven, since they are all unified in the linear graph embedding framework (Yan et al. 2007). For example, the proof for LDA holds the value of $\mathbf {Tr}(P^\mathrm{T}SP)$ fixed, where $S=S_\mathrm{b} + S_\mathrm{w}$, and then maximizes $\mathbf {Tr}(P^\mathrm{T}S_\mathrm{b}P)$ in the same way as was done for PCA.