Low-Rank Transfer Learning

Abstract

Real-world visual data are expensive to label for the purpose of training supervised learning algorithms. Leverage of auxiliary databases with well labeled data for the new task may save considerable labeling efforts. However, data in the auxiliary databases are often obtained under conditions that differ from those in the new task. Transfer learning provides techniques for transferring learned knowledge from a source domain to a target domain by mitigating the divergence. In this chapter, we discuss transfer learning in a generalized subspace where each target sample can be represented by some combination of source samples under a low-rank constraint. Under this constraint, the underlying structure of both source and target domains are considered in the knowledge transfer, which brings in three benefits: First, good alignment between domains is ensured in that only relevant data in some subspace of the source domain are used to reconstruct the data in the target domain. Second, the discriminative power of the source domain is naturally passed on to the target domain. Third, noisy information will be filtered out in the knowledge transfer. Extensive experiments on synthetic data, and important computer vision problems, e.g., face recognition application, visual domain adaptation for object recognition, demonstrate the superiority of the proposed approach over the existing, well-established methods.

© 2014 Springer. Reprinted, with permission, from International Journal of Computer Vision, August 2014, Volume 109, Issue 1–2, pp. 74–93.

Appendix

1.1 Proof of Theorem 2

Suppose $X_\text{ s }$ and $X_\text{ t }$ are strictly drawn from $\mathcal {S}_i$ and $\mathcal {T}_i$. We use $Y_\text{ s } = P^\text{ T } X_\text{ s }$ and $Y_\text{ t } = P^\text{ T } X_\text{ t }$ to denote their low-dimensional representations in the subspace $P$. Both $Y_\text{ s }$ and $Y_\text{ t }$ are of size $m \times n$. Therefore, the energy of PCA in the source domain $\mathcal {S}_i$ is:

$$\begin{aligned} F(P, \mathcal {S}_i) = \frac{1}{n-1} \left\| Y_\text{ s } - Y_\text{ s }*\frac{1}{n}\mathbf {e}\mathbf {e^\text{ T }}\right\| _F^2, \end{aligned}$$

(16)

where $\mathbf {e}$ is $\{\mathbf {e}_i = 1 | i = 1,2, \ldots , n\}$. For simplicity, we remove the constant term $\frac{1}{n-1}$ and replace $\frac{1}{n}\mathbf {e}\mathbf {e^\text{ T }}$ with the matrix $C$. Then the energy of PCA in $\mathcal {S}_i$ and $\mathcal {T}_i$ can be rewritten as:

$$\begin{aligned} F(P, \mathcal {S}_i)&= \left\| Y_\text{ s } - Y_\text{ s }*C\right\| _F^2.\end{aligned}$$

(17)

$$\begin{aligned} F(P, \mathcal {T}_i)&= \left\| Y_\text{ s } Z_i - Y_\text{ s }*C Z_i \right\| _F^2. \end{aligned}$$

(18)

Since $\widehat{Z}_i$ is non-singular, we have $\widehat{Z}_i\widehat{Z}_i^{-1} = \mathbf {I}$ and the above function can be rewritten as:

$$\begin{aligned} F(P, \mathcal {S}_i)&= \Vert Y_\text{ s } \widehat{Z}_i\widehat{Z}_i^{-1} - Y_\text{ s }*C \widehat{Z}_i\widehat{Z}_i^{-1} \Vert _F^2\nonumber \\&= \Vert \left( Y_\text{ s } \widehat{Z}_i - Y_\text{ s }*C \widehat{Z}_i \right) \widehat{Z}_i^{-1} \Vert _F^2\nonumber \\&\le \Vert Y_\text{ s } \widehat{Z}_i - Y_\text{ s }*C \widehat{Z}_i \Vert _F^2 \Vert \widehat{Z}_i^{-1} \Vert _F^2. \end{aligned}$$

(19)

Note that $\Vert Y_\text{ s } \widehat{Z}_i - Y_\text{ s }\,*\,C \widehat{Z}_i \Vert _F^2$ is the PCA energy of the data from the target domain that has been perturbed by $\widehat{Z}_i$. Therefore, $F(P, \widehat{\mathcal {T}}_i) = \Vert Y_\text{ s } \widehat{Z}_i - Y_\text{ s }*C \widehat{Z}_i \Vert _F^2$. Combining that with the inequality in (19) results in:

$$\begin{aligned} F(P, \mathcal {\widehat{T}}_i) \ge F(P, \mathcal {S}_i) \Vert \widehat{Z}_i^{-1}\Vert _F^{-2}. \end{aligned}$$

(20)

If we add $F(P, \mathcal {{T}}_i)$ to both sides of Inequality (20), then we derive Inequality (8).   $\blacksquare$

There are three points to be made. First, the difference between $F(P, \mathcal {T}_i)$ and $F(P, \widehat{\mathcal {T}}_i)$ is that the first one is the PCA energy of $Y_\text{ s } Z_i$ while the second one is the PCA energy of $Y_\text{ s } \widehat{Z}_i$ where $\widehat{Z}_i = Z_i + \gamma \mathbf {I}$. Compared with $Y_\text{ s } Z_i$, $Y_\text{ s } \widehat{Z}_i$ adds a small term to each vector. However, this will not cause a significant change in $F(P, \mathcal {T}_i)$. Therefore, term $\xi$ in Theorem 2 will not be very large. Second, although we only compare the PCA energy of $\mathcal {S}_i$ and $\mathcal {T}_i$ that are drawn from one subspace, this theorem is easily extended to any other subspace included in the subspace union. Finally, other subspace learning methods can be similarly proven, since they are all unified in the linear graph embedding framework [60]. For example, the proof for LDA holds the value of $\mathbf {Tr}(P^\text{ T }SP)$ fixed, where $S=S_\text{ b } + S_\text{ w }$, and then maximizes $\mathbf {Tr}(P^\text{ T }S_\text{ b }P)$ in the same way as was done for PCA.