Sparse Illumination Learning and Transfer for Single-Sample Face Recognition with Image Corruption and Misalignment

Abstract

Single-sample face recognition is one of the most challenging problems in face recognition. We propose a novel algorithm to address this problem based on a sparse representation based classification (SRC) framework. The new algorithm is robust to image misalignment and pixel corruption, and is able to reduce required gallery images to one sample per class. To compensate for the missing illumination information traditionally provided by multiple gallery images, a sparse illumination learning and transfer (SILT) technique is introduced. The illumination in SILT is learned by fitting illumination examples of auxiliary face images from one or more additional subjects with a sparsely-used illumination dictionary. By enforcing a sparse representation of the query image in the illumination dictionary, the SILT can effectively recover and transfer the illumination and pose information from the alignment stage to the recognition stage. Our extensive experiments have demonstrated that the new algorithms significantly outperform the state of the art in the single-sample regime and with less restrictions. In particular, the single-sample face alignment accuracy is comparable to that of the well-known Deformable SRC algorithm using multiple gallery images per class. Furthermore, the face recognition accuracy exceeds those of the SRC and Extended SRC algorithms using hand labeled alignment initialization.

Appendix

We proof Theorem 1 in this Appendix. First, eliminating the variable $E$ of problem (10) with

$$\begin{aligned} E= {D} - \overline{{V}}\otimes {\mathbf {1}}^T - \overline{{C}}{H}, \end{aligned}$$

(21)

Problem (10) can then be equivalently written as

$$\begin{aligned} \min _{\overline{{V}},\overline{{C}}, {H}} \,\,\Vert {D} - \overline{{V}}\otimes {\mathbf {1}}^T - \overline{{C}}{H}\Vert _F^2,\,\mathrm{s.t.\,} \overline{{C}}^T\overline{{C}} = {I}. \end{aligned}$$

(22)

As a basic result in least squares (Horn and Johnson 1985), the optimal ${H}$ can be written as

$$\begin{aligned} {H}^* = \overline{{C}}^T({D} - \overline{{V}}\otimes {\mathbf {1}}^T), \end{aligned}$$

(23)

for any $\overline{{V}}\in \mathbb {R}^{m\times p}$ and any $\overline{{C}}\in \mathbb {R}^{m\times k}$ such that $\overline{{C}}^T\overline{{C}} = {I}$. Substituting ${H}^*$ into (22) yields

$$\begin{aligned} \min _{\overline{{V}},\overline{{C}}} \,\,\Vert {P}_{\overline{{C}}}^\perp ({D} - \overline{{V}}\otimes {\mathbf {1}}^T)\Vert _F^2,\,\mathrm{s.t.\,} \overline{{C}}^T\overline{{C}} = {I}, \end{aligned}$$

(24)

where ${P}_{\overline{{C}}}^\perp = {I} - \overline{{C}}\overline{{C}}^T$ denotes the orthogonal complement projector of $\overline{{C}}$. It is also easy to show from (24) that a solution of $\overline{{V}}$ is

$$\begin{aligned}{}[\overline{{V}}^*]_i = \frac{1}{n} {D}_i{\mathbf {1}},\,i=1,...,p. \end{aligned}$$

(25)

Note that the solution $[\overline{{V}}^*]_i$ presents the mean vector of the data matrix ${D}_i$ corresponding to subject $i$. Furthermore, by letting ${U} = {D} - \overline{{V}}^*\otimes {\mathbf {1}}^T$, problem (24) becomes $\min _{\overline{{C}}^T\overline{{C}} = {I}} \mathrm{trace}({U}^T{P}_{\overline{{C}}}^\perp {U})$, and it is equivalent to

$$\begin{aligned} \overline{{C}}^* = \mathrm{arg}\max _{\overline{{C}}^T\overline{{C}} = {I}} \mathrm{trace}(\overline{{C}}^T{U}{U}^T\overline{{C}}). \end{aligned}$$

(26)

By Horn and Johnson (1985), an optimal solution $\overline{{C}}^*$ is known to be the $k$ principal eigenvector matrix of ${U}{U}^T$; i.e.,

$$\begin{aligned} \overline{{C}}^* = [~ \varvec{q}_1( {U}{U}^T ), \varvec{q}_2( {U}{U}^T ), \ldots , \varvec{q}_{k}( {U}{U}^T ) ~]. \end{aligned}$$

(27)

Hence, the problem solution (13) simply follows from (21), (23) (25), and (27). $\square$