Face-Specific Data Augmentation for Unconstrained Face Recognition

Abstract

We identify two issues as key to developing effective face recognition systems: maximizing the appearance variations of training images and minimizing appearance variations in test images. The former is required to train the system for whatever appearance variations it will ultimately encounter and is often addressed by collecting massive training sets with millions of face images. The latter involves various forms of appearance normalization for removing distracting nuisance factors at test time and making test faces easier to compare. We describe novel, efficient face-specific data augmentation techniques and show them to be ideally suited for both purposes. By using knowledge of faces, their 3D shapes, and appearances, we show the following: (a) We can artificially enrich training data for face recognition with face-specific appearance variations. (b) This synthetic training data can be efficiently produced online, thereby reducing the massive storage requirements of large-scale training sets and simplifying training for many appearance variations. Finally, (c) The same, fast data augmentation techniques can be applied at test time to reduce appearance variations and improve face representations. Together, with additional technical novelties, we describe a highly effective face recognition pipeline which, at the time of submission, obtains state-of-the-art results across multiple benchmarks. Portions of this paper were previously published by Masi et al. (European conference on computer vision, Springer, pp 579–596, 2016b, International conference on automatic face and gesture recognition, 2017).

Appendices

Preparing Generic 3D Heads and Backgrounds

When rendering novel views of faces, similarly to Masi et al. (2016b), we use ten generic 3D face shapes ${\mathscr {S}} = \{ \mathbf {S}_1,\ldots ,\mathbf {S}_{10} \}$ from the Basel Face Set (Paysan et al. 2009) to model head shape variations. These 3D faces are all aligned with each other and so can easily be manipulated and segmented: Only one shape needs to be manipulated and any modifications to this shape can easily be transfered to all others. Thus, selecting the 3D eye regions in these models to avoid cross-eyed results (Hassner et al. 2015) only needs to be done once.

These models only represent the facial region of the head. Masi et al. (2016b) therefore rendered only partial head views, without backgrounds, and this is presumably why Taigman et al. (2014) and Hassner et al. (2015) only used tight bounding boxes around the face center.

To allow rendering of the entire head and background, we leverage the alignment of these heads to modify them by completing the head shapes and adding a background plane. This is performed by stitching the ten 3D models to an additional, generic 3D structure containing head, ears, and neck and adding a plane representing a flat background.

The process of combining 3D faces to 3D heads is described in Fig. 12. We use the generic 3D head of Zhu et al. (2016). We remove its facial region and exchange it with the 3D faces of Paysan et al. (2009). To allow blending with different 3D faces (e.g., Fig. 12a), varying in sizes and shapes, we maintain an overlap belt with radius r = 2cm (Fig. 12b). Given an input 3D face (Fig. 12a), we merge it onto the head model using soft boundary blending. We first detect points on the overlap region of the face model. Each point $\mathbf {X}$ is then assigned a soft blending weight w:

$$\begin{aligned} w = \frac{1}{2} - \frac{1}{2} \cos \Big (\frac{\pi d}{r}\Big ), \end{aligned}$$

(5)

where d is the distance to the boundary. Next, $\mathbf {X}$ is adjusted to the new 3D position $\mathbf {X}'$ by:

$$\begin{aligned} \mathbf {X}' = w \mathbf {X} + (1 - w) \mathbf {P}_X, \end{aligned}$$

(6)

where $\mathbf {P}_X$ is the closest 3D point from the head. The result is a complete 3D face model (Fig. 12c). Ostensibly, an alternative to this method would be to scan new models, with complete 3D heads. Besides requiring less labor for scanning and alignment, the method described above was selected in order to minimize the differences between our recognition system and the one used by Masi et al. (2016b), including the use of the same 3D face shapes from in their system.

Fig. 12

Preparing generic 3D models. Head added to a generic 3D face along with two planes for background

To additionally preserve the background, we simply add two planes to the 3D model: one positioned just behind the head and another, perpendicular plane, on its right. This second plane is used to represent the background when the input face is rendered to a profile view, in which case the first plane is mapped to a line. Figure 12d shows the models we produced using one of the 3D face shapes of Paysan et al. (2009). Figure 12e shows the rendered view of this generic face from the profile pose used by our system.

Fast 3D Rendering Snippet Code

Given an input image ${\mathbf {I}}$ containing a face in unconstrained settings, we use the following simple procedure to render it to a desired new view using ${\mathbf {U}}$. The code in Fig. 13 for a Python code example explains the process.

Fig. 13

Python code snippet for 3D rendering at 2D image warping speed