InterFace: A software package for face image warping, averaging, and principal components analysis
- 1.6k Downloads
We describe InterFace, a software package for research in face recognition. The package supports image warping, reshaping, averaging of multiple face images, and morphing between faces. It also supports principal components analysis (PCA) of face images, along with tools for exploring the “face space” produced by PCA. The package uses a simple graphical user interface, allowing users to perform these sophisticated image manipulations without any need for programming knowledge. The program is available for download in the form of an app, which requires that users also have access to the (freely available) MATLAB Runtime environment.
KeywordsFace processing Morphing Principal components analysis
Psychological research in face perception has benefited greatly from advances in computer graphics. For example, morphing software allows us to test hypotheses about the ways faces are recognized by creating high-quality images that blend pictures in sophisticated ways. That technique can be employed by psychologists with a wide range of interests—for example, the perception of identity, expression, or social attributes (e.g., Beale & Keil, 1995; Calder, Young, Perrett, Etcoff, & Rowland, 1996; Young et al., 1997). Other image manipulation techniques allow us to alter images in ways that we predict will affect judgments of gender, age, race, or any number of other psychologically relevant dimensions (e.g., Busey, 1998; Oosterhof & Todorov, 2008; Stewart et al., 2012; Walker & Tanaka, 2003). In short, the facility to manipulate images in well-specified ways opens up the opportunity to design perceptual experiments that were impossible in the era before widely available graphical computers.
A second benefit of computational graphics is the ability it brings to analyze large sets of face images. Rather than generating novel pictures, some research questions can best be addressed by a statistical analysis of large sets of unmodified images—for example, when asking which physical properties of faces predict consistent social attributions (Nestor, Plaut, & Behrmann, 2013; Scheuchenpflug, 1999; Tredoux, 2002). Of course, these two approaches are related—if the analysis of image sets throws up a statistical regularity (let us say, a systematic difference between kindly and threatening faces), then it should be possible to use this finding to manipulate a novel set of images—perhaps rendering them more kindly or more threatening.
Although these techniques have been very prevalent in the past 20 years of face-processing research, they are not widely available to the whole research community. Laboratories specializing in such research have typically developed in-house bespoke software, and its use normally requires programming ability. Furthermore, because labs do not typically generate software for use by others, these programs are not generally user-friendly or well-documented enough to make sharing easy, even between collaborating partners. Although some face-related software is commonly available (e.g., Fantamorph—www.fantamorph.com; or Psychomorph—Tiddeman, Burt, & Perrett, 2001), the programs tend to be highly specific in their function, or not straightforward for novice users.
Shape and texture remapping Any face can be warped to a different shape. For example, the face of one person can be manipulated to the shape of a second person, or to a shape that is particularly masculine, friendly or smiling, and so forth. Faces can also be morphed together to produce blends.
Facial averaging Many faces can be averaged together. This technique can be used to observe regularities in different groups—for example, an average of faces that have been rated friendly might be compared to an average of faces that have been rated fierce. Alternatively, many different images of the same person might be combined to provide a single, average version of that person.
Principal components analysis This is a technique for extracting the key dimensions of face images. The program delivers these dimensions for further analysis, and also provides an interactive “reconstruction” tool, allowing users to manipulate the facial components independently—a facility that is useful for projects aiming to understand how different sources of facial information are coded in images.
The InterFace manual provides detailed instructions and examples. Here we will describe the approach taken in the design of the software, and illustrate some of its uses at a conceptual level. We will describe its main properties, though many more detailed features are available to users of the software.
Main concepts: Shape and texture
We next consider the texture of a face. “Texture” is a shorthand label for all the information in a face that is not carried by the position of the key (fiducial) points. This includes information about the reflectance properties, the lighting and surface information, and information due to the camera characteristics. To consider this information separately from shape, all faces are warped to a standard shape. InterFace provides a standard shape that can be used, but users also have the option to define their own template for this purpose. The key issue is that within any set of faces under analysis, the “texture” of each face will be defined as the image resulting from morphing the original to a standard shape. We refer to the resulting images as shape-free faces (Craw, 1995; Craw & Cameron, 1991). This is because shape does not discriminate between the faces in the set—following the standardization, they all have the same shape. Figure 1 illustrates the separation of a particular image into its constituent shape and texture (for further examples, see Hancock, Burton, & Bruce, 1996; Itz, Schweinberger, Schulz, & Kaufmann, 2014; Schulz, Kaufmann, Walther, & Schweinberger, 2012).
Use of the software
Reshaping a face
A number of different types of averages can be created within InterFace, each with potential uses in psychological research.
Since the shape of faces is coded as an ordered set of x-y-coordinates, it is straightforward to compute the average of any set of these. The average fiducial points for particular sets can be useful in a number of research settings. For example, how does the average shape of a set of men differ from the average of a set of women? Such questions have previously been answered through laborious measurement (Bruce et al., 1993; Burton, Bruce, & Dench, 1993). However, this software delivers the ability to compute and display shape averages very easily.
The construction of face averages is also very useful in other settings. For example, the standard shape template in InterFace was derived as the average of a large and diverse set of faces. But some research questions might require normalization by more restrictive criteria—that is, norms based on a single sex, race, or age of a face. Similarly, all of the uses of face reshaping, described in the previous section, could be used with average face shapes. One might want to ask how a European face looks when it is morphed to the average shape of a set of Chinese faces, or the average of a set of men or women, young or old people. In short, this facility allows one to derive “norms” on which to base further image analysis.
Principal components analysis: PCA
PCA has become a very important part of face perception research (Kirby & Sirovich, 1990; Phillips, Moon, Rizvi, & Rauss, 2000; Turk & Pentland, 1991). The technique provides a statistical description of a set of face images, by extracting the dimensions of variability (eigenvectors, or “eigenfaces”) in the order of the variances they explain. Thus, early components capture gross variation in the image set, and later components capture more fine-grained variation. This technique, along with other, related techniques such as factor analysis, is useful in data reduction when a relatively small number of dimensions captures a large proportion of the set variance.
PCA is especially popular in psychological face research because it provides an operationalization of face space: a space with metric dimensions into which faces can be placed (Valentine, 1991). The typical use of PCA takes a large number of faces to derive a relatively small number of dimensions, in which any face image can be described, as either a set of coordinates in that space or (equivalently) a weighted sum of the eigenvectors. For an introduction to this technique, see Valentin, Abdi, and O’Toole (1994), or for a full mathematical account, see Gong, McKenna, and Psarrou (2000).
InterFace provides the facility to carry out PCA straightforwardly. As with the techniques above, it requires a set of faces that have been landmarked. The program performs separate PCAs on the shape and texture of the set, and users are prompted to specify how many components they wish to extract (up to a maximum of the set size minus one). Following PCA, eigenvectors and the corresponding eigenvalues are written to files, which can then be used in subsequent analysis of the original or novel images (see below). The texture eigenvectors are also represented in an image file, giving a visualization of each component. Reconstruction values of each of the contributing face images are also written to file (i.e., their values on the derived dimensions), along with measures of reconstruction error (e.g., the cosine between the original and the reconstruction).
Having derived a novel set of dimensions with which to describe faces, it is possible to use these faces in a number of ways. Most simply, one can ask whether the distribution of the faces in PC space has any correspondence with human face perception. The data files derived from the PCA—which give a location in space for each contributing image—are simple text files that can be analyzed in any way the researcher wishes. This approach has been used to examine notions of face similarity and distinctiveness—asking whether the faces that are close together in PC space are those that human perceivers find most similar (e.g., Nestor, Plaut, & Behrmann, 2013; Scheuchenpflug, 1999; Tredoux, 2002), or whether the faces perceived as “distinctive” by human viewers are those that lie in sparsely populated regions of space (e.g., Burton, Bruce, & Hancock, 1999; Hancock et al., 1996; O’Toole, Deffenbacher, Valentin, & Abdi, 1994).
A further possibility is to use the PC space to “reconstruct” novel face images. A property of the technique is that the novel space can be used to represent any image (as long as it is the same size as the originals). This property is at the heart of PCA as a tool for face identification. In short, the images for recognition are “reconstructed” in the low-dimensional PC space and then compared to known faces. If a novel image lies sufficiently close to a known face in this space, then the novel face is taken as being recognized (e.g., Moon & Phillips, 2001; Turk & Pentland, 1991). InterFace provides the facility to code any novel image in the PC space derived from a previous PCA, and thus supports this use.
We now provide an example of using InterFace for carrying out a PCA in realistic research.
Example: Within-person PCA
The traditional use of PCA in face recognition research is to use images of different people to extract the major ways in which faces vary (e.g., Moon & Phillips, 2001; O’Toole, Abdi, Deffenbacher, & Valentin, 1993; Zhao, Chellappa, Phillips, & Rosenfeld, 2003). The intuition behind this approach is that a statistical description of real face images is more likely to reveal the true underlying dimensions of “face space” than is an intuitive language-based factorization relying on easily labelled metrics such as “distance between the eyes” or “width of the mouth”.
Step 1: Creating a set of images In this example, we used 30 ambient/unconstrained images of the same identity, collected from personal photographs.
Step 2: Landmarking We next manually landmarked each of the images using the InterFace tool (see Fig. 2). This process creates two new files for each original face: a shape file and (optionally) a texture file (see above). These are stored in separate shape and texture directories.
Step 3: PCA The InterFace program was used to conduct a PCA on the 30 images. In this example, we requested all possible 29 dimensions of shape and similarly all 29 of texture. This process created a number of new files, including the eigenvectors of the PCA, the associated eigenvalues, and the reconstruction values (coefficients) needed to code each original face in the new 29 + 29 dimensional space.
- Step 4: Reconstruction After running a PCA on a set of faces, we can reconstruct those images using a simple weighted sum of the resulting components/eigenvectors. An example is shown in Fig. 8. This tool is useful to gain an intuitive understanding of the reconstruction, but we can also interrogate the quality of the representation more formally.
The first texture component (which explains the largest amount of variance) represents a general change in the brightness of the photographs. The first shape component captures head rotation along the longitudinal axis (“roll”), as well as a slight change in camera distance. These components are, of course, specific to the images of this identity, so other sets will likely depict different transformations for their components. What seems to be common to all within-person PCAs that we have tried is that the first three shape dimensions tend to describe rigid head rotations in three-dimensional space in some order/combination (Burton et al., 2016; Jenkins & Burton, 2011). This remains a topic of ongoing research.
We have provided an outline of the main features of InterFace, a software package intended to support researchers in face recognition. We have shown that the package offers a number of standard image manipulation tools (shape-warping and morphing), as well as statistical analysis (PCA and a visualization tool). We have outlined some potential uses of this tool in psychological research, and we hope it will support further work in this field.
This tool can be downloaded from https://www.york.ac.uk/psychology/interface. It contains a runtime app, which can be used on either PC or Mac, and is accompanied by a software guide describing all of its features.
The research leading to these results received funding from the European Research Council under the European Union’s Seventh Framework Programme (FP/2007-2013)/ERC Grant Agreement No. 323262, and from the Economic and Social Research Council, UK [ES/J022950/1].
- Aishwarya, P., & Marcus, K. (2010). Face recognition using multiple eigenface subspaces. Journal of Engineering and Technology Research, 2, 139–143.Google Scholar
- Beymer, D. (1995). Vectorizing face images by interleaving shape and texture computations (MIT AI Lab memo 1537). Cambridge, MA: Massachusetts Institute of Technology.Google Scholar
- Bruce, V., Burton, A. M., Hanna, E., Healey, P., Mason, O., Coombes, A., . . . Linney, A. (1993). Sex discrimination: How do we tell the difference between male and female faces? Perception, 22, 131–152.Google Scholar
- Craw, I. (1995). A manifold model of face and object recognition. In T. Valentine (Ed.), Cognitive and computational aspects of face recognition: Explorations in face space (pp. 183–203). London, UK: Routledge.Google Scholar
- Craw, I., & Cameron, P. (1991). Parameterising images for recognition and reconstruction. In P. Mowforth (Ed.), Proceedings of the British Machine Vision Conference (pp. 367–370). Heidelberg, Germany: Springer.Google Scholar