Markov Chain Monte Carlo for Automated Face Image Analysis

Abstract

We present a novel fully probabilistic method to interpret a single face image with the 3D Morphable Model. The new method is based on Bayesian inference and makes use of unreliable image-based information. Rather than searching a single optimal solution, we infer the posterior distribution of the model parameters given the target image. The method is a stochastic sampling algorithm with a propose-and-verify architecture based on the Metropolis–Hastings algorithm. The stochastic method can robustly integrate unreliable information and therefore does not rely on feed-forward initialization. The integrative concept is based on two ideas, a separation of proposal moves and their verification with the model (Data-Driven Markov Chain Monte Carlo), and filtering with the Metropolis acceptance rule. It does not need gradients and is less prone to local optima than standard fitters. We also introduce a new collective likelihood which models the average difference between the model and the target image rather than individual pixel differences. The average value shows a natural tendency towards a normal distribution, even when the individual pixel-wise difference is not Gaussian. We employ the new fitting method to calculate posterior models of 3D face reconstructions from single real-world images. A direct application of the algorithm with the 3D Morphable Model leads us to a fully automatic face recognition system with competitive performance on the Multi-PIE database without any database adaptation.

Appendices

Appendix 1: The Face Model

Face model The matrix $\mathbf {U}$ contains the principal components. The diagonal matrix $\mathbf {D}$ is modified slightly compared to a standard PCA where it would contain the eigenvalues $\lambda _i$ of the covariance matrix $\mathbf {\Sigma }= \mathbf {U} \tilde{\mathbf {D}}^2 \mathbf {U}^T$. We modify $\mathbf {D}$ to correspond with the proposed Maximum-Likelihood estimators from Tipping and Bishop (1999)

$$\begin{aligned} \mathbf {D}^2 = \tilde{\mathbf {D}}^2 - \sigma ^2 \mathbf {I} \end{aligned}$$

(27)

We estimate the missing standard deviation parameter $\sigma$ of the PPCA model as the Root Mean Square (RMS) reconstruction error of 3D faces, for both shape and texture. Reconstructing the 10 BFM out-of-sample faces, we obtain RMS reconstruction errors of ${{\hat{\sigma }}}_S = 0.61$ mm for the shape part and ${{\hat{\sigma }}}_C = 0.047$ for the color. Note that all color values are RGB floating point numbers in the interval [0, 1].

In order to better adapt the model to real images, we adapt only the face, without ears and throat. A rendering of the mean face of the masked model can be found in Fig. 2. We recalculate the statistics using only the restricted face mask to keep the model statistically valid and orthogonal.

All model parts of our software concerning statistical shape modeling are implemented using the Statismo framework (Lüthi et al. 2012).

Scene model The pinhole camera is modeled with focal length f and an offset $\mathbf {o}$ of the principal point within the image plane of size $w \times h$ pixels. The complete 3D-to-2D projection is then

$$\begin{aligned} \mathbf {x}_{2D}&= {\mathcal {P}} \circ T \circ R_Z \circ R_Y \circ R_X \circ \left( \mathbf {x}_{3D} \right) \end{aligned}$$

(28)

$$\begin{aligned} {\mathcal {P}} \left( \mathbf {r} \right)&= \begin{bmatrix} wf r_x / r_z + o_x \\ hf r_y/r_z + o_y \end{bmatrix}. \end{aligned}$$

(29)

Illumination The radiance $p_i$ of a point i on the face surface with normal $\mathbf {n}_i$ and albedo $a_i$ can be expressed using an expansion into real Spherical Harmonics basis functions $Y_{lm}$

$$\begin{aligned} p_i = a_i \sum _{l=0}^2 \sum _{m=-l}^l Y_{lm}(\mathbf {n}_i) L_{lm} k_l. \end{aligned}$$

(30)

The above equation is per color channel. The expansion of the environment map is captured in the illumination parameters $L_{lm}$, whereas the expansion of the Lambert reflectance kernel is given by $k_l$. For details, including the coefficient values $k_l$ of the expansion, refer to Basri and Jacobs (2003).

The final image is produced by rasterization of all triangles in the face model. We use a Phong shading approach with a varying, interpolated normal for each pixel.

Illumination estimation As the light model (30) is linear for a given geometry, the illumination expansion coefficients $L_{lm}^c$ for each color channel c are estimated solving a linear system (least squares) with entries for each vertex i

$$\begin{aligned} \sum _{l'=1}^9 Y_{l'}(\mathbf {n}_i) k_{l'} a_i^c L_{l'}^c = p_i^c. \end{aligned}$$

(31)

We solve the above system on 1000 randomly selected visible vertices i.

Product likelihood normalization The distribution is centered at the color value of the synthetic image and normalized to account for the truncation due to limited intensity values through

$$\begin{aligned} N= \int _{0}^{1} \exp \left( -\frac{\left||t - I_i(\theta ) \right||^2}{2 \sigma ^2} \right) dt_R dt_G dt_B \end{aligned}$$

(32)

which can be calculated using the error function. The normalization can also be replaced by a standard Gaussian normalization as an approximation if the standard deviation is much smaller than the range of bounds and the color channels are neither saturated nor zero. This is a valid assumption for the majority of typical face images.

Appendix 2: Random Walk Proposals

The standard random walk proposal type is a Gaussian update:

$$\begin{aligned} Q:~~\theta '&= \theta + d \qquad d \sim {\mathcal {N}}(d|0, \sigma ). \end{aligned}$$

(33)

Camera model The proposals change three Euler angles of rotation, three directions of translation, the principal point in the image plane and the focal length. All of these are updated independently, only one at a time, using a selected variance for each. Additionally, 3D rotation proposals are compensated for unwanted movements of the face within the image plane such that it is kept at a fixed position in the image.

Table 11 Random walk proposals: $\sigma$ is the standard deviation of the normal distribution, centered at the current location. $\lambda$ designates mixture coefficients of the different scales coarse (C), intermediate (I) and fine (F). The values are obtained empirically

Face model The updates of the 3DMM’s shape and texture models consist of two types of parameter variations. First, there is the addition of uncorrelated Gaussian noise to all parameters. Second, there is a scaling of the total parameter vector length with a log-normal distribution (distance from mean, “caricature”). Proposals are generated by

$$\begin{aligned} Q_S:~~{\varvec{\theta }}'_{\text {S}}&= {\varvec{\theta }}_{\text {S}} + d \qquad d \sim {\mathcal {N}}(d|0, \sigma _{\text {S}}\mathbf {I}) \end{aligned}$$

(34)

$$\begin{aligned} {\varvec{\theta }}'_{\text {S}}&= {\varvec{\theta }}_{\text {S}} \times \lambda \qquad \lambda \sim \log {\mathcal {N}}(1, \sigma _{\text {SL}}). \end{aligned}$$

(35)

Ilumination The illumination coefficients are updated with a mixture of a perturbation, an intensity and a color proposal. The perturbation is a standard independent Gaussian acting on all coefficients at once. The intensity proposal scales all coefficients by a factor drawn from a log-normal distribution and the color proposal keeps the intensity constant while perturbing the coefficients.

Table 11 contains a detailed overview.