Subspace Procrustes Analysis

Abstract

Procrustes analysis (PA) has been a popular technique to align and build 2-D statistical models of shapes. Given a set of 2-D shapes PA is applied to remove rigid transformations. Later, a non-rigid 2-D model is computed by modeling the residual (e.g., PCA). Although PA has been widely used, it has several limitations for modeling 2-D shapes: occluded landmarks and missing data can result in local minima solutions, and there is no guarantee that the 2-D shapes provide a uniform sampling of the 3-D space of rotations for the object. To address previous issues, this paper proposes subspace PA (SPA). Given several instances of a 3-D object, SPA computes the mean and a 2-D subspace that can model rigid and non-rigid deformations of the 3-D object. We propose a discrete (DSPA) and continuous (CSPA) formulation for SPA, assuming that 3-D samples of an object are provided. DSPA extends the traditional PA, and produces unbiased 2-D models by uniformly sampling different views of the 3-D object. CSPA provides a continuous approach to uniformly sample the space of 3-D rotations, being more efficient in space and time. We illustrate the benefits of SPA in two different applications. First, SPA is used to learn 2-D face and body models from 3-D datasets. Experiments on the FaceWarehouse and CMU motion capture (MoCap) datasets show the benefits of our 2-D models against the state-of-the-art PA approaches and conventional 3-D models. Second, SPA learns an unbiased 2-D model from CMU MoCap dataset and it is used to estimate the human pose on the Leeds Sports dataset.

Appendices

Appendix 1: Vec-transpose

Vec-transpose $\mathbf{A}^{(p)}$ is a linear operator that generalizes vectorization and transposition operators (Marimont and Wandell 1992; Minka 2000). It reshapes matrix $\mathbf{A}\in \mathbb {R}^{m \times n}$ by vectorizing each ith block of p rows, and rearranging it as the ith column of the reshaped matrix, such that $\mathbf{A}^{(p)} \in \mathbb {R}^{pn \times \frac{m}{p}}$,

$$\begin{aligned} \begin{bmatrix}{a_{11}}&{a_{12}}&{a_{13}}\\{a_{21}}&{a_{22}}&{a_{23}}\\{a_{31}}&{a_{32}}&{a_{33}}\\{a_{41}}&{a_{42}}&{a_{43}}\\{a_{51}}&{a_{52}}&{a_{53}}\\{a_{61}}&{a_{62}}&{a_{63}}\end{bmatrix}^{(2)} = \begin{bmatrix}{a_{11}}&{a_{31}}&{a_{51}}\\{a_{21}}&{a_{41}}&{a_{61}}\\{a_{12}}&{a_{32}}&{a_{52}}\\{a_{22}}&{a_{42}}&{a_{62}}\\{a_{13}}&{a_{33}}&{a_{53}}\\{a_{23}}&{a_{43}}&{a_{63}}\end{bmatrix} ,\\ \begin{bmatrix}{a_{11}}&{a_{12}}&{a_{13}}\\ {a_{21}}&{a_{22}}&{a_{23}}\\ {a_{31}}&{a_{32}}&{a_{33}}\\ {a_{41}}&{a_{42}}&{a_{43}}\\ {a_{51}}&{a_{52}}&{a_{53}}\\ {a_{61}}&{a_{62}}&{a_{63}}\end{bmatrix}^{(3)} = \begin{bmatrix}{a_{11}}&{a_{41}}\\ {a_{21}}&{a_{51}}\\{a_{31}}&{a_{61}}\\{a_{12}}&{a_{42}}\\{a_{22}}&{a_{52}}\\{a_{32}}&{a_{62}}\\{a_{13}}&{a_{43}}\\{a_{23}}&{a_{53}}\\{a_{33}}&{a_{63}}\end{bmatrix} . \end{aligned}$$

Note that $(\mathbf{A}^{(p)})^{(p)} = \mathbf{A}$ and $\mathbf{A}^{(m)} = {{\mathrm{vec}}}(\mathbf{A})$. A useful rule for pulling a matrix out of nested Kronecker products is, $((\mathbf{B}\mathbf{A})^{(p)} \mathbf{C})^{(p)}$ $= (\mathbf{C}^T \otimes \mathbf{I}_p)\mathbf{B}\mathbf{A}= (\mathbf{B}^{(p)} \mathbf{C})^{(p)} \mathbf{A}$ , which leads to $(\mathbf{C}^T \otimes \mathbf{I}_2)\mathbf{B}= (\mathbf{B}^{(2)} \mathbf{C})^{(2)}$ .

Appendix 2: CSPA Formulation

In this Appendix, we detail the steps from Eqs. (21) to (24), as well as the definition of the covariance matrix, introduced in Sect. 3.

Given the value of $\mathbf{M}^*$ and the optimal expression of $\mathbf{A}(\varvec{\omega })^*_i$ from Eq. (10), we substitute them in Eq. (21) resulting in:

$$\begin{aligned}&E_{{{\mathrm{CSPA}}}}(\mathbf{B},\mathbf{c}(\varvec{\omega })_i) = \sum _{i=1}^{n} \int _{\varvec{\varOmega }}\nonumber \\&\quad \left\| \mathbf{P}(\varvec{\omega }) \mathbf{D}_i - \mathbf{P}(\varvec{\omega }) \mathbf{D}_i \mathbf{H}- (\mathbf{c}(\varvec{\omega })_i^T \otimes \mathbf{I}_2) \mathbf{B}^{(2)}\right\| ^{2}_{F} d\varvec{\omega }, \end{aligned}$$

(29)

where $\mathbf{H}= \mathbf{M}^{*T}(\mathbf{M}^*\mathbf{M}^{*T})^{-1}\mathbf{M}^*$ and $\mathbf{D}_i \in \mathbb {R}^{3 \times \ell }$. Then,

$$\begin{aligned}&E_{{{\mathrm{CSPA}}}}(\mathbf{B},\mathbf{c}(\varvec{\omega })_i) = \sum _{i=1}^{n} \int _{\varvec{\varOmega }}\nonumber \\&\quad \left\| {\mathbf{P}(\varvec{\omega }) \mathbf{D}_i ( \mathbf{I}_{\ell } - \mathbf{H}) - (\mathbf{c}(\varvec{\omega })_i^T \otimes \mathbf{I}_2) \mathbf{B}^{(2)}}\right\| ^{2}_{F} d\varvec{\omega }\end{aligned}$$

(30)

leads us to Eqs. (23) and (24), where $\bar{\mathbf{D}}_i = \mathbf{D}_i (\mathbf{I}_\ell - \mathbf{H})$ and $\bar{\mathbf{d}}_i = {{\mathrm{vec}}}(\bar{\mathbf{D}}_i)$. From Eq. (24), solving $\frac{{\partial E_{{{\mathrm{CSPA}}}}}}{{\partial \mathbf{c}(\varvec{\omega })_i}} = 0$ we find:

$$\begin{aligned} \mathbf{c}(\varvec{\omega })^*_i = (\mathbf{B}^T \mathbf{B})^{-1}\mathbf{B}^T(\mathbf{I}_{\ell } \otimes \mathbf{P}(\omega )) \bar{\mathbf{d}}_i . \end{aligned}$$

(31)

The substitution of $\mathbf{c}(\varvec{\omega })^*_i$ in Eq. (24) results in:

$$\begin{aligned} E_{{{\mathrm{CSPA}}}}(\mathbf{B})&= \sum _{i=1}^{n} \int _{\varvec{\varOmega }} \Big \Vert (\mathbf{I}_\ell \otimes \mathbf{P}(\varvec{\omega }))\bar{\mathbf{d}}_i -\mathbf{B}(\mathbf{B}^T \mathbf{B})^{-1}\nonumber \\&\quad \times \mathbf{B}^T(\mathbf{I}_{\ell } \otimes \mathbf{P}(\omega )) \bar{\mathbf{d}}_i\Big \Vert ^{2}_{2} d\varvec{\omega }\end{aligned}$$

(32)

$$\begin{aligned}&= \sum _{i=1}^{n} \int _{\varvec{\varOmega }} \Big \Vert \left( \mathbf{I}- \mathbf{B}(\mathbf{B}^T \mathbf{B})^{-1}\mathbf{B}^T\right) \nonumber \\&\quad \times (\mathbf{I}_{\ell } \otimes \mathbf{P}(\omega )) \bar{\mathbf{d}}_i\Big \Vert ^{2}_{2} d\varvec{\omega }\end{aligned}$$

(33)

$$\begin{aligned}&= \sum _{i=1}^{n} \int _{\varvec{\varOmega }} {{\mathrm{tr}}}\Big [ \left( \mathbf{I}- \mathbf{B}(\mathbf{B}^T \mathbf{B})^{-1}\mathbf{B}^T\right) (\mathbf{I}_{\ell } \otimes \mathbf{P}(\omega )) \bar{\mathbf{d}}_i\nonumber \\&\quad \times \left( (\mathbf{I}_{\ell } \otimes \mathbf{P}(\omega )) \bar{\mathbf{d}}_i\right) ^T\Big ] d\varvec{\omega }\end{aligned}$$

(34)

$$\begin{aligned}&= {{\mathrm{tr}}}\left[ \left( \mathbf{I}- \mathbf{B}(\mathbf{B}^T \mathbf{B})^{-1}\mathbf{B}^T\right) \varvec{\varSigma }\right] , \end{aligned}$$

(35)

where:

$$\begin{aligned} \varvec{\varSigma }= \int _{\varvec{\varOmega }} (\mathbf{I}_{\ell } \otimes \mathbf{P}(\omega )) \left( \sum _{i=1}^{n}\bar{\mathbf{d}}_i \bar{\mathbf{d}}^T_i \right) (\mathbf{I}_{\ell } \otimes \mathbf{P}(\omega ))^T d\varvec{\omega }. \end{aligned}$$

(36)

We can find the global optima of Eq. (35) by solving the eigenvalue problem, $\varvec{\varSigma }\mathbf{B}= \mathbf{B}\varvec{\varLambda }$, where $\varvec{\varSigma }$ is the covariance matrix and $\varvec{\varLambda }$ are the eigenvalues corresponding to columns of $\mathbf{B}$. However, the definite integral in $\varvec{\varSigma }$ is data dependent. To be able to compute the integral off-line, we need to rearrange the elements in $\varvec{\varSigma }$. Using vectorization and vec-transpose operator⁸:

$$\begin{aligned} \varvec{\varSigma }&= \left( {{\mathrm{vec}}}{\left[ \varvec{\varSigma }\right] }\right) ^{(2\ell )} \end{aligned}$$

(37)

$$\begin{aligned}&= \left( {{\mathrm{vec}}}{\left[ \int _{\varvec{\varOmega }} (\mathbf{I}_{\ell } \otimes \mathbf{P}(\omega )) \left( \sum _{i=1}^{n}\bar{\mathbf{d}}_i \bar{\mathbf{d}}^T_i \right) (\mathbf{I}_{\ell } \otimes \mathbf{P}(\omega ))^T d\varvec{\omega }\right] }\right) ^{(2\ell )} \end{aligned}$$

(38)

$$\begin{aligned}&= \left( \left( \int _{\varvec{\varOmega }} (\mathbf{I}_{\ell } \!\otimes \!\mathbf{P}(\omega )) \!\otimes \! (\mathbf{I}_{\ell } \otimes \mathbf{P}(\omega )) d\varvec{\omega }\right) {{\mathrm{vec}}}{\left[ \sum _{i=1}^{n}\bar{\mathbf{d}}_i \bar{\mathbf{d}}^T_i \!\right] }\!\right) ^{(2\ell )} , \end{aligned}$$

(39)

which finally leads to:

$$\begin{aligned} \varvec{\varSigma }= \left( (\mathbf{I}_\ell \otimes \mathbf{Y}) {{\mathrm{vec}}}\left[ \sum _{i=1}^{n} \bar{\mathbf{d}}_{ij} \bar{\mathbf{d}}_{ij}^T\right] \right) ^{(2\ell )} , \end{aligned}$$

(40)

where the definite integral $\mathbf{Y}= \int _{\varvec{\varOmega }} \mathbf{P}(\varvec{\omega }) \otimes (\mathbf{I}_\ell \otimes \mathbf{P}(\varvec{\omega })) d\varvec{\omega }\in \mathbb {R}^{4\ell \times 9\ell }$ can be computed off-line.