Modal Space: A Physics-Based Model for Sequential Estimation of Time-Varying Shape from Monocular Video

Abstract

This paper describes two sequential methods for recovering the camera pose together with the 3D shape of highly deformable surfaces from a monocular video. The nonrigid 3D shape is modeled as a linear combination of mode shapes with time-varying weights that define the shape at each frame and are estimated on-the-fly. The low-rank constraint is combined with standard smoothness priors to optimize the model parameters over a sliding window of image frames. We propose to obtain a physics-based shape basis using the initial frames on the video to code the time-varying shape along the sequence, reducing the problem from trilinear to bilinear. To this end, the 3D shape is discretized by means of a soup of elastic triangular finite elements where we apply a force balance equation. This equation is solved using modal analysis via a simple eigenvalue problem to obtain a shape basis that encodes the modes of deformation. Even though this strategy can be applied in a wide variety of scenarios, when the observations are denser, the solution can become prohibitive in terms of computational load. We avoid this limitation by proposing two efficient coarse-to-fine approaches that allow us to easily deal with dense 3D surfaces. This results in a scalable solution that estimates a small number of parameters per frame and could potentially run in real time. We show results on both synthetic and real videos with ground truth 3D data, while robustly dealing with artifacts such as noise and missing data.

Rotation Update on the Stiefel Manifold

The camera rotation $\mathbf {Q}_{i}^{t}$ is subject to orthonormality constraints, and a closed-form update is not possible. The rotation matrix lies exactly on a smooth manifold based on the orthogonal group SO(3), where it is possible to generalize a Riemannian-Newton algorithm [19, 49]. We define the problem as that of minimizing the function $\mathcal {B}\left( \mathbf {Q}^{t}_{i} \right)$, where $\mathbf {Q}^{t}_{i}$ is constrained to the set of matrices such that ${\mathbf {Q}^{t}_{i}}^{\top }\mathbf {Q}^{t}_{i}=\mathbf {I}$, i.e., a Stiefel matrix. In this work, we use the Riemannian manifold optimization to update the rotation matrices. First, we rewrite the expected negative log-likelihood function Eq. (21) dropping the dependence on $\sigma ^{2}$ and $\mathcal {B}\left( \mathbf {Q}^{t}_{i} \right)$ can be expressed as:

$$\begin{aligned} \mathop {\hbox {arg min}}\limits _{\mathbf {Q}^{t}_{i}\in SO(3)} \, \, \sum _{i=f-\mathcal {W}+1}^{f} \sum _{\varrho \in \mathcal {V}} \mathbb {E} \left[ \Vert \mathbf {w}_{i\varrho }-\varvec{\Uppi }\mathbf {Q}^{t}_{i} \tilde{\mathcal {S}}_{\varrho } \tilde{\varvec{\gamma }}_{i} -\mathbf {t}_{i} \Vert _{\mathcal {F}}^{2} \right] \end{aligned}$$

(23)

where $\mathbf {Q}^{t}_{i}\in SO(3)$ and its tangent $\varDelta _{Q^{t}_{i}}\in T_{Q^{t}}(SO(3))$ can be expressed as $\varDelta _{Q^{t}_{i}}=\mathbf {Q}^{t}_{i}\left[ \varvec{\delta }\right] _{\times }$ with $\left[ \varvec{\delta }\right] _{\times }$ being the skew-symmetric matrix. On SO(3), the geodesic at $\mathbf {Q}^{t}_{i}$ in the tangent direction can be expressed by means of the Rodrigues’ rotation formula:

$$\begin{aligned} \mathbf {Q}^{t+1}\big (\hat{\varvec{\delta }},\alpha \big )= \mathbf {Q}^{t}\left( \mathbf {I}_{3}+\left[ \hat{\varvec{\delta }}\right] _{\times } \sin (\alpha )+ \left[ \hat{\varvec{\delta }}\right] _{\times }^{2}\left( 1-\cos (\alpha )\right) \right) \end{aligned}$$

(24)

where $\left[ \varvec{\delta }\right] _{\times }\in \mathfrak {so}(3)$ is the Lie algebra of SO(3) the group and $\left[ \varvec{\delta }\right] _{\times }=\alpha \left[ \hat{\varvec{\delta }}\right] _{\times }$. This explicit formula for geodesics is necessary for computing the gradient $d\mathcal {B}(\varDelta _{Q^{t}_{i}})$ and the Hessian $Hess \,\mathcal {B}(\varDelta _{Q^{t}_{i}},\varDelta _{Q^{t}_{i}})$ of the cost function Eq. (23) along the geodesics on the manifold. Given the previous definition, we can obtain both the gradient and Hessian in a tangent direction $\varDelta _{Q_{i}^{t}}$ as:

$$\begin{aligned}&d \,\mathcal {B}(\varDelta _{Q_{i}^{t}})\equiv \left. {\frac{d \,\mathcal {B}\left( \mathbf {Q}_{i}^{t}(\alpha ) \right) }{d\alpha }} \right| _{\alpha =0} \nonumber \\&\quad =\left( \mathbf {R}_{i}^{t}\sum _{\varrho \in \mathcal {V}} \left( \tilde{\mathcal {S}}_{\varrho }\tilde{\varvec{\phi }}_{i}\tilde{\mathcal {S}}_{\varrho }^{\top } \right) -\sum _{\varrho \in \mathcal {V}}\left( \left( \mathbf {w}_{i\varrho }-\mathbf {t}_{i}\right) \tilde{\varvec{\mu }}_{i}^{\top } \tilde{\mathcal {S}}_{\varrho }^{\top }\right) \right) \varDelta _{R_{i}^{t}}^{\top } \end{aligned}$$

(25)

$$\begin{aligned}&Hess \,\mathcal {B}(\varDelta _{Q_{i}^{t}},\varDelta _{Q_{i}^{t}})\equiv \left. \frac{d^{2} \,\mathcal {B}\left( \mathbf {Q}_{i}^{t}(\alpha ) \right) }{d\alpha ^{2}}\right| _{\alpha =0}\nonumber \\&\quad =\left( \mathbf {R}_{i}^{t}\sum _{\varrho \in \mathcal {V}} \left( \tilde{\mathcal {S}}_{\varrho }\tilde{\varvec{\phi }}_{i}\tilde{\mathcal {S}}_{\varrho }^{\top } \right) - \sum _{\varrho \in \mathcal {M}}\left( \left( \mathbf {w}_{i\varrho }-\mathbf {t}_{i}\right) \tilde{\varvec{\mu }}_{i}^{\top } \tilde{\mathcal {S}}_{\varrho }^{\top }\right) \right) \varDelta _{Q_{i}^{t}}^{\top }\mathbf {Q}_{i}^{t} \varDelta _{R_{i}^{t}}^{\top }\nonumber \\&\qquad +\varDelta _{R_{i}^{t}}\sum _{\varrho \in \mathcal {V}} \left( \tilde{\mathcal {S}}_{\varrho }\tilde{\varvec{\phi }}_{i}\tilde{\mathcal {S}}_{\varrho }^{\top } \right) \varDelta _{R_{i}^{t}}^{\top } \end{aligned}$$

(26)

where $\varDelta _{R_{i}^{t}}=\varvec{\Uppi }\varDelta _{Q_{i}^{t}}$ are the first two rows of a full tangent vector. The expectations are $\tilde{\varvec{\mu }}_{i}=\mathbb {E}\big [\tilde{\varvec{\gamma }}_{i} \big ]$ and $\tilde{\varvec{\phi }}_{i} = \mathbb {E} \Big [\tilde{\varvec{\gamma }}_{i} \tilde{\varvec{\gamma }}_{i}^{\top }\Big ]$. The Hessian can be obtained by polarizing $Hess \,\mathcal {B}(\varDelta _{Q_{i}^{t}},\varDelta _{Q_{i}^{t}})$ [29, 49]. Assuming that the Hessian is nondegenerate, we compute the optimal updating vector for a generalized Newton method as $\varDelta _{Q_{i}^{t}}=-Hess^{-1}G$, where G is the gradient on the manifold. To compute the Hessian, we use an orthonormal basis $\mathbf {E}_{b}$ of the tangent space on SO(3). For simplicity, we can choose the standard basis $\mathbf {e}_{b}$ for $\mathbb R^{3}$. The Hessian matrix $\mathbf {H}$ and gradient vector $\mathbf {g}$ can be obtained as:

$$\begin{aligned} \mathbf {g}_{b}= & {} d \,\mathcal {B}(\mathbf {E}_{b}), \end{aligned}$$

(27)

$$\begin{aligned} \mathbf {H}_{bc}= & {} Hess \,\mathcal {B}(\mathbf {E}_{b},\mathbf {E}_{c}). \end{aligned}$$

(28)

Finally, the optimal updating vector can be computed as $\varDelta _{Q_{i}^{t}}=-Hess^{-1}G=\mathbf {Q}_{i}^{t}\left[ \varvec{\delta }\right] _{\times }$ and to move it in the tangent direction along the geodesic on SO(3). The outline of the algorithm is shown in Algorithm 1.

Next, the noise variance and the translation vector can be updated in an on-line manner as:

$$\begin{aligned} \sigma ^2= & {} \frac{1}{2\varrho \mathcal {W}} \sum _{i=f-\mathcal {W}+1}^{f} \sum _{\varrho \in \mathcal {V}}\Big ( - 2\left( \mathbf {w}_{i\varrho }- \bar{\mathbf {t}}_{i}\right) ^{\top } \mathbf {G}_{i} \tilde{\mathcal {S}}_\varrho \tilde{\varvec{\mu }}_{i}\\&+ \Vert \mathbf {w}_{i\varrho } - \bar{\mathbf {t}}_{i}\Vert ^{2} + \text {tr}\left( \tilde{\mathcal {S}}_\varrho ^{\top } \mathbf {G}_{i}^{\top } \mathbf {G}_{i}\tilde{\mathcal {S}}_\varrho \tilde{\varvec{\phi }}_{i} \right) \Big ), \\ \mathbf {t}_{i}= & {} \frac{1}{\varrho } \sum _{\varrho \in \mathcal {V}} \left( \mathbf {w}_{i\varrho } -\mathbf {R}_{i}\tilde{\mathcal {S}}_{\varrho } \tilde{\varvec{\mu }}_{i}\right) . \end{aligned}$$