Modal Space: A Physics-Based Model for Sequential Estimation of Time-Varying Shape from Monocular Video

Abstract

This paper describes two sequential methods for recovering the camera pose together with the 3D shape of highly deformable surfaces from a monocular video. The nonrigid 3D shape is modeled as a linear combination of mode shapes with time-varying weights that define the shape at each frame and are estimated on-the-fly. The low-rank constraint is combined with standard smoothness priors to optimize the model parameters over a sliding window of image frames. We propose to obtain a physics-based shape basis using the initial frames on the video to code the time-varying shape along the sequence, reducing the problem from trilinear to bilinear. To this end, the 3D shape is discretized by means of a soup of elastic triangular finite elements where we apply a force balance equation. This equation is solved using modal analysis via a simple eigenvalue problem to obtain a shape basis that encodes the modes of deformation. Even though this strategy can be applied in a wide variety of scenarios, when the observations are denser, the solution can become prohibitive in terms of computational load. We avoid this limitation by proposing two efficient coarse-to-fine approaches that allow us to easily deal with dense 3D surfaces. This results in a scalable solution that estimates a small number of parameters per frame and could potentially run in real time. We show results on both synthetic and real videos with ground truth 3D data, while robustly dealing with artifacts such as noise and missing data.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Notes

  1. 1.

    In this work, we denote shape functions as the approximation functions that interpolate the solution using the discrete values obtained at the mesh nodes.

  2. 2.

    We use the motion capture data from: http://cvlab.epfl.ch/data/dsr. These data were acquired with a Vicon motion capture system. It contains one sequence of a deforming piece of cloth (serviette data) and one sequence of a deforming piece of cardboard (carton data).

  3. 3.

    Video available from: http://hamlyn.doc.ic.ac.uk/vision.

References

  1. 1.

    Agudo, A., Calvo, B., Montiel, J.M.M.: FEM models to code non-rigid EKF monocular SLAM. In: IEEE International Workshop on Dynamic Shape Capture and Analysis, pp. 1586–1593 (2011)

  2. 2.

    Agudo, A., Calvo, B., Montiel, J.M.M.: 3D reconstruction of non-rigid surfaces in real-time using wedge elements. In: Workshop on non-rigid shape analysis and deformable image alignment, pp 113–122 (2012a)

  3. 3.

    Agudo, A., Calvo, B., Montiel, J.M.M.: Finite element based sequential bayesian non-rigid structure from motion. In: IEEE conference on computer vision and pattern recognition, pp. 1418–1425 (2012b)

  4. 4.

    Agudo, A., Agapito, L., Calvo, B., Montiel, J.M.M.: Good vibrations: a modal analysis approach for sequential non-rigid structure from motion. In: IEEE conference on computer vision and pattern recognition, pp. 1558–1565 (2014a)

  5. 5.

    Agudo, A., Montiel, J.M.M., Agapito, L., Calvo, B.: Online dense non-rigid 3D shape and camera motion recovery. In: British Machine Vision Conference (2014b)

  6. 6.

    Agudo, A., Moreno-Noguer, F., Calvo, B., Montiel, J.M.M.: Sequential non-rigid structure from motion using physical priors. IEEE Trans. Pattern Anal. Mach. Intell. 38(5), 979–994 (2016)

    Article  Google Scholar 

  7. 7.

    Akhter, I., Sheikh, Y., Khan, S.: In defense of orthonormality constraints for nonrigid structure from motion. In: IEEE conference on computer vision and pattern recognition, pp. 1534–1541 (2009)

  8. 8.

    Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Trajectory space: a dual representation for nonrigid structure from motion. IEEE Trans. Pattern Anal. Mach. Intell. 33(7), 1442–1456 (2011)

    Article  Google Scholar 

  9. 9.

    Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S., Sayd, P.: Coarse-to-fine low-rank structure-from-motion. In: IEEE Conference on computer vision and pattern recognition, pp. 1–8 (2008)

  10. 10.

    Bathe, K.J.: Finite Element Procedures in Engineering Analysis. Prentice-Hall, Upper Saddle River (1982)

    Google Scholar 

  11. 11.

    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: International conference and exhibition on computer graphics and interactive techniques, pp. 187–194 (1999)

  12. 12.

    Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3D shape from image streams. In: IEEE conference on computer vision and pattern recognition, pp. 690–696 (2000)

  13. 13.

    Chhatkuli, A., Pizarro, D., Bartoli, A.: Non-rigid shape-from-motion for isometric surfaces using infinitesimal planarity. In: British Machine Vision Conference (2014)

  14. 14.

    Cohen, L., Cohen, I.: Finite-element methods for active contour models and balloons for 2D and 3D images. IEEE Trans. Pattern Anal. Mach. Intell. 15(11), 1131–1147 (1993)

    Article  Google Scholar 

  15. 15.

    Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. Int. J. Comput. Vis. 107(2), 101–122 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  16. 16.

    Davison, A.J., Reid, I.D., Molton, N.D., Stasse, O.: MonoSLAM: real-time single camera SLAM. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 1052–1067 (2007)

    Article  Google Scholar 

  17. 17.

    Del Bue, A., Llado, X., Agapito, L.: Non-rigid metric shape and motion recovery from uncalibrated images using priors. In: IEEE conference on computer vision and pattern recognition, pp 1191–1198 (2006)

  18. 18.

    Demmel, J.W., Philadelphia: Applied Numerical Linear Algebra. SIAM, Philadelphia (1997)

    Google Scholar 

  19. 19.

    Edelman, A., Arias, T., Smith, S.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20(20), 303–353 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  20. 20.

    Fayad, J., Agapito, L., Del Bue, A.: Piecewise quadratic reconstruction of non-rigid surfaces from monocular sequences. In: European conference on computer vision, pp. 297–310 (2010)

  21. 21.

    Garg, R., Roussos, A., Agapito, L.: Dense variational reconstruction of non-rigid surfaces from monocular video. In: IEEE conference on computer vision and pattern recognition, pp. 1272–1279 (2013a)

  22. 22.

    Garg, R., Roussos, A., Agapito, L.: A variational approach to video registration with subspace constraints. Int. J. Comput. Vis. 104(3), 286–314 (2013b)

    Article  MATH  MathSciNet  Google Scholar 

  23. 23.

    Ghahramani, Z., Hinton, G.E (1996) The EM algorithm for mixtures of factor analyzers. Technical Report CRG-TR-96-1, University of Toronto (2013a)

  24. 24.

    Golub, G., Van Loa, C.: Matrix Computations. Johns Hopkins University Pess, Baltimore (1996)

    Google Scholar 

  25. 25.

    Gotardo, P.F.U., Martinez, A.M.: Kernel non-rigid structure from motion. In: International conference on computer vision, pp. 802 – 809 (2011)

  26. 26.

    Kita, Y.: Elastic-model driven analysis of several views of a deformable cylindrical object. IEEE Trans. Pattern Anal. Mach. Intell. 18(12), 1150–1162 (1996)

    Article  Google Scholar 

  27. 27.

    Klein, G., Murray, D.W.: Parallel tracking and mapping for small AR workspaces. In: International symposium on mixed and augmented reality, pp. 225–234 (2007)

  28. 28.

    Lee, M., Cho, J., Choi, C.H., Oh, S.: Procrustean normal distribution for non-rigid structure from motion. In: IEEE conference on computer vision and pattern recognition, pp. 1280–1287 (2013)

  29. 29.

    Ma, Y., Kosecka, J., Sastry, S.: Optimization criteria and geometric algorithms for motion and structure estimation. Int. J. Comput. Vis. 44(3), 219–249 (1999)

    Article  MATH  Google Scholar 

  30. 30.

    Malti, A., Hartley, R., Bartoli, A., Kim, J.H.: Monocular template-based 3D reconstruction of extensible surfaces with local linear elasticity. In: IEEE conference on computer vision and pattern recognition, pp. 1522–1529 (2013)

  31. 31.

    Marques, M., Costeira, J.: Optimal shape from estimation with missing and degenerate data. In: IEEE workshop on motion and video computing, pp. 1–6 (2008)

  32. 32.

    Metaxas, D., Terzopoulos, D.: Constrained deformable superquadrics and nonrigid motion tracking. IEEE Trans. Pattern Anal. Mach. Intell. 15(6), 580–591 (1993)

    Article  Google Scholar 

  33. 33.

    Moreno-Noguer, F., Porta, J.M.: Probabilistic simultaneous pose and non-rigid shape recovery. In: IEEE conference on computer vision and pattern recognition, pp. 1289–1296 (2011)

  34. 34.

    Mouragnon, E., Lhuillier, M., Dhome, M., Dekeyser, F., Sayd, P.: Generic and real-time structure from motion using local bundle adjustment. Image Vis. Comput. 27(8), 1178–1193 (2009)

    Article  Google Scholar 

  35. 35.

    Nastar, C., Ayache, N.: Fast segmentation, tracking and analysis of deformable objects. In: IEEE international conference on computer vision, pp. 275–279 (1993)

  36. 36.

    Newcome, R., Davison, A.J.: Live dense reconstruction with a single moving camera. In: IEEE conference on computer vision and pattern recognition, pp. 1498–1505 (2010)

  37. 37.

    Newcome, R., Lovegrove, S., Davison, A.J.: DTAM: Dense tracking and mapping in real-time. In: IEEE international conference on computer vision, pp. 2320–2327 (2011)

  38. 38.

    Ostlund, J., Varol, A., Ngo, D.T., Fua, P.: Laplacian meshes for monocular 3D shape recovery. In: European conference on computer vision, pp. 412–425 (2012)

  39. 39.

    Paladini, M., Del Bue, A., Stosic, M., Dodig, M., Xavier, J., Agapito, L.: Factorization for non-rigid and articulated structure using metric projections. In: IEEE conference on computer vision and pattern recognition, pp. 2898–2905 (2009)

  40. 40.

    Paladini, M., Bartoli, A., Agapito, L.: Sequential non rigid structure from motion with the 3D implicit low rank shape model. In: European Conference on computer vision, pp. 15–28 (2010)

  41. 41.

    Park, H.S., Shiratori, T., Matthews, I., Sheikh, Y.: 3D reconstruction of a moving point from a series of 2D projections. In: European conference on computer vision, pp. 158–171 (2010)

  42. 42.

    Pentland, A., Horowitz, B.: Recovery of nonrigid motion and structure. IEEE Trans. Pattern Anal. Mach. Intell 13(7), 730–742 (1991)

    Article  Google Scholar 

  43. 43.

    Pentland, A., Sclaroff, S.: Closed-form solutions for physically based shape modeling and recognition. IEEE Trans. Pattern Anal. Mach. Intell 13(21), 715–729 (1991)

    Article  Google Scholar 

  44. 44.

    Roweis, S.: EM algorithms for PCA and SPCA. In: Neural information processing systems, pp. 626–632 (1998)

  45. 45.

    Russell, C., Fayad, J., Agapito, L.: Energy based multiple model fitting for non-rigid structure from motion. In: IEEE conference on computer vision and pattern recognition, pp. 3009–3016 (2011)

  46. 46.

    Salzmann, M., Fua, P.: Reconstructing sharply folding surfaces: A convex formulation. In: IEEE conference on computer vision and pattern recognition, pp. 1054–1061 (2009)

  47. 47.

    Salzmann, M., Urtasun, R., Fua, P.: Local deformation models for monocular 3D shape recovery. In: IEEE conference on computer vision and pattern recognition, pp. 1–8 (2008)

  48. 48.

    Sclaroff, S., Pentland, A.: Modal matching for correspondence and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 17(6), 545–561 (1995)

    Article  Google Scholar 

  49. 49.

    Shaji, A., Chandran, S.: Riemannian manifold optimisation for non-rigid structure from motion. In: Workshop on non-rigid shape analysis and deformable image alignment, pp. 1–6 (2008)

  50. 50.

    Tao, H., Huang, T.S.: Connected vibrations: a modal analysis approach for non-rigid motion tracking. In: IEEE conference on computer vision and pattern recognition, pp. 735–740 (1998)

  51. 51.

    Tao, L., Mein, S.J., Quan, W., Matuszewski, B.J.: Recursive non-rigid structure from motion with online learned shape prior. Computer Vision and Image Understanding 117(10), 1287–1298 (2013)

    Article  Google Scholar 

  52. 52.

    Taylor, J., Jepson, A.D., Kutulakos, K.N.: Non-rigid structure from locally-rigid motion. In: IEEE conference on computer vision and pattern recognition, pp. 2761–2768 (2010)

  53. 53.

    Tipping, M.E., Bishop, C.M.: Mixtures of probabilistic principal component analysers. Neural Comput. 11(2), 443–482 (1999)

    Article  Google Scholar 

  54. 54.

    Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization approach. Int. J. Comput. Vis. 9(2), 137–154 (1992)

    Article  Google Scholar 

  55. 55.

    Torresani, L., Yang, D.B., Alexander, E.J., Bregler, C.: Tracking and modelling non-rigid objects with rank constraints. In: IEEE conference on computer vision and pattern recognition, pp. 493–500 (2001)

  56. 56.

    Torresani, L., Hertzmann, A., Bregler, C.: Learning non-rigid 3D shape from 2D motion. In: Neural information processing systems, pp. 1555–1562 (2004)

  57. 57.

    Torresani, L., Hertzmann, A., Bregler, C.: Nonrigid structure-from-motion: estimating shape and motion with hierarchical priors. IEEE Trans. Pattern Anal. Mach. Intell. 30(5), 878–892 (2008)

    Article  Google Scholar 

  58. 58.

    Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment: a modern synthesis. Vis. Algorithms 1883, 298–372 (2000)

    Google Scholar 

  59. 59.

    Tsap, L.V., Goldof, D.B., Sarkar, S.: Nonrigid motion analysis based on dynamic refinement of finite element models. IEEE Trans. Pattern Anal. Mach. Intell. 22(5), 526–543 (2000)

    Article  Google Scholar 

  60. 60.

    Varol, A., Salzmann, M., Tola, E., Fua, P.: Template-free monocular reconstruction of deformable surfaces. In: IEEE international conference on computer vision, pp. 1811–1818 (2009)

  61. 61.

    Vicente, S., Agapito, L.: Soft inextensibility constraints for template-free non-rigid reconstruction. In: European conference on computer vision, pp. 426–440 (2012)

  62. 62.

    Wendel, A., Maurer, M., Graber, G., Pock, T., Bischof, H.: Dense reconstruction on-the-fly. In: IEEE conference on computer vision and pattern recognition, pp. 1450–1457 (2012)

  63. 63.

    Woodbury, M.A.: Inverting modified matrices. Statistical Research Group, Memorandum Report No 42 (1950)

  64. 64.

    Yeoh, O.H.: Some forms of the strain energy function for rubber. Rubber Chem. Technol. 66(5), 754–771 (1993)

    Article  Google Scholar 

  65. 65.

    Young, A., Axel, L.: Non-rigid wall motion using MR tagging. In: IEEE conference on computer vision and pattern recognition, pp. 399–404 (1993)

  66. 66.

    Zienkiewicz, O.C., Taylor, R.L.: The Finite Element Method. Vol. 1: Basic Formulation and Linear Problems. McGraw-Hill, London (1989)

    Google Scholar 

Download references

Acknowledgments

This work was partly supported by the MINECO projects RT-SLAM DPI2015-67275-P, RobInstruct TIN2014-58178-R and Keratocono DPI2014-54981-R; by the SecondHands project funded by the EU Horizon 2020 Research and Innovation programme under grant agreement No 643950; by the ERA-net CHISTERA projects VISEN PCIN-2013-047 and I-DRESS PCIN-2015-147; and by a scholarship FPU12/04886 from the Spanish MECD. We thank R. Garg for his optical flow data and M. A. Ariza for the stretching data.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Antonio Agudo.

Additional information

Part of this work was done while A. Agudo was at UCL.

Rotation Update on the Stiefel Manifold

Rotation Update on the Stiefel Manifold

The camera rotation \(\mathbf {Q}_{i}^{t}\) is subject to orthonormality constraints, and a closed-form update is not possible. The rotation matrix lies exactly on a smooth manifold based on the orthogonal group SO(3), where it is possible to generalize a Riemannian-Newton algorithm [19, 49]. We define the problem as that of minimizing the function \(\mathcal {B}\left( \mathbf {Q}^{t}_{i} \right) \), where \(\mathbf {Q}^{t}_{i}\) is constrained to the set of matrices such that \({\mathbf {Q}^{t}_{i}}^{\top }\mathbf {Q}^{t}_{i}=\mathbf {I}\), i.e., a Stiefel matrix. In this work, we use the Riemannian manifold optimization to update the rotation matrices. First, we rewrite the expected negative log-likelihood function Eq. (21) dropping the dependence on \(\sigma ^{2}\) and \(\mathcal {B}\left( \mathbf {Q}^{t}_{i} \right) \) can be expressed as:

$$\begin{aligned} \mathop {\hbox {arg min}}\limits _{\mathbf {Q}^{t}_{i}\in SO(3)} \, \, \sum _{i=f-\mathcal {W}+1}^{f} \sum _{\varrho \in \mathcal {V}} \mathbb {E} \left[ \Vert \mathbf {w}_{i\varrho }-\varvec{\Uppi }\mathbf {Q}^{t}_{i} \tilde{\mathcal {S}}_{\varrho } \tilde{\varvec{\gamma }}_{i} -\mathbf {t}_{i} \Vert _{\mathcal {F}}^{2} \right] \end{aligned}$$
(23)

where \(\mathbf {Q}^{t}_{i}\in SO(3)\) and its tangent \(\varDelta _{Q^{t}_{i}}\in T_{Q^{t}}(SO(3))\) can be expressed as \(\varDelta _{Q^{t}_{i}}=\mathbf {Q}^{t}_{i}\left[ \varvec{\delta }\right] _{\times }\) with \(\left[ \varvec{\delta }\right] _{\times }\) being the skew-symmetric matrix. On SO(3), the geodesic at \(\mathbf {Q}^{t}_{i}\) in the tangent direction can be expressed by means of the Rodrigues’ rotation formula:

$$\begin{aligned} \mathbf {Q}^{t+1}\big (\hat{\varvec{\delta }},\alpha \big )= \mathbf {Q}^{t}\left( \mathbf {I}_{3}+\left[ \hat{\varvec{\delta }}\right] _{\times } \sin (\alpha )+ \left[ \hat{\varvec{\delta }}\right] _{\times }^{2}\left( 1-\cos (\alpha )\right) \right) \end{aligned}$$
(24)

where \(\left[ \varvec{\delta }\right] _{\times }\in \mathfrak {so}(3)\) is the Lie algebra of SO(3) the group and \(\left[ \varvec{\delta }\right] _{\times }=\alpha \left[ \hat{\varvec{\delta }}\right] _{\times }\). This explicit formula for geodesics is necessary for computing the gradient \(d\mathcal {B}(\varDelta _{Q^{t}_{i}})\) and the Hessian \(Hess \,\mathcal {B}(\varDelta _{Q^{t}_{i}},\varDelta _{Q^{t}_{i}})\) of the cost function Eq. (23) along the geodesics on the manifold. Given the previous definition, we can obtain both the gradient and Hessian in a tangent direction \(\varDelta _{Q_{i}^{t}}\) as:

$$\begin{aligned}&d \,\mathcal {B}(\varDelta _{Q_{i}^{t}})\equiv \left. {\frac{d \,\mathcal {B}\left( \mathbf {Q}_{i}^{t}(\alpha ) \right) }{d\alpha }} \right| _{\alpha =0} \nonumber \\&\quad =\left( \mathbf {R}_{i}^{t}\sum _{\varrho \in \mathcal {V}} \left( \tilde{\mathcal {S}}_{\varrho }\tilde{\varvec{\phi }}_{i}\tilde{\mathcal {S}}_{\varrho }^{\top } \right) -\sum _{\varrho \in \mathcal {V}}\left( \left( \mathbf {w}_{i\varrho }-\mathbf {t}_{i}\right) \tilde{\varvec{\mu }}_{i}^{\top } \tilde{\mathcal {S}}_{\varrho }^{\top }\right) \right) \varDelta _{R_{i}^{t}}^{\top } \end{aligned}$$
(25)
$$\begin{aligned}&Hess \,\mathcal {B}(\varDelta _{Q_{i}^{t}},\varDelta _{Q_{i}^{t}})\equiv \left. \frac{d^{2} \,\mathcal {B}\left( \mathbf {Q}_{i}^{t}(\alpha ) \right) }{d\alpha ^{2}}\right| _{\alpha =0}\nonumber \\&\quad =\left( \mathbf {R}_{i}^{t}\sum _{\varrho \in \mathcal {V}} \left( \tilde{\mathcal {S}}_{\varrho }\tilde{\varvec{\phi }}_{i}\tilde{\mathcal {S}}_{\varrho }^{\top } \right) - \sum _{\varrho \in \mathcal {M}}\left( \left( \mathbf {w}_{i\varrho }-\mathbf {t}_{i}\right) \tilde{\varvec{\mu }}_{i}^{\top } \tilde{\mathcal {S}}_{\varrho }^{\top }\right) \right) \varDelta _{Q_{i}^{t}}^{\top }\mathbf {Q}_{i}^{t} \varDelta _{R_{i}^{t}}^{\top }\nonumber \\&\qquad +\varDelta _{R_{i}^{t}}\sum _{\varrho \in \mathcal {V}} \left( \tilde{\mathcal {S}}_{\varrho }\tilde{\varvec{\phi }}_{i}\tilde{\mathcal {S}}_{\varrho }^{\top } \right) \varDelta _{R_{i}^{t}}^{\top } \end{aligned}$$
(26)

where \(\varDelta _{R_{i}^{t}}=\varvec{\Uppi }\varDelta _{Q_{i}^{t}}\) are the first two rows of a full tangent vector. The expectations are \(\tilde{\varvec{\mu }}_{i}=\mathbb {E}\big [\tilde{\varvec{\gamma }}_{i} \big ]\) and \(\tilde{\varvec{\phi }}_{i} = \mathbb {E} \Big [\tilde{\varvec{\gamma }}_{i} \tilde{\varvec{\gamma }}_{i}^{\top }\Big ]\). The Hessian can be obtained by polarizing \(Hess \,\mathcal {B}(\varDelta _{Q_{i}^{t}},\varDelta _{Q_{i}^{t}})\) [29, 49]. Assuming that the Hessian is nondegenerate, we compute the optimal updating vector for a generalized Newton method as \(\varDelta _{Q_{i}^{t}}=-Hess^{-1}G\), where G is the gradient on the manifold. To compute the Hessian, we use an orthonormal basis \(\mathbf {E}_{b}\) of the tangent space on SO(3). For simplicity, we can choose the standard basis \(\mathbf {e}_{b}\) for \(\mathbb R^{3}\). The Hessian matrix \(\mathbf {H}\) and gradient vector \(\mathbf {g}\) can be obtained as:

$$\begin{aligned} \mathbf {g}_{b}= & {} d \,\mathcal {B}(\mathbf {E}_{b}), \end{aligned}$$
(27)
$$\begin{aligned} \mathbf {H}_{bc}= & {} Hess \,\mathcal {B}(\mathbf {E}_{b},\mathbf {E}_{c}). \end{aligned}$$
(28)

Finally, the optimal updating vector can be computed as \(\varDelta _{Q_{i}^{t}}=-Hess^{-1}G=\mathbf {Q}_{i}^{t}\left[ \varvec{\delta }\right] _{\times }\) and to move it in the tangent direction along the geodesic on SO(3). The outline of the algorithm is shown in Algorithm 1.

figuree

Next, the noise variance and the translation vector can be updated in an on-line manner as:

$$\begin{aligned} \sigma ^2= & {} \frac{1}{2\varrho \mathcal {W}} \sum _{i=f-\mathcal {W}+1}^{f} \sum _{\varrho \in \mathcal {V}}\Big ( - 2\left( \mathbf {w}_{i\varrho }- \bar{\mathbf {t}}_{i}\right) ^{\top } \mathbf {G}_{i} \tilde{\mathcal {S}}_\varrho \tilde{\varvec{\mu }}_{i}\\&+ \Vert \mathbf {w}_{i\varrho } - \bar{\mathbf {t}}_{i}\Vert ^{2} + \text {tr}\left( \tilde{\mathcal {S}}_\varrho ^{\top } \mathbf {G}_{i}^{\top } \mathbf {G}_{i}\tilde{\mathcal {S}}_\varrho \tilde{\varvec{\phi }}_{i} \right) \Big ), \\ \mathbf {t}_{i}= & {} \frac{1}{\varrho } \sum _{\varrho \in \mathcal {V}} \left( \mathbf {w}_{i\varrho } -\mathbf {R}_{i}\tilde{\mathcal {S}}_{\varrho } \tilde{\varvec{\mu }}_{i}\right) . \end{aligned}$$

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Agudo, A., Montiel, J.M.M., Agapito, L. et al. Modal Space: A Physics-Based Model for Sequential Estimation of Time-Varying Shape from Monocular Video. J Math Imaging Vis 57, 75–98 (2017). https://doi.org/10.1007/s10851-016-0668-2

Download citation

Keywords

  • Sequential nonrigid structure from motion
  • Dense reconstruction
  • Modal analysis
  • Finite elements