Advertisement

Journal of Mathematical Imaging and Vision

, Volume 57, Issue 1, pp 75–98 | Cite as

Modal Space: A Physics-Based Model for Sequential Estimation of Time-Varying Shape from Monocular Video

  • Antonio Agudo
  • José M. Martínez Montiel
  • Lourdes Agapito
  • Begoña Calvo
Article

Abstract

This paper describes two sequential methods for recovering the camera pose together with the 3D shape of highly deformable surfaces from a monocular video. The nonrigid 3D shape is modeled as a linear combination of mode shapes with time-varying weights that define the shape at each frame and are estimated on-the-fly. The low-rank constraint is combined with standard smoothness priors to optimize the model parameters over a sliding window of image frames. We propose to obtain a physics-based shape basis using the initial frames on the video to code the time-varying shape along the sequence, reducing the problem from trilinear to bilinear. To this end, the 3D shape is discretized by means of a soup of elastic triangular finite elements where we apply a force balance equation. This equation is solved using modal analysis via a simple eigenvalue problem to obtain a shape basis that encodes the modes of deformation. Even though this strategy can be applied in a wide variety of scenarios, when the observations are denser, the solution can become prohibitive in terms of computational load. We avoid this limitation by proposing two efficient coarse-to-fine approaches that allow us to easily deal with dense 3D surfaces. This results in a scalable solution that estimates a small number of parameters per frame and could potentially run in real time. We show results on both synthetic and real videos with ground truth 3D data, while robustly dealing with artifacts such as noise and missing data.

Keywords

Sequential nonrigid structure from motion Dense reconstruction Modal analysis Finite elements 

Notes

Acknowledgments

This work was partly supported by the MINECO projects RT-SLAM DPI2015-67275-P, RobInstruct TIN2014-58178-R and Keratocono DPI2014-54981-R; by the SecondHands project funded by the EU Horizon 2020 Research and Innovation programme under grant agreement No 643950; by the ERA-net CHISTERA projects VISEN PCIN-2013-047 and I-DRESS PCIN-2015-147; and by a scholarship FPU12/04886 from the Spanish MECD. We thank R. Garg for his optical flow data and M. A. Ariza for the stretching data.

Supplementary material

References

  1. 1.
    Agudo, A., Calvo, B., Montiel, J.M.M.: FEM models to code non-rigid EKF monocular SLAM. In: IEEE International Workshop on Dynamic Shape Capture and Analysis, pp. 1586–1593 (2011)Google Scholar
  2. 2.
    Agudo, A., Calvo, B., Montiel, J.M.M.: 3D reconstruction of non-rigid surfaces in real-time using wedge elements. In: Workshop on non-rigid shape analysis and deformable image alignment, pp 113–122 (2012a)Google Scholar
  3. 3.
    Agudo, A., Calvo, B., Montiel, J.M.M.: Finite element based sequential bayesian non-rigid structure from motion. In: IEEE conference on computer vision and pattern recognition, pp. 1418–1425 (2012b)Google Scholar
  4. 4.
    Agudo, A., Agapito, L., Calvo, B., Montiel, J.M.M.: Good vibrations: a modal analysis approach for sequential non-rigid structure from motion. In: IEEE conference on computer vision and pattern recognition, pp. 1558–1565 (2014a)Google Scholar
  5. 5.
    Agudo, A., Montiel, J.M.M., Agapito, L., Calvo, B.: Online dense non-rigid 3D shape and camera motion recovery. In: British Machine Vision Conference (2014b)Google Scholar
  6. 6.
    Agudo, A., Moreno-Noguer, F., Calvo, B., Montiel, J.M.M.: Sequential non-rigid structure from motion using physical priors. IEEE Trans. Pattern Anal. Mach. Intell. 38(5), 979–994 (2016)CrossRefGoogle Scholar
  7. 7.
    Akhter, I., Sheikh, Y., Khan, S.: In defense of orthonormality constraints for nonrigid structure from motion. In: IEEE conference on computer vision and pattern recognition, pp. 1534–1541 (2009)Google Scholar
  8. 8.
    Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Trajectory space: a dual representation for nonrigid structure from motion. IEEE Trans. Pattern Anal. Mach. Intell. 33(7), 1442–1456 (2011)CrossRefGoogle Scholar
  9. 9.
    Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S., Sayd, P.: Coarse-to-fine low-rank structure-from-motion. In: IEEE Conference on computer vision and pattern recognition, pp. 1–8 (2008)Google Scholar
  10. 10.
    Bathe, K.J.: Finite Element Procedures in Engineering Analysis. Prentice-Hall, Upper Saddle River (1982)Google Scholar
  11. 11.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: International conference and exhibition on computer graphics and interactive techniques, pp. 187–194 (1999)Google Scholar
  12. 12.
    Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3D shape from image streams. In: IEEE conference on computer vision and pattern recognition, pp. 690–696 (2000)Google Scholar
  13. 13.
    Chhatkuli, A., Pizarro, D., Bartoli, A.: Non-rigid shape-from-motion for isometric surfaces using infinitesimal planarity. In: British Machine Vision Conference (2014)Google Scholar
  14. 14.
    Cohen, L., Cohen, I.: Finite-element methods for active contour models and balloons for 2D and 3D images. IEEE Trans. Pattern Anal. Mach. Intell. 15(11), 1131–1147 (1993)CrossRefGoogle Scholar
  15. 15.
    Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. Int. J. Comput. Vis. 107(2), 101–122 (2014)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Davison, A.J., Reid, I.D., Molton, N.D., Stasse, O.: MonoSLAM: real-time single camera SLAM. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 1052–1067 (2007)CrossRefGoogle Scholar
  17. 17.
    Del Bue, A., Llado, X., Agapito, L.: Non-rigid metric shape and motion recovery from uncalibrated images using priors. In: IEEE conference on computer vision and pattern recognition, pp 1191–1198 (2006)Google Scholar
  18. 18.
    Demmel, J.W., Philadelphia: Applied Numerical Linear Algebra. SIAM, Philadelphia (1997)CrossRefGoogle Scholar
  19. 19.
    Edelman, A., Arias, T., Smith, S.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20(20), 303–353 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  20. 20.
    Fayad, J., Agapito, L., Del Bue, A.: Piecewise quadratic reconstruction of non-rigid surfaces from monocular sequences. In: European conference on computer vision, pp. 297–310 (2010)Google Scholar
  21. 21.
    Garg, R., Roussos, A., Agapito, L.: Dense variational reconstruction of non-rigid surfaces from monocular video. In: IEEE conference on computer vision and pattern recognition, pp. 1272–1279 (2013a)Google Scholar
  22. 22.
    Garg, R., Roussos, A., Agapito, L.: A variational approach to video registration with subspace constraints. Int. J. Comput. Vis. 104(3), 286–314 (2013b)CrossRefzbMATHMathSciNetGoogle Scholar
  23. 23.
    Ghahramani, Z., Hinton, G.E (1996) The EM algorithm for mixtures of factor analyzers. Technical Report CRG-TR-96-1, University of Toronto (2013a)Google Scholar
  24. 24.
    Golub, G., Van Loa, C.: Matrix Computations. Johns Hopkins University Pess, Baltimore (1996)Google Scholar
  25. 25.
    Gotardo, P.F.U., Martinez, A.M.: Kernel non-rigid structure from motion. In: International conference on computer vision, pp. 802 – 809 (2011)Google Scholar
  26. 26.
    Kita, Y.: Elastic-model driven analysis of several views of a deformable cylindrical object. IEEE Trans. Pattern Anal. Mach. Intell. 18(12), 1150–1162 (1996)CrossRefGoogle Scholar
  27. 27.
    Klein, G., Murray, D.W.: Parallel tracking and mapping for small AR workspaces. In: International symposium on mixed and augmented reality, pp. 225–234 (2007)Google Scholar
  28. 28.
    Lee, M., Cho, J., Choi, C.H., Oh, S.: Procrustean normal distribution for non-rigid structure from motion. In: IEEE conference on computer vision and pattern recognition, pp. 1280–1287 (2013)Google Scholar
  29. 29.
    Ma, Y., Kosecka, J., Sastry, S.: Optimization criteria and geometric algorithms for motion and structure estimation. Int. J. Comput. Vis. 44(3), 219–249 (1999)CrossRefzbMATHGoogle Scholar
  30. 30.
    Malti, A., Hartley, R., Bartoli, A., Kim, J.H.: Monocular template-based 3D reconstruction of extensible surfaces with local linear elasticity. In: IEEE conference on computer vision and pattern recognition, pp. 1522–1529 (2013)Google Scholar
  31. 31.
    Marques, M., Costeira, J.: Optimal shape from estimation with missing and degenerate data. In: IEEE workshop on motion and video computing, pp. 1–6 (2008)Google Scholar
  32. 32.
    Metaxas, D., Terzopoulos, D.: Constrained deformable superquadrics and nonrigid motion tracking. IEEE Trans. Pattern Anal. Mach. Intell. 15(6), 580–591 (1993)CrossRefGoogle Scholar
  33. 33.
    Moreno-Noguer, F., Porta, J.M.: Probabilistic simultaneous pose and non-rigid shape recovery. In: IEEE conference on computer vision and pattern recognition, pp. 1289–1296 (2011)Google Scholar
  34. 34.
    Mouragnon, E., Lhuillier, M., Dhome, M., Dekeyser, F., Sayd, P.: Generic and real-time structure from motion using local bundle adjustment. Image Vis. Comput. 27(8), 1178–1193 (2009)CrossRefGoogle Scholar
  35. 35.
    Nastar, C., Ayache, N.: Fast segmentation, tracking and analysis of deformable objects. In: IEEE international conference on computer vision, pp. 275–279 (1993)Google Scholar
  36. 36.
    Newcome, R., Davison, A.J.: Live dense reconstruction with a single moving camera. In: IEEE conference on computer vision and pattern recognition, pp. 1498–1505 (2010)Google Scholar
  37. 37.
    Newcome, R., Lovegrove, S., Davison, A.J.: DTAM: Dense tracking and mapping in real-time. In: IEEE international conference on computer vision, pp. 2320–2327 (2011)Google Scholar
  38. 38.
    Ostlund, J., Varol, A., Ngo, D.T., Fua, P.: Laplacian meshes for monocular 3D shape recovery. In: European conference on computer vision, pp. 412–425 (2012)Google Scholar
  39. 39.
    Paladini, M., Del Bue, A., Stosic, M., Dodig, M., Xavier, J., Agapito, L.: Factorization for non-rigid and articulated structure using metric projections. In: IEEE conference on computer vision and pattern recognition, pp. 2898–2905 (2009)Google Scholar
  40. 40.
    Paladini, M., Bartoli, A., Agapito, L.: Sequential non rigid structure from motion with the 3D implicit low rank shape model. In: European Conference on computer vision, pp. 15–28 (2010)Google Scholar
  41. 41.
    Park, H.S., Shiratori, T., Matthews, I., Sheikh, Y.: 3D reconstruction of a moving point from a series of 2D projections. In: European conference on computer vision, pp. 158–171 (2010)Google Scholar
  42. 42.
    Pentland, A., Horowitz, B.: Recovery of nonrigid motion and structure. IEEE Trans. Pattern Anal. Mach. Intell 13(7), 730–742 (1991)CrossRefGoogle Scholar
  43. 43.
    Pentland, A., Sclaroff, S.: Closed-form solutions for physically based shape modeling and recognition. IEEE Trans. Pattern Anal. Mach. Intell 13(21), 715–729 (1991)CrossRefGoogle Scholar
  44. 44.
    Roweis, S.: EM algorithms for PCA and SPCA. In: Neural information processing systems, pp. 626–632 (1998)Google Scholar
  45. 45.
    Russell, C., Fayad, J., Agapito, L.: Energy based multiple model fitting for non-rigid structure from motion. In: IEEE conference on computer vision and pattern recognition, pp. 3009–3016 (2011)Google Scholar
  46. 46.
    Salzmann, M., Fua, P.: Reconstructing sharply folding surfaces: A convex formulation. In: IEEE conference on computer vision and pattern recognition, pp. 1054–1061 (2009)Google Scholar
  47. 47.
    Salzmann, M., Urtasun, R., Fua, P.: Local deformation models for monocular 3D shape recovery. In: IEEE conference on computer vision and pattern recognition, pp. 1–8 (2008)Google Scholar
  48. 48.
    Sclaroff, S., Pentland, A.: Modal matching for correspondence and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 17(6), 545–561 (1995)CrossRefGoogle Scholar
  49. 49.
    Shaji, A., Chandran, S.: Riemannian manifold optimisation for non-rigid structure from motion. In: Workshop on non-rigid shape analysis and deformable image alignment, pp. 1–6 (2008)Google Scholar
  50. 50.
    Tao, H., Huang, T.S.: Connected vibrations: a modal analysis approach for non-rigid motion tracking. In: IEEE conference on computer vision and pattern recognition, pp. 735–740 (1998)Google Scholar
  51. 51.
    Tao, L., Mein, S.J., Quan, W., Matuszewski, B.J.: Recursive non-rigid structure from motion with online learned shape prior. Computer Vision and Image Understanding 117(10), 1287–1298 (2013)CrossRefGoogle Scholar
  52. 52.
    Taylor, J., Jepson, A.D., Kutulakos, K.N.: Non-rigid structure from locally-rigid motion. In: IEEE conference on computer vision and pattern recognition, pp. 2761–2768 (2010)Google Scholar
  53. 53.
    Tipping, M.E., Bishop, C.M.: Mixtures of probabilistic principal component analysers. Neural Comput. 11(2), 443–482 (1999)CrossRefGoogle Scholar
  54. 54.
    Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization approach. Int. J. Comput. Vis. 9(2), 137–154 (1992)CrossRefGoogle Scholar
  55. 55.
    Torresani, L., Yang, D.B., Alexander, E.J., Bregler, C.: Tracking and modelling non-rigid objects with rank constraints. In: IEEE conference on computer vision and pattern recognition, pp. 493–500 (2001)Google Scholar
  56. 56.
    Torresani, L., Hertzmann, A., Bregler, C.: Learning non-rigid 3D shape from 2D motion. In: Neural information processing systems, pp. 1555–1562 (2004)Google Scholar
  57. 57.
    Torresani, L., Hertzmann, A., Bregler, C.: Nonrigid structure-from-motion: estimating shape and motion with hierarchical priors. IEEE Trans. Pattern Anal. Mach. Intell. 30(5), 878–892 (2008)CrossRefGoogle Scholar
  58. 58.
    Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment: a modern synthesis. Vis. Algorithms 1883, 298–372 (2000)Google Scholar
  59. 59.
    Tsap, L.V., Goldof, D.B., Sarkar, S.: Nonrigid motion analysis based on dynamic refinement of finite element models. IEEE Trans. Pattern Anal. Mach. Intell. 22(5), 526–543 (2000)CrossRefGoogle Scholar
  60. 60.
    Varol, A., Salzmann, M., Tola, E., Fua, P.: Template-free monocular reconstruction of deformable surfaces. In: IEEE international conference on computer vision, pp. 1811–1818 (2009)Google Scholar
  61. 61.
    Vicente, S., Agapito, L.: Soft inextensibility constraints for template-free non-rigid reconstruction. In: European conference on computer vision, pp. 426–440 (2012)Google Scholar
  62. 62.
    Wendel, A., Maurer, M., Graber, G., Pock, T., Bischof, H.: Dense reconstruction on-the-fly. In: IEEE conference on computer vision and pattern recognition, pp. 1450–1457 (2012)Google Scholar
  63. 63.
    Woodbury, M.A.: Inverting modified matrices. Statistical Research Group, Memorandum Report No 42 (1950)Google Scholar
  64. 64.
    Yeoh, O.H.: Some forms of the strain energy function for rubber. Rubber Chem. Technol. 66(5), 754–771 (1993)CrossRefGoogle Scholar
  65. 65.
    Young, A., Axel, L.: Non-rigid wall motion using MR tagging. In: IEEE conference on computer vision and pattern recognition, pp. 399–404 (1993)Google Scholar
  66. 66.
    Zienkiewicz, O.C., Taylor, R.L.: The Finite Element Method. Vol. 1: Basic Formulation and Linear Problems. McGraw-Hill, London (1989)Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Antonio Agudo
    • 1
  • José M. Martínez Montiel
    • 2
  • Lourdes Agapito
    • 3
  • Begoña Calvo
    • 2
  1. 1.Institut de Robòtica i Informàtica Industrial (CSIC-UPC)BarcelonaSpain
  2. 2.I3A-Universidad de ZaragozaZaragozaSpain
  3. 3.University College LondonLondonUK

Personalised recommendations