Separable Spatiotemporal Priors for Convex Reconstruction of Time-Varying 3D Point Clouds

  • Tomas Simon
  • Jack Valmadre
  • Iain Matthews
  • Yaser Sheikh
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8691)

Abstract

Reconstructing 3D motion data is highly under-constrained due to several common sources of data loss during measurement, such as projection, occlusion, or miscorrespondence. We present a statistical model of 3D motion data, based on the Kronecker structure of the spatiotemporal covariance of natural motion, as a prior on 3D motion. This prior is expressed as a matrix normal distribution, composed of separable and compact row and column covariances. We relate the marginals of the distribution to the shape, trajectory, and shape-trajectory models of prior art. When the marginal shape distribution is not available from training data, we show how placing a hierarchical prior over shapes results in a convex MAP solution in terms of the trace-norm. The matrix normal distribution, fit to a single sequence, outperforms state-of-the-art methods at reconstructing 3D motion data in the presence of significant data loss, while providing covariance estimates of the imputed points.

Keywords

Matrix normal trace-norm spatiotemporal missing data 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Akhter, I., Simon, T., Matthews, I., Khan, S., Sheikh, Y.: Bilinear spatiotemporal basis models. In: TOG (2012)Google Scholar
  2. 2.
    Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Nonrigid structure from motion in trajectory space. In: NIPS (2008)Google Scholar
  3. 3.
    Allen, G., Tibshirani, R.: Transposable regularized covariance models with an application to missing data imputation. Annals of Applied Statistics (2010)Google Scholar
  4. 4.
    Angst, R., Pollefeys, M.: A unified view on deformable shape factorizations. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VI. LNCS, vol. 7577, pp. 682–695. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  5. 5.
    Angst, R., Zach, C., Pollefeys, M.: The generalized trace-norm and its application to structure-from-motion problems. In: ICCV (2011)Google Scholar
  6. 6.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning (2011)Google Scholar
  7. 7.
    Brand, M.: A direct method for 3d factorization of nonrigid motion observed in 2d. In: ICCV (2005)Google Scholar
  8. 8.
    Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3d shape from image streams. In: CVPR (2000)Google Scholar
  9. 9.
    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models, their training and application. In: CVIU (1995)Google Scholar
  10. 10.
    Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. In: CVPR (2002)Google Scholar
  11. 11.
    Del Bue, A., Lladó, X., Agapito, L.: Non-rigid face modelling using shape priors. In: Analysis & Modelling of Faces & Gestures (2005)Google Scholar
  12. 12.
    Dutilleul, P.: The mle algorithm for the matrix normal distribution. Statistical Computation and Simulation (1999)Google Scholar
  13. 13.
    Fayad, J., Del Bue, A., Agapito, L., Aguiar, P.: Non-rigid structure from motion using quadratic deformation models. In: BMVC (2009)Google Scholar
  14. 14.
    Garg, R., Roussos, A., Agapito, L.: Dense variational reconstruction of non-rigid surfaces from monocular video. In: CVPR (2013)Google Scholar
  15. 15.
    Gotardo, P., Martinez, A.: Computing smooth time trajectories for camera and deformable shape in structure from motion with occlusion. PAMI (2011)Google Scholar
  16. 16.
    Joo, H., Park, H., Sheikh, Y.: Optimal visibility estimation for large-scale dynamic 3d reconstruction. In: CVPR (2014)Google Scholar
  17. 17.
    Lee, M., Cho, J., Choi, C., Oh, S.: Procrustean normal distribution for non-rigid structure from motion. In: CVPR (2013)Google Scholar
  18. 18.
    Lee, M., Choi, C., Oh, S.: A procrustean markov process for non-rigid structure recovery. In: CVPR (2014)Google Scholar
  19. 19.
    Metaxas, D., Terzopoulos, D.: Shape and nonrigid motion estimation through physics-based synthesis. PAMI (1993)Google Scholar
  20. 20.
    Olsen, S., Bartoli, A.A.: Implicit non-rigid structure-from-motion with priors. Journal of Mathematical Imaging and Vision (2008)Google Scholar
  21. 21.
    Park, H.S., Shiratori, T., Matthews, I., Sheikh, Y.: 3D reconstruction of a moving point from a series of 2D projections. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 158–171. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  22. 22.
    Park, S.I., Hodgins, J.K.: Data-driven modeling of skin and muscle deformation. In: TOG (2008)Google Scholar
  23. 23.
    Pentland, A., Horowitz, B.: Recovery of nonrigid motion & structure. PAMI (1993)Google Scholar
  24. 24.
    Recht, B., Fazel, M., Parrillo, P.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. In: SIAM (2010)Google Scholar
  25. 25.
    Russell, C., Fayad, J., Agapito, L.: Energy based multiple model fitting for non-rigid structure from motion. In: CVPR (2011)Google Scholar
  26. 26.
    Salzmann, M., Urtasun, R.: Physically-based motion models for 3d tracking: A convex. formulation. In: ICCV (2011)Google Scholar
  27. 27.
    Sidenbladh, H., Black, M.J., Fleet, D.J.: Stochastic tracking of 3D human figures using 2D image motion. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1843, pp. 702–718. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  28. 28.
    Srebro, N., Rennie, J., Jaakkola, T.: Maximum margin matrix factorizations. In: NIPS (2005)Google Scholar
  29. 29.
    Strang, G.: The discrete cosine transform. SIAM Review (1999)Google Scholar
  30. 30.
    Taylor, J., Jepson, A., Kutulakos, K.: Non-rigid structure from locally-rigid motion. In: CVPR (2010)Google Scholar
  31. 31.
    Terzopoulos, D., Witkin, A., Kass, M.: Constraints on deformable models: Recovering 3d shape and nonrigid motion. Artificial Intelligence (1988)Google Scholar
  32. 32.
    Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. IJCV (1992)Google Scholar
  33. 33.
    Torresani, L., Hertzmann, A., Bregler, C.: Non-rigid structure-from-motion: Estimating shape and motion with hierarchical priors. PAMI (2008)Google Scholar
  34. 34.
    Valmadre, J., Lucey, S.: A general trajectory prior for non-rigid reconstruction. In: CVPR (2012)Google Scholar
  35. 35.
    Vidal, R., Abretske, D.: Nonrigid shape and motion from multiple perspective views. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 205–218. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  36. 36.
    Xiao, J., Chai, J.-X., Kanade, T.: A closed-form solution to non-rigid shape and motion recovery. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 573–587. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  37. 37.
    Xiong, X., De La Torre, F.: Supervised descent method and its applications to face alignment. In: CVPR (2013)Google Scholar
  38. 38.
    Yan, J., Pollefeys, M.: A factorization-based approach to articulated motion recovery. In: CVPR (2005)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Tomas Simon
    • 1
  • Jack Valmadre
    • 2
    • 3
  • Iain Matthews
    • 4
    • 1
  • Yaser Sheikh
    • 1
  1. 1.Carnegie Mellon UniversityUSA
  2. 2.Queensland University of TechnologyAustralia
  3. 3.Commonwealth Scientific and Industrial Research OrganisationAustralia
  4. 4.Disney Research PittsburghUSA

Personalised recommendations