Separable Spatiotemporal Priors for Convex Reconstruction of Time-Varying 3D Point Clouds

  • Tomas Simon
  • Jack Valmadre
  • Iain Matthews
  • Yaser Sheikh
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8691)


Reconstructing 3D motion data is highly under-constrained due to several common sources of data loss during measurement, such as projection, occlusion, or miscorrespondence. We present a statistical model of 3D motion data, based on the Kronecker structure of the spatiotemporal covariance of natural motion, as a prior on 3D motion. This prior is expressed as a matrix normal distribution, composed of separable and compact row and column covariances. We relate the marginals of the distribution to the shape, trajectory, and shape-trajectory models of prior art. When the marginal shape distribution is not available from training data, we show how placing a hierarchical prior over shapes results in a convex MAP solution in terms of the trace-norm. The matrix normal distribution, fit to a single sequence, outperforms state-of-the-art methods at reconstructing 3D motion data in the presence of significant data loss, while providing covariance estimates of the imputed points.


Matrix normal trace-norm spatiotemporal missing data 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Akhter, I., Simon, T., Matthews, I., Khan, S., Sheikh, Y.: Bilinear spatiotemporal basis models. In: TOG (2012)Google Scholar
  2. 2.
    Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Nonrigid structure from motion in trajectory space. In: NIPS (2008)Google Scholar
  3. 3.
    Allen, G., Tibshirani, R.: Transposable regularized covariance models with an application to missing data imputation. Annals of Applied Statistics (2010)Google Scholar
  4. 4.
    Angst, R., Pollefeys, M.: A unified view on deformable shape factorizations. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VI. LNCS, vol. 7577, pp. 682–695. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  5. 5.
    Angst, R., Zach, C., Pollefeys, M.: The generalized trace-norm and its application to structure-from-motion problems. In: ICCV (2011)Google Scholar
  6. 6.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning (2011)Google Scholar
  7. 7.
    Brand, M.: A direct method for 3d factorization of nonrigid motion observed in 2d. In: ICCV (2005)Google Scholar
  8. 8.
    Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3d shape from image streams. In: CVPR (2000)Google Scholar
  9. 9.
    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models, their training and application. In: CVIU (1995)Google Scholar
  10. 10.
    Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. In: CVPR (2002)Google Scholar
  11. 11.
    Del Bue, A., Lladó, X., Agapito, L.: Non-rigid face modelling using shape priors. In: Analysis & Modelling of Faces & Gestures (2005)Google Scholar
  12. 12.
    Dutilleul, P.: The mle algorithm for the matrix normal distribution. Statistical Computation and Simulation (1999)Google Scholar
  13. 13.
    Fayad, J., Del Bue, A., Agapito, L., Aguiar, P.: Non-rigid structure from motion using quadratic deformation models. In: BMVC (2009)Google Scholar
  14. 14.
    Garg, R., Roussos, A., Agapito, L.: Dense variational reconstruction of non-rigid surfaces from monocular video. In: CVPR (2013)Google Scholar
  15. 15.
    Gotardo, P., Martinez, A.: Computing smooth time trajectories for camera and deformable shape in structure from motion with occlusion. PAMI (2011)Google Scholar
  16. 16.
    Joo, H., Park, H., Sheikh, Y.: Optimal visibility estimation for large-scale dynamic 3d reconstruction. In: CVPR (2014)Google Scholar
  17. 17.
    Lee, M., Cho, J., Choi, C., Oh, S.: Procrustean normal distribution for non-rigid structure from motion. In: CVPR (2013)Google Scholar
  18. 18.
    Lee, M., Choi, C., Oh, S.: A procrustean markov process for non-rigid structure recovery. In: CVPR (2014)Google Scholar
  19. 19.
    Metaxas, D., Terzopoulos, D.: Shape and nonrigid motion estimation through physics-based synthesis. PAMI (1993)Google Scholar
  20. 20.
    Olsen, S., Bartoli, A.A.: Implicit non-rigid structure-from-motion with priors. Journal of Mathematical Imaging and Vision (2008)Google Scholar
  21. 21.
    Park, H.S., Shiratori, T., Matthews, I., Sheikh, Y.: 3D reconstruction of a moving point from a series of 2D projections. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 158–171. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  22. 22.
    Park, S.I., Hodgins, J.K.: Data-driven modeling of skin and muscle deformation. In: TOG (2008)Google Scholar
  23. 23.
    Pentland, A., Horowitz, B.: Recovery of nonrigid motion & structure. PAMI (1993)Google Scholar
  24. 24.
    Recht, B., Fazel, M., Parrillo, P.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. In: SIAM (2010)Google Scholar
  25. 25.
    Russell, C., Fayad, J., Agapito, L.: Energy based multiple model fitting for non-rigid structure from motion. In: CVPR (2011)Google Scholar
  26. 26.
    Salzmann, M., Urtasun, R.: Physically-based motion models for 3d tracking: A convex. formulation. In: ICCV (2011)Google Scholar
  27. 27.
    Sidenbladh, H., Black, M.J., Fleet, D.J.: Stochastic tracking of 3D human figures using 2D image motion. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1843, pp. 702–718. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  28. 28.
    Srebro, N., Rennie, J., Jaakkola, T.: Maximum margin matrix factorizations. In: NIPS (2005)Google Scholar
  29. 29.
    Strang, G.: The discrete cosine transform. SIAM Review (1999)Google Scholar
  30. 30.
    Taylor, J., Jepson, A., Kutulakos, K.: Non-rigid structure from locally-rigid motion. In: CVPR (2010)Google Scholar
  31. 31.
    Terzopoulos, D., Witkin, A., Kass, M.: Constraints on deformable models: Recovering 3d shape and nonrigid motion. Artificial Intelligence (1988)Google Scholar
  32. 32.
    Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. IJCV (1992)Google Scholar
  33. 33.
    Torresani, L., Hertzmann, A., Bregler, C.: Non-rigid structure-from-motion: Estimating shape and motion with hierarchical priors. PAMI (2008)Google Scholar
  34. 34.
    Valmadre, J., Lucey, S.: A general trajectory prior for non-rigid reconstruction. In: CVPR (2012)Google Scholar
  35. 35.
    Vidal, R., Abretske, D.: Nonrigid shape and motion from multiple perspective views. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 205–218. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  36. 36.
    Xiao, J., Chai, J.-X., Kanade, T.: A closed-form solution to non-rigid shape and motion recovery. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 573–587. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  37. 37.
    Xiong, X., De La Torre, F.: Supervised descent method and its applications to face alignment. In: CVPR (2013)Google Scholar
  38. 38.
    Yan, J., Pollefeys, M.: A factorization-based approach to articulated motion recovery. In: CVPR (2005)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Tomas Simon
    • 1
  • Jack Valmadre
    • 2
    • 3
  • Iain Matthews
    • 4
    • 1
  • Yaser Sheikh
    • 1
  1. 1.Carnegie Mellon UniversityUSA
  2. 2.Queensland University of TechnologyAustralia
  3. 3.Commonwealth Scientific and Industrial Research OrganisationAustralia
  4. 4.Disney Research PittsburghUSA

Personalised recommendations