International Journal of Computer Vision

, Volume 96, Issue 2, pp 252–276 | Cite as

Optimal Metric Projections for Deformable and Articulated Structure-from-Motion

  • Marco Paladini
  • Alessio Del Bue
  • João Xavier
  • Lourdes Agapito
  • Marko Stošić
  • Marija Dodig
Article

Abstract

This paper describes novel algorithms for recovering the 3D shape and motion of deformable and articulated objects purely from uncalibrated 2D image measurements using a factorisation approach. Most approaches to deformable and articulated structure from motion require to upgrade an initial affine solution to Euclidean space by imposing metric constraints on the motion matrix. While in the case of rigid structure the metric upgrade step is simple since the constraints can be formulated as linear, deformability in the shape introduces non-linearities. In this paper we propose an alternating bilinear approach to solve for non-rigid 3D shape and motion, associated with a globally optimal projection step of the motion matrices onto the manifold of metric constraints. Our novel optimal projection step combines into a single optimisation the computation of the orthographic projection matrix and the configuration weights that give the closest motion matrix that satisfies the correct block structure with the additional constraint that the projection matrix is guaranteed to have orthonormal rows (i.e. its transpose lies on the Stiefel manifold). This constraint turns out to be non-convex. The key contribution of this work is to introduce an efficient convex relaxation for the non-convex projection step. Efficient in the sense that, for both the cases of deformable and articulated motion, the proposed relaxations turned out to be exact (i.e. tight) in all our numerical experiments. The convex relaxations are semi-definite (SDP) or second-order cone (SOCP) programs which can be readily tackled by popular solvers. An important advantage of these new algorithms is their ability to handle missing data which becomes crucial when dealing with real video sequences with self-occlusions. We show successful results of our algorithms on synthetic and real sequences of both deformable and articulated data. We also show comparative results with state of the art algorithms which reveal that our new methods outperform existing ones.

Keywords

Non-rigid structure from motion Articulated structure from motion Convex optimization 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Supplementary material

11263_2011_468_MOESM1_ESM.pdf (105 kb)
(PDF 106 kB)

References

  1. Aanæs, H., & Kahl, F. (2002). Estimation of deformable structure and motion. In Workshop on vision and modelling of dynamic scenes, Copenhagen, Denmark. Google Scholar
  2. Akhter, I., Sheikh, Y., Khan, S., & Kanade, T. (2008). Nonrigid structure from motion in trajectory space. In Neural information processing systems. Google Scholar
  3. Akhter, I., Sheikh, Y., & Khan, S. (2009). In defense of orthonormality constraints for nonrigid structure from motion. In Proc. IEEE conference on computer vision and pattern recognition, Miami, Florida. Google Scholar
  4. Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S., & Sayd, P. (2008). Coarse-to-fine low-rank structure-from-motion. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska. Google Scholar
  5. Brand, M. (2005). A direct method for 3D factorization of nonrigid motion observed in 2D. In Proc. IEEE conference on computer vision and pattern recognition, San Diego, California. Google Scholar
  6. Bregler, C., Hertzmann, A., & Biermann, H. (2000). Recovering non-rigid 3D shape from image streams. In Proc. IEEE conference on computer vision and pattern recognition, Hilton Head, South Carolina. Google Scholar
  7. Buchanan, A. M., & Fitzgibbon, A. (2005). Damped Newton algorithms for matrix factorization with missing data. In Proc. IEEE conference on computer vision and pattern recognition: Vol. 2. San Diego, California. Google Scholar
  8. Del Bue, A. (2008). A factorization approach to structure from motion with shape priors. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska. Google Scholar
  9. Del Bue, A., Lladó, X., & Agapito, L. (2006). Non-rigid metric shape and motion recovery from uncalibrated images using priors. In Proc. IEEE conference on computer vision and pattern recognition, New York, NY. Google Scholar
  10. Del Bue, A., Smeraldi, F., & Agapito, L. (2007). Non-rigid structure from motion using ranklet–based tracking and non-linear optimization. Image and Vision Computing, 25(3). doi: 10.1016/j.imavis.2005.10.004.
  11. Dodig, M., Stošić, M., & Xavier, J. (2009). On minimizing a quadratic function on Stiefel manifolds (Tech. rept.). Instituto de Sistemas e Robotica. Available at http://users.isr.ist.utl.pt/~jxavier/ctech.pdf.
  12. Edelman, A., Arias, T. A., & Smith, S. T. (1999). The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications, 20(2). doi: 10.1137/S0895479895290954.
  13. Fayad, J., Del Bue, A., Agapito, L., & Aguiar, P. (2009). Non-rigid structure from motion using quadratic deformation models. In British machine vision conference, London, UK. Google Scholar
  14. Hartley, R., & Vidal, R. (2008). Perspective nonrigid shape and motion recovery. In Proc. European conference on computer vision. Google Scholar
  15. Marques, M., & Costeira, J. (2008). Optimal shape from motion estimation with missing and degenerate data. In WMVC ’08: proceedings of the 2008 IEEE workshop on motion and video computing (pp. 1–6). Washington: IEEE Computer Society. CrossRefGoogle Scholar
  16. Marques, M., & Costeira, J. (2009). Estimating 3D shape from degenerate sequences with missing data. Computer Vision and Image Understanding, 113(2). doi: 10.1016/j.cviu.2008.09.004.
  17. Paladini, M., Del Bue, A., Stošić, M., Dodig, M., Xavier, J., & Agapito, L. (2009). Factorization for non-rigid and articulated structure using metric projections. In Proc. IEEE conference on computer vision and pattern recognition, Miami, Florida. Google Scholar
  18. Rabaud, V., & Belongie, S. (2008). Re-thinking non-rigid structure from motion. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska. Google Scholar
  19. Sturm, J. F. (1999). Using SeDuMi 1.02, A Matlab toolbox for optimization over symmetric cones. Optimization Methods and Software, 11–12, 625–653. CrossRefMathSciNetGoogle Scholar
  20. Tomasi, C., & Kanade, T. (1992). Shape and motion from image streams under orthography: a factorization approach. International Journal of Computer Vision, 9(2). doi: 10.1007/BF00129684.
  21. Torresani, L., Yang, D., Alexander, E., & Bregler, C. (2001). Tracking and modeling non-rigid objects with rank constraints. In: Proc. IEEE conference on computer vision and pattern recognition, Kauai, Hawaii. Google Scholar
  22. Torresani, L., Hertzmann, A., & Bregler, C. (2008). Non-rigid structure-from-motion: estimating shape and motion with hierarchical priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(5). doi: 10.1109/TPAMI.2007.70752.
  23. Tresadern, P., & Reid, I. (2005). Articulated structure from motion by factorization. In Proc. IEEE conference on computer vision and pattern recognition: Vol. 2. San Diego, California. Google Scholar
  24. Wang, G., & Wu, Q. M. J. (2008). Quasi-perspective projection model: theory and application to structure and motion factorization from uncalibrated image sequences. International Journal of Computer Vision. doi: 10.1007/s11263-009-0267-4.
  25. Wang, G., Tsui, H. T., & Wu, Q. M. J. (2008). Rotation constrained power factorization for structure from motion of nonrigid objects. Pattern Recognition Letters, 29(1). doi: 10.1016/j.patrec.2007.09.004.
  26. Xiao, J., & Kanade, T. (2005). Uncalibrated perspective reconstruction of deformable structures. In Proc. 10th international conference on computer vision, Beijing, China. Google Scholar
  27. Xiao, J., Chai, J., & Kanade, T. (2006). A closed-form solution to non-rigid shape and motion recovery. International Journal of Computer Vision, 67(2). doi: 10.1007/s11263-005-3962-9.
  28. Yan, J., & Pollefeys, M. (2008). A factorization-based approach for articulated non-rigid shape, motion and kinematic chain recovery from video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(5). doi: 10.1109/TPAMI.2007.70739.

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Marco Paladini
    • 3
  • Alessio Del Bue
    • 1
  • João Xavier
    • 2
  • Lourdes Agapito
    • 3
  • Marko Stošić
    • 2
  • Marija Dodig
    • 4
  1. 1.IIT—Istituto Italiano di TecnologiaGenovaItaly
  2. 2.Instituto de Sistemas e Robotica (ISR), Instituto Superior Técnico (IST)Technical University of LisbonLisboaPortugal
  3. 3.School of Electronic Engineering and Computer ScienceQueen Mary University of LondonLondonUK
  4. 4.CELCUniversidade de LisboaLisboaPortugal

Personalised recommendations