Skip to main content
Log in

Optimal Metric Projections for Deformable and Articulated Structure-from-Motion

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

This paper describes novel algorithms for recovering the 3D shape and motion of deformable and articulated objects purely from uncalibrated 2D image measurements using a factorisation approach. Most approaches to deformable and articulated structure from motion require to upgrade an initial affine solution to Euclidean space by imposing metric constraints on the motion matrix. While in the case of rigid structure the metric upgrade step is simple since the constraints can be formulated as linear, deformability in the shape introduces non-linearities. In this paper we propose an alternating bilinear approach to solve for non-rigid 3D shape and motion, associated with a globally optimal projection step of the motion matrices onto the manifold of metric constraints. Our novel optimal projection step combines into a single optimisation the computation of the orthographic projection matrix and the configuration weights that give the closest motion matrix that satisfies the correct block structure with the additional constraint that the projection matrix is guaranteed to have orthonormal rows (i.e. its transpose lies on the Stiefel manifold). This constraint turns out to be non-convex. The key contribution of this work is to introduce an efficient convex relaxation for the non-convex projection step. Efficient in the sense that, for both the cases of deformable and articulated motion, the proposed relaxations turned out to be exact (i.e. tight) in all our numerical experiments. The convex relaxations are semi-definite (SDP) or second-order cone (SOCP) programs which can be readily tackled by popular solvers. An important advantage of these new algorithms is their ability to handle missing data which becomes crucial when dealing with real video sequences with self-occlusions. We show successful results of our algorithms on synthetic and real sequences of both deformable and articulated data. We also show comparative results with state of the art algorithms which reveal that our new methods outperform existing ones.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aanæs, H., & Kahl, F. (2002). Estimation of deformable structure and motion. In Workshop on vision and modelling of dynamic scenes, Copenhagen, Denmark.

    Google Scholar 

  • Akhter, I., Sheikh, Y., Khan, S., & Kanade, T. (2008). Nonrigid structure from motion in trajectory space. In Neural information processing systems.

    Google Scholar 

  • Akhter, I., Sheikh, Y., & Khan, S. (2009). In defense of orthonormality constraints for nonrigid structure from motion. In Proc. IEEE conference on computer vision and pattern recognition, Miami, Florida.

    Google Scholar 

  • Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S., & Sayd, P. (2008). Coarse-to-fine low-rank structure-from-motion. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska.

    Google Scholar 

  • Brand, M. (2005). A direct method for 3D factorization of nonrigid motion observed in 2D. In Proc. IEEE conference on computer vision and pattern recognition, San Diego, California.

    Google Scholar 

  • Bregler, C., Hertzmann, A., & Biermann, H. (2000). Recovering non-rigid 3D shape from image streams. In Proc. IEEE conference on computer vision and pattern recognition, Hilton Head, South Carolina.

    Google Scholar 

  • Buchanan, A. M., & Fitzgibbon, A. (2005). Damped Newton algorithms for matrix factorization with missing data. In Proc. IEEE conference on computer vision and pattern recognition: Vol. 2. San Diego, California.

    Google Scholar 

  • Del Bue, A. (2008). A factorization approach to structure from motion with shape priors. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska.

    Google Scholar 

  • Del Bue, A., Lladó, X., & Agapito, L. (2006). Non-rigid metric shape and motion recovery from uncalibrated images using priors. In Proc. IEEE conference on computer vision and pattern recognition, New York, NY.

    Google Scholar 

  • Del Bue, A., Smeraldi, F., & Agapito, L. (2007). Non-rigid structure from motion using ranklet–based tracking and non-linear optimization. Image and Vision Computing, 25(3). doi:10.1016/j.imavis.2005.10.004.

  • Dodig, M., Stošić, M., & Xavier, J. (2009). On minimizing a quadratic function on Stiefel manifolds (Tech. rept.). Instituto de Sistemas e Robotica. Available at http://users.isr.ist.utl.pt/~jxavier/ctech.pdf.

  • Edelman, A., Arias, T. A., & Smith, S. T. (1999). The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications, 20(2). doi:10.1137/S0895479895290954.

  • Fayad, J., Del Bue, A., Agapito, L., & Aguiar, P. (2009). Non-rigid structure from motion using quadratic deformation models. In British machine vision conference, London, UK.

    Google Scholar 

  • Hartley, R., & Vidal, R. (2008). Perspective nonrigid shape and motion recovery. In Proc. European conference on computer vision.

    Google Scholar 

  • Marques, M., & Costeira, J. (2008). Optimal shape from motion estimation with missing and degenerate data. In WMVC ’08: proceedings of the 2008 IEEE workshop on motion and video computing (pp. 1–6). Washington: IEEE Computer Society.

    Chapter  Google Scholar 

  • Marques, M., & Costeira, J. (2009). Estimating 3D shape from degenerate sequences with missing data. Computer Vision and Image Understanding, 113(2). doi:10.1016/j.cviu.2008.09.004.

  • Paladini, M., Del Bue, A., Stošić, M., Dodig, M., Xavier, J., & Agapito, L. (2009). Factorization for non-rigid and articulated structure using metric projections. In Proc. IEEE conference on computer vision and pattern recognition, Miami, Florida.

    Google Scholar 

  • Rabaud, V., & Belongie, S. (2008). Re-thinking non-rigid structure from motion. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska.

    Google Scholar 

  • Sturm, J. F. (1999). Using SeDuMi 1.02, A Matlab toolbox for optimization over symmetric cones. Optimization Methods and Software, 11–12, 625–653.

    Article  MathSciNet  Google Scholar 

  • Tomasi, C., & Kanade, T. (1992). Shape and motion from image streams under orthography: a factorization approach. International Journal of Computer Vision, 9(2). doi:10.1007/BF00129684.

  • Torresani, L., Yang, D., Alexander, E., & Bregler, C. (2001). Tracking and modeling non-rigid objects with rank constraints. In: Proc. IEEE conference on computer vision and pattern recognition, Kauai, Hawaii.

    Google Scholar 

  • Torresani, L., Hertzmann, A., & Bregler, C. (2008). Non-rigid structure-from-motion: estimating shape and motion with hierarchical priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(5). doi:10.1109/TPAMI.2007.70752.

  • Tresadern, P., & Reid, I. (2005). Articulated structure from motion by factorization. In Proc. IEEE conference on computer vision and pattern recognition: Vol. 2. San Diego, California.

    Google Scholar 

  • Wang, G., & Wu, Q. M. J. (2008). Quasi-perspective projection model: theory and application to structure and motion factorization from uncalibrated image sequences. International Journal of Computer Vision. doi:10.1007/s11263-009-0267-4.

  • Wang, G., Tsui, H. T., & Wu, Q. M. J. (2008). Rotation constrained power factorization for structure from motion of nonrigid objects. Pattern Recognition Letters, 29(1). doi:10.1016/j.patrec.2007.09.004.

  • Xiao, J., & Kanade, T. (2005). Uncalibrated perspective reconstruction of deformable structures. In Proc. 10th international conference on computer vision, Beijing, China.

    Google Scholar 

  • Xiao, J., Chai, J., & Kanade, T. (2006). A closed-form solution to non-rigid shape and motion recovery. International Journal of Computer Vision, 67(2). doi:10.1007/s11263-005-3962-9.

  • Yan, J., & Pollefeys, M. (2008). A factorization-based approach for articulated non-rigid shape, motion and kinematic chain recovery from video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(5). doi:10.1109/TPAMI.2007.70739.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Lourdes Agapito.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

(PDF 106 kB)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Paladini, M., Del Bue, A., Xavier, J. et al. Optimal Metric Projections for Deformable and Articulated Structure-from-Motion. Int J Comput Vis 96, 252–276 (2012). https://doi.org/10.1007/s11263-011-0468-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-011-0468-5

Keywords

Navigation