Optimal Metric Projections for Deformable and Articulated Structure-from-Motion
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
This paper describes novel algorithms for recovering the 3D shape and motion of deformable and articulated objects purely from uncalibrated 2D image measurements using a factorisation approach. Most approaches to deformable and articulated structure from motion require to upgrade an initial affine solution to Euclidean space by imposing metric constraints on the motion matrix. While in the case of rigid structure the metric upgrade step is simple since the constraints can be formulated as linear, deformability in the shape introduces non-linearities. In this paper we propose an alternating bilinear approach to solve for non-rigid 3D shape and motion, associated with a globally optimal projection step of the motion matrices onto the manifold of metric constraints. Our novel optimal projection step combines into a single optimisation the computation of the orthographic projection matrix and the configuration weights that give the closest motion matrix that satisfies the correct block structure with the additional constraint that the projection matrix is guaranteed to have orthonormal rows (i.e. its transpose lies on the Stiefel manifold). This constraint turns out to be non-convex. The key contribution of this work is to introduce an efficient convex relaxation for the non-convex projection step. Efficient in the sense that, for both the cases of deformable and articulated motion, the proposed relaxations turned out to be exact (i.e. tight) in all our numerical experiments. The convex relaxations are semi-definite (SDP) or second-order cone (SOCP) programs which can be readily tackled by popular solvers. An important advantage of these new algorithms is their ability to handle missing data which becomes crucial when dealing with real video sequences with self-occlusions. We show successful results of our algorithms on synthetic and real sequences of both deformable and articulated data. We also show comparative results with state of the art algorithms which reveal that our new methods outperform existing ones.
- Aanæs, H., & Kahl, F. (2002). Estimation of deformable structure and motion. In Workshop on vision and modelling of dynamic scenes, Copenhagen, Denmark.
- Akhter, I., Sheikh, Y., Khan, S., & Kanade, T. (2008). Nonrigid structure from motion in trajectory space. In Neural information processing systems.
- Akhter, I., Sheikh, Y., & Khan, S. (2009). In defense of orthonormality constraints for nonrigid structure from motion. In Proc. IEEE conference on computer vision and pattern recognition, Miami, Florida.
- Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S., & Sayd, P. (2008). Coarse-to-fine low-rank structure-from-motion. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska.
- Brand, M. (2005). A direct method for 3D factorization of nonrigid motion observed in 2D. In Proc. IEEE conference on computer vision and pattern recognition, San Diego, California.
- Bregler, C., Hertzmann, A., & Biermann, H. (2000). Recovering non-rigid 3D shape from image streams. In Proc. IEEE conference on computer vision and pattern recognition, Hilton Head, South Carolina.
- Buchanan, A. M., & Fitzgibbon, A. (2005). Damped Newton algorithms for matrix factorization with missing data. In Proc. IEEE conference on computer vision and pattern recognition: Vol. 2. San Diego, California.
- Del Bue, A. (2008). A factorization approach to structure from motion with shape priors. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska.
- Del Bue, A., Lladó, X., & Agapito, L. (2006). Non-rigid metric shape and motion recovery from uncalibrated images using priors. In Proc. IEEE conference on computer vision and pattern recognition, New York, NY.
- Del Bue, A., Smeraldi, F., & Agapito, L. (2007). Non-rigid structure from motion using ranklet–based tracking and non-linear optimization. Image and Vision Computing, 25(3). doi:10.1016/j.imavis.2005.10.004.
- Dodig, M., Stošić, M., & Xavier, J. (2009). On minimizing a quadratic function on Stiefel manifolds (Tech. rept.). Instituto de Sistemas e Robotica. Available at http://users.isr.ist.utl.pt/~jxavier/ctech.pdf.
- Edelman, A., Arias, T. A., & Smith, S. T. (1999). The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications, 20(2). doi:10.1137/S0895479895290954.
- Fayad, J., Del Bue, A., Agapito, L., & Aguiar, P. (2009). Non-rigid structure from motion using quadratic deformation models. In British machine vision conference, London, UK.
- Hartley, R., & Vidal, R. (2008). Perspective nonrigid shape and motion recovery. In Proc. European conference on computer vision.
- Marques, M., & Costeira, J. (2008). Optimal shape from motion estimation with missing and degenerate data. In WMVC ’08: proceedings of the 2008 IEEE workshop on motion and video computing (pp. 1–6). Washington: IEEE Computer Society. CrossRef
- Marques, M., & Costeira, J. (2009). Estimating 3D shape from degenerate sequences with missing data. Computer Vision and Image Understanding, 113(2). doi:10.1016/j.cviu.2008.09.004.
- Paladini, M., Del Bue, A., Stošić, M., Dodig, M., Xavier, J., & Agapito, L. (2009). Factorization for non-rigid and articulated structure using metric projections. In Proc. IEEE conference on computer vision and pattern recognition, Miami, Florida.
- Rabaud, V., & Belongie, S. (2008). Re-thinking non-rigid structure from motion. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska.
- Sturm, J. F. (1999). Using SeDuMi 1.02, A Matlab toolbox for optimization over symmetric cones. Optimization Methods and Software, 11–12, 625–653. CrossRef
- Tomasi, C., & Kanade, T. (1992). Shape and motion from image streams under orthography: a factorization approach. International Journal of Computer Vision, 9(2). doi:10.1007/BF00129684.
- Torresani, L., Yang, D., Alexander, E., & Bregler, C. (2001). Tracking and modeling non-rigid objects with rank constraints. In: Proc. IEEE conference on computer vision and pattern recognition, Kauai, Hawaii.
- Torresani, L., Hertzmann, A., & Bregler, C. (2008). Non-rigid structure-from-motion: estimating shape and motion with hierarchical priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(5). doi:10.1109/TPAMI.2007.70752.
- Tresadern, P., & Reid, I. (2005). Articulated structure from motion by factorization. In Proc. IEEE conference on computer vision and pattern recognition: Vol. 2. San Diego, California.
- Wang, G., & Wu, Q. M. J. (2008). Quasi-perspective projection model: theory and application to structure and motion factorization from uncalibrated image sequences. International Journal of Computer Vision. doi:10.1007/s11263-009-0267-4.
- Wang, G., Tsui, H. T., & Wu, Q. M. J. (2008). Rotation constrained power factorization for structure from motion of nonrigid objects. Pattern Recognition Letters, 29(1). doi:10.1016/j.patrec.2007.09.004.
- Xiao, J., & Kanade, T. (2005). Uncalibrated perspective reconstruction of deformable structures. In Proc. 10th international conference on computer vision, Beijing, China.
- Xiao, J., Chai, J., & Kanade, T. (2006). A closed-form solution to non-rigid shape and motion recovery. International Journal of Computer Vision, 67(2). doi:10.1007/s11263-005-3962-9.
- Yan, J., & Pollefeys, M. (2008). A factorization-based approach for articulated non-rigid shape, motion and kinematic chain recovery from video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(5). doi:10.1109/TPAMI.2007.70739.
- Optimal Metric Projections for Deformable and Articulated Structure-from-Motion
International Journal of Computer Vision
Volume 96, Issue 2 , pp 252-276
- Cover Date
- Print ISSN
- Online ISSN
- Springer US
- Additional Links
- Non-rigid structure from motion
- Articulated structure from motion
- Convex optimization
- Industry Sectors
- Author Affiliations
- 3. School of Electronic Engineering and Computer Science, Queen Mary University of London, London, UK
- 1. IIT—Istituto Italiano di Tecnologia, Genova, Italy
- 2. Instituto de Sistemas e Robotica (ISR), Instituto Superior Técnico (IST), Technical University of Lisbon, Lisboa, Portugal
- 4. CELC, Universidade de Lisboa, Lisboa, Portugal