Multilinear Factorizations for Multi-Camera Rigid Structure from Motion Problems
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
Camera networks have gained increased importance in recent years. Existing approaches mostly use point correspondences between different camera views to calibrate such systems. However, it is often difficult or even impossible to establish such correspondences. But even without feature point correspondences between different camera views, if the cameras are temporally synchronized then the data from the cameras are strongly linked together by the motion correspondence: all the cameras observe the same motion. The present article therefore develops the necessary theory to use this motion correspondence for general rigid as well as planar rigid motions. Given multiple static affine cameras which observe a rigidly moving object and track feature points located on this object, what can be said about the resulting point trajectories? Are there any useful algebraic constraints hidden in the data? Is a 3D reconstruction of the scene possible even if there are no point correspondences between the different cameras? And if so, how many points are sufficient? Is there an algorithm which warrants finding the correct solution to this highly non-convex problem? This article addresses these questions and thereby introduces the concept of low-dimensional motion subspaces. The constraints provided by these motion subspaces enable an algorithm which ensures finding the correct solution to this non-convex reconstruction problem. The algorithm is based on multilinear analysis, matrix and tensor factorizations. Our new approach can handle extreme configurations, e.g. a camera in a camera network tracking only one single point. Results on synthetic as well as on real data sequences act as a proof of concept for the presented insights.
- Aguiar, P. M. Q., Xavier, J. M. F.,& Stosic, M. (2008). Spectrally optimal factorization of incomplete matrices. In IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer Society.
- Akhter, I., Sheikh, Y., Khan, S.,& Kanade, T. (2011). Trajectory space: A dual representation for nonrigid structure from motion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(7), 1442–1456. doi:10.1109/TPAMI.2010.201.
- Angst, R.,& Pollefeys, M. (2009). Static multi-camera factorization using rigid motion. In Proceedings of IEEE international conference on computer vision 2009 (ICCV ’09), Washington, DC (pp. 1203–1210). IEEE Computer Society.
- Angst, R.,& Pollefeys, M. (2010). 5D motion subspaces for planar motions. In Proceedings of the 11th European conference on computer vision conference on computer vision 2010 (ECCV’10): Part III (pp. 144–157). Berlin: Springer.
- Brand, M. (2001). Morphable 3D models from video. In IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 2, pp. 456–463). IEEE Computer Society.
- Brand, M. (2005). A direct method for 3D factorization of nonrigid motion observed in 2D. In IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 2, pp. 122–128). IEEE Computer Society.
- Bregler, C., Hertzmann, A.,& Biermann, H. (2000). Recovering non-rigid 3D shape from image streams. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2690–2696). IEEE Computer Society.
- Bue, A. D.,& de Agapito, L. (2006). Non-rigid stereo factorization. International Journal of Computer Vision, 66(2), 193–207.
- Carroll, J.,& Chang, J. J. (1970). Analysis of individual differences in multidimensional scaling via an n-way generalization of Eckart–Young decomposition. Psychometrika, 35(3), 283–319. CrossRef
- Chen, P. (2008). Optimization algorithms on subspaces: Revisiting missing data problem in low-rank matrix. International Journal of Computer Vision, 80(1), 125–142.
- Daniilidis, K. (1999). Hand-eye calibration using dual quaternions. International Journal of Robotics Research, 18(3), 286–298.
- Guerreiro, R. F. C.,& Aguiar, P. M. Q. (2002). 3D structure from video streams with partially overlapping images. In International conference on image processing (ICIP) (Vol. 3, pp. 897–900).
- Harshman, R. (1970). Foundations of the parafac procedure: Models and conditions for an explanatory multi-modal factor analysis. Working papers in phonetics, Vol. 16.
- Hartley, R.,& Schaffalitzky, F. (2004). Power factorization: 3D reconstruction with missing or uncertain data. In Japan-Australia workshop on computer vision.
- Hartley, R. I.,& Zisserman, A. (2004). Multiple view geometry in computer vision (2nd ed.). Cambridge: Cambridge University Press. ISBN: 0521540518. CrossRef
- Kolda, T. G.,& Bader, B. W. (2009). Tensor decompositions and applications. SIAM Review, 51(3), 455–500. doi:10.1137/07070111X.
- Kumar, R. K., Ilie, A., Frahm, J. M.,& Pollefeys, M. (2008). Simple calibration of non-overlapping cameras with a mirror. In IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer Society.
- Lathauwer, L. D., Moor, B.,& Vandewalle, J. (2000). A multilinear singular value decomposition. SIAM Journal of Matrix Analysis and Applications, 21(4), 1253–1278. doi:10.1137/S0895479896305696.
- Li, J.,& Chellappa, R. (2005). A factorization method for structure from planar motion. In 7th IEEE workshop on applications of computer vision/IEEE workshop on motion and video computing (WACV/MOTION) (pp. 154–159). IEEE Computer Society.
- Magnus, J. R.,& Neudecker, H. (1999). Matrix differential calculus with applications in statistics and econometrics (2nd ed.). New York: Wiley.
- Sturm, P. F.,& Triggs, B. (1996). A factorization based algorithm for multi-image projective structure and motion. In Buxton, B. F.,& Cipolla, R. (Eds.). European conference on computer vision (ECCV) (Vol. 2, pp. 709–720). Lecture Notes in Computer Science, Vol. 1065. Berlin: Springer
- Svoboda, T., Martinec, D.,& Pajdla, T. (2005). A convenient multicamera self-calibration for virtual environments. PRESENCE: Teleoperators and Virtual Environments, 14(4), 407–422.
- Tomasi, C.,& Kanade, T. (1992). Shape and motion from image streams under orthography: A factorization method. International Journal of Computer Vision, 9(2), 137–154.
- Torresani, L., Yang, D. B., Alexander, E. J.,& Bregler, C. (2001). Tracking and modeling non-rigid objects with rank constraints. In IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 1, pp. 493–500). IEEE Computer Society.
- Tresadern, P. A.,& Reid, I. D. (2005). Articulated structure from motion by factorization. In: IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 2, pp. 1110–1115). IEEE Computer Society.
- Tron, R.,& Vidal, R. (2007). A benchmark for the comparison of 3-D motion segmentation algorithms. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer Society.
- Tucker, L. (1966). Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3), 279–311.
- Vidal, R.,& Oliensis, J. (2002). Structure from planar motions with small baselines. In Heyden, A., Sparr, G., Nielsen, M.,& Johansen, P. (eds). European conference on computer vision (ECCV) (Vol. 2, pp. 383–398). Lecture Notes in Computer Science, Vol. 2351. Berlin: Springer.
- Wang, G., Tsui, H. T.,& Wu, Q. M. J. (2008). Rotation constrained power factorization for structure from motion of nonrigid objects. Pattern Recognition Letters, 29(1), 72–80.
- Wagner, D.,& Schmalstieg, D. (2007). Artoolkitplus for pose tracking on mobile devices. In Proceedings of 12th computer vision winter workshop.
- Wolf, L.,& Zomet, A. (2006). Wide baseline matching between unsynchronized video sequences. International Journal of Computer Vision, 68(1), 43–52.
- Xiao, J., Chai, J.,& Kanade, T. (2004). A closed-form solution to non-rigid shape and motion recovery. In Pajdla, T.,& Matas, J. (Eds.). European conference on computer vision (ECCV) (Vol. 4, pp. 573–587). Lecture Notes in Computer Science, Vol. 3024. Berlin: Springer.
- Yan, J.,& Pollefeys, M. (2008). A factorization-based approach for articulated nonrigid shape, motion and kinematic chain recovery from video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(5), 865–877.
- Zelnik-Manor, L.,& Irani, M. (2006). On single-sequence and multi-sequence factorizations. International Journal of Computer Vision, 67(3), 313–326.
- Multilinear Factorizations for Multi-Camera Rigid Structure from Motion Problems
International Journal of Computer Vision
Volume 103, Issue 2 , pp 240-266
- Cover Date
- Print ISSN
- Online ISSN
- Springer US
- Additional Links
- Computer vision
- 3D reconstruction
- Structure from motion
- Multilinear factorizations
- Tensor algebra
- Industry Sectors