International Journal of Computer Vision

, Volume 103, Issue 2, pp 240–266 | Cite as

Multilinear Factorizations for Multi-Camera Rigid Structure from Motion Problems

Article
  • 742 Downloads

Abstract

Camera networks have gained increased importance in recent years. Existing approaches mostly use point correspondences between different camera views to calibrate such systems. However, it is often difficult or even impossible to establish such correspondences. But even without feature point correspondences between different camera views, if the cameras are temporally synchronized then the data from the cameras are strongly linked together by the motion correspondence: all the cameras observe the same motion. The present article therefore develops the necessary theory to use this motion correspondence for general rigid as well as planar rigid motions. Given multiple static affine cameras which observe a rigidly moving object and track feature points located on this object, what can be said about the resulting point trajectories? Are there any useful algebraic constraints hidden in the data? Is a 3D reconstruction of the scene possible even if there are no point correspondences between the different cameras? And if so, how many points are sufficient? Is there an algorithm which warrants finding the correct solution to this highly non-convex problem? This article addresses these questions and thereby introduces the concept of low-dimensional motion subspaces. The constraints provided by these motion subspaces enable an algorithm which ensures finding the correct solution to this non-convex reconstruction problem. The algorithm is based on multilinear analysis, matrix and tensor factorizations. Our new approach can handle extreme configurations, e.g. a camera in a camera network tracking only one single point. Results on synthetic as well as on real data sequences act as a proof of concept for the presented insights.

Keywords

Computer vision 3D reconstruction Structure from motion Multilinear factorizations Tensor algebra  

References

  1. Aguiar, P. M. Q., Xavier, J. M. F.,& Stosic, M. (2008). Spectrally optimal factorization of incomplete matrices. In IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer Society.Google Scholar
  2. Akhter, I., Sheikh, Y., Khan, S.,& Kanade, T. (2011). Trajectory space: A dual representation for nonrigid structure from motion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(7), 1442–1456. doi:10.1109/TPAMI.2010.201.Google Scholar
  3. Angst, R.,& Pollefeys, M. (2009). Static multi-camera factorization using rigid motion. In Proceedings of IEEE international conference on computer vision 2009 (ICCV ’09), Washington, DC (pp. 1203–1210). IEEE Computer Society.Google Scholar
  4. Angst, R.,& Pollefeys, M. (2010). 5D motion subspaces for planar motions. In Proceedings of the 11th European conference on computer vision conference on computer vision 2010 (ECCV’10): Part III (pp. 144–157). Berlin: Springer.Google Scholar
  5. Brand, M. (2001). Morphable 3D models from video. In IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 2, pp. 456–463). IEEE Computer Society.Google Scholar
  6. Brand, M. (2005). A direct method for 3D factorization of nonrigid motion observed in 2D. In IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 2, pp. 122–128). IEEE Computer Society.Google Scholar
  7. Bregler, C., Hertzmann, A.,& Biermann, H. (2000). Recovering non-rigid 3D shape from image streams. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2690–2696). IEEE Computer Society.Google Scholar
  8. Bue, A. D.,& de Agapito, L. (2006). Non-rigid stereo factorization. International Journal of Computer Vision, 66(2), 193–207.Google Scholar
  9. Carroll, J.,& Chang, J. J. (1970). Analysis of individual differences in multidimensional scaling via an n-way generalization of Eckart–Young decomposition. Psychometrika, 35(3), 283–319.MATHCrossRefGoogle Scholar
  10. Chen, P. (2008). Optimization algorithms on subspaces: Revisiting missing data problem in low-rank matrix. International Journal of Computer Vision, 80(1), 125–142.Google Scholar
  11. Daniilidis, K. (1999). Hand-eye calibration using dual quaternions. International Journal of Robotics Research, 18(3), 286–298.Google Scholar
  12. Guerreiro, R. F. C.,& Aguiar, P. M. Q. (2002). 3D structure from video streams with partially overlapping images. In International conference on image processing (ICIP) (Vol. 3, pp. 897–900).Google Scholar
  13. Harshman, R. (1970). Foundations of the parafac procedure: Models and conditions for an explanatory multi-modal factor analysis. Working papers in phonetics, Vol. 16.Google Scholar
  14. Hartley, R.,& Schaffalitzky, F. (2004). Power factorization: 3D reconstruction with missing or uncertain data. In Japan-Australia workshop on computer vision.Google Scholar
  15. Hartley, R. I.,& Zisserman, A. (2004). Multiple view geometry in computer vision (2nd ed.). Cambridge: Cambridge University Press. ISBN: 0521540518.MATHCrossRefGoogle Scholar
  16. Kolda, T. G.,& Bader, B. W. (2009). Tensor decompositions and applications. SIAM Review, 51(3), 455–500. doi:10.1137/07070111X.Google Scholar
  17. Kumar, R. K., Ilie, A., Frahm, J. M.,& Pollefeys, M. (2008). Simple calibration of non-overlapping cameras with a mirror. In IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer Society.Google Scholar
  18. Lathauwer, L. D., Moor, B.,& Vandewalle, J. (2000). A multilinear singular value decomposition. SIAM Journal of Matrix Analysis and Applications, 21(4), 1253–1278. doi:10.1137/S0895479896305696.Google Scholar
  19. Li, J.,& Chellappa, R. (2005). A factorization method for structure from planar motion. In 7th IEEE workshop on applications of computer vision/IEEE workshop on motion and video computing (WACV/MOTION) (pp. 154–159). IEEE Computer Society.Google Scholar
  20. Magnus, J. R.,& Neudecker, H. (1999). Matrix differential calculus with applications in statistics and econometrics (2nd ed.). New York: Wiley.MATHGoogle Scholar
  21. Sturm, P. F.,& Triggs, B. (1996). A factorization based algorithm for multi-image projective structure and motion. In Buxton, B. F.,& Cipolla, R. (Eds.). European conference on computer vision (ECCV) (Vol. 2, pp. 709–720). Lecture Notes in Computer Science, Vol. 1065. Berlin: SpringerGoogle Scholar
  22. Svoboda, T., Martinec, D.,& Pajdla, T. (2005). A convenient multicamera self-calibration for virtual environments. PRESENCE: Teleoperators and Virtual Environments, 14(4), 407–422.Google Scholar
  23. Tomasi, C.,& Kanade, T. (1992). Shape and motion from image streams under orthography: A factorization method. International Journal of Computer Vision, 9(2), 137–154.Google Scholar
  24. Torresani, L., Yang, D. B., Alexander, E. J.,& Bregler, C. (2001). Tracking and modeling non-rigid objects with rank constraints. In IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 1, pp. 493–500). IEEE Computer Society.Google Scholar
  25. Tresadern, P. A.,& Reid, I. D. (2005). Articulated structure from motion by factorization. In: IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 2, pp. 1110–1115). IEEE Computer Society.Google Scholar
  26. Tron, R.,& Vidal, R. (2007). A benchmark for the comparison of 3-D motion segmentation algorithms. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer Society. Google Scholar
  27. Tucker, L. (1966). Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3), 279–311.Google Scholar
  28. Vidal, R.,& Oliensis, J. (2002). Structure from planar motions with small baselines. In Heyden, A., Sparr, G., Nielsen, M.,& Johansen, P. (eds). European conference on computer vision (ECCV) (Vol. 2, pp. 383–398). Lecture Notes in Computer Science, Vol. 2351. Berlin: Springer.Google Scholar
  29. Wang, G., Tsui, H. T.,& Wu, Q. M. J. (2008). Rotation constrained power factorization for structure from motion of nonrigid objects. Pattern Recognition Letters, 29(1), 72–80.Google Scholar
  30. Wagner, D.,& Schmalstieg, D. (2007). Artoolkitplus for pose tracking on mobile devices. In Proceedings of 12th computer vision winter workshop.Google Scholar
  31. Wolf, L.,& Zomet, A. (2006). Wide baseline matching between unsynchronized video sequences. International Journal of Computer Vision, 68(1), 43–52.Google Scholar
  32. Xiao, J., Chai, J.,& Kanade, T. (2004). A closed-form solution to non-rigid shape and motion recovery. In Pajdla, T.,& Matas, J. (Eds.). European conference on computer vision (ECCV) (Vol. 4, pp. 573–587). Lecture Notes in Computer Science, Vol. 3024. Berlin: Springer.Google Scholar
  33. Yan, J.,& Pollefeys, M. (2008). A factorization-based approach for articulated nonrigid shape, motion and kinematic chain recovery from video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(5), 865–877.Google Scholar
  34. Zelnik-Manor, L.,& Irani, M. (2006). On single-sequence and multi-sequence factorizations. International Journal of Computer Vision, 67(3), 313–326.Google Scholar

Copyright information

© Springer Science+Business Media New York 2012

Authors and Affiliations

  1. 1.ETH ZürichZürichSwitzerland

Personalised recommendations