Advertisement

Quasi-perspective Projection Model: Theory and Application to Structure and Motion Factorization from Uncalibrated Image Sequences

Abstract

This paper addresses the problem of factorization-based 3D reconstruction from uncalibrated image sequences. Previous studies on structure and motion factorization are either based on simplified affine assumption or general perspective projection. The affine approximation is widely adopted due to its simplicity, whereas the extension to perspective model suffers from recovering projective depths. To fill the gap between simplicity of affine and accuracy of perspective model, we propose a quasi-perspective projection model for structure and motion recovery of rigid and nonrigid objects based on factorization framework. The novelty and contribution of this paper are as follows. Firstly, under the assumption that the camera is far away from the object with small lateral rotations, we prove that the imaging process can be modeled by quasi-perspective projection, which is more accurate than affine model from both geometrical error analysis and experimental studies. Secondly, we apply the model to establish a framework of rigid and nonrigid factorization under quasi-perspective assumption. Finally, we propose an Extended Cholesky Decomposition to recover the rotation part of the Euclidean upgrading matrix. We also prove that the last column of the upgrading matrix corresponds to a global scale and translation of the camera thus may be set freely. The proposed method is validated and evaluated extensively on synthetic and real image sequences and improved results over existing schemes are observed.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 199

This is the net price. Taxes to be calculated in checkout.

References

  1. Bascle, B., & Blake, A. (1998). Separability of pose and expression in facial tracing and animation. In Proceedings of the international conference on computer vision (pp. 323–328) 1998.

  2. Brand, M. (2001). Morphable 3D models from video. In Proceedings of IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 456–463) 2001.

  3. Brand, M. (2005). A direct method for 3D factorization of nonrigid motion observed in 2d. In Proceedings of IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 122–128) 2005.

  4. Bregler, C., Hertzmann, A., & Biermann, H. (2000). Recovering non-rigid 3D shape from image streams. In Proceedings of IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 690–696) 2000.

  5. Buchanan, A. M., & Fitzgibbon, A. W. (2005). Damped newton algorithms for matrix factorization with missing data. In Proceedings of IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 316–322) 2005.

  6. Chen, P. (2008). Optimization algorithms on subspaces: revisiting missing data problem in low-rank matrix. International Journal of Computer Vision, 80(1), 125–142.

  7. Christy, S., & Horaud, R. (1996). Euclidean shape and motion from multiple perspective views by affine iterations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(11), 1098–1104.

  8. Costeira, J., & Kanade, T. (1998). A multibody factorization method for independent moving objects. International Journal of Computer Vision, 29(3), 159–179.

  9. Del Bue, A., Smeraldi, F., & de Agapito, L. (2004). Non-rigid structure from motion using nonparametric tracking and non-linear optimization. In IEEE workshop in articulated and nonrigid motion ANM04, held in conjunction with CVPR2004 (pp. 8–15), June 2004.

  10. Del Bue, A., Lladó, X., & de Agapito, L. (2006). Non-rigid metric shape and motion recovery from uncalibrated images using priors. In Proceedings of IEEE conference on computer vision and pattern recognition (Vol. 1, pp. 1191–1198) 2006.

  11. Han, M., & Kanade, T. (2000). Creating 3D models with uncalibrated cameras. In Proceedings of IEEE computer society workshop on the application of computer vision (WACV2000), December 2000.

  12. Hartley, R. (1997). Kruppa’s equations derived from the fundamental matrix. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(2), 133–135.

  13. Hartley, R., & Schaffalizky, F. (2003). Powerfactorization: 3D reconstruction with missing or uncertain data. In Australia-Japan advanced workshop on computer vision, 2003.

  14. Hartley, R., & Vidal, R. (2008). Perspective nonrigid shape and motion recovery. In ECCV (1), Lecture notes in computer science : Vol. 5302 (pp. 276–289). Berlin: Springer.

  15. Hartley, R., & Zisserman, A. (2004). Multiple view geometry in computer vision (2nd edn.). Cambridge: Cambridge University Press.

  16. Heyden, A., & Åström, K. (1997) Euclidean reconstruction from image sequences with varying and unknown focal length and principal point. In IEEE conference on computer vision and pattern recognition (pp. 438–443) 1997.

  17. Heyden, A., Berthilsson, R., & Sparr, G. (1999). An iterative factorization method for projective structure and motion from image sequences. Image and Vision Computing, 17(13), 981–991.

  18. Li, T., Kallem, V., Singaraju, D., & Vidal, R. (2007). Projective factorization of multiple rigid-body motions. In IEEE conference on computer vision and pattern recognition, 2007.

  19. Luong, Q., & Faugeras, O. (1997). Self-calibration of a moving camera from point correspondences and fundamental matrices. International Journal of Computer Vision, 22(3), 261–289.

  20. Mahamud, S., & Hebert, M. (2000). Iterative projective reconstruction from multiple views. In IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 430–437) 2000.

  21. Maybank, S., & Faugeras, O. (1992). A theory of self-calibration of a moving camera. International Journal of Computer Vision, 8(2), 123–151.

  22. Oliensis, J., & Hartley, R. (2007). Iterative extensions of the Sturm/Triggs algorithm: convergence and nonconvergence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(12), 2217–2233.

  23. Poelman, C., & Kanade, T. (1997). A paraperspective factorization method for shape and motion recovery. IEEE Transactions on Pattern and Analysis and Machine Intelligence, 19(3), 206–218.

  24. Pollefeys, M., Koch, R., & Van Gool, L. (1999). Self-calibration and metric reconstruction in spite of varying and unknown intrinsic camera parameters. International Journal of Computer Vision, 32(1), 7–25.

  25. Quan, L. (1996). Self-calibration of an affine camera from multiple views. International Journal of Computer Vision, 19(1), 93–105.

  26. Rabaud, V., & Belongie, S. (2008). Re-thinking non-rigid structure from motion. In IEEE conference on computer vision and pattern recognition, 2008.

  27. Sturm, P. F., & Triggs, B. (1996). A factorization based algorithm for multi-image projective structure and motion. In European conference on computer vision (2) (pp. 709–720) 1996.

  28. Tomasi, C., & Kanade, T. (1992). Shape and motion from image streams under orthography: a factorization method. International Journal of Computer Vision, 9(2), 137–154.

  29. Torr, P. H. S., Zisserman, A., & Maybank, S. J. (1998). Robust detection of degenerate configurations while estimating the fundamental matrix. Computer Vision and Image Understanding, 71(3), 312–333.

  30. Torresani, L., Yang, D. B., Alexander, E. J., & Bregler, C. (2001). Tracking and modeling non-rigid objects with rank constraints. In IEEE conference on computer vision and pattern recognition (Vol. 1, pp. 493–500) 2001.

  31. Torresani, L., Hertzmann, A., & Bregler, C. (2008). Nonrigid structure-from-motion: Estimating shape and motion with hierarchical priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(5), 878–892.

  32. Triggs, B. (1996). Factorization methods for projective structure and motion. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 845–851). San Francisco, California, USA, 1996.

  33. Vidal, R., & Abretske, D. (2006). Nonrigid shape and motion from multiple perspective views. In European conference on computer vision (2). Lecture notes in computer science : Vol. 3952 (pp. 205–218). Berlin: Springer.

  34. Vidal, R., Tron, R., & Hartley, R. (2008). Multiframe motion segmentation with missing data using powerfactorization and GPCA. International Journal of Computer Vision, 79(1), 85–105.

  35. Wang, G. (2006). A hybrid system for feature matching based on SIFT and epipolar constraints. (Tech. Rep.). Department of Electrical and Computer Engineering, University of Windsor.

  36. Wang, G., Tsui, H.-T., & Wu, J. (2008). Rotation constrained power factorization for structure from motion of nonrigid objects. Pattern Recognition Letters, 29(1), 72–80.

  37. Wang, G., & Wu, Q. J. (2008a). Stratification approach for 3D euclidean reconstruction of nonrigid objects from uncalibrated image sequences. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 38(1), 90–101.

  38. Wang, G., & Wu, J. (2008b). Quasi-perspective projection with applications to 3D factorization from uncalibrated image sequences. In IEEE conference on computer vision and pattern recognition, 2008.

  39. Wang, G., Wu, J., & Zhang, W. (2008). Camera self-calibration and three dimensional reconstruction under quasi-perspective projection. In Proceedings Canadian conference on computer and robot vision (pp. 129–136) 2008.

  40. Xiao, J., & Kanade, T. (2005). Uncalibrated perspective reconstruction of deformable structures. In Proceedings of the international conference on computer vision (Vol. 2, pp. 1075–1082) 2005.

  41. Xiao, J., Chai, J., & Kanade, T. (2006). A closed-form solution to non-rigid shape and motion recovery. International Journal of Computer Vision, 67(2), 233–246.

  42. Yan, J., & Pollefeys, M. (2005). A factorization-based approach to articulated motion recovery. In IEEE conference on computer vision and pattern recognition (2) (pp. 815–821) 2005.

  43. Yan, J., & Pollefeys, M. (2008). A factorization-based approach for articulated nonrigid shape, motion and kinematic chain recovery from video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(5), 865–877.

Download references

Author information

Correspondence to Guanghui Wang.

Additional information

The work is supported in part by Natural Sciences and Engineering Research Council of Canada, and the National Natural Science Foundation of China under Grant No. 60575015.

Electronic Supplementary Material

Below is the link to the electronic supplementary material. (AVI 4.79 MB)

Below is the link to the electronic supplementary material. (AVI 4.42 MB)

Below is the link to the electronic supplementary material. (AVI 4.73 MB)

Below is the link to the electronic supplementary material. (AVI 4.70 MB)

Below is the link to the electronic supplementary material. (AVI 4.79 MB)

Below is the link to the electronic supplementary material. (AVI 4.42 MB)

Below is the link to the electronic supplementary material. (AVI 4.73 MB)

Below is the link to the electronic supplementary material. (AVI 4.70 MB)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Wang, G., Wu, Q.M.J. Quasi-perspective Projection Model: Theory and Application to Structure and Motion Factorization from Uncalibrated Image Sequences. Int J Comput Vis 87, 213–234 (2010) doi:10.1007/s11263-009-0267-4

Download citation

Keywords

  • Structure from motion
  • Computational models of vision
  • Quasi-perspective projection
  • Imaging geometry
  • Matrix factorization
  • Singular value decomposition
  • Euclidean reconstruction