International Journal of Computer Vision

, Volume 115, Issue 2, pp 115–135

3D Trajectory Reconstruction under Perspective Projection

  • Hyun Soo Park
  • Takaaki Shiratori
  • Iain Matthews
  • Yaser Sheikh
Article

Abstract

We present an algorithm to reconstruct the 3D trajectory of a moving point from its correspondence in a collection of temporally non-coincidental 2D perspective images, given the time of capture that produced each image and the relative camera poses at each time instant. Triangulation-based solutions do not apply, as multiple views of the point may not exist at each time instant. We represent a 3D trajectory using a linear combination of compact trajectory basis vectors, such as the discrete cosine transform basis, that have been shown to approximate object independence. We note that such basis vectors are also coordinate independent, which allows us to directly use camera poses estimated from stationary areas in the scene (in contrast to nonrigid structure from motion techniques where cameras are simultaneously estimated). This reduces the reconstruction optimization to a linear least squares problem, allowing us to robustly handle missing data that often occur due to motion blur, texture deformation, and self occlusion. We present an algorithm to determine the number of trajectory basis vectors, individually for each trajectory via a cross validation scheme and refine the solution by minimizing the geometric error. The relationship between point and camera motion can cause degeneracies to occur. We geometrically analyze the problem by studying the relationship of the camera motion, point motion, and trajectory basis vectors. We define the reconstructability of a 3D trajectory under projection, and show that the estimate approaches the ground truth when reconstructability approaches infinity. This analysis enables us to precisely characterize cases when accurate reconstruction is achievable. We present qualitative results for the reconstruction of several real-world scenes from a series of 2D projections where high reconstructability can be guaranteed, and report quantitative results on motion capture sequences.

Keywords

Dynamic 3D reconstruction Trajectory triangulation  Trajectory space Reconstructability 

References

  1. Akhter, I., Sheikh, Y., & Khan, S. (2009). In defense of orthonormality constraints for nonrigid structure from motion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  2. Akhter, I., Sheikh, Y., Khan, S., & Kanade, T. (2008). Nonrigid structure from motion in trajectory space. In Advances in Neural Information Processing Systems.Google Scholar
  3. Akhter, I., Sheikh, Y., Khan, S., & Kanade, T. (2011). Trajectory space: A dual representation for nonrigid structure from motion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(7), 1442–1456.CrossRefGoogle Scholar
  4. Avidan, S., & Shashua, A. (2000). Trajectory triangulation: 3D reconstruction of moving points from a monocular image sequence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22, 348–357.CrossRefGoogle Scholar
  5. Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S. I., & Sayd, P. (2008). Coarse-to-fine low-rank structure-from-motion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  6. Blanz, V., & Vetter, T. (1999). A morphable model for the synthesis of 3D faces. In ACM transactions on Graphics (SIGGRAPH).Google Scholar
  7. Brand, M. (2001). Morphable 3D models from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  8. Brand, M. (2005). A direct method for 3D factorization of nonrigid motion observed in 2D. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  9. Bregler, C., Hertzmann, A., & Biermann, H. (1999). Recovering non-rigid 3D shape from image streams. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  10. Dai, Y., Li, H., & He, M. (2012). A simple prior-free method for non-rigid structure-from-motion factorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  11. Del Bue, A. (2008). A factorization approach to structure from motion with shape priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  12. Del Bue, A., Llad, X., & Agapito, L. (2006). Non-rigid metric shape and motion recovery from uncalibrated images using priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  13. Faugeras, O., Luong, Q.-T., & Papadopoulou, T. (2001). The geometry of multiple images: The laws that govern the formation of images of a scene and some of their applications. Cambridge: MIT Press.Google Scholar
  14. Fayad, J., Agapito, L., & Del Bue, A. (2010). Piecewise quadratic reconstruction of non-rigid surface from monocular sequences. In Proceedings of the European Conference on Computer Vision.Google Scholar
  15. Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381–395.MathSciNetCrossRefGoogle Scholar
  16. Gotardo, P. F. U., & Martinez, A. M. (2011). Computing smooth time-trajectories for camera and deformable shape in structure from motion with occlusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(10), 2051–2065.CrossRefGoogle Scholar
  17. Hamidi, M., & Pearl, J. (1976). Comparison of the cosine and Fourier transforms of Markov-I signal. IEEE Transactions on Acoustics, Speech, and Signal Processing, 24, 428–429.MathSciNetCrossRefGoogle Scholar
  18. Hartley, R., & Zisserman, A. (2004). Multiple view geometry in computer vision (2nd ed.). Cambridge: Cambridge University Press.MATHCrossRefGoogle Scholar
  19. Hartley, R. (1997). In defense of the eight-point algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19, 580–593.CrossRefGoogle Scholar
  20. Hartley, R., & Vidal, R. (2008). Perspective nonrigid shape and motion recovery. In Proceedings of the European Conference on Computer Vision.Google Scholar
  21. Kaminski, J. Y., & Teicher, M. (2004). A general framework for trajectory triangulation. Journal of Mathematical Imaging and Vision, 21(1), 27–41.MathSciNetCrossRefGoogle Scholar
  22. Lladó, X., Del Bue, A., & Agapito, L. (2010). Non-rigid metric reconstruction from perspective cameras. Image and Vision Computing, 28(9), 1339–1353.CrossRefGoogle Scholar
  23. Longuet-Higgins, H. C. (1981). A computer algorithm for reconstructing a scene from two projections. Nature, 293, 133–135.CrossRefGoogle Scholar
  24. Lourakis, M. I. A., & Argyros, A. A. (2009). SBA: A software package for generic sparse bundle adjustment. ACM Transactions on Mathematical Software, 36(1), 1–30.MathSciNetCrossRefGoogle Scholar
  25. Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.CrossRefGoogle Scholar
  26. Ma, Y., Soatto, S., Kosecka, J., & Sastry, S. S. (2003). An invitation to 3-D vision: From images to geometric models. New York: Springer.Google Scholar
  27. Moreno-Noguer, F., Lepetit, V., & Fua, P. (2007). EPnP: Efficient perspective-n-point camera pose estimation. In Proceedings of the International Conference on Computer Vision.Google Scholar
  28. Olsen, S., & Bartoli, A. (2007). Using priors for improving generalization in non-rigid structure-from-motion. In Proceedings of British Machine Vision Conference.Google Scholar
  29. Östlund, J., Varol, A., Ngo, D. T., & Fua, P. (2012). Laplacian meshes for monocular 3D shape recovery. In Proceedings of the European Conference on Computer Vision.Google Scholar
  30. Ozden, K. E., Cornelis, K., Eychen, L. V., & Gool, L. V. (2004). Reconstructing 3D trajectories of independently moving objects using generic constraints. Computer Vision and Image Understanding, 93, 1453–1471.Google Scholar
  31. Paladini, M., Del Bue, A., Stosic, M., Dodig, M., Xavier, J., & Agapito, L. (2009). Factorization for non-rigid and articulated structure using metric projections. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  32. Park, H. S., Shiratori, T., Matthews, I., & Sheikh, Y. (2010). 3D reconstruction of a moving point from a series of 2D projections. In Proceedings of the European Conference on Computer Vision.Google Scholar
  33. Salzmann, M., Pilet, J., Ilic, S., & Fua, P. (2007). Surface deformation models for nonrigid 3D shape recovery. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(7), 1481–1487.CrossRefGoogle Scholar
  34. Shashua, A., & Wolf, L. (2000). Homography tensors: On algebraic entities that represent three views of static or moving planar points. In Proceedings of the European Conference on Computer Vision.Google Scholar
  35. Sidenbladh, H., Black, M. J., & Fleet, D. J. (2000). Stochastic tracking of 3d human figures using 2D image motion. In Proceedings of the European Conference on Computer Vision.Google Scholar
  36. Snavely, N., Seitz, S. M., & Szeliski, R. (2006). Photo tourism: Exploring photo collections in 3D. ACM Transactions on Graphics (SIGGRAPH).Google Scholar
  37. Taylor, J., Jepson, A. D., & Kutulakos, K. N. (2010). Non-rigid structure from locally-rigid motion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  38. Tomasi, C., & Kanade, T. (1992). Shape and motion from image streams under orthography: A factorization method. International Journal of Computer Vision, 9(2), 137–154.CrossRefGoogle Scholar
  39. Torresani, L., Yang, D., Alexander, G., & Bregler, C. (2001). Tracking and modeling non-rigid objects with rank constraints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  40. Torresani, L., & Bregler, C. (2002). Space-time tracking. In Proceedings of the European Conference on Computer Vision.Google Scholar
  41. Torresani, L., Hertzmann, A., & Bregler, C. (2008). Nonrigid structure-from-motion: Estimating shape and motion with hierarchical priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30, 878–892.Google Scholar
  42. Torresani, L., Hertzmann, A., & Bregler, C. (2003). Learning non-rigid 3D shape from 2D motion. In Advances in Neural Information Processing Systems.Google Scholar
  43. Valmadre, J., & Lucey, S. (2012). General trajectory prior for non-rigid reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  44. Vidal, R., & Abretske, D. (2006). Nonrigid shape and motion from multiple perspective views. In Proceedings of the European Conference on Computer Vision.Google Scholar
  45. Vidal, R., & Hartley, R. (2004). Motion segmentation with missing data by powerfactorization and generalized pca. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  46. Wexler, Y., & Shashua, A. (2000). On the synthesis of dynamic scenes from reference views. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  47. Wolf, L., & Shashua, A. (2002). On projection matrices \({\cal P}^{k} \rightarrow {\cal P}^{2}, k =3, \ldots \), 6, and their applications in computer vision. International Journal of Computer Vision, 48(1), 53–67.MATHCrossRefGoogle Scholar
  48. Xiao, J., & Kanade, T. (2004). Non-rigid shape and motion recovery: Degenerate deformations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  49. Xiao, J., Chai, J., & Kanade, T. (2006). A closed-form solution to non-rigid shape and motion recovery. International Journal of Computer Vision, 67(2), 233–246.CrossRefGoogle Scholar
  50. Yan, J., & Pollefeys, M. (2005). A factorization-based approach to articulated motion recovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  51. Zhu, S., Zhang, L., & Smith, B. M. (2010). Model evolution: An incremental approach to non-rigid structure from motion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Hyun Soo Park
    • 1
  • Takaaki Shiratori
    • 2
  • Iain Matthews
    • 3
  • Yaser Sheikh
    • 1
  1. 1.Carnegie Mellon UniversityPittsburghUSA
  2. 2.Microsoft Research AsiaBeijingChina
  3. 3.Disney Research PittsburghPittsburghUSA

Personalised recommendations