A ToF-Aided Approach to 3D Mesh-Based Reconstruction of Isometric Surfaces

  • S. Jafar Hosseini
  • Helder Araujo
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9443)


In this paper, we investigate structure-from-motion (SfM) for surfaces that deform isometrically. Our SfM framework is intended for the estimation of both the 3D surface and the camera motion at one time through a template-based approach founded on the combination of a ToF sensor and a conventional RGB camera. The objective is to take advantage of depth maps acquired by the ToF sensor so that a considerable enhancement can be achieved in the reconstruction of the non-rigid structure using the high-resolution images captured by means of the RGB camera. A triangular mesh is adopted to represent isometric surfaces. The depth of a sparse set of 3D feature points spread all over the surface will be obtained with the help of the ToF camera, thereby enabling the recovery of the depth of the mesh vertices using a multivariate linear system. Subsequently, a non-linear constraint is formed based on the projected length of each edge of the mesh. A second non-linear constraint is then used for minimizing re-projection errors. These constraints are finally incorporated into an optimization scheme to solve for structure and motion. Experimental results show that the proposed approach has good performance even if only a low-resolution depth image is used.


Structure from motion Isometric surface ToF camera 3D reconstruction 


  1. 1.
    Aans, H., Kahl, F.: Estimation of deformable structure and motion. In: Workshop on Vision and Modelling of Dynamic Scenes, ECCV, Denmark (2002)Google Scholar
  2. 2.
    Del-Bue. A., Llad, X., Agapito, L.: Non-rigid metric shape and motion recovery from uncalibrated images using priors. In: IEEE Conference on Computer Vision and Pattern Recognition, New York (2006)Google Scholar
  3. 3.
    Salzmann, M., Hartley, R., Fua, P.: Convex optimization for deformable surface 3-D tracking. In: IEEE International Conference on Computer Vision (2007)Google Scholar
  4. 4.
    Salzmann, M., Fua, P.: Reconstructing sharply folding surfaces: a convex formulation. In: IEEE Conference on Computer Vision and Pattern Recognition (2007)Google Scholar
  5. 5.
    Salzmann, M., Urtasun R., Fua, P.: Local deformation models for monocular 3D shape recovery. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1054–1061 (2008)Google Scholar
  6. 6.
    Gumerov, N., Zandifar, A., Duraiswami, R., Davis, L.S.: Structure of applicable surfaces from single views. In: Pajdla, T., Matas, J.G. (eds.) ECCV 2004. LNCS, vol. 3023, pp. 482–496. Springer, Heidelberg (2004) CrossRefGoogle Scholar
  7. 7.
    Prasad, M., Zisserman, A., Fitzgibbon, A.W.: Single view reconstruction of curved surfaces. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1345–1354 (2006)Google Scholar
  8. 8.
    Shen, S., Shi, W., Liu, Y.: Monocular 3-D tracking of inextensible deformable surfaces under L2-norm. IEEE Trans. Image Process. 19, 512–521 (2010)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Perriollat, M., Hartley, R., Bartoli, A.: Monocular template-based reconstruction of inextensible surfaces. Int. J. Comput. Vis. 95(2), 124–137 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Salzmann, M., Moreno-Noguer, F., Lepetit, V., Fua, P.: Closed-form solution to non-rigid 3D surface registration. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part IV. LNCS, vol. 5305, pp. 581–594. Springer, Heidelberg (2008) CrossRefGoogle Scholar
  11. 11.
    Metaxas, D., Terzopoulos, D.: Constrained deformable superquadrics and nonrigid motion tracking. PAMI 15, 580–591 (1993)CrossRefGoogle Scholar
  12. 12.
    Torresani, L., Hertzmann A., Bregler C.: Learning non-rigid 3d shape from 2d motion. In: NIPS, pp. 580–591 (2003)Google Scholar
  13. 13.
    Llado, X., Bue A.D., Agapito, L.: Non-rigid 3D factorization for projective reconstruction. In: BMVC (2005)Google Scholar
  14. 14.
    White, R., Forsyth, D.: Combining cues: shape from shading and texture. In: CVPR (2006)Google Scholar
  15. 15.
    Diebel, J., Thrun, S.: An application of markov random fields to range sensing. In: Proceedings of NIPS (2005)Google Scholar
  16. 16.
    Yang, R., Davis, J., Nister, D.: Spatial-Depth Super Resolution for Range Images. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2007, Minneapolis, pp. 1–8 (2007)Google Scholar
  17. 17.
    Kim, H., Tai, Y.W., Brown, M.S.: High quality depth map upsampling for 3D-TOF cameras. In: IEEE International Conference on Inso Kweon Computer Vision (ICCV), Barcelona, pp. 1623–1630 (2011)Google Scholar
  18. 18.
    Brunet, F., Hartley, R., Bartoli, A., Navab, N., Malgouyres, R.: Monocular template-based reconstruction of smooth and inextensible surfaces. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part III. LNCS, vol. 6494, pp. 52–66. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  19. 19.
    Akhter, I., Sheikh, Y., Khan, S.: In defense of orthonormality constraints for nonrigid structure from motion. In: CVPR, pp. 1534–1541 (2009)Google Scholar
  20. 20.
    Xiao, J., Chai, J., Kanade, T.: A closed-form solution to non-rigid shape and motion recovery. In: Pajdla, T., Matas, J.G. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 573–587. Springer, Heidelberg (2004) CrossRefGoogle Scholar
  21. 21.
    Zhou, H., Li, X., Sadka, A.H.: Nonrigid structure-from-motion from 2-D images using markov chain monte carlo. MultMed 14(1), 168–177 (2012)Google Scholar
  22. 22.
    Srivastava, S., Saxena, A., Theobalt, C., Thrun, S.: Rapid interactive 3D reconstruction from a single image. In: VMV, pp. 9–28 (2009)Google Scholar
  23. 23.
    Paladini, M., Bue, A.D., Stosic, M., Dodig, M., Xavier, J., Agapito, L.: Factorization for non-rigid and articulated structure using metric projections. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2898–2905 (2009)Google Scholar
  24. 24.
    Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. In: CVPR, pp. 2018–2025 (2012)Google Scholar
  25. 25.
    Kim, Y.M., Theobalt, C., Diebel, J., Kosecka, J., Miscusik, B., Thrun, S.: Multi-view image and tof sensor fusion for dense 3D reconstruction. In: Computer Vision Workshops (ICCV Workshops), pp. 1542–1549 (2009)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Institute of Systems and Robotics, Department of Electrical and Computer EngineeringUniversity of CoimbraCoimbraPortugal

Personalised recommendations