Template-Free 3D Reconstruction of Poorly-Textured Nonrigid Surfaces

  • Xuan WangEmail author
  • Mathieu Salzmann
  • Fei Wang
  • Jizhong Zhao
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9911)


Two main classes of approaches have been studied to perform monocular nonrigid 3D reconstruction: Template-based methods and Non-rigid Structure from Motion techniques. While the first ones have been applied to reconstruct poorly-textured surfaces, they assume the availability of a 3D shape model prior to reconstruction. By contrast, the second ones do not require such a shape template, but, instead, rely on points being tracked throughout a video sequence, and are thus ill-suited to handle poorly-textured surfaces. In this paper, we introduce a template-free approach to reconstructing a poorly-textured, deformable surface. To this end, we leverage surface isometry and formulate 3D reconstruction as the joint problem of non-rigid image registration and depth estimation. Our experiments demonstrate that our approach yields much more accurate 3D reconstructions than state-of-the-art techniques.


Non-rigid 3D reconstruction Poorly-textured surfaces Template-free shape estimation 



This work was supported in part by Natural Science Foundation of China (No. 61231018, No. 61273366), National Science and Technology Support Program (2015BAH31F01) and Program of Introducing Talents of Discipline to University under grant B13043. Part of this work was performed while X. Wang and M. Salzmann were respectively visiting and affiliated with NICTA, Canberra.

Supplementary material (6 mb)
Supplementary material 1 (zip 6099 KB)


  1. 1.
    Salzmann, M., Urtasun, R.: Beyond feature points: structured prediction for monocular non-rigid 3D reconstruction. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7575, pp. 245–259. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33765-9_18 Google Scholar
  2. 2.
    Ngo, T.D., Park, S., Jorstad, A.A., Crivellaro, A., Yoo, C., Fua, P.: Dense image registration and deformable surface reconstruction in presence of occlusions and minimal texture. In: ICCV (2015)Google Scholar
  3. 3.
    Chhatkuli, A., Pizarro, D., Bartoli, A.: Stable template-based isometric 3D reconstruction in all imaging conditions by linear least-squares. In: CVPR (2014)Google Scholar
  4. 4.
    Perriollat, M., Hartley, R., Bartoli, A.: Monocular template-based reconstruction of inextensible surfaces. Int. J. Comput. Vis. (IJCV) 95(2), 124–127 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Salzmann, M., Urtasun, R., Fua, P.: Local deformation models for monocular 3D shape recovery. In: CVPR (2008)Google Scholar
  6. 6.
    Bartoli, A., Gerard, Y., Chadebecq, F., Collins, T.: On template-based reconstruction from a single view: analytical solutions and proofs of well-posedness for developable, isometric and conformal surfaces. In: CVPR (2012)Google Scholar
  7. 7.
    Bartoli, A., Collins, T.: Template-based isometric deformable 3D reconstruction with sampling-based focal length self-calibration. In: CVPR (2013)Google Scholar
  8. 8.
    Bartoli, A., Pizarro, D., Collins, T.: A robust analytical solution to isometric shape-from-template with focal length calibration. In: ICCV (2013)Google Scholar
  9. 9.
    Salzmann, M., Fua, P.: Reconstructing sharply folding surfaces: a convex formulation. In: CVPR (2009)Google Scholar
  10. 10.
    Salzmann, M., Urtasun, R.: Combining discriminative and generative methods for 3D deformable surface and articulated pose reconstruction. In: CVPR (2010)Google Scholar
  11. 11.
    Yu, R., Russell, C., Campbell, N., Agapito, L.: Direct, dense, and deformable: template-based non-rigid 3D reconstruction from RGB video. In: ICCV (2015)Google Scholar
  12. 12.
    Malti, A., Bartoli, A., Hartley, R.: A linear least-squares solution to elastic shape-from-template. In: CVPR (2015)Google Scholar
  13. 13.
    Ngo, T.D., Östlund, J., Fua, P.: Template-based monocular 3D shape recovery using laplacian meshes. IEEE Trans. Pattern Anal. Mach. Intell. 38(1), 172–187 (2016)CrossRefGoogle Scholar
  14. 14.
    Russell, C., Yu, R., Agapito, L.: Video-popup: Monocular 3D reconstruction of dynamic scenes. In: ECCV (2014)Google Scholar
  15. 15.
    Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3D shape from image streams. In: CVPR (2000)Google Scholar
  16. 16.
    Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Nonrigid structure from motion in trajectory space. In: NIPS (2008)Google Scholar
  17. 17.
    Taylor, J., Jepson, A., Kutulakos, K.: Non-rigid structure from locally-rigid motion. In: CVPR (2010)Google Scholar
  18. 18.
    Dai, Y., Li, H., He, M.: A simple prior-free method for nonrigid structure from motion factorization. In: CVPR (2012)Google Scholar
  19. 19.
    Vicente, S., Agapito, L.: Soft inextensibility constraints for template-free non-rigid reconstruction. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 426–440. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33712-3_31 Google Scholar
  20. 20.
    Chhatkuliand, A., Pizarro, D., Bartoli, A.: Non-rigid shape-from-motion for isometric surfaces using infinitesimal planarity. In: BMVC (2014)Google Scholar
  21. 21.
    Garg, R., Roussos, A., Agapito, L.: Dense variational reconstruction of non-rigid surfaces from monocular video. In: CVPR (2013)Google Scholar
  22. 22.
    Kass, M., Witkin, A., Terzopoulos, D.: Snakes: active contour models. Int. J. Comput. Vis. (IJCV) 1(4), 321–331 (1988)CrossRefzbMATHGoogle Scholar
  23. 23.
    Fua, P., Leclerc, Y.G.: Object-centered surface reconstruction: combining multi-image stereo and shading. Int. J. Comput. Vis. (IJCV) 16(1), 35–56 (1995)CrossRefGoogle Scholar
  24. 24.
    Greminger, M., Nelson, B.: Deformable object tracking using the boundary element method. In: CVPR (2003)Google Scholar
  25. 25.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1407, pp. 484–498. Springer, Heidelberg (1998). doi: 10.1007/BFb0054760 Google Scholar
  26. 26.
    Matthews, I., Baker, S.: Active appearance models revisited. Int. J. Comput. Vis. (IJCV) 60(2), 135–164 (2004)CrossRefGoogle Scholar
  27. 27.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: SIGGRAPH (1999)Google Scholar
  28. 28.
    Xiao, J., Baker, S., Matthews, I., Kanade, T.: Real-time combined 2D+3D active appearance models. In: CVPR (2004)Google Scholar
  29. 29.
    Blanz, V., Basso, C., Poggio, T., Vetter, T.: Reanimating faces in images and video. In: Eurographics (2003)Google Scholar
  30. 30.
    Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. Int. J. Comput. Vis. (IJCV) 9(2), 137–154 (1992)CrossRefGoogle Scholar
  31. 31.
    Garg, R., Roussos, A., Agapito, L.: A variational approach to video registration with subspace constraints. Int. J. Comput. Vis. (IJCV) 104(3), 286–314 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Woodford, O., Torr, P., Reid, I., Fitzgibbon, A.: Global stereo reconstruction under second order smoothness priorsGoogle Scholar
  33. 33.
    Lempitsky, V., Rother, C., Roth, S., Blake, A.: Fusion moves for markov random field optimization. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 32(8), 1392–1405 (2010)CrossRefGoogle Scholar
  34. 34.
    Ishikawa, H.: Higher-order clique reduction in binary graph cut. In: CVPR (2009)Google Scholar
  35. 35.
    Rother, C., Kolmogorov, V., Lempitsky, V., Szummer, M.: Optimizing binary MRFs via extended roof duality. In: CVPR (2007)Google Scholar
  36. 36.
    Ishikawa, H.: Higher-order gradient descent by fusion-move graph cut. In: ICCV (2009)Google Scholar
  37. 37.

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Xuan Wang
    • 1
    Email author
  • Mathieu Salzmann
    • 2
  • Fei Wang
    • 1
  • Jizhong Zhao
    • 1
  1. 1.Xi’an Jiaotong UniversityXi’anChina
  2. 2.CVLab, EPFLZurichSwitzerland

Personalised recommendations