Skip to main content
SpringerLink
Log in

Non-Rigid Structure-from-Motion and Shading

  • Chapter
  • First Online:
Advances in Photometric 3D-Reconstruction

Part of the book series: Advances in Computer Vision and Pattern Recognition ((ACVPR))

  • 842 Accesses

Abstract

We show how photometric and motion-based approaches can be combined to reconstruct the 3D shape of deformable objects from monocular images. We start by motivating the problem using real-world applications. We give a comprehensive overview of the state-of-the-art approaches and discuss their limitations for practical use in these applications. We then introduce the problem of Non-Rigid Structure-from-Motion and Shading (NRSfMS), where photometric and geometric information is used for reconstruction, without prior knowledge about the shape of the deformable object. We present in detail the first technical solution to NRSfMS and close the chapter with the main remaining open problems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The code is available at http://igt.ip.uca.fr/~ab/code_and_datasets/index.php (Matlab SfT Toolbox).

  2. 2.

    The datasets are available at http://igt.ip.uca.fr/~ab/code_and_datasets/index.php.

References

  1. 3Dflow (2017) 3DF Zephyr. https://www.3dflow.net

  2. Agisoft (2014) PhotoScan version 1.2.3 build 2331. http://www.agisoft.com

  3. Ahmed AH, Farag AA (2007) Shape from shading under various imaging conditions. In: CVPR

    Google Scholar 

  4. Akhter I, Sheikh Y, Khan S, Kanade T (2009) Nonrigid structure from motion in trajectory space. In: NIPS

    Google Scholar 

  5. Barron JT, Malik J (2015) Shape, illumination, and reflectance from shading. IEEE Trans Pattern Anal Mach Intell 37(8):1670–1687

    Google Scholar 

  6. Bartoli A, Gérard Y, Chadebecq F, Collins T, Pizarro D (2015) Shape-from-template. IEEE Trans Pattern Anal Mach Intell 37(10):2099–2118

    Google Scholar 

  7. Bartoli A, Özgür E (2016) A perspective on non-isometric shape-from-template. In: ISMAR

    Google Scholar 

  8. Beeler T, Bickel B, Beardsley P, Sumner B, Gross M (2010) High-quality single-shot capture of facial geometry. In: SIGGRAPH

    Google Scholar 

  9. Belhumeur PN, Kriegman DJ, Yuille AL (1997) The bas-relief ambiguity. In: CVPR

    Google Scholar 

  10. Bell S, Bala K, Snavely N (2014) Intrinsic images in the wild. ACM Trans Graph (SIGGRAPH) 33(4)

    Google Scholar 

  11. Bregler C, Hertzmann A, Biermann H (2000) Recovering non-rigid 3D shape from image streams. In: CVPR

    Google Scholar 

  12. Brostow GJ, Hernández C, Vogiatzis G, Stenger B, Cipolla R (2001) Video normals from colored lights. IEEE Trans Pattern Anal Mach Intell 33(10):2104–2114

    Google Scholar 

  13. Brunet F, Hartley R, Bartoli A (2014) Monocular template-based 3D surface reconstruction: convex inextensible and nonconvex isometric methods. Comput Vis Image Underst 125:138–154 August

    Google Scholar 

  14. Chhatkuli A, Pizarro D, Bartoli A (2014) Non-rigid shape-from-motion for isometric surfaces using infinitesimal planarity. In: BMVC

    Google Scholar 

  15. Chhatkuli A, Pizarro D, Bartoli A, Collins T (2017) A stable analytical framework for isometric shape-from-template by surface integration. IEEE Trans Pattern Anal Mach Intell 39(5):833–850

    Google Scholar 

  16. Chhatkuli A, Pizarro D, Collins T, Bartoli A (2016) Inextensible non-rigid shape-from-motion by second-order cone programming. In: CVPR

    Google Scholar 

  17. Collins T, Bartoli A (2015) Realtime shape-from-template: system and applications. In: ISMAR

    Google Scholar 

  18. Dai Y, Li H, He M (2014) A simple prior-free method for non-rigid structure-from-motion factorization. Int J Comput Vis 107(2):101–122

    MathSciNet  MATH  Google Scholar 

  19. David 3D Scanner (2014) http://www.david-3d.com/en/products/david4

  20. Durou J-D, Falcone M, Sagona M (2008) Numerical methods for shape-from-shading: a new survey with benchmarks. Comput Vis Image Underst 109(1):22–43 January

    Google Scholar 

  21. Ecker A, Jepson AD (2010) Polynomial shape from shading. In: CVPR

    Google Scholar 

  22. Falcone M, Sagona M, Seghini A (2003) A scheme for the shape-from-shading model with “black shadows”. Numerical mathematics and advanced applications, pp 503–512

    Google Scholar 

  23. Fayad J, Agapito L, Del Bue A (2010) Piecewise quadratic reconstruction of non-rigid surfaces from monocular sequences. In: ECCV

    Google Scholar 

  24. Fukunaga K, Hostetler L (1975) The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Trans Inf Theory 21(1):32–40

    Google Scholar 

  25. Gallardo M (2018) Contributions to monocular deformable 3D reconstruction: curvilinear objects and multiple visual cues. Theses, Université Clermont Auvergne. https://tel.archives-ouvertes.fr/tel-01930477

  26. Gallardo M, Collins T, Bartoli A (2016) Can we jointly register and reconstruct creased surfaces by shape-from-template accurately? In: ECCV

    Google Scholar 

  27. Gallardo M, Collins T, Bartoli A (2016) Using shading and a 3D template to reconstruct complex surface deformations. In: BMVC

    Google Scholar 

  28. Gallardo M, Collins T, Bartoli A (2017) Dense non-rigid structure-from-motion and shading with unknown albedos. In: ICCV

    Google Scholar 

  29. Garg R, Roussos A, Agapito L (2013) Dense variational reconstruction of non-rigid surfaces from monocular video. In: CVPR

    Google Scholar 

  30. Gotardo PFU, Martinez AM (2011) Kernel non-rigid structure from motion. In: ICCV

    Google Scholar 

  31. Gotardo PFU, Simon T, Sheikh Y, Matthews I (2015) Photogeometric scene flow for high-detail dynamic 3D reconstruction. In: ICCV

    Google Scholar 

  32. Gumerov N, Zandifar A, Duraiswami R, Davis LS (2004) Structure of applicable surfaces from single views. In: ECCV

    Google Scholar 

  33. Haefner B, Quéau Y, Möllenhoff T, Cremers D (2018) Fight ill-posedness with ill-posedness: single-shot variational depth super-resolution from shading. In: CVPR

    Google Scholar 

  34. Haefner B, Ye Z, Gao M, Wu T, Quéau Y, Cremers D (2019) Variational uncalibrated photometric stereo under general lighting. In: ICCV

    Google Scholar 

  35. Haouchine N, Dequidt J, Berger MO, Cotin S (2014) Single view augmentation of 3D elastic objects. In: ISMAR

    Google Scholar 

  36. Hartley RI, Zisserman A (2003) Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, Cambridge

    Google Scholar 

  37. Horn BKP (1970) Shape from shading: a method for obtaining the shape of a smooth shape of a smooth opaque object from one view. Technical report, Cambridge, MA, USA

    Google Scholar 

  38. Horn BKP (1989) Shape from shading, pp 123–171

    Google Scholar 

  39. Ikeuchi K, Horn BKP (1981) Numerical shape from shading and occluding boundaries. Artif Intell 17(1):141–184

    Google Scholar 

  40. Innmann M, Zollhöfer M, Nießner M, Theobalt C, Stamminger M (2016) VolumeDeform: real-time volumetric non-rigid reconstruction. In: ECCV

    Google Scholar 

  41. Jin H, Cremers D, Wang D, Prados E, Yezzi A, Soatto S (2008) 3-D reconstruction of shaded objects from multiple images under unknown illumination. Int J Comput Vis 76(3):245–256

    Google Scholar 

  42. Kim K, Torii A, Okutomi M (2016) Multi-view inverse rendering under arbitrary illumination and albedo. In: ECCV

    Google Scholar 

  43. Kimmel R, Bruckstein AM (1994) Global shape-from-shading. In: ICPR

    Google Scholar 

  44. Koo B, Özgür E, Le Roy B, Buc E, Bartoli A (2017) Deformable registration of a preoperative 3D liver volume to a laparoscopy image using contour and shading cues. In: MICCAI

    Google Scholar 

  45. Langguth F, Sunkavalli K, Hadap S, Goesele M (2016) Shading-aware Multi-view Stereo. In: ECCV

    Google Scholar 

  46. Lee KM, Kuo C-C (1997) Shape from shading with a generalized reflectance map model. Comput Vis Image Underst 67(2):143–160

    Google Scholar 

  47. Lee KM, Kuo CCJ (1993) Shape from shading with a linear triangular element surface model. IEEE Trans Pattern Anal Mach Intell 15(8):815–822

    Google Scholar 

  48. Liu-Yin Q, Yu R, Agapito L, Fitzgibbon A, Russell C (2016) Better together: joint reasoning for non-rigid 3D reconstruction with specularities and shading. In: BMVC

    Google Scholar 

  49. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Google Scholar 

  50. Magnenat S, Ngo DT, Zund F, Ryffel M, Noris G, Rothlin G, Marra A, Nitti M, Fua P, Gross M, Sumner R (2015) Live texturing of augmented reality characters from colored drawings. IEEE Trans Vis Comput Graph 21(11):1201–1210

    Google Scholar 

  51. Malti A, Bartoli A (2014) Combining conformal deformation and cook-torrance shading for 3D reconstruction in laparoscopy. IEEE Trans Biol Eng 61(6):1684–1692

    Google Scholar 

  52. Malti A, Bartoli A, Collins T (2011) A pixel-based approach to template-based monocular 3D reconstruction of deformable surfaces. In: Proceedings of the IEEE international workshop on dynamic shape capture and analysis at ICCV

    Google Scholar 

  53. Malti A, Bartoli A, Hartley RI (2015) A linear least-squares solution to elastic shape-from-template. In: CVPR

    Google Scholar 

  54. Moreno-Noguer F, Salzmann M, Lepetit V, Fua P (2009) Capturing 3D stretchable surfaces from single images in closed form. In: CVPR

    Google Scholar 

  55. Newcombe RA, Fox D, Seitz SM (2015) DynamicFusion: reconstruction and tracking of non-rigid scenes in real-time. In: CVPR

    Google Scholar 

  56. Ngo TD, Östlund J, Fua P (2016) Template-based monocular 3D shape recovery using laplacian meshes. IEEE Trans Pattern Anal Mach Intell 38(1):172–187

    Google Scholar 

  57. Ngo TD, Park S, Jorstad AA, Crivellaro A, Yoo C, Fua P (2015) Dense image registration and deformable surface reconstruction in presence of occlusions and minimal texture. In: ICCV

    Google Scholar 

  58. Okatani T, Deguchi K (1996) Shape reconstruction from an endoscope image by SfS technique for a point light source at the projection Center. Comput Vis Image Underst 66(2):119–131 July

    Google Scholar 

  59. Özgür E, Bartoli A (2017) Particle-SfT: a provably-convergent, fast shape-from-template algorithm. Int J Comput Vis 123(2):184–205 June

    Google Scholar 

  60. Parashar S, Pizarro D, Bartoli A (2016) Isometric non-rigid shape-from-motion in linear time. In: CVPR

    Google Scholar 

  61. Pentland AP (1984) Local shading analysis. IEEE Trans Pattern Anal Mach Intell 6(2):170–187

    Google Scholar 

  62. Pentland AP (1988) Shape information from shading: a theory about human perception. In: ICCV

    Google Scholar 

  63. Perriollat M, Hartley R, Bartoli A (2011) Monocular template-based reconstruction of inextensible surfaces. Int J Comput Vis 95(2):124–137 November

    MathSciNet  MATH  Google Scholar 

  64. Prados E, Faugeras O (2005) Shape from shading: a well-posed problem? In: CVPR

    Google Scholar 

  65. Pumarola A, Agudo A, Porzi L, Sanfeliu A, Lepetit V, Moreno-Noguer F (2018) Geometry-aware network for non-rigid shape prediction from a single view. In: CVPR

    Google Scholar 

  66. Quéau Y, Mélou J, Castan F, Cremers D, Durou J (2017) A variational approach to shape-from-shading under natural illumination. In: EMMCVPR

    Google Scholar 

  67. Richardson E, Sela M, Kimmel R (2016) 3D face reconstruction by learning from synthetic data. In: 3DV

    Google Scholar 

  68. Richter SR, Roth S (2015) Discriminative shape from shading in uncalibrated illumination. In: CVPR

    Google Scholar 

  69. Rouy E, Tourin A (1992) A viscosity solutions approach to shape-from-shading. SIAM J Numer Anal 29(3):867–884 June

    MathSciNet  MATH  Google Scholar 

  70. Salzmann M, Fua P (2009) Reconstructing sharply folding surfaces: a convex formulation. In: CVPR

    Google Scholar 

  71. Salzmann M, Fua P (2011) Linear local models for monocular reconstruction of deformable surfaces. IEEE Trans Pattern Anal Mach Intell 33(5):931–944

    Google Scholar 

  72. Salzmann M, Pilet J, Ilic S, Fua P (2007) Surface deformation models for nonrigid 3D shape recovery. IEEE Trans Pattern Anal Mach Intell 29(8):1481–1487

    Google Scholar 

  73. Sorkine O, Alexa M (2007) As-rigid-as-possible surface modeling. In: Symposium on geometry processing, pp 109–116

    Google Scholar 

  74. Sundaram N, Brox T, Keutzer K (2010) Dense point trajectories by GPU-accelerate large displacement optical flow. In: ECCV

    Google Scholar 

  75. Tankus A, Sochen N, Yeshurun Y (2005) Shape-from-shading under perspective projection. Int J Comput Vis 63(1):21–43 June

    Google Scholar 

  76. Taylor J, Jepson AD, Kutulakos K (2010) Non-rigid structure from locally-rigid motion. In: CVPR

    Google Scholar 

  77. Thies J, Zollhöfer M, Stamminger M, Theobalt C, Nießner M (2016) Face2Face: real-time face capture and reenactment of RGB videos. In: CVPR

    Google Scholar 

  78. Tsai P-S, Shah M (1994) Shape from shading using linear approximation. Image Vis Comput 12(8):487–498

    Google Scholar 

  79. Valgaerts L, Wu C, Bruhn A, Seidel H-P, Theobalt C (2012) Lightweight binocular facial performance capture under uncontrolled lighting. In: SIGGRAPH

    Google Scholar 

  80. Varol A, Salzmann M, Fua P, Urtasun R (2012) A constrained latent variable model. In: CVPR

    Google Scholar 

  81. Varol A, Salzmann M, Tola E, Fua P (2009) Template-free monocular reconstruction of deformable surfaces. In: ICCV

    Google Scholar 

  82. Varol A, Shaji A, Salzmann M, Fua P (2012) Monocular 3D reconstruction of locally textured surfaces. IEEE Trans Pattern Anal Mach Intell 34(6)

    Google Scholar 

  83. Vicente S, Agapito L (2012) Soft inextensibility constraints for template-free non-rigid reconstruction. In: ECCV

    Google Scholar 

  84. Vicente S, Agapito L (2013) Balloon shapes: reconstructing and deforming objects with volume from images. In: 3DV

    Google Scholar 

  85. Wang X, Salzmann M, Wang F, Zhao J (2016) Template-free 3D reconstruction of poorly-textured nonrigid surfaces. In: ECCV

    Google Scholar 

  86. Wu C, Narasimhan SG, Jaramaz B (2010) A multi-image shape-from-shading framework for near-lighting perspective endoscopes. Int J Comput Vis 86(2):211–228

    MathSciNet  Google Scholar 

  87. Wu C, Varanasi K, Liu Y, Seidel HP, Theobalt C (2011) Shading-based dynamic shape refinement from multi-view video under general illumination. In: ICCV

    Google Scholar 

  88. Xiong Y, Chakrabarti A, Basri R, Gortler SJ, Jacobs DW, Zickler T (2015) From shading to local shape. IEEE Trans Pattern Anal Mach Intell 37(1)

    Google Scholar 

  89. Yu R, Russell C, Campbell NDF, Agapito L (2015) Direct, dense, and deformable: template-based non-rigid 3D reconstruction from RGB video. In: ICCV

    Google Scholar 

  90. Zhang Z (1997) Parameter estimation techniques: a tutorial with application to conic fitting. Image Vis Comput 15(1):59–76

    Google Scholar 

  91. Zhou Z, Wu Z, Tan P (2013) Multi-view photometric stereo with spatially varying isotropic materials. In: CVPR

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. EnCoV, Institut Pascal, UMR 6602, CNRS/UBP/SIGMA, 63000, Clermont-Ferrand, France

    Mathias Gallardo

  2. IRCAD and IHU-Strasbourg, 1 Place de l’Hôpital, 67000, Strasbourg, France

    Toby Collins

  3. EnCoV, Institut Pascal, UMR 6602, CNRS/UBP/SIGMA, 63000, Clermont-Ferrand, France

    Adrien Bartoli

Authors
  1. Mathias Gallardo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Toby Collins
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adrien Bartoli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mathias Gallardo .

Editor information

Editors and Affiliations

  1. IRIT, UMR CNRS 5505, University of Toulouse, Toulouse, France

    Jean-Denis Durou

  2. Department of Mathematics, Sapienza University of Rome, Rome, Italy

    Maurizio Falcone

  3. GREYC Laboratory, CNRS, Caen, France

    Yvain Quéau

  4. Department of Mathematics and Applications “Renato Caccioppoli”, University of Naples Federico II, Naples, Italy

    Silvia Tozza

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gallardo, M., Collins, T., Bartoli, A. (2020). Non-Rigid Structure-from-Motion and Shading. In: Durou, JD., Falcone, M., Quéau, Y., Tozza, S. (eds) Advances in Photometric 3D-Reconstruction. Advances in Computer Vision and Pattern Recognition. Springer, Cham. https://doi.org/10.1007/978-3-030-51866-0_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-51866-0_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-51865-3

  • Online ISBN: 978-3-030-51866-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation