Skip to main content
SpringerLink
Log in

Template-Based 3D Reconstruction of Non-rigid Deformable Object from Monocular Video

  • 3DR Express
  • Published:
3D Research

Abstract

In this paper, we propose a template-based 3D surface reconstruction system of non-rigid deformable objects from monocular video sequence. Firstly, we generate a semi-dense template of the target object with structure from motion method using a subsequence video. This video can be captured by rigid moving camera orienting the static target object or by a static camera observing the rigid moving target object. Then, with the reference template mesh as input and based on the framework of classical template-based methods, we solve an energy minimization problem to get the correspondence between the template and every frame to get the time-varying mesh to present the deformation of objects. The energy terms combine photometric cost, temporal and spatial smoothness cost as well as as-rigid-as-possible cost which can enable elastic deformation. In this paper, an easy and controllable solution to generate the semi-dense template for complex objects is presented. Besides, we use an effective iterative Schur based linear solver for the energy minimization problem. The experimental evaluation presents qualitative deformation objects reconstruction results with real sequences. Compare against the results with other templates as input, the reconstructions based on our template have more accurate and detailed results for certain regions. The experimental results show that the linear solver we used performs better efficiency compared to traditional conjugate gradient based solver.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Whelan, T., Leutenegger, S., Salas-Moreno, R. F., Glocker, B., & Davison, A. J. (2015). ElasticFusion: Dense SLAM without a pose graph. In Proceedings of robotics: Science and systems. Rome, Italy.

  2. Whelan, T., Kaess, M., Johannsson, H., Fallon, M., Leonard, J. J., & McDonald, J. (2015). Real-time large-scale dense RGB-D SLAM with volumetric fusion. The International Journal of Robotics Research, 34(4–5), 598–626.

    Article  Google Scholar 

  3. Whelan, T., Kaess, M., Fallon, M., Johannsson, H., Leonard, J., & McDonald, J. (2012). Kintinuous: Spatially extended kinectfusion. Robotics and Autonomous Systems, 69(1), 3–14.

    Google Scholar 

  4. Newcombe, R. A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., et al (2011). KinectFusion: Real-time dense surface mapping and tracking. In Proceedings of mixed and augmented reality (ISMAR), 10th IEEE international symposium. Basel, Switzerland.

  5. Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., et al. (2011). KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of 24th annual ACM symposium on user interface software and technology. CA, USA.

  6. Engel, J., Schöps, T., & Cremers, D. (2014). LSD-SLAM: Large-scale direct monocular SLAM. In Proceedings of European conference on computer vision. Zurich, Switzerland.

  7. Engel, J., Sturm, J., & Cremers, D. (2013). Semi-dense visual odometry for a monocular camera. In Proceedings of IEEE international conference on computer vision. Sydney, Australia.

  8. Newcombe, R. A., Lovegrove, S. J., & Davison, A. J. (2011). DTAM: dense tracking and mapping in real-time. In Proceedings of IEEE international conference on computer vision. Barcelona, Spain.

  9. Stühmer, J., Gumhold, S., & Cremers, D. (2010). Real-time dense geometry from a handheld camera. In Proceedings of Dagm conference on pattern recognition. Darmstadt, Germany.

  10. Keller, M., Lefloch, D., Lambers, M., & Izadi, S. (2013). Real-time 3D reconstruction in dynamic scenes using point-based fusion. In Proceedings of joint 3dim/3dpvt conference. Scotland, UK.

  11. Zollhöfer, M., Nießner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., et al. (2014). Real-time Non-rigid reconstruction using an RGB-D Camera. ACM Transactions on Graphics (TOG), 33(4), 156.

    Article  MATH  Google Scholar 

  12. Newcombe, R. A., Fox, D., & Seitz, S. M. (2015). DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of computer vision and pattern recognition. Boston, MA, USA.

  13. Innmann, M., Zollhöfer, M., Nießner, M., Theobalt, C., & Stamminger, M. (2016). Volumedeform: Real-time volumetric non-rigid reconstruction. In Proceedings of European conference on computer vision. Amsterdam, The Netherlands.

  14. Qi, L. Y., Fitzgibbon, A., & Agapito, L. (2016). Better Together: Joint reasoning for non-rigid 3D reconstruction with specularities and shading. In Proceedings of BMVC. York, UK.

  15. Garg, R., Roussos, A., & Agapito, L. (2013). Dense variational reconstruction of non-rigid surfaces from monocular video. In Proceedings of computer vision and pattern recognition. Portland, Oregon, USA.

  16. Yu, R., Russell, C., Campbell, N. D. F., & Agapito, L. (2015). Direct, dense, and deformable: Template-based non-rigid 3D reconstruction from rgb video. In Proceedings of IEEE international conference on computer vision. Santiago, Chile.

  17. Paladini, M., Bartoli, A., & Agapito, L. (2010). Sequential non-rigid structure-from-motion with the 3D-implicit low-rank shape model. In Proceedings of European conference on computer vision. Crete, Greece.

  18. Agudo, A., Agapito, L., Calvo, B., & Montiel, J. M. M. (2014). Good vibrations: A modal analysis approach for sequential non-rigid structure from motion. In Proceedings of computer vision and pattern recognition. Columbus, Ohio.

  19. Agudo, A., Montiel, J. M. M., Agapito, L., et al. (2014). Online dense non-rigid 3D shape and camera motion recovery. In Proceedings of British machine vision conference. Nottingham, UK.

  20. Agudo, A., Moreno-Noguer, F., Calvo, B., & Montiel, J. M. M. (2016). Real-time 3D reconstruction of non-rigid shapes with a single moving camera. Computer Vision and Image Understanding, 153, 37–54.

    Article  Google Scholar 

  21. Bronte, S., Paladini, M., Bergasa, L. M., Agapito, L., & Arroyo, R. (2014). Real-time sequential model-based non-rigid SFM proceedings of Ieee/rsj international conference on intelligent robots and systems. Chicago, IL, USA.

  22. Agudo, A., Calvo, B., & Montiel, J. M. M. (2012). 3D reconstruction of non-rigid surfaces in real-time using wedge elements (pp. 113–122). Berlin: Springer.

    Google Scholar 

  23. Malti, A., Hartley, R., Bartoli, A., & Kim, J. H. (2013). Monocular template-based 3D reconstruction of extensible surfaces with local linear elasticity. In Proceedings of computer vision and pattern recognition. Portland, Oregon, USA.

  24. Chadebecq, F. (2012). On template-based reconstruction from a single view: Analytical solutions and proofs of well posedness for developable, isometric and conformal surfaces. In Proceedings of computer vision and pattern recognition. Providence, RI, USA.

  25. Bregler, C., Hertzmann, A., & Biermann, H. (2000). Recovering non-rigid 3D shape from image streams. In Proceedings of computer vision and pattern recognition. Hilton Head, SC, USA.

  26. Dai, Y., Li, H., & He, M. (2014). A simple prior-free method for non-rigid structure-from-motion factorization. International Journal of Computer Vision, 107(2), 101–122.

    Article  MathSciNet  MATH  Google Scholar 

  27. Suwajanakorn, S., Kemelmacher-Shlizerman, I., & Seitz, S. M. (2014). Total moving face reconstruction proceedings of European conference on computer vision. Zurich, Switzerland.

  28. Garrido, P., Valgaert, L., Wu, C., & Theobalt, C. (2013). Reconstructing detailed dynamic face geometry from monocular video. ACM Transactions on Graphics, 32(6), 1–10.

    Article  Google Scholar 

  29. Pawar, A., Zhang, Y., Jia, Y., et al. (2016). Adaptive FEM-based nonrigid image registration using truncated hierarchical B-splines. Computers and Mathematics with Applications, 72(8), 2028–2040.

    Article  MathSciNet  MATH  Google Scholar 

  30. Pawar, A., Zhang, Y. J., Anitescu, C., et al. (2018). DTHB3D reg: Dynamic truncated hierarchical B-spline based 3D nonrigid image registration. Communications in Computational Physics, 23(3), 877–898.

    Google Scholar 

  31. Ling, C. C., Cosmin, A., Zhang, Y., et al. (2017). Two and three dimensional image registration based on B-spline composition and level sets. Communications in Computational Physics, 21(2), 600–622.

    Article  MathSciNet  Google Scholar 

  32. Jia, Y., Zhang, Y., & Rabczuk, T. (2015). A novel dynamic multilevel technique for image registration. Computers and Mathematics with Applications, 69(9), 909–925.

    Article  MathSciNet  Google Scholar 

  33. Lehmann, E., & Casella, G. (1998). Theory of Point Estimation. Technometrics, 41(3), 274.

    MATH  Google Scholar 

  34. Agarwal, S., & Mierle, K. (2012). Ceres solver. http://www.ceres-solver.org/.

  35. Wu, C. (2011). VisualSFM: a visual structure from motion system. http://ccwu.me/vsfm/.

  36. Goesele, M., Snavely, N., Curless, B., Hoppe, H., & Seitz, S. M. (2007). Multi-view Stereo for community photo collections. In Proceedings of IEEE international conference on computer vision. Rio de Janeiro, Brazil.

  37. Fuhrmann, S., Langguth, F., & Goesele, M. (2014). MVE-A Multi-view reconstruction environment. Proceedings of Eurographics workshop on graphics and cultural heritage (pp. 11–19).

  38. Fuhrmann, S., & Goesele, M. (2014). Floating scale surface reconstruction. ACM Transactions on Graphics, 33(4), 46.

    Article  MATH  Google Scholar 

  39. Cignoni, P., Corsini, M., & Ranzuglia, G. (2008). Meshlab: an open-source 3d mesh processing system. Ercim News, 73(45–46), 6.

    Google Scholar 

  40. Varol, A., Salzmann, M., Fua, P., & Urtasun, R. (2012). A constrained latent variable model. In Proceedings of IEEE conference on computer vision and pattern recognition. Providence, Rhode Island, USA.

  41. Oikonomidis, I., Kyriazis, N., & Argyros, A. (2011). Efficient model-based 3d tracking of hand articulations using Kinect. In Proceedings of the British machine vision conference.

  42. Buonamici, F., Carfagni, M., Furferi, R., et al. (2017). Reverse engineering of mechanical parts: A template-based approach. Journal of Computational Design and Engineering, 5(2), 145–159.

    Article  Google Scholar 

  43. Ferrara, P., Piva, A., Argenti, F., et al. (2017). Wide-angle and long-range real time pose estimation: a comparison between monocular and stereo vision systems. Journal of Visual Communication and Image Representation, 48, 159–168.

    Article  Google Scholar 

Download references

Acknowledgements

The financial support provided by China Scholarship Council during a visit of Yang Liu to German Aerospace Center (DLR) is acknowledged.

Author information

Authors and Affiliations

  1. National Space Science Center, University of Chinese Academy of Sciences, Beijing, 100190, China

    Yang Liu, Xiaodong Peng, Wugen Zhou & Bo Liu

  2. Simulation and Software Technology, German Aerospace Center (DLR), 38108, Brunswick, Germany

    Yang Liu & Andreas Gerndt

Authors
  1. Yang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaodong Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wugen Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bo Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andreas Gerndt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Peng, X., Zhou, W. et al. Template-Based 3D Reconstruction of Non-rigid Deformable Object from Monocular Video. 3D Res 9, 21 (2018). https://doi.org/10.1007/s13319-018-0174-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13319-018-0174-y

Keywords

Search

Navigation