Advertisement

Computational Visual Media

, Volume 4, Issue 4, pp 287–303 | Cite as

FusionMLS: Highly dynamic 3D reconstruction with consumer-grade RGB-D cameras

  • Siim Meerits
  • Diego Thomas
  • Vincent Nozick
  • Hideo Saito
Open Access
Research Article
  • 230 Downloads

Abstract

Multi-view dynamic three-dimensional reconstruction has typically required the use of custom shutter-synchronized camera rigs in order to capture scenes containing rapid movements or complex topology changes. In this paper, we demonstrate that multiple unsynchronized low-cost RGB-D cameras can be used for the same purpose. To alleviate issues caused by unsynchronized shutters, we propose a novel depth frame interpolation technique that allows synchronized data capture from highly dynamic 3D scenes. To manage the resulting huge number of input depth images, we also introduce an efficient moving least squares-based volumetric reconstruction method that generates triangle meshes of the scene. Our approach does not store the reconstruction volume in memory, making it memory-efficient and scalable to large scenes. Our implementation is completely GPU based and works in real time. The results shown herein, obtained with real data, demonstrate the effectiveness of our proposed method and its advantages compared to state-of-the-art approaches.

Keywords

3D reconstruction RGB-D cameras motion capture GPU 

Supplementary material

Supplementary material, approximately 67.7 MB.

References

  1. [1]
    De Reu, J.; Plets, G.; Verhoeven, G.; Smedt, P. D.; Bats, M.; Cherrett´e, B.; Maeyer, W. D.; Deconynck, J.; Herremans, D.; Laloo, P.; Meirvenne, M. V.; Clercq, W. D. Towards a three-dimensional costeffective registration of the archaeological heritage. Journal of Archaeological Science Vol. 40, No. 2, 1108–1121, 2013.CrossRefGoogle Scholar
  2. [2]
    Rong, Y.; Zheng, Y.; Shao, T.; Yang, Y.; Zhou, K. An interactive approach for functional prototype recovery from a single RGBD image. Computational Visual Media Vol. 2, No. 1, 87–96, 2016.CrossRefGoogle Scholar
  3. [3]
    Chen, K.; Lai, Y.-K.; Hu, S.-M. 3D indoor scene modeling from RGB-D data: A survey. Computational Visual Media Vol. 1, No. 4, 267–278, 2015.CrossRefGoogle Scholar
  4. [4]
    Orts-Escolano, S.; Rhemann, C.; Fanello, S.; Chang, W.; Kowdle, A.; Degtyarev, Y.; Kim, D.; Davidson, P. L.; Khamis, S.; Dou, M.; Tankovich, V.; Loop, C.; Cai, Q.; Chou, P. A.; Mennicken, S.; Valentin, J.; Pradeep, V.; Wang, S.; Kang, S. B.; Kohli, P.; Lutchyn, Y.; Keskin, C.; Izadi, S. Holoportation: Virtual 3D teleportation in real-time. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 741–754, 2016.CrossRefGoogle Scholar
  5. [5]
    Zollhöfer, M.; Nießner, M.; Izadi, S.; Rehmann, C.; Zach, C.; Fisher, M.; Wu, C.; Fitzgibbon, A.; Loop, C.; Theobalt, C.; Stamminger, M. Real-time nonrigid reconstruction using an RGB-D camera. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 156, 2014.Google Scholar
  6. [6]
    Newcombe, R. A.; Fox, D.; Seitz, S. M. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 343–352, 2015.Google Scholar
  7. [7]
    Innmann, M.; Zollhöfer, M.; Nießner, M.; Theobalt, C.; Stamminger, M. VolumeDeform: Real-time volumetric non-rigid reconstruction. In: Computer Vision–ECCV 2016. Lecture Notes in Computer Science, Vol. 9912. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 362–379, 2016.zbMATHGoogle Scholar
  8. [8]
    Yu, T.; Guo, K.; Xu, F.; Dong, Y.; Su, Z.; Zhao, J.; Li, J.; Dai, Q.; Liu, Y. BodyFusion: Real-time capture of human motion and surface geometry using a single depth camera. In: Proceedings of the IEEE International Conference on Computer Vision, 910–919, 2017.Google Scholar
  9. [9]
    Collet, A.; Chuang, M.; Sweeney, P.; Gillett, D.; Evseev, D.; Calabrese, D.; Hoppe, H.; Kirk, A.; Sullivan, S. High-quality streamable free-viewpoint video. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 69, 2015.Google Scholar
  10. [10]
    Dou, M.; Khamis, S.; Degtyarev, Y.; Davidson, P.; Fanello, S. R.; Kowdle, A.; Escolano, S. O.; Rhemann, C.; Kim, D.; Taylor, J.; Kohli, P.; Tankovich, V.; Izadi, S. Fusion4D: Real-time performance capture of challenging scenes. ACM Transactions on Graphics Vol. 35, No. 4, Article No. 114, 2016.Google Scholar
  11. [11]
    Dou, M.; Davidson, P.; Fanello, S. R.; Khamis, S.; Kowdle, A.; Rhemann, C.; Tankovich, V.; Izadi, S. Motion2fusion: Real-time volumetric performance capture. ACM Transactions on Graphics Vol. 36, No. 6, Article No. 246, 2017.Google Scholar
  12. [12]
    Nießner, M.; Zollhöfer, M.; Izadi, S.; Stamminger, M. Real-time 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics Vol. 32, No. 6, Article No. 169, 2013.Google Scholar
  13. [13]
    Berger, M.; Tagliasacchi, A.; Seversky, L. M.; Alliez, P.; Guennebaud, G.; Levine, J. A.; Sharf, A.; Silva, C. T. A survey of surface reconstruction from point clouds. Computer Graphics Forum Vol. 36, No. 1, 301–329, 2017.CrossRefGoogle Scholar
  14. [14]
    Li, Z.; Ji, Y.; Yang, W.; Ye, J.; Yu, J. Robust 3D human motion reconstruction via dynamic template construction. In: Proceedings of the International Conference on 3D Vision, 496–505, 2017.Google Scholar
  15. [15]
    Wang, K.; Zhang, G.; Xia, S. Templateless non-rigid reconstruction and motion tracking with a single RGBD camera. IEEE Transactions on Image Processing Vol. 26, No. 12, 5966–5979, 2017.MathSciNetCrossRefGoogle Scholar
  16. [16]
    Curless, B.; Levoy, M. A volumetric method for building complex models from range images. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, 303–312, 1996.Google Scholar
  17. [17]
    Guo, K.; Xu, F.; Yu, T.; Liu, X.; Dai, Q.; Liu, Y. Real-time geometry, albedo, and motion reconstruction using a single RGB-D camera. ACM Transactions on Graphics Vol. 36, No. 3, Article No. 32, 2017.Google Scholar
  18. [18]
    Zhang, H.; Xu, F. MixedFusion: Real-time reconstruction of an indoor scene with dynamic objects. IEEE Transactions on Visualization and Computer Graphics DOI: 10.1109/TVCG.2017.2786233, 2018.Google Scholar
  19. [19]
    Kazhdan, M.; Hoppe, H. Screened Poisson surface reconstruction. ACM Transactions on Graphics Vol. 32, No. 3, Article No. 29, 2013.Google Scholar
  20. [20]
    Wang, R.; Wei, L.; Vouga, E.; Huang, Q.; Ceylan, D.; Medioni, G.; Li, H. Capturing dynamic textured surfaces of moving targets. In: Computer Vision–ECCV 2016. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 271–288, 2016.Google Scholar
  21. [21]
    Alexiadis, D. S.; Zioulis, N.; Zarpalas, D.; Daras P. Fast deformable model-based human performance capture and FVV using consumer-grade RGB-D sensors. Pattern Recognition Vol. 79, 260–278, 2018.CrossRefGoogle Scholar
  22. [22]
    Alexiadis, D. S.; Chatzitofis, A.; Zioulis, N.; Zoidi, O.; Louizis, G.; Zarpalas, D.; Daras, P. An integrated platform for live 3D human reconstruction and motion capturing. IEEE Transactions on Circuits and Systems for Video Technology Vol. 27, No. 4, 798–813, 2017.CrossRefGoogle Scholar
  23. [23]
    Kuster, C.; Bazin, J.-C.; Öztireli, C.; Deng, T.; Martin, T.; Popa, T.; Gross, M. Spatio-temporal geometry fusion for multiple hybrid cameras using moving least squares surfaces. Computer Graphics Forum Vol. 33, No. 2, 1–10, 2014.CrossRefGoogle Scholar
  24. [24]
    Meerits, S.; Nozick, V.; Saito, H. Real-time scene reconstruction and triangle mesh generation using multiple RGB-D cameras. Journal of Real-Time Image Processing DOI: 10.1007/s11554-017-0736-x, 2017.Google Scholar
  25. [25]
    Turk, G.; Levoy, M. Zippered polygon meshes from range images. In: Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques, 311–318, 1994.Google Scholar
  26. [26]
    Yan, Z.; Xiang, X. Scene flow estimation: A survey. arXiv preprint arXiv:1612.02590, 2016.Google Scholar
  27. [27]
    Li, L.; Xiang, S.; Yang, Y.; Yu, L. Multicamera interference cancellation of time-of-flight (TOF) cameras. In: Proceedings of the IEEE International Conference on Image Processing, 556–560, 2015.Google Scholar
  28. [28]
    Guennebaud, G.; Gross, M. Algebraic point set surfaces. ACM Transactions on Graphics Vol. 26, No. 3, Article No. 23, 2007.Google Scholar
  29. [29]
    Chen, J.; Bautembach, D.; Izadi, S. Scalable real-time volumetric surface reconstruction. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 113, 2013.Google Scholar
  30. [30]
    Steinbrücker, F.; Sturm, J.; Cremers, D. Volumetric 3D mapping in real-time on a CPU. In: Proceedings of the IEEE International Conference on Robotics and Automation, 2021–2028, 2014.Google Scholar
  31. [31]
    Alexa, M.; Adamson, A. On normals and projection operators for surfaces defined by point sets. In: Proceedings of the First Eurographics Conference on Point-Based Graphics, 149–155, 2004.Google Scholar
  32. [32]
    Lorensen, W. E.; Cline, H. E. Marching cubes: A high resolution 3D surface construction algorithm. ACM SIGGRAPH Computer Graphics Vol. 21, No. 4, 163–169, 1987.CrossRefGoogle Scholar
  33. [33]
    Everitt, C. Interactive order-independent transparency. White paper, nVIDIA Vol. 2, No. 6, 7, 2001.Google Scholar

Copyright information

© The Author(s) 2018

Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (https://doi.org/creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Other papers from this open access journal are available free of charge from https://doi.org/www.springer.com/journal/41095. To submit a manuscript, please go to https://doi.org/www.editorialmanager.com/cvmj.

Authors and Affiliations

  • Siim Meerits
    • 1
  • Diego Thomas
    • 2
  • Vincent Nozick
    • 3
    • 4
  • Hideo Saito
    • 1
  1. 1.Department of Information and Computer ScienceKeio UniversityYokohamaJapan
  2. 2.Department of Advanced Information TechnologyKyushu UniversityFukuokaJapan
  3. 3.LIGM, UMR 8049Université Paris-Est Marne-la-ValléeChamps-sur-MarneFrance
  4. 4.Japanese–French Laboratory for InformaticsCNRSTokyoJapan

Personalised recommendations