Multi-view dynamic three-dimensional reconstruction has typically required the use of custom shutter-synchronized camera rigs in order to capture scenes containing rapid movements or complex topology changes. In this paper, we demonstrate that multiple unsynchronized low-cost RGB-D cameras can be used for the same purpose. To alleviate issues caused by unsynchronized shutters, we propose a novel depth frame interpolation technique that allows synchronized data capture from highly dynamic 3D scenes. To manage the resulting huge number of input depth images, we also introduce an efficient moving least squares-based volumetric reconstruction method that generates triangle meshes of the scene. Our approach does not store the reconstruction volume in memory, making it memory-efficient and scalable to large scenes. Our implementation is completely GPU based and works in real time. The results shown herein, obtained with real data, demonstrate the effectiveness of our proposed method and its advantages compared to state-of-the-art approaches.
De Reu, J.; Plets, G.; Verhoeven, G.; Smedt, P. D.; Bats, M.; Cherrett´e, B.; Maeyer, W. D.; Deconynck, J.; Herremans, D.; Laloo, P.; Meirvenne, M. V.; Clercq, W. D. Towards a three-dimensional costeffective registration of the archaeological heritage. Journal of Archaeological Science Vol. 40, No. 2, 1108–1121, 2013.
Rong, Y.; Zheng, Y.; Shao, T.; Yang, Y.; Zhou, K. An interactive approach for functional prototype recovery from a single RGBD image. Computational Visual Media Vol. 2, No. 1, 87–96, 2016.
Chen, K.; Lai, Y.-K.; Hu, S.-M. 3D indoor scene modeling from RGB-D data: A survey. Computational Visual Media Vol. 1, No. 4, 267–278, 2015.
Orts-Escolano, S.; Rhemann, C.; Fanello, S.; Chang, W.; Kowdle, A.; Degtyarev, Y.; Kim, D.; Davidson, P. L.; Khamis, S.; Dou, M.; Tankovich, V.; Loop, C.; Cai, Q.; Chou, P. A.; Mennicken, S.; Valentin, J.; Pradeep, V.; Wang, S.; Kang, S. B.; Kohli, P.; Lutchyn, Y.; Keskin, C.; Izadi, S. Holoportation: Virtual 3D teleportation in real-time. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 741–754, 2016.
Zollhöfer, M.; Nießner, M.; Izadi, S.; Rehmann, C.; Zach, C.; Fisher, M.; Wu, C.; Fitzgibbon, A.; Loop, C.; Theobalt, C.; Stamminger, M. Real-time nonrigid reconstruction using an RGB-D camera. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 156, 2014.
Newcombe, R. A.; Fox, D.; Seitz, S. M. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 343–352, 2015.
Innmann, M.; Zollhöfer, M.; Nießner, M.; Theobalt, C.; Stamminger, M. VolumeDeform: Real-time volumetric non-rigid reconstruction. In: Computer Vision–ECCV 2016. Lecture Notes in Computer Science, Vol. 9912. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 362–379, 2016.
Yu, T.; Guo, K.; Xu, F.; Dong, Y.; Su, Z.; Zhao, J.; Li, J.; Dai, Q.; Liu, Y. BodyFusion: Real-time capture of human motion and surface geometry using a single depth camera. In: Proceedings of the IEEE International Conference on Computer Vision, 910–919, 2017.
Collet, A.; Chuang, M.; Sweeney, P.; Gillett, D.; Evseev, D.; Calabrese, D.; Hoppe, H.; Kirk, A.; Sullivan, S. High-quality streamable free-viewpoint video. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 69, 2015.
Dou, M.; Khamis, S.; Degtyarev, Y.; Davidson, P.; Fanello, S. R.; Kowdle, A.; Escolano, S. O.; Rhemann, C.; Kim, D.; Taylor, J.; Kohli, P.; Tankovich, V.; Izadi, S. Fusion4D: Real-time performance capture of challenging scenes. ACM Transactions on Graphics Vol. 35, No. 4, Article No. 114, 2016.
Dou, M.; Davidson, P.; Fanello, S. R.; Khamis, S.; Kowdle, A.; Rhemann, C.; Tankovich, V.; Izadi, S. Motion2fusion: Real-time volumetric performance capture. ACM Transactions on Graphics Vol. 36, No. 6, Article No. 246, 2017.
Nießner, M.; Zollhöfer, M.; Izadi, S.; Stamminger, M. Real-time 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics Vol. 32, No. 6, Article No. 169, 2013.
Berger, M.; Tagliasacchi, A.; Seversky, L. M.; Alliez, P.; Guennebaud, G.; Levine, J. A.; Sharf, A.; Silva, C. T. A survey of surface reconstruction from point clouds. Computer Graphics Forum Vol. 36, No. 1, 301–329, 2017.
Li, Z.; Ji, Y.; Yang, W.; Ye, J.; Yu, J. Robust 3D human motion reconstruction via dynamic template construction. In: Proceedings of the International Conference on 3D Vision, 496–505, 2017.
Wang, K.; Zhang, G.; Xia, S. Templateless non-rigid reconstruction and motion tracking with a single RGBD camera. IEEE Transactions on Image Processing Vol. 26, No. 12, 5966–5979, 2017.
Curless, B.; Levoy, M. A volumetric method for building complex models from range images. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, 303–312, 1996.
Guo, K.; Xu, F.; Yu, T.; Liu, X.; Dai, Q.; Liu, Y. Real-time geometry, albedo, and motion reconstruction using a single RGB-D camera. ACM Transactions on Graphics Vol. 36, No. 3, Article No. 32, 2017.
Zhang, H.; Xu, F. MixedFusion: Real-time reconstruction of an indoor scene with dynamic objects. IEEE Transactions on Visualization and Computer Graphics DOI: 10.1109/TVCG.2017.2786233, 2018.
Kazhdan, M.; Hoppe, H. Screened Poisson surface reconstruction. ACM Transactions on Graphics Vol. 32, No. 3, Article No. 29, 2013.
Wang, R.; Wei, L.; Vouga, E.; Huang, Q.; Ceylan, D.; Medioni, G.; Li, H. Capturing dynamic textured surfaces of moving targets. In: Computer Vision–ECCV 2016. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 271–288, 2016.
Alexiadis, D. S.; Zioulis, N.; Zarpalas, D.; Daras P. Fast deformable model-based human performance capture and FVV using consumer-grade RGB-D sensors. Pattern Recognition Vol. 79, 260–278, 2018.
Alexiadis, D. S.; Chatzitofis, A.; Zioulis, N.; Zoidi, O.; Louizis, G.; Zarpalas, D.; Daras, P. An integrated platform for live 3D human reconstruction and motion capturing. IEEE Transactions on Circuits and Systems for Video Technology Vol. 27, No. 4, 798–813, 2017.
Kuster, C.; Bazin, J.-C.; Öztireli, C.; Deng, T.; Martin, T.; Popa, T.; Gross, M. Spatio-temporal geometry fusion for multiple hybrid cameras using moving least squares surfaces. Computer Graphics Forum Vol. 33, No. 2, 1–10, 2014.
Meerits, S.; Nozick, V.; Saito, H. Real-time scene reconstruction and triangle mesh generation using multiple RGB-D cameras. Journal of Real-Time Image Processing DOI: 10.1007/s11554-017-0736-x, 2017.
Turk, G.; Levoy, M. Zippered polygon meshes from range images. In: Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques, 311–318, 1994.
Yan, Z.; Xiang, X. Scene flow estimation: A survey. arXiv preprint arXiv:1612.02590, 2016.
Li, L.; Xiang, S.; Yang, Y.; Yu, L. Multicamera interference cancellation of time-of-flight (TOF) cameras. In: Proceedings of the IEEE International Conference on Image Processing, 556–560, 2015.
Guennebaud, G.; Gross, M. Algebraic point set surfaces. ACM Transactions on Graphics Vol. 26, No. 3, Article No. 23, 2007.
Chen, J.; Bautembach, D.; Izadi, S. Scalable real-time volumetric surface reconstruction. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 113, 2013.
Steinbrücker, F.; Sturm, J.; Cremers, D. Volumetric 3D mapping in real-time on a CPU. In: Proceedings of the IEEE International Conference on Robotics and Automation, 2021–2028, 2014.
Alexa, M.; Adamson, A. On normals and projection operators for surfaces defined by point sets. In: Proceedings of the First Eurographics Conference on Point-Based Graphics, 149–155, 2004.
Lorensen, W. E.; Cline, H. E. Marching cubes: A high resolution 3D surface construction algorithm. ACM SIGGRAPH Computer Graphics Vol. 21, No. 4, 163–169, 1987.
Everitt, C. Interactive order-independent transparency. White paper, nVIDIA Vol. 2, No. 6, 7, 2001.
Siim Meerits received his B.Sc. degree in physics from Tartu University, Estonia, in 2010. He continued at Keio University, Japan, receiving his M.Sc.Eng. degree in computer science in 2015. Currently he is in the Ph.D. program at the same institution. His research interests include computer vision, particularly 3D reconstruction, and augmented reality.
Diego Thomas received his master degree in informatics and applied mathematics from the Ecole Nationale Superieure d’Informatique et de Mathematiques Appliquees de Grenoble, France, in 2008. He received his Ph.D. degree from the Graduate University for Advanced Studies in 2012. After two years as a JSPS postdoc at Kyushu University, he is now assistant professor at the Laboratory for Image and Media Understanding at Kyushu University, Fukuoka, Japan. His research interests include 3D image registration, 3D reconstruction, and photometric analysis.
Vincent Nozick received his Ph.D. degree in computer sciences in 2006 from Université Paris-Est Marne-la-Vallée, France. In 2006, he was laureate of a Lavoisier fellowship for a post-doc position in the laboratory of Prof. Hideo Saito, Keio University. Since 2008, he has been a tenured “maître de conférences” (assistance/associate professor) at Université Paris-Est Marne-la-Vallée, France. He served as a headmaster of the Imac Engineering School from 2011 to 2013. He held a “délégation CNRS” position from 2016 to 2018 at the Japanese French Laboratory for Informatics (JFLI), at Keio University, NII and The University of Tokyo, Japan. In addition to computer vision applications, his research interests include digital image forensics and geometric algebra.
Hideo Saito received his Ph.D. degree in electrical engineering from Keio University, Japan, in 1992. Since then, he has been on the Faculty of Science and Technology, Keio University. From 1997 to 1999, he joined the Virtualized Reality Project in the Robotics Institute at Carnegie Mellon University as a visiting researcher. Since 2006, he has been a full professor in the Department of Information and Computer Science, Keio University. His recent activities for academic conferences include being Program Chair of ACCV2014, General Chair of ISMAR2015, and a Program Chair of ISMAR2016. His research interests include computer vision and pattern recognition and their applications to augmented reality, virtual reality, and human–robotics interaction.
Electronic supplementary material
About this article
Cite this article
Meerits, S., Thomas, D., Nozick, V. et al. FusionMLS: Highly dynamic 3D reconstruction with consumer-grade RGB-D cameras. Comp. Visual Media 4, 287–303 (2018). https://doi.org/10.1007/s41095-018-0121-0
- 3D reconstruction
- RGB-D cameras
- motion capture