Topology-Change-Aware Volumetric Fusion for Dynamic Scene Reconstruction

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12361)


Topology change is a challenging problem for 4D reconstruction of dynamic scenes. In the classic volumetric fusion-based framework, a mesh is usually extracted from the TSDF volume as the canonical surface representation to help estimating deformation field. However, the surface and Embedded Deformation Graph (EDG) representations bring conflicts under topology changes since the surface mesh has fixed-connectivity but the deformation field can be discontinuous. In this paper, the classic framework is re-designed to enable 4D reconstruction of dynamic scene under topology changes, by introducing a novel structure of Non-manifold Volumetric Grid to the re-design of both TSDF and EDG, which allows connectivity updates by cell splitting and replication. Experiments show convincing reconstruction results for dynamic scenes of topology changes, as compared to the state-of-the-art methods.


Reconstruction Topology change Fusion Dynamic scene 



This research is partially supported by National Science Foundation (2007661). The opinions expressed are solely those of the authors, and do not necessarily represent those of the National Science Foundation.

Supplementary material

504471_1_En_16_MOESM1_ESM.pdf (360 kb)
Supplementary material 1 (pdf 360 KB)


  1. 1.
    Baran, I., Vlasic, D., Grinspun, E., Popović, J.: Semantic deformation transfer. In: ACM SIGGRAPH 2009 Papers, pp. 1–6 (2009)Google Scholar
  2. 2.
    Bertholet, P., Ichim, A.E., Zwicker, M.: Temporally consistent motion segmentation from RGB-D video. Comput. Graph. Forum 37, 118–134 (2018)CrossRefGoogle Scholar
  3. 3.
    Black, M.J., Rangarajan, A.: On the unification of line processes, outlier rejection, and robust statistics with applications in early vision. Int. J. Comput. Vis. 19(1), 57–91 (1996). Scholar
  4. 4.
    Bojsen-Hansen, M., Li, H., Wojtan, C.: Tracking surfaces with evolving topology. ACM Trans. Graph. 31(4) (2012). Article no. 53–1Google Scholar
  5. 5.
    Chen, X., Feng, J., Bechmann, D.: Mesh sequence morphing. Comput. Graph. Forum 35, 179–190 (2016)CrossRefGoogle Scholar
  6. 6.
    Collet, A., et al.: High-quality streamable free-viewpoint video. ACM Trans. Graph. (ToG) 34(4), 69 (2015)CrossRefGoogle Scholar
  7. 7.
    Curless, B., Levoy, M.: A volumetric method for building complex models from range images. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996, pp. 303–312. ACM (1996)Google Scholar
  8. 8.
    Digne, J., Cohen-Steiner, D., Alliez, P., de Goes, F., Desbrun, M.: Feature-preserving surface reconstruction and simplification from defect-laden point sets. J. Math. Imaging Vis. 48(2), 369–382 (2013). Scholar
  9. 9.
    Dou, M., et al.: Motion2fusion: real-time volumetric performance capture. ACM Trans. Graph. (TOG) 36(6), 246 (2017)CrossRefGoogle Scholar
  10. 10.
    Dou, M., et al.: Fusion4D: real-time performance capture of challenging scenes. ACM Trans. Graph. 35(4), 114 (2016)CrossRefGoogle Scholar
  11. 11.
    Enright, D., Marschner, S., Fedkiw, R.: Animation and rendering of complex water surfaces. In: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, pp. 736–744 (2002)Google Scholar
  12. 12.
    Fröhlich, S., Botsch, M.: Example-driven deformations based on discrete shells. Comput. Graph. Forum 30, 2246–2257 (2011)CrossRefGoogle Scholar
  13. 13.
    Gao, L., Chen, S.Y., Lai, Y.K., Xia, S.: Data-driven shape interpolation and morphing editing. Comput. Graph. Forum 36, 19–31 (2017)CrossRefGoogle Scholar
  14. 14.
    Gao, L., Lai, Y.K., Huang, Q.X., Hu, S.M.: A data-driven approach to realistic shape morphing. Comput. Graph. Forum 32, 449–457 (2013)CrossRefGoogle Scholar
  15. 15.
    Gao, W., Tedrake, R.: SurfelWarp: efficient non-volumetric single view dynamic reconstruction. arXiv preprint arXiv:1904.13073 (2019)
  16. 16.
    Garg, R., Roussos, A., Agapito, L.: A variational approach to video registration with subspace constraints. Int. J. Comput. Vis. 104(3), 286–314 (2013). Scholar
  17. 17.
    Golla, T., Kneiphof, T., Kuhlmann, H., Weinmann, M., Klein, R.: Temporal upsampling of point cloud sequences by optimal transport for plant growth visualization. Comput. Graph. Forum (2020)Google Scholar
  18. 18.
    Guo, K., Xu, F., Yu, T., Liu, X., Dai, Q., Liu, Y.: Real-time geometry, albedo, and motion reconstruction using a single RGB-D camera. ACM Trans. Graph. (TOG) 36(3), 32 (2017)CrossRefGoogle Scholar
  19. 19.
    Innmann, M., Zollhöfer, M., Nießner, M., Theobalt, C., Stamminger, M.: VolumeDeform: real-time volumetric non-rigid reconstruction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 362–379. Springer, Cham (2016). Scholar
  20. 20.
    Kowdle, A., et al.: The need 4 speed in real-time dense visual tracking. ACM Trans. Graph. 37(6), 220:1–220:14 (2018)Google Scholar
  21. 21.
    Letouzey, A., Boyer, E.: Progressive shape models. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 190–197. IEEE (2012)Google Scholar
  22. 22.
    Li, C., Zhao, Z., Guo, X.: ArticulatedFusion: real-time reconstruction of motion, geometry and segmentation using a single depth camera. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 324–340. Springer, Cham (2018). Scholar
  23. 23.
    Li, H., et al.: Temporally coherent completion of dynamic shapes. ACM Trans. Graph. (TOG) 31(1), 1–11 (2012)CrossRefGoogle Scholar
  24. 24.
    Li, H., Yu, J., Ye, Y., Bregler, C.: Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. 32(4) (2013). Article no. 42–1Google Scholar
  25. 25.
    Mitchell, N., Aanjaneya, M., Setaluri, R., Sifakis, E.: Non-manifold level sets: a multivalued implicit surface representation with applications to self-collision processing. ACM Trans. Graph. (TOG) 34(6), 247 (2015)CrossRefGoogle Scholar
  26. 26.
    Molino, N., Bao, Z., Fedkiw, R.: A virtual node algorithm for changing mesh topology during simulation. ACM Trans. Graph. (TOG) 23, 385–392 (2004)CrossRefGoogle Scholar
  27. 27.
    Myronenko, A., Song, X.: Point set registration: coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2262–2275 (2010)CrossRefGoogle Scholar
  28. 28.
    Newcombe, R.A., Fox, D., Seitz, S.M.: DynamicFusion: reconstruction and tracking of non-rigid scenes in real-time. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 343–352 (2015)Google Scholar
  29. 29.
    Osher, S., Fedkiw, R.P.: Level Set Methods and Dynamic Implicit Surfaces, vol. 200. Springer, New York (2005)zbMATHGoogle Scholar
  30. 30.
    Oswald, M.R., Stühmer, J., Cremers, D.: Generalized connectivity constraints for spatio-temporal 3D reconstruction. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 32–46. Springer, Cham (2014). Scholar
  31. 31.
    Pons-Moll, G., Baak, A., Helten, T., Müller, M., Seidel, H.P., Rosenhahn, B.: Multisensor-fusion for 3D full-body human motion capture. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 663–670. IEEE (2010)Google Scholar
  32. 32.
    Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: 3DIM, vol. 1, pp. 145–152 (2001)Google Scholar
  33. 33.
    Slavcheva, M., Baust, M., Cremers, D., Ilic, S.: KillingFusion: non-rigid 3D reconstruction without correspondences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1386–1395 (2017)Google Scholar
  34. 34.
    Slavcheva, M., Baust, M., Ilic, S.: SobolevFusion: 3D reconstruction of scenes undergoing free non-rigid motion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2646–2655 (2018)Google Scholar
  35. 35.
    Solomon, J., et al.: Convolutional Wasserstein distances: efficient optimal transportation on geometric domains. ACM Trans. Graph. (TOG) 34(4), 1–11 (2015)CrossRefGoogle Scholar
  36. 36.
    Sumner, R.W., Schmid, J., Pauly, M.: Embedded deformation for shape manipulation. ACM Trans. Graph. 26(3) (2007) Google Scholar
  37. 37.
    Tkach, A., Pauly, M., Tagliasacchi, A.: Sphere-meshes for real-time hand modeling and tracking. ACM Trans. Graph. (TOG) 35(6), 222 (2016)CrossRefGoogle Scholar
  38. 38.
    Tsoli, A., Argyros, A.A.: Tracking deformable surfaces that undergo topological changes using an RGB-D camera. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 333–341. IEEE (2016)Google Scholar
  39. 39.
    Von-Tycowicz, C., Schulz, C., Seidel, H.P., Hildebrandt, K.: Real-time nonlinear shape interpolation. ACM Trans. Graph. (TOG) 34(3), 1–10 (2015)CrossRefGoogle Scholar
  40. 40.
    Xu, D., Zhang, H., Wang, Q., Bao, H.: Poisson shape interpolation. Graph. Models 68(3), 268–281 (2006)CrossRefGoogle Scholar
  41. 41.
    Xu, W., Salzmann, M., Wang, Y., Liu, Y.: Deformable 3D fusion: from partial dynamic 3D observations to complete 4D models. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2183–2191 (2015)Google Scholar
  42. 42.
    Yu, T., et al.: BodyFusion: real-time capture of human motion and surface geometry using a single depth camera. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 910–919 (2017)Google Scholar
  43. 43.
    Yu, T., et al.: DoubleFusion: real-time capture of human performances with inner body shapes from a single depth sensor. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7287–7296 (2018)Google Scholar
  44. 44.
    Yuan, Q., Li, G., Xu, K., Chen, X., Huang, H.: Space-time co-segmentation of articulated point cloud sequences. Comput. Graph. Forum 35, 419–429 (2016)CrossRefGoogle Scholar
  45. 45.
    Zampogiannis, K., Fermuller, C., Aloimonos, Y.: Topology-aware non-rigid point cloud registration. IEEE Trans. Pattern Anal. Mach. Intell. (2019) Google Scholar
  46. 46.
    Zollhöfer, M., et al.: Real-time non-rigid reconstruction using an RGB-D camera. ACM Trans. Graph. (ToG) 33(4), 156 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Computer ScienceThe University of Texas at DallasRichardsonUSA

Personalised recommendations