Capturing Dynamic Textured Surfaces of Moving Targets

  • Ruizhe Wang
  • Lingyu Wei
  • Etienne Vouga
  • Qixing Huang
  • Duygu Ceylan
  • Gérard Medioni
  • Hao Li
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9911)


We present an end-to-end system for reconstructing complete watertight and textured models of moving subjects such as clothed humans and animals, using only three or four handheld sensors. The heart of our framework is a new pairwise registration algorithm that minimizes, using a particle swarm strategy, an alignment error metric based on mutual visibility and occlusion. We show that this algorithm reliably registers partial scans with as little as 15 % overlap without requiring any initial correspondences, and outperforms alternative global registration algorithms. This registration algorithm allows us to reconstruct moving subjects from free-viewpoint video produced by consumer-grade sensors, without extensive sensor calibration, constrained capture volume, expensive arrays of cameras, or templates of the subject geometry.


Range image registration Particle swarm optimization Dynamic surface reconstruction Free-viewpoint video Moving target Texture reconstruction 



We thank Jieqi Jiang, Xiang Ao, Jin Xu, Mingfai Wong, Bor-Jeng Chen and Anh Tran for being our capture models. This research is supported in part by Adobe, Oculus & Facebook, Sony, Pelican Imaging, Panasonic, Embodee, Huawei, the Google Faculty Research Award, The Okawa Foundation Research Grant, the Office of Naval Research (ONR)/U.S. Navy, under award number N00014-15-1-2639, the Office of the Director of National Intelligence (ODNI), and Intelligence Advanced Research Projects Activity (IARPA), under contract number 2014-14071600010. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purpose notwithstanding any copyright annotation thereon.

Supplementary material (21 mb)
Supplementary material 1 (mov 21468 KB)
419982_1_En_17_MOESM2_ESM.pdf (2.3 mb)
Supplementary material 2 (pdf 2352 KB)


  1. 1.
    de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.P., Thrun, S.: Performance capture from sparse multi-view video. In: ACM SIGGRAPH, pp. 98:1–98:10. ACM, New York (2008)Google Scholar
  2. 2.
    Ahmed, N., Theobalt, C., Dobrev, P., Seidel, H.P., Thrun, S.: Robust fusion of dynamic shape and normal capture for high-quality reconstruction of time-varying geometry. In: IEEE CVPR, pp. 1–8, June 2008Google Scholar
  3. 3.
    Aiger, D., Mitra, N.J., Cohen-Or, D.: 4-points congruent sets for robust pairwise surface registration. In: ACM Transactions on Graphics (TOG), vol. 27, p. 85. ACM (2008)Google Scholar
  4. 4.
    Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., Davis, J.: Scape: shape completion and animation of people. ACM Trans. Graph. 24(3), 408–416 (2005)CrossRefGoogle Scholar
  5. 5.
    Bogo, F., Black, M.J., Loper, M., Romero, J.: Detailed full-body reconstructions of moving people from monocular RGB-D sequences, pp. 2300–2308, December 2015Google Scholar
  6. 6.
    Bojsen-Hansen, M., Li, H., Wojtan, C.: Tracking surfaces with evolving topology. ACM Trans. Graph. (SIGGRAPH 2012) 31(4), 53:1–53:10 (2012)Google Scholar
  7. 7.
    Chen, C.S., Hung, Y.P., Cheng, J.B.: Ransac-based darces: a new approach to fast automatic registration of partially overlapping range images. IEEE Trans. Pattern Anal. Mach. Intell. 21(11), 1229–1234 (1999)CrossRefGoogle Scholar
  8. 8.
    Chen, Y., Medioni, G.: Object modeling by registration of multiple range images. In: ICRA, pp. 2724–2729. IEEE (1991)Google Scholar
  9. 9.
    Chuang, M., Luo, L., Brown, B.J., Rusinkiewicz, S., Kazhdan, M.: Estimating the laplace-beltrami operator by restricting 3D functions. In: Computer Graphics Forum, vol. 28, pp. 1475–1484. Wiley Online Library (2009)Google Scholar
  10. 10.
    Chung, D.H., Yun, I.D., Lee, S.U.: Registration of multiple-range views using the reverse-calibration technique. Pattern Recogn. 31(4), 457–464 (1998)CrossRefGoogle Scholar
  11. 11.
    Collet, A., Chuang, M., Sweeney, P., Gillett, D., Evseev, D., Calabrese, D., Hoppe, H., Kirk, A., Sullivan, S.: High-quality streamable free-viewpoint video. In: ACM SIGGRAPH, vol. 34, pp. 69:1–69:13. ACM, July 2015Google Scholar
  12. 12.
    Debevec, P.: The light stages and their applications to photoreal digital actors. In: SIGGRAPH Asia, Singapore, November 2012Google Scholar
  13. 13.
    Dou, M., Taylor, J., Fuchs, H., Fitzgibbon, A., Izadi, S.: 3D scanning deformable objects with a single rgbd sensor. In: IEEE CVPR, pp. 493–501, June 2015Google Scholar
  14. 14.
    Franco, J., Lapierre, M., Boyer, E.: Visual shapes of silhouette sets. In: Third International Symposium on 3D Data Processing, Visualization, and Transmission, pp. 397–404, June 2006Google Scholar
  15. 15.
    Gelfand, N., Mitra, N.J., Guibas, L.J., Pottmann, H.: Robust global registration. In: Symposium on Geometry Processing, vol. 2, p. 5 (2005)Google Scholar
  16. 16.
    Hernández, C., Schmitt, F., Cipolla, R.: Silhouette coherence for camera calibration under circular motion. IEEE Trans. Pattern Anal. Mach. Intell. 29(2), 343–349 (2007)CrossRefGoogle Scholar
  17. 17.
    Horn, B.K.: Extended gaussian images. Proc. IEEE 72(12), 1671–1686 (1984)CrossRefGoogle Scholar
  18. 18.
    Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., Fitzgibbon, A.: Kinectfusion: real-time 3D reconstruction and interaction using a moving depth camera. In: UIST, pp. 559–568. ACM, New York (2011)Google Scholar
  19. 19.
    Johnson, A.E., Hebert, M.: Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Trans. Pattern Anal. Mach. Intell. 21(5), 433–449 (1999)CrossRefGoogle Scholar
  20. 20.
    Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proceedings of the Fourth Eurographics Symposium on Geometry Processing, vol. 7 (2006)Google Scholar
  21. 21.
    Kennedy, J.: Particle swarm optimization. In: Sammut, C., Webb, G.I. (eds.) Encyclopedia of Machine Learning, pp. 760–766. Springer, New York (2010)Google Scholar
  22. 22.
    Li, H., Adams, B., Guibas, L.J., Pauly, M.: Robust single-view geometry and motion reconstruction. In: ACM SIGGRAPH Asia, SIGGRAPH Asia 2009, pp. 175:1–175:10. ACM, New York (2009)Google Scholar
  23. 23.
    Li, H., Luo, L., Vlasic, D., Peers, P., Popović, J., Pauly, M., Rusinkiewicz, S.: Temporally coherent completion of dynamic shapes. ACM TOG 31(1), 2:1–2:11 (2012)CrossRefGoogle Scholar
  24. 24.
    Li, H., Vouga, E., Gudym, A., Luo, L., Barron, J.T., Gusev, G.: 3D self-portraits. In: ACM SIGGRAPH Asia, vol. 32, pp. 187:1–187:9. ACM, November 2013Google Scholar
  25. 25.
    Li, H., Yu, J., Ye, Y., Bregler, C.: Realtime facial animation with on-the-fly correctives. In: ACM SIGGRAPH, vol. 32, pp. 42:1–42:10. ACM, July 2013Google Scholar
  26. 26.
    Liu, Y., Ye, G., Wang, Y., Dai, Q., Theobalt, C.: Human performance capture using multiple handheld kinects. In: Shao, L., Han, J., Kohli, P., Zhang, Z. (eds.) Computer Vision and Machine Learning with RGB-D Sensors, pp. 91–108. Springer International Publishing, Cham (2014)Google Scholar
  27. 27.
    Makadia, A., Patterson, A., Daniilidis, K.: Fully automatic registration of 3D point clouds. In: 2006 IEEE Conference on CVPR, vol. 1, pp. 1297–1304. IEEE (2006)Google Scholar
  28. 28.
    Matusik, W., Buehler, C., Raskar, R., Gortler, S.J., McMillan, L.: Image-based visual hulls. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2000, pp. 369–374. ACM Press/Addison-Wesley Publishing Co., New York (2000)Google Scholar
  29. 29.
    Mellado, N., Aiger, D., Mitra, N.J.: Super 4pcs fast global pointcloud registration via smart indexing. In: Computer Graphics Forum, vol. 33, pp. 205–215. Wiley Online Library (2014)Google Scholar
  30. 30.
    Moezzi, S., Tai, L.C., Gerard, P.: Virtual view generation for 3D digital video. MultiMedia, IEEE 4(1), 18–26 (1997)CrossRefGoogle Scholar
  31. 31.
    Newcombe, R.A., Fox, D., Seitz, S.M.: Dynamicfusion: reconstruction and tracking of non-rigid scenes in real-time. In: IEEE CVPR, June 2015Google Scholar
  32. 32.
    Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Tracking the articulated motion of two strongly interacting hands. In: IEEE CVPR, pp. 1862–1869. IEEE (2012)Google Scholar
  33. 33.
    Qian, C., Sun, X., Wei, Y., Tang, X., Sun, J.: Realtime and robust hand tracking from depth. In: IEEE CVPR, pp. 1106–1113. IEEE (2014)Google Scholar
  34. 34.
    Rusinkiewicz, S., Levoy, M.: Efficient variants of the icp algorithm. In: 3-D Digital Imaging and Modeling, pp. 145–152. IEEE (2001)Google Scholar
  35. 35.
    Rusu, R.B., Blodow, N., Beetz, M.: Fast point feature histograms (fpfh) for 3D registration. In: 2009 IEEE International Conference on Robotics and Automation, pp. 3212–3217. IEEE (2009)Google Scholar
  36. 36.
    Rusu, R.B., Blodow, N., Marton, Z.C., Beetz, M.: Aligning point cloud views using persistent feature histograms. In: 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3384–3391. IEEE (2008)Google Scholar
  37. 37.
    Rusu, R.B., Cousins, S.: 3D is here: point cloud library (pcl). In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp. 1–4. IEEE (2011)Google Scholar
  38. 38.
    Shilane, P., Min, P., Kazhdan, M., Funkhouser, T.: The princeton shape benchmark. In: Shape modeling applications, 2004. Proceedings, pp. 167–178. IEEE (2004)Google Scholar
  39. 39.
    Shoemake, K.: Uniform random rotations. In: Graphics Gems III, pp. 124–132. Academic Press Professional, Inc. (1992)Google Scholar
  40. 40.
    Silva, L., Bellon, O.R., Boyer, K.L.: Precision range image registration using a robust surface interpenetration measure and enhanced genetic algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 27(5), 762–776 (2005)CrossRefzbMATHGoogle Scholar
  41. 41.
    Starck, J., Hilton, A.: Surface capture for performance-based animation. IEEE Comput. Graph. Appl. 27(3), 21–31 (2007)CrossRefGoogle Scholar
  42. 42.
    Süßmuth, J., Winter, M., Greiner, G.: Reconstructing animated meshes from time-varying point clouds. In: SGP, SGP 2008, pp. 1469–1476 (2008)Google Scholar
  43. 43.
    Tevs, A., Berner, A., Wand, M., Ihrke, I., Bokeloh, M., Kerber, J., Seidel, H.P.: Animation cartography—intrinsic reconstruction of shape and motion. ACM TOG 31(2), 12:1–12:15 (2012)CrossRefGoogle Scholar
  44. 44.
    Tong, J., Zhou, J., Liu, L., Pan, Z., Yan, H.: Scanning 3D full human bodies using kinects. IEEE TVCG 18(4), 643–650 (2012)Google Scholar
  45. 45.
    Vlasic, D., Baran, I., Matusik, W., Popović, J.: Articulated mesh animation from multi-view silhouettes. In: ACM SIGGRAPH, SIGGRAPH 2008, pp. 97:1–97:9. ACM, New York (2008)Google Scholar
  46. 46.
    Vlasic, D., Peers, P., Baran, I., Debevec, P., Popović, J., Rusinkiewicz, S., Matusik, W.: Dynamic shape capture using multi-view photometric stereo. In: ACM SIGGRAPH Asia, SIGGRAPH Asia 2009. pp. 174:1–174:11 (2009)Google Scholar
  47. 47.
    Wand, M., Adams, B., Ovsjanikov, M., Berner, A., Bokeloh, M., Jenke, P., Guibas, L., Seidel, H.P., Schilling, A.: Efficient reconstruction of nonrigid shape and motion from real-time 3D scanner data. ACM TOG 28(2), 15:1–15:15 (2009)CrossRefGoogle Scholar
  48. 48.
    Wand, M., Jenke, P., Huang, Q., Bokeloh, M., Guibas, L., Schilling, A.: Reconstruction of deforming geometry from time-varying point clouds. In: SGP, SGP 2007, pp. 49–58 (2007)Google Scholar
  49. 49.
    Wang, R., Choi, J., Medioni, G.: 3D modeling from wide baseline range scans using contour coherence. In: 2014 IEEE Conference on CVPR, pp. 4018–4025 (2014)Google Scholar
  50. 50.
    Wei, L., Huang, Q., Ceylan, D., Vouga, E., Li, H.: Dense human body correspondences using convolutional networks. In: IEEE CVPR. IEEE (2016)Google Scholar
  51. 51.
    Wu, C., Stoll, C., Valgaerts, L., Theobalt, C.: On-set performance capture of multiple actors with a stereo camera. ACM Trans. Graph. 32(6), 161:1–161:11 (2013)Google Scholar
  52. 52.
    Wu, C., Varanasi, K., Liu, Y., Seidel, H.P., Theobalt, C.: Shading-based dynamic shape refinement from multi-view video under general illumination, pp. 1108–1115. IEEE, November 2011Google Scholar
  53. 53.
    Yang, J., Li, H., Jia, Y.: Go-icp: Solving 3D registration efficiently and globally optimally. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1457–1464. IEEE (2013)Google Scholar
  54. 54.
    Ye, G., Deng, Y., Hasler, N., Ji, X., Dai, Q., Theobalt, C.: Free-viewpoint video of human actors using multiple handheld kinects. IEEE Trans. Cybern. 43(5), 1370–1382 (2013)CrossRefGoogle Scholar
  55. 55.
    Zhang, Z.: Iterative point matching for registration of free-form curves and surfaces. IJCV 13(2), 119–152 (1994)CrossRefGoogle Scholar
  56. 56.
    Zollhöfer, M., Nießner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., Stamminger, M.: Real-time non-rigid reconstruction using an rgb-d camera. In: ACM SIGGRAPH, vol. 33, pp. 156:1–156:12. ACM, New York, July 2014Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Ruizhe Wang
    • 1
  • Lingyu Wei
    • 1
  • Etienne Vouga
    • 2
  • Qixing Huang
    • 2
    • 3
  • Duygu Ceylan
    • 4
  • Gérard Medioni
    • 1
  • Hao Li
    • 1
  1. 1.University of Southern CaliforniaLos AngelesUSA
  2. 2.University of Texas at AustinAustinUSA
  3. 3.Toyota Technological Institute at ChicagoChicagoUSA
  4. 4.Adobe ResearchSan JoseUSA

Personalised recommendations