Nonrigid Surface Registration and Completion from RGBD Images

  • Weipeng Xu
  • Mathieu Salzmann
  • Yongtian Wang
  • Yue Liu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8690)


Nonrigid surface registration is a challenging problem that suffers from many ambiguities. Existing methods typically assume the availability of full volumetric data, or require a global model of the surface of interest. In this paper, we introduce an approach to nonrigid registration that performs on relatively low-quality RGBD images and does not assume prior knowledge of the global surface shape. To this end, we model the surface as a collection of patches, and infer the patch deformations by performing inference in a graphical model. Our representation lets us fill in the holes in the input depth maps, thus essentially achieving surface completion. Our experimental evaluation demonstrates the effectiveness of our approach on several sequences, as well as its robustness to missing data and occlusions.


Nonrigid registration surface completion RGBD images 

Supplementary material

978-3-319-10605-2_5_MOESM1_ESM.mpg (379 kb)
Electronic Supplementary Material (MPG 380 KB)
978-3-319-10605-2_5_MOESM2_ESM.mpg (438 kb)
Electronic Supplementary Material (MPG 439 KB)
978-3-319-10605-2_5_MOESM3_ESM.mpg (1.1 mb)
Electronic Supplementary Material (MPG 1,108 KB)
978-3-319-10605-2_5_MOESM4_ESM.mpg (1.8 mb)
Electronic Supplementary Material (MPG 1,825 KB)
978-3-319-10605-2_5_MOESM5_ESM.mpg (1.8 mb)
Electronic Supplementary Material (MPG 1,841 KB)
978-3-319-10605-2_5_MOESM6_ESM.mpg (1.1 mb)
Electronic Supplementary Material (MPG 1,158 KB)


  1. 1.
    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(11), 2274–2282 (2012)CrossRefGoogle Scholar
  2. 2.
    Bartoli, A., Pizarro, D., Collins, T.: A robust analytical solution to isometric shape-from-template with focal length calibration. In: ICCV (2013)Google Scholar
  3. 3.
    Bronstein, A.M., Bronstein, M.M., Kimmel, R., Mahmoudi, M., Sapiro, G.: A Gromov-Hausdorff Framework with Diffusion Geometry for Topologically-Robust Non-Rigid Shape Matching. IJCV (2010)Google Scholar
  4. 4.
    Cagniart, C., Boyer, E., Ilic, S.: Free-form mesh tracking: a patch-based approach. In: CVPR (2010)Google Scholar
  5. 5.
    Cui, Y., Schuon, S., Chan, D., Thrun, S., Theobalt, C.: 3d shape scanning with a time-of-flight camera. In: CVPR (2010)Google Scholar
  6. 6.
    Dou, M., Fuchs, H., Frahm, J.M.: Scanning and tracking dynamic objects with commodity depth cameras. In: ISMAR (2013)Google Scholar
  7. 7.
    Endres, F., Hess, J., Engelhard, N., Sturm, J., Cremers, D., Burgard, W.: An evaluation of the rgb-d slam system. In: ICRA (2012)Google Scholar
  8. 8.
    Gall, J., Stoll, C., de Aguiar, E., Theobalt, C., Rosenhahn, B., Seidel, H.P.: Motion capture using joint skeleton tracking and surface estimation. In: CVPR (2009)Google Scholar
  9. 9.
    Garg, R., Roussos, A., Agapito, L.: Dense Variational Reconstruction of Non-Rigid Surfaces from Monocular Video. In: CVPR (2013)Google Scholar
  10. 10.
    Hadfield, S., Bowden, R.: Kinecting the dots: Particle based scene flow from depth sensors. In: ICCV (2011)Google Scholar
  11. 11.
    Herbst, E., Ren, X., Fox, D.: Rgb-d flow: Dense 3-d motion estimation using color and depth. In: ICRA (2013)Google Scholar
  12. 12.
    Hornacek, M., Rhemann, C., Gelautz, M., Rother, C.: Depth super resolution by rigid body self-similarity in 3d. In: CVPR (2013)Google Scholar
  13. 13.
    Huang, P., Budd, C., Hilton, A.: Global temporal registration of multiple non-rigid surface sequences. In: CVPR (2011)Google Scholar
  14. 14.
    Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., Fitzgibbon, A.: Kinectfusion: Real-time 3d reconstruction and interaction using a moving depth camera. In: UIST (2011)Google Scholar
  15. 15.
    Jordt, A., Koch, R.: Fast tracking of deformable objects in depth and colour video. In: BMVC (2011)Google Scholar
  16. 16.
    Jordt, A., Koch, R.: Direct model-based tracking of 3d object deformations in depth and color video. IJCV (2013)Google Scholar
  17. 17.
    Keller, M., Lefloch, D., Lambers, M., Izadi, S., Weyrich, T., Kolb, A.: Real-time 3d reconstruction in dynamic scenes using point-based fusion. In: 3DV (2013)Google Scholar
  18. 18.
    Lempitsky, V., Rother, C., Roth, S., Blake, A.: Fusion moves for markov random field optimization. PAMI (2010)Google Scholar
  19. 19.
    Letouzey, A., Boyer, E.: Progressive shape models. In: CVPR (2012)Google Scholar
  20. 20.
    Li, H., Sumner, R.W., Pauly, M.: Global correspondence optimization for non-rigid registration of depth scans. In: SGP (2008)Google Scholar
  21. 21.
    Mac Aodha, O., Campbell, N.D.F., Nair, A., Brostow, G.J.: Patch based synthesis for single depth image super-resolution. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part III. LNCS, vol. 7574, pp. 71–84. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  22. 22.
    Malti, A., Hartley, R., Bartoli, A., Kim, J.H.: Monocular template-based 3d reconstruction of extensible surfaces with local linear elasticity. In: CVPR (2013)Google Scholar
  23. 23.
    Myronenko, A., Song, X.: Point set registration: Coherent point drift. PAMI (2010)Google Scholar
  24. 24.
    Park, J., Kim, H., Tai, Y.W., Brown, M.S., Kweon, I.: High quality depth map upsampling for 3d-tof cameras. In: ICCV (2011)Google Scholar
  25. 25.
    Rouhani, M., Sappa, A.D.: Non-rigid shape registration: A single linear least squares framework. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VII. LNCS, vol. 7578, pp. 264–277. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  26. 26.
    Russell, C., Fayad, J., Agapito, L.: Energy Based Multiple Model Fitting for NRSFM. In: CVPR (2011)Google Scholar
  27. 27.
    Salzmann, M., Fua, P.: Deformable Surface 3D Reconstruction from Monocular Images. Morgan Kaufmann (2010)Google Scholar
  28. 28.
    Salzmann, M., Fua, P.: Linear Local Deformation Models for Monocular Reconstruction of Deformable Surfaces. PAMI (2011)Google Scholar
  29. 29.
    Santa, Z., Kato, Z.: Correspondence-less non-rigid registration of triangular surface meshes. In: CVPR (2013)Google Scholar
  30. 30.
    Schuon, S., Theobalt, C., Davis, J., Thrun, S.: Lidarboost: Depth superresolution for tof 3d shape scanning. In: CVPR (2009)Google Scholar
  31. 31.
    Taylor, J., Shotton, J., Sharp, T., Fitzgibbon, A.: The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation. In: CVPR (2012)Google Scholar
  32. 32.
    Torresani, L., Hertzmann, A., Bregler, C.: Nonrigid Structure-From-Motion: Estimating Shape and Motion with Hierarchical Priors. PAMI (2008)Google Scholar
  33. 33.
    Ulusoy, A.O., Biris, O., Mundy, J.L.: Dynamic probabilistic volumetric models. In: ICCV (2013)Google Scholar
  34. 34.
    Varol, A., Salzmann, M., Tola, E., Fua, P.: Template-Free Monocular Reconstruction of Deformable Surfaces. In: ICCV (2009)Google Scholar
  35. 35.
    Vicente, S., Agapito, L.: Soft inextensibility constraints for template-free non-rigid reconstruction. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part III. LNCS, vol. 7574, pp. 426–440. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  36. 36.
    Vlasic, D., Baran, I., Matusik, W., Popovic, J.: Articulated mesh animation from multi-view silhouettes. In: SIGGRAPH (2008)Google Scholar
  37. 37.
    Weikersdorfer, D., Gossow, D., Beetz, M.: Depth-adaptive superpixels. In: ICPR (2012)Google Scholar
  38. 38.
    Yang, Q., Yang, R., Davis, J., Nister, D.: Spatial-depth super resolution for range images. In: CVPR (2007)Google Scholar
  39. 39.
    Ye, G., Liu, Y., Hasler, N., Ji, X., Dai, Q., Theobalt, C.: Performance capture of interacting characters with handheld kinects. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part II. LNCS, vol. 7573, pp. 828–841. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  40. 40.
    Zeng, Y., Wang, C., Wang, Y., Gu, X., Samaras, D., Paragios, N.: Dense non-rigid surface registration using high-order graph matching. In: CVPR (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Weipeng Xu
    • 1
    • 2
  • Mathieu Salzmann
    • 2
    • 3
  • Yongtian Wang
    • 1
  • Yue Liu
    • 1
  1. 1.School of OptoelectronicsBeijing Institute of Technology (BIT)China
  2. 2.NICTACanberraAustralia
  3. 3.Australian National University (ANU)Australia

Personalised recommendations