Reconstructing Articulated Rigged Models from RGB-D Videos

  • Dimitrios Tzionas
  • Juergen Gall
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9915)


Although commercial and open-source software exist to reconstruct a static object from a sequence recorded with an RGB-D sensor, there is a lack of tools that build rigged models of articulated objects that deform realistically and can be used for tracking or animation. In this work, we fill this gap and propose a method that creates a fully rigged model of an articulated object from depth data of a single sensor. To this end, we combine deformable mesh tracking, motion segmentation based on spectral clustering and skeletonization based on mean curvature flow. The fully rigged model then consists of a watertight mesh, embedded skeleton, and skinning weights.


Kinematic model learning Skeletonization Rigged model acquisition Deformable tracking Spectral clustering Mean curvature flow 



The authors acknowledge financial support by the DFG Emmy Noether program (GA 1927/1-1).


  1. 1.
    Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohli, P., Shotton, J., Hodges, S., Fitzgibbon, A.: Kinectfusion: real-time dense surface mapping and tracking. In: International Symposium on Mixed and Augmented Reality (ISMAR) (2011)Google Scholar
  2. 2.
    Sturm, J., Bylow, E., Kahl, F., Cremers, D.: Copyme3d: Scanning and printing persons in 3d. In: German Conference on Pattern Recognition (GCPR) (2013)Google Scholar
  3. 3.
    Baran, I., Popović, J.: Automatic rigging and animation of 3d characters. ACM Trans. Graph. (TOG) 26(3), 72 (2007)CrossRefGoogle Scholar
  4. 4.
    Pillai, S., Walter, M.R., Teller, S.: Learning articulated motions from visual demonstration. In: Robotics: Science and Systems (RSS) (2014)Google Scholar
  5. 5.
    Stoll, C., Gall, J., de Aguiar, E., Thrun, S., Theobalt, C.: Video-based reconstruction of animatable human characters. ACM Trans. Graph. (TOG) 29(6), 139: 1–139: 10 (2010)CrossRefGoogle Scholar
  6. 6.
    De Aguiar, E., Theobalt, C., Thrun, S., Seidel, H.P.: Automatic conversion of mesh animations into skeleton-based animations. Comput. Graph. Forum (CGF) 27(2), 389–397 (2008)CrossRefGoogle Scholar
  7. 7.
    Liu, Y., Gall, J., Stoll, C., Dai, Q., Seidel, H.P., Theobalt, C.: Markerless motion capture of multiple characters using multiview image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 35(11), 2720–2735 (2013)CrossRefGoogle Scholar
  8. 8.
    Yan, J., Pollefeys, M.: Automatic kinematic chain building from feature trajectories of articulated objects. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2006)Google Scholar
  9. 9.
    Yan, J., Pollefeys, M.: A factorization-based approach for articulated nonrigid shape, motion and kinematic chain recovery from video. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 30(5), 865–877 (2008)CrossRefGoogle Scholar
  10. 10.
    Ross, D.A., Tarlow, D., Zemel, R.S.: Learning articulated structure and motion. Int. J. Comput. Vis. (IJCV) 88(2), 214–237 (2010)CrossRefGoogle Scholar
  11. 11.
    Chang, H.J., Demiris, Y.: Unsupervised learning of complex articulated kinematic structures combining motion and skeleton information. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  12. 12.
    Fayad, J., Russell, C., Agapito, L.: Automated articulated structure and 3d shape recovery from point correspondences. In: International Conference on Computer Vision (ICCV) (2011)Google Scholar
  13. 13.
    Sturm, J., Pradeep, V., Stachniss, C., Plagemann, C., Konolige, K., Burgard, W.: Learning kinematic models for articulated objects. In: International Joint Conference on Artificial Intelligence (IJCAI) (2009)Google Scholar
  14. 14.
    Sturm, J., Stachniss, C., Burgard, W.: A probabilistic framework for learning kinematic models of articulated objects. J. Artif. Intell. Res. (JAIR) 41(2), 477–626 (2011)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Yücer, K., Wang, O., Sorkine-Hornung, A., Sorkine-Hornung, O.: Reconstruction of articulated objects from a moving camera. In: ICCVW (2015)Google Scholar
  16. 16.
    Katz, D., Kazemi, M., Bagnell, A.J., Stentz, A.: Interactive segmentation, tracking, and kinematic modeling of unknown 3d articulated objects. In: IEEE International Conference on Robotics and Automation (ICRA) (2013)Google Scholar
  17. 17.
    Martín-Martín, R., Höfer, S., Brock, O.: An integrated approach to visual perception of articulated objects. In: IEEE International Conference on Robotics and Automation (ICRA) (2016)Google Scholar
  18. 18.
    Tresadern, P., Reid, I.: Articulated structure from motion by factorization. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2005)Google Scholar
  19. 19.
    Skanect: Accessed 19 Aug 2016
  20. 20.
    MeshLab: Accessed 19 Aug 2016
  21. 21.
    Jones, M.J., Rehg, J.M.: Statistical color models with application to skin detection. Int. J. Comput. Vis. (IJCV) 46(1), 81–96 (2002)CrossRefzbMATHGoogle Scholar
  22. 22.
    Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. Int. J. Comput. Vis. (IJCV) 81(1), 24–52 (2009)CrossRefGoogle Scholar
  23. 23.
    Holzer, S., Rusu, R.B., Dixon, M., Gedikli, S., Navab, N.: Adaptive neighborhood selection for real-time surface normal estimation from organized point cloud data using integral images. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2012)Google Scholar
  24. 24.
    Botsch, M., Sorkine, O.: On linear variational surface deformation methods. IEEE Trans. Vis. Comput. Graph. (TVCG) 14(1), 213–230 (2008)CrossRefGoogle Scholar
  25. 25.
    Gall, J., Stoll, C., De Aguiar, E., Theobalt, C., Rosenhahn, B., Seidel, H.P.: Motion capture using joint skeleton tracking and surface estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2009)Google Scholar
  26. 26.
    Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., Gall, J.: Capturing hands in action using discriminative salient points and physics simulation. Int. J. Comput. Vis. (IJCV) 118, 172–193 (2016)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Pons-Moll, G., Rosenhahn, B.: Model-based pose estimation. In: Moeslund, T.B., Hilton, A., Krüger, V., Sigal, L. (eds.) Visual Analysis of Humans: Looking at People, pp. 139–170. Springer, London (2011). doi: 10.1007/978-0-85729-997-0_9 CrossRefGoogle Scholar
  28. 28.
    Brox, T., Malik, J.: Object segmentation by long term analysis of point trajectories. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6315, pp. 282–295. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15555-0_21 CrossRefGoogle Scholar
  29. 29.
    Ng, A.Y., Jordan, M.I., Weiss, Y.: On spectral clustering: analysis and an algorithm. In: Advances in Neural Information Processing Systems NIPS (2002)Google Scholar
  30. 30.
    Tagliasacchi, A., Alhashim, I., Olson, M., Zhang, H.: Mean curvature skeletons. Comput. Graph. Forum (CGF) 31, 1735–1744 (2012)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.University of BonnBonnGermany
  2. 2.MPI for Intelligent SystemsTübingenGermany

Personalised recommendations