NASA Neural Articulated Shape Approximation

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12352)


Efficient representation of articulated objects such as human bodies is an important problem in computer vision and graphics. To efficiently simulate deformation, existing approaches represent 3D objects using polygonal meshes and deform them using skinning techniques. This paper introduces neural articulated shape approximation (NASA), an alternative framework that enables representation of articulated deformable objects using neural indicator functions that are conditioned on pose. Occupancy testing using NASA is straightforward, circumventing the complexity of meshes and the issue of water-tightness. We demonstrate the effectiveness of NASA for 3D tracking applications, and discuss other potential extensions.


3D deep learning Neural object representation Articulated objects Deformation Skinning Occupancy Neural implicit functions 

Supplementary material

Supplementary material 1 (mp4 25693 KB)

504444_1_En_36_MOESM2_ESM.pdf (722 kb)
Supplementary material 2 (pdf 721 KB)


  1. 1.
    Anguelov, D., Koller, D., Pang, H.C., Srinivasan, P., Thrun, S.: Recovering articulated object models from 3D range data. In: Uncertainty in Artificial Intelligence (2004)Google Scholar
  2. 2.
    Atzmon, M., Lipman, Y.: SAL: sign agnostic learning of shapes from raw data. arXiv preprint arXiv:1911.10414 (2019)
  3. 3.
    Bailey, S.W., Otte, D., Dilorenzo, P., O’Brien, J.F.: Fast and deep deformation approximations. In: SIGGRAPH (2018)Google Scholar
  4. 4.
    Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., Black, M.J.: Keep It SMPL: automatic estimation of 3D human pose and shape from a single image. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 561–578. Springer, Cham (2016). Scholar
  5. 5.
    Bogo, F., Romero, J., Pons-Moll, G., Black, M.J.: Dynamic FAUST: registering human bodies in motion. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  6. 6.
    Bouaziz, S., Tagliasacchi, A., Pauly, M.: Sparse iterative closest point. In: SGP (2013)Google Scholar
  7. 7.
    Chang, A.X., et al.: Shapenet: an information-rich 3D model repository. arXiv:1512.03012 (2015)
  8. 8.
    Chen, Z., Yin, K., Fisher, M., Chaudhuri, S., Zhang, H.: Bae-net: branched autoencoder for shape co-segmentation. In: ICCV (2019)Google Scholar
  9. 9.
    Chen, Z., Zhang, H.: Learning implicit fields for generative shape modeling. In: CVPR (2019)Google Scholar
  10. 10.
    Chibane, J., Alldieck, T., Pons-Moll, G.: Implicit functions in feature space for 3D shape reconstruction and completion. In: CVPR (2020)Google Scholar
  11. 11.
    Crane, K., Weischedel, C., Wardetzky, M.: Geodesics in heat: a new approach to computing distance based on heat flow. ACM TOG 32(5), 1–11 (2013)CrossRefGoogle Scholar
  12. 12.
    Deng, B., Genova, K., Yazdani, S., Bouaziz, S., Hinton, G., Tagliasacchi, A.: Cvxnet: learnable convex decomposition. In: CVPR (2020)Google Scholar
  13. 13.
    Deng, B., Kornblith, S., Hinton, G.: Cerberus: a multi-headed derenderer. arXiv:1905.11940 (2019)
  14. 14.
    Dou, M., et al.: Fusion4d: real-time performance capture of challenging scenes. ACM TOG 35(4), 1–13 (2016)CrossRefGoogle Scholar
  15. 15.
    Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3D object reconstruction from a single image. In: CVPR (2017)Google Scholar
  16. 16.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Distance transforms of sampled functions. Theory Comput. 8(1), 415–428 (2012)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Gao, L., et al.: SDM-NET: deep generative network for structured deformable mesh. ACM TOG 38(6), 1–15 (2019)Google Scholar
  18. 18.
    Genova, K., Cole, F., Sud, A., Sarna, A., Funkhouser, T.: Deep structured implicit functions. In: CVPR (2019)Google Scholar
  19. 19.
    de Goes, F., Goldenstein, S., Velho, L.: A hierarchical segmentation of articulated bodies. In: SGP (2008)Google Scholar
  20. 20.
    Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: Atlasnet: a papier-mâché approach to learning 3D surface generation. arXiv preprint arXiv:1802.05384 (2018)
  21. 21.
    Huang, Q., Koltun, V., Guibas, L.: Joint shape segmentation with linear programming. ACM TOG (2011)Google Scholar
  22. 22.
    Ioannou, Y., Robertson, D., Cipolla, R., Criminisi, A.: Deep roots: improving CNN efficiency with hierarchical filter groups. In: CVPR (2017)Google Scholar
  23. 23.
    Jacobson, A., Deng, Z., Kavan, L., Lewis, J.: Skinning: real-time shape deformation. In: ACM SIGGRAPH Courses (2014)Google Scholar
  24. 24.
    Jacobson, A., Kavan, L., Sorkine-Hornung, O.: Robust inside-outside segmentation using generalized winding numbers. ACM TOG 32(4), 1–12 (2013)CrossRefGoogle Scholar
  25. 25.
    James, D.L.: Twigg, C.D.: Skinning mesh animations. In: SIGGRAPH (2005)Google Scholar
  26. 26.
    Joseph-Rivlin, M., Zvirin, A., Kimmel, R.: Momen(e)t: flavor the moments in learning to classify shapes. In: CVPR Workshops (2019)Google Scholar
  27. 27.
    Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: CVPR (2018)Google Scholar
  28. 28.
    Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  29. 29.
    Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)Google Scholar
  30. 30.
    Le, B.H., Deng, Z.: Smooth skinning decomposition with rigid bones. ACM TOG 31(6), 1–10 (2012)CrossRefGoogle Scholar
  31. 31.
    Lewis, J.P., Cordner, M., Fong, N.: Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation. In: SIGGRAPH (2000)Google Scholar
  32. 32.
    Lin, M.C., Manocha, U.D., Cohen, J.: Collision detection: algorithms and applications (1996)Google Scholar
  33. 33.
    Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. In: SIGGRAPH Asia (2015)Google Scholar
  34. 34.
    Lorenz, D., Bereska, L., Milbich, T., Ommer, B.: Unsupervised part-based disentangling of object shape and appearance. arXiv:1903.06946 (2019)
  35. 35.
    Mahmood, N., Ghorbani, N., Troje, N.F., Pons-Moll, G., Black, M.J.: AMASS: archive of motion capture as surface shapes. In: ICCV (2019)Google Scholar
  36. 36.
    Melax, S., Keselman, L., Orsten, S.: Dynamics based 3D skeletal hand tracking. In: Graphics Interface (2013)Google Scholar
  37. 37.
    Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: learning 3D reconstruction in function space. arXiv:1812.03828 (2018)
  38. 38.
    Omran, M., Lassner, C., Pons-Moll, G., Gehler, P., Schiele, B.: Neural body fitting: unifying deep learning and model based human pose and shape estimation. In: International Conference on 3D Vision (3DV) (2018)Google Scholar
  39. 39.
    Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: CVPR (2019)Google Scholar
  40. 40.
    Pavlakos, G., Zhu, L., Zhou, X., Daniilidis, K.: Learning to estimate 3D human pose and shape from a single color image. In: CVPR (2018)Google Scholar
  41. 41.
    Remelli, E., Tkach, A., Tagliasacchi, A., Pauly, M.: Low-dimensionality calibration through local anisotropic scaling for robust hand model personalization. In: ICCV (2017)Google Scholar
  42. 42.
    Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: PIFu: pixel-aligned implicit function for high-resolution clothed human digitization. In: CVPR (2019)Google Scholar
  43. 43.
    Samet, H.: Applications of Spatial Data Structures: Computer Graphics, Image Processing, and GIS. Addison-Wesley Longman Publishing Co., Inc., Boston (1990)Google Scholar
  44. 44.
    Shen, J., et al.: The phong surface: efficient 3D model fitting using lifted optimization (2020)Google Scholar
  45. 45.
    Shotton, J., et al.: Real-time human pose recognition in parts from single depth images. In: CVPR (2011)Google Scholar
  46. 46.
    Sitzmann, V., Martel, J.N.P., Bergman, A.W., Lindell, D.B., Wetzstein, G.: Implicit neural representations with periodic activation functions (2020)Google Scholar
  47. 47.
    Tagliasacchi, A., Bouaziz, S.: Dynamic 2D/3D registration. In: Proceedings of Symposium on Geometry Processing (Technical Course Notes) (2018)Google Scholar
  48. 48.
    Tagliasacchi, A., Schröder, M., Tkach, A., Bouaziz, S., Botsch, M., Pauly, M.: Robust articulated-ICP for real-time hand tracking. In: SGP (2015)Google Scholar
  49. 49.
    Tancik, M., et al.: Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006.10739 (2020)
  50. 50.
    Tatarchenko, M., Richter, S.R., Ranftl, R., Li, Z., Koltun, V., Brox, T.: What do single-view 3D reconstruction networks learn? In: CVPR (2019)Google Scholar
  51. 51.
    Taylor, J., et al.: Articulated distance fields for ultra-fast tracking of hands interacting. ACM Trans. Graph. (TOG) 36(6), 1–12 (2017)CrossRefGoogle Scholar
  52. 52.
    Tkach, A., Tagliasacchi, A., Remelli, E., Pauly, M., Fitzgibbon, A.: Online generative model personalization for hand tracking. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 36(6), 1–11 (2017)Google Scholar
  53. 53.
    Tung, H.Y., Tung, H.W., Yumer, E., Fragkiadaki, K.: Self-supervised learning of motion capture. In: Advances in Neural Information Processing Systems, pp. 5236–5246 (2017)Google Scholar
  54. 54.
    Valentin, J., Keskin, C., Pidlypenskyi, P., Makadia, A., Sud, A., Bouaziz, S.: Tensorflow graphics: computer graphics meets deep learning (2019)Google Scholar
  55. 55.
    Wang, H., Sridhar, S., Huang, J., Valentin, J., Song, S., Guibas, L.J.: Normalized object coordinate space for category-level 6D object pose and size estimation. In: CVPR (2019)Google Scholar
  56. 56.
    Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: DISN: deep implicit surface network for high-quality single-view 3D reconstruction. In: NeurIPS (2019)Google Scholar
  57. 57.
    Zhou, Y., Barnes, C., Lu, J., Yang, J., Li, H.: On the continuity of rotation representations in neural networks. In: CVPR (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Google ResearchMountain ViewUSA
  2. 2.MPI for Informatics, Saarland Informatics CampusSaarbrückenGermany
  3. 3.University of TorontoTorontoCanada

Personalised recommendations