Advertisement

DEMEA: Deep Mesh Autoencoders for Non-rigidly Deforming Objects

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12349)

Abstract

Mesh autoencoders are commonly used for dimensionality reduction, sampling and mesh modeling. We propose a general-purpose DEep MEsh Autoencoder (DEMEA) which adds a novel embedded deformation layer to a graph-convolutional mesh autoencoder. The embedded deformation layer (EDL) is a differentiable deformable geometric proxy which explicitly models point displacements of non-rigid deformations in a lower dimensional space and serves as a local rigidity regularizer. DEMEA decouples the parameterization of the deformation from the final mesh resolution since the deformation is defined over a lower dimensional embedded deformation graph. We perform a large-scale study on four different datasets of deformable objects. Reasoning about the local rigidity of meshes using EDL allows us to achieve higher-quality results for highly deformable objects, compared to directly regressing vertex positions. We demonstrate multiple applications of DEMEA, including non-rigid 3D reconstruction from depth and shading cues, non-rigid surface tracking, as well as the transfer of deformations over different meshes.

Keywords

Auto-encoding Embedded deformation Non-rigid tracking 

Notes

Acknowledgments

This work was supported by the ERC Consolidator Grant 4DReply (770784), the Max Planck Center for Visual Computing and Communications (MPC-VCC), and an Oculus research grant.

Supplementary material

504439_1_En_35_MOESM1_ESM.pdf (5.6 mb)
Supplementary material 1 (pdf 5774 KB)

References

  1. 1.
    Abadi, M., et al.: TensorFlow: Large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/
  2. 2.
    Bagautdinov, T., Wu, C., Saragih, J., Sheikh, Y., Fua, P.: Modeling facial geometry using compositional vaes (2018)Google Scholar
  3. 3.
    Bednařík, J., Fua, P., Salzmann, M.: Learning to reconstruct texture-less deformable surfaces. In: International Conference on 3D Vision (3DV) (2018)Google Scholar
  4. 4.
    Bogo, F., Romero, J., Pons-Moll, G., Black, M.J.: Dynamic FAUST: registering human bodies in motion. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  5. 5.
    Boscaini, D., Masci, J., Rodoià, E., Bronstein, M.: Learning shape correspondence with anisotropic convolutional neural networks. In: International Conference on Neural Information Processing Systems (NIPS) (2016)Google Scholar
  6. 6.
    Bouritsas, G., Bokhnyak, S., Ploumpis, S., Bronstein, M., Zafeiriou, S.: Neural 3D morphable models: spiral convolutional networks for 3D shape representation learning and generation. In: International Conference on Computer Vision (ICCV) (2019)Google Scholar
  7. 7.
    Bruna, J., Zaremba, W., Szlam, A., LeCun, Y.: Spectral networks and locally connected networks on graphs. CoRR abs/1312.6203 (2013)Google Scholar
  8. 8.
    Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., Ranzuglia, G.: MeshLab: an open-source mesh processing tool. In: Scarano, V., Chiara, R.D., Erra, U. (eds.) Eurographics Italian Chapter Conference. The Eurographics Association (2008).  https://doi.org/10.2312/LocalChapterEvents/ItalChap/ItalianChapConf2008/129-136
  9. 9.
    Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional neural networks on graphs with fast localized spectral filtering. In: International Conference on Neural Information Processing Systems (NIPS) (2016)Google Scholar
  10. 10.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Computer Vision and Pattern Recognition (CVPR) (2009)Google Scholar
  11. 11.
    Fuentes-Jimenez, D., Casillas-Perez, D., Pizarro, D., Collins, T., Bartoli, A.: Deep Shape-from-Template: Wide-Baseline, Dense and Fast Registration and Deformable Reconstruction from a Single Image. arXiv e-prints (2018)Google Scholar
  12. 12.
    Gao, L., Yang, J., Qiao, Y.L., Lai, Y.K., Rosin, P.L., Xu, W., Xia, S.: Automatic unpaired shape deformation transfer. ACM Trans. Graph. (TOG) 37(6), 1–15 (2018)CrossRefGoogle Scholar
  13. 13.
    Garland, M., Heckbert, P.S.: Surface simplification using quadric error metrics. In: ACM SIGGRAPH (1997)Google Scholar
  14. 14.
    Golyanik, V., Shimada, S., Varanasi, K., Stricker, D.: Hdm-net: monocular non-rigid 3D reconstruction with learned deformation model. In: International Conference on Virtual Reality and Augmented Reality (EuroVR) (2018)Google Scholar
  15. 15.
    Groueix, T., Fisher, M., Kim, V.G., Russell, B., Aubry, M.: AtlasNet: a Papier-Mâché approach to learning 3D surface generation. In: Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  16. 16.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  17. 17.
    Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Jack, D., et al.: Learning free-form deformations for 3D object reconstruction. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11362, pp. 317–333. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-20890-5_21CrossRefGoogle Scholar
  19. 19.
    Kanazawa, A., Tulsiani, S., Efros, A.A., Malik, J.: Learning category-specific mesh reconstruction from image collections. In: European Conference on Computer Vision (ECCV) (2018)Google Scholar
  20. 20.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)Google Scholar
  21. 21.
    Kurenkov, A., et al.: Deformnet: Free-form deformation network for 3D shape reconstruction from a single image. In: Winter Conference on Applications of Computer Vision (WACV) (2018)Google Scholar
  22. 22.
    Li, H., Adams, B., Guibas, L.J., Pauly, M.: Robust single-view geometry and motion reconstruction. In: ACM SIGGRAPH Asia (2009)Google Scholar
  23. 23.
    Litany, O., Bronstein, A., Bronstein, M., Makadia, A.: Deformable shape completion with graph convolutional autoencoders. In: Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  24. 24.
    Loper, M., Mahmood, N., Black, M.J.: Mosh: motion and shape capture from sparse markers. ACM Trans. Graph. (TOG) 33, 1–13 (2014)CrossRefGoogle Scholar
  25. 25.
    Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: a skinned multi-person linear model. ACM Trans. Graph. (TOG) 34, 1–16 (2015)CrossRefGoogle Scholar
  26. 26.
    Malik, J., et al.: Deephps: end-to-end estimation of 3D hand pose and shape by learning from synthetic depth. In: International Conference on 3D Vision (3DV) (2018)Google Scholar
  27. 27.
    Masci, J., Boscaini, D., Bronstein, M.M., Vandergheynst, P.: Geodesic convolutional neural networks on riemannian manifolds. In: International Conference on Computer Vision Workshop (ICCVW) (2015)Google Scholar
  28. 28.
    Monti, F., Boscaini, D., Masci, J., Rodola, E., Svoboda, J., Bronstein, M.M.: Geometric deep learning on graphs and manifolds using mixture model CNNs. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  29. 29.
    Niepert, M., Ahmed, M., Kutzkov, K.: Learning convolutional neural networks for graphs. In: International Conference on Machine Learning (ICML) (2016)Google Scholar
  30. 30.
    Pumarola, A., Agudo, A., Porzi, L., Sanfeliu, A., Lepetit, V., Moreno-Noguer, F.: Geometry-aware network for non-rigid shape prediction from a single view. In: Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  31. 31.
    Ranjan, A., Bolkart, T., Sanyal, S., Black, M.J.: Generating 3D faces using convolutional mesh autoencoders. In: European Conference on Computer Vision (ECCV) (2018)Google Scholar
  32. 32.
    Shimada, S., Golyanik, V., Theobalt, C., Stricker, D.: IsMo-GAN: adversarial learning for monocular non-rigid 3D reconstruction. In: Computer Vision and Pattern Recognition Workshops (CVPRW) (2019)Google Scholar
  33. 33.
    Sinha, A., Unmesh, A., Huang, Q., Ramani, K.: Surfnet: generating 3D shape surfaces using deep residual networks. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  34. 34.
    Sorkine, O., Alexa, M.: As-rigid-as-possible surface modeling. In: Eurographics Symposium on Geometry Processing (SGP) (2007)Google Scholar
  35. 35.
    Sumner, R.W., Schmid, J., Pauly, M.: Embedded deformation for shape manipulation. In: ACM SIGGRAPH (2007)Google Scholar
  36. 36.
    Tan, Q., Gao, L., Lai, Y.K., Xia, S.: Variational autoencoders for deforming 3D mesh models. In: Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  37. 37.
    Tan, Q., Gao, L., Lai, Y.K., Yang, J., Xia, S.: Mesh-based autoencoders for localized deformation component analysis. In: AAAI Conference on Artificial Intelligence (AAAI) (2018)Google Scholar
  38. 38.
    Verma, N., Boyer, E., Verbeek, J.: FeaStNet: Feature-steered graph convolutions for 3D shape analysis. In: Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  39. 39.
    Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.G.: Pixel2mesh: generating 3D mesh models from single RGB images. In: European Conference on Computer Vision (ECCV) (2018)Google Scholar
  40. 40.
    Yi, L., Su, H., Guo, X., Guibas, L.: Syncspeccnn: synchronized spectral CNN for 3D shape segmentation. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  41. 41.
    Zollhöfer, M., et al.: Real-time non-rigid reconstruction using an RGB-D camera. ACM Trans. Graph. (TOG) 33, 1–12 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Max Planck Institute for InformaticsSaarbrückenGermany
  2. 2.Stanford UniversityStanfordUSA

Personalised recommendations