Advertisement

PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations

Conference paper
  • 886 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12361)

Abstract

Implicit surface representations, such as signed-distance functions, combined with deep learning have led to impressive models which can represent detailed shapes of objects with arbitrary topology. Since a continuous function is learned, the reconstructions can also be extracted at any arbitrary resolution. However, large datasets such as ShapeNet are required to train such models.

In this paper, we present a new mid-level patch-based surface representation. At the level of patches, objects across different categories share similarities, which leads to more generalizable models. We then introduce a novel method to learn this patch-based representation in a canonical space, such that it is as object-agnostic as possible. We show that our representation trained on one category of objects from ShapeNet can also well represent detailed shapes from any other category. In addition, it can be trained using much fewer shapes, compared to existing approaches. We show several applications of our new representation, including shape interpolation and partial point cloud completion. Due to explicit control over positions, orientations and scales of patches, our representation is also more controllable compared to object-level representations, which enables us to deform encoded shapes non-rigidly.

Keywords

Implicit functions Patch-based surface representation Intra-object class generalizability 

Notes

Acknowledgements

This work was supported by the ERC Consolidator Grant 4DReply (770784), and an Oculus research grant.

Supplementary material

504471_1_En_18_MOESM1_ESM.pdf (2.5 mb)
Supplementary material 1 (pdf 2556 KB)

Supplementary material 2 (mp4 75507 KB)

References

  1. 1.
    Atzmon, M., Lipman, Y.: Sal: sign agnostic learning of shapes from raw data. In: Computer Vision and Pattern Recognition (CVPR) (2020)Google Scholar
  2. 2.
    Bogo, F., Romero, J., Pons-Moll, G., Black, M.J.: Dynamic FAUST: registering human bodies in motion. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  3. 3.
    Chang, A.X., et al.: ShapeNet: an information-rich 3D model repository. arXiv preprint arXiv:1512.03012 (2015)
  4. 4.
    Chen, Z., Zhang, H.: Learning implicit fields for generative shape modeling. In: Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  5. 5.
    Choy, C.B., Xu, D., Gwak, J.Y., Chen, K., Savarese, S.: 3D-R2N2: a unified approach for single and multi-view 3D object reconstruction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 628–644. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46484-8_38CrossRefGoogle Scholar
  6. 6.
    Curless, B., Levoy, M.: A volumetric method for building complex models from range images. In: SIGGRAPH (1996)Google Scholar
  7. 7.
    Deng, B., Genova, K., Yazdani, S., Bouaziz, S., Hinton, G., Tagliasacchi, A.: CvxNets: learnable convex decomposition. In: Advances in Neural Information Processing Systems Workshops (2019)Google Scholar
  8. 8.
    Deng, B., Lewis, J., Jeruzalski, T., Pons-Moll, G., Hinton, G., Norouzi, M., Tagliasacchi, A.: Nasa: neural articulated shape approximation (2020)Google Scholar
  9. 9.
    Deprelle, T., Groueix, T., Fisher, M., Kim, V., Russell, B., Aubry, M.: Learning elementary structures for 3D shape generation and matching. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)Google Scholar
  10. 10.
    Genova, K., Cole, F., Sud, A., Sarna, A., Funkhouser, T.: Local deep implicit functions for 3D shape. In: Computer Vision and Pattern Recognition (CVPR) (2020)Google Scholar
  11. 11.
    Genova, K., Cole, F., Vlasic, D., Sarna, A., Freeman, W.T., Funkhouser, T.: Learning shape templates with structured implicit functions. In: International Conference on Computer Vision (ICCV) (2019)Google Scholar
  12. 12.
    Groueix, T., Fisher, M., Kim, V., Russell, B., Aubry, M.: A papier-mache approach to learning 3D surface generation. In: Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  13. 13.
    Kato, H., Ushiku, Y., Harada, T.: Neural 3D mesh renderer. In: Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  14. 14.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)Google Scholar
  15. 15.
    Liu, S., Saito, S., Chen, W., Li, H.: Learning to infer implicit surfaces without 3D supervision. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)Google Scholar
  16. 16.
    Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. In: Conference on Computer Graphics and Interactive Techniques (1987)Google Scholar
  17. 17.
    Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: learning 3D reconstruction in function space. In: Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  18. 18.
    Michalkiewicz, M., Pontes, J.K., Jack, D., Baktashmotlagh, M., Eriksson, A.: Implicit surface representations as layers in neural networks. In: International Conference on Computer Vision (ICCV) (2019)Google Scholar
  19. 19.
    Niemeyer, M., Mescheder, L., Oechsle, M., Geiger, A.: Occupancy flow: 4D reconstruction by learning particle dynamics. In: International Conference on Computer Vision (CVPR) (2019)Google Scholar
  20. 20.
    Ohtake, Y., Belyaev, A., Alexa, M., Turk, G., Seidel, H.P.: Multi-level partition of unity implicits. In: ACM Transactions on Graphics (TOG) (2003)Google Scholar
  21. 21.
    Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  22. 22.
    Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)Google Scholar
  23. 23.
    Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems (NeurIPS) (2017)Google Scholar
  24. 24.
    Riegler, G., Osman Ulusoy, A., Geiger, A.: OctNet: learning deep 3D representations at high resolutions. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  25. 25.
    Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: PIFu: pixel-aligned implicit function for high-resolution clothed human digitization. In: International Conference on Computer Vision (ICCV) (2019)Google Scholar
  26. 26.
    Salimans, T., Kingma, D.P.: Weight normalization: a simple reparameterization to accelerate training of deep neural networks. In: Advances in Neural Information Processing Systems (NeurIPS) (2016)Google Scholar
  27. 27.
    Shimada, S., Golyanik, V., Tretschk, E., Stricker, D., Theobalt, C.: DispVoxNets: non-rigid point set alignment with supervised learning proxies. In: International Conference on 3D Vision (3DV) (2019)Google Scholar
  28. 28.
    Sitzmann, V., Zollhöfer, M., Wetzstein, G.: Scene representation networks: continuous 3D-structure-aware neural scene representations. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)Google Scholar
  29. 29.
    Stutz, D., Geiger, A.: Learning 3D shape completion under weak supervision. Int. J. Comput. Vision (IJCV) 31, 1–10 (2018)Google Scholar
  30. 30.
    Tretschk, E., Tewari, A., Zollhöfer, M., Golyanik, V., Theobalt, C.: DEMEA: deep mesh autoencoders for non-rigidly deforming objects. In: European Conference on Computer Vision (ECCV) (2020)Google Scholar
  31. 31.
    Tulsiani, S., Su, H., Guibas, L.J., Efros, A.A., Malik, J.: Learning shape abstractions by assembling volumetric primitives. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  32. 32.
    Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.-G.: Pixel2Mesh: generating 3D mesh models from single RGB images. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 55–71. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01252-6_4CrossRefGoogle Scholar
  33. 33.
    Williams, F., Parent-Levesque, J., Nowrouzezahrai, D., Panozzo, D., Moo Yi, K., Tagliasacchi, A.: Voronoinet: general functional approximators with local support. In: Computer Vision and Pattern Recognition Workshops (CVPRW) (2020)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Max Planck Institute for Informatics, Saarland Informatics CampusSaarbrückenGermany
  2. 2.Facebook Reality LabsPittsburghUSA

Personalised recommendations