Advertisement

Multimodal Shape Completion via Conditional Generative Adversarial Networks

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12349)

Abstract

Several deep learning methods have been proposed for completing partial data from shape acquisition setups, i.e., filling the regions that were missing in the shape. These methods, however, only complete the partial shape with a single output, ignoring the ambiguity when reasoning the missing geometry. Hence, we pose a multi-modal shape completion problem, in which we seek to complete the partial shape with multiple outputs by learning a one-to-many mapping. We develop the first multimodal shape completion method that completes the partial shape via conditional generative modeling, without requiring paired training data. Our approach distills the ambiguity by conditioning the completion on a learned multimodal distribution of possible results. We extensively evaluate the approach on several datasets that contain varying forms of shape incompleteness, and compare among several baseline methods and variants of our methods qualitatively and quantitatively, demonstrating the merit of our method in completing partial shapes with both diversity and quality.

Keywords

Shape completion Multimodal mapping Conditional generative adversarial network 

Notes

Acknowledgements

We thank the anonymous reviewers for their valuable comments. This work was supported in part by National Key R&D Program of China (2018YFB1403901, 2019YFF0302902) and NSFC (61902007).

Supplementary material

504439_1_En_17_MOESM1_ESM.pdf (2.1 mb)
Supplementary material 1 (pdf 2102 KB)

References

  1. 1.
    Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L.: Learning representations and generative models for 3D point clouds. In: International Conference on Machine Learning (ICML), pp. 40–49 (2018)Google Scholar
  2. 2.
    Alec, R., Luke, M., Soumith, C.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: International Conference on Learning Representations (ICLR) (2016)Google Scholar
  3. 3.
    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International Conference on Machine Learning (ICML), pp. 214–223 (2017)Google Scholar
  4. 4.
    Chang, A.X., et al.: ShapeNet: an Information-Rich 3D Model Repository. Technical report, arXiv:1512.03012 [cs.GR], Stanford University – Princeton University – Toyota Technological Institute at Chicago (2015)
  5. 5.
    Chen, X., Chen, B., Mitra, N.J.: Unpaired point cloud completion on real scans using adversarial training. In: International Conference on Learning Representations (ICLR) (2020)Google Scholar
  6. 6.
    Chen, Z., Zhang, H.: Learning implicit fields for generative shape modeling. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5939–5948 (2019)Google Scholar
  7. 7.
    Dai, A., Ruizhongtai Qi, C., Nießner, M.: Shape completion using 3D-encoder-predictor CNNs and shape synthesis. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5868–5877 (2017)Google Scholar
  8. 8.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 2672–2680 (2014)Google Scholar
  9. 9.
    Groueix, T., Fisher, M., Kim, V.G., Russell, B., Aubry, M.: AtlasNet: a Papier-Mâché approach to learning 3D surface generation. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  10. 10.
    Guerrero, P., Kleiman, Y., Ovsjanikov, M., Mitra, N.J.: PCPNET learning local shape properties from raw point clouds. Comput. Graph. Forum. 37, 75–85 (2018)CrossRefGoogle Scholar
  11. 11.
    Gurumurthy, S., Agrawal, S.: High fidelity semantic shape completion for point clouds using latent optimization, pp. 1099–1108. IEEE (2019)Google Scholar
  12. 12.
    Han, X., Li, Z., Huang, H., Kalogerakis, E., Yu, Y.: High-resolution shape completion using deep neural networks for global structure and local geometry inference. In: International Conference on Computer Vision (ICCV), pp. 85–93 (2017)Google Scholar
  13. 13.
    Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)Google Scholar
  14. 14.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2014)Google Scholar
  15. 15.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4681–4690 (2017)Google Scholar
  16. 16.
    Li, J., Chen, B.M., Hee Lee, G.: SO-Net: self-organizing network for point cloud analysis. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9397–9406 (2018)Google Scholar
  17. 17.
    Li, Y., Bu, R., Sun, M., Wu, W., Di, X., Chen, B.: PointCNN: convolution on X-transformed points. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 820–830 (2018)Google Scholar
  18. 18.
    Liu, M., Sheng, L., Yang, S., Shao, J., Hu, S.M.: Morphing and sampling network for dense point cloud completion. In: Association for the Advancement of Artificial Intelligence (AAAI) (2019)Google Scholar
  19. 19.
    Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: International Conference on Computer Vision (ICCV), pp. 2794–2802 (2017)Google Scholar
  20. 20.
    Mo, K., et al.: PartNet: a large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019Google Scholar
  21. 21.
    Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 165–174 (2019)Google Scholar
  22. 22.
    Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 652–660 (2017)Google Scholar
  23. 23.
    Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 5099–5108 (2017)Google Scholar
  24. 24.
    Sharma, A., Grau, O., Fritz, M.: VConv-DAE: deep volumetric shape learning without object labels. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9915, pp. 236–250. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-49409-8_20CrossRefGoogle Scholar
  25. 25.
    Smolensky, P.: Information processing in dynamical systems: foundations of harmony theory. Technical report. Colorado University at Boulder Department of Computer Science (1986)Google Scholar
  26. 26.
    Stutz, D., Geiger, A.: Learning 3D shape completion under weak supervision. Int. J. Comput. Vis. (IJCV), 1–20 (2018)Google Scholar
  27. 27.
    Su, H., et al.: SplatNet: sparse lattice networks for point cloud processing. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2530–2539 (2018)Google Scholar
  28. 28.
    Thanh Nguyen, D., Hua, B.S., Tran, K., Pham, Q.H., Yeung, S.K.: A field model for repairing 3D shapes. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5676–5684 (2016)Google Scholar
  29. 29.
    Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: International Conference on Machine Learning (ICML), pp. 1096–1103 (2008)Google Scholar
  30. 30.
    Wang, W., Huang, Q., You, S., Yang, C., Neumann, U.: Shape inpainting using 3D generative adversarial network and recurrent convolutional networks. In: International Conference on Computer Vision (ICCV), pp. 2298–2306 (2017)Google Scholar
  31. 31.
    Wu, R., Zhuang, Y., Xu, K., Zhang, H., Chen, B.: PQ-NET: a generative part Seq2Seq network for 3D shapes. arXiv preprint arXiv:1911.10949 (2019)
  32. 32.
    Wu, Z., et al.: 3D ShapeNets: a deep representation for volumetric shapes. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1912–1920 (2015)Google Scholar
  33. 33.
    Yang, B., Rosa, S., Markham, A., Trigoni, N., Wen, H.: 3D object dense reconstruction from a single depth view. arXiv preprint arXiv:1802.00411 1(2), 6 (2018)
  34. 34.
    Yin, K., Huang, H., Cohen-Or, D., Zhang, H.: P2P-NET: bidirectional point displacement net for shape transform. ACM Trans. Graph. (TOG) 37(4), 1–13 (2018)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Yu, L., Li, X., Fu, C.-W., Cohen-Or, D., Heng, P.-A.: EC-Net: an edge-aware point set consolidation network. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 398–414. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01234-2_24CrossRefGoogle Scholar
  36. 36.
    Yu, L., Li, X., Fu, C.W., Cohen-Or, D., Heng, P.A.: PU-Net: point cloud upsampling network. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2790–2799 (2018)Google Scholar
  37. 37.
    Yuan, W., Khot, T., Held, D., Mertz, C., Hebert, M.: PCN: point completion network. In: International Conference on 3D Vision (3DV), pp. 728–737 (2018)Google Scholar
  38. 38.
    Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 3391–3401 (2017)Google Scholar
  39. 39.
    Zhu, J.Y., et al.: Toward multimodal image-to-image translation. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 465–476 (2017)Google Scholar
  40. 40.
    Zhu, J.Y., et al.: Visual object networks: image generation with disentangled 3D representations. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 118–129 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Center on Frontiers of Computing StudiesPeking UniversityBeijingChina
  2. 2.Shandong UniversityQingdaoChina

Personalised recommendations