3D Object Completion via Class-Conditional Generative Adversarial Network

  • Yu-Chieh Chen
  • Daniel Stanley Tan
  • Wen-Huang Cheng
  • Kai-Lung HuaEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11296)


Many robotic tasks require accurate shape models in order to properly grasp or interact with objects. However, it is often the case that sensors produce incomplete 3D models due to several factors such as occlusion or sensor noise. To address this problem, we propose a semi-supervised method that can recover the complete the shape of a broken or incomplete 3D object model. We formulated a hybrid of 3D variational autoencoder (VAE) and generative adversarial network (GAN) to recover the complete voxelized 3D object. Furthermore, we incorporated a separate classifier in the GAN framework, making it a three player game instead of two which helps stabilize the training of the GAN as well as guides the shape completion process to follow the object class labels. Our experiments show that our model produces 3D object reconstructions with high-similarity to the ground truth and outperforms several baselines in both quantitative and qualitative evaluations.


Object reconstruction Shape completion Generative adversarial network Object classification 


  1. 1.
    Bao, J., Chen, D., Wen, F., Li, H., Hua, G.: CVAE-GAN: fine-grained image generation through asymmetric training. CoRR, abs/1703.10155 5 (2017)Google Scholar
  2. 2.
    Chang, A.X., et al.: ShapeNet: an information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)
  3. 3.
    Choi, S., Zhou, Q.Y., Miller, S., Koltun, V.: A large dataset of object scans. arXiv preprint arXiv:1602.02481 (2016)
  4. 4.
    Chongxuan, L., Xu, T., Zhu, J., Zhang, B.: Triple generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 4091–4101 (2017)Google Scholar
  5. 5.
    Dai, A., Qi, C.R., Nießner, M.: Shape completion using 3D-Encoder-Predictor CNNs and shape synthesis. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 3 (2017)Google Scholar
  6. 6.
    Denton, E.L., Chintala, S., Fergus, R., et al.: Deep generative image models using a Laplacian pyramid of adversarial networks. In: Advances in Neural Information Processing Systems, pp. 1486–1494 (2015)Google Scholar
  7. 7.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  8. 8.
    He, F.L., Wang, Y.C.F., Hua, K.L.: Self-learning approach to color demosaicking via support vector regression. In: International Conference on Image Processing (ICIP). IEEE (2012)Google Scholar
  9. 9.
    Hua, K.L., Zhang, R., Comer, M., Pollak, I.: Inter frame video compression with large dictionaries of tilings: algorithms for tiling selection and entropy coding. IEEE Trans. Circ. Syst. Video Technol. 22(8), 1136–1149 (2012)CrossRefGoogle Scholar
  10. 10.
    Kinga, D., Adam, J.B.: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR), vol. 5 (2015)Google Scholar
  11. 11.
    Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: International Conference on Learning Representations (ICLR) (2014)Google Scholar
  12. 12.
    Larsen, A.B.L., Sønderby, S.K., Larochelle, H., Winther, O.: Autoencoding beyond pixels using a learned similarity metric. arXiv preprint arXiv:1512.09300 (2015)
  13. 13.
    Li, C., Wand, M.: Precomputed real-time texture synthesis with Markovian generative adversarial networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 702–716. Springer, Cham (2016). Scholar
  14. 14.
    Li, H.C., et al.: Dependency-aware quality-differentiated wireless video multicast. In: 2013 IEEE Wireless Communications and Networking Conference (WCNC), pp. 2226–2231. IEEE (2013)Google Scholar
  15. 15.
    Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  16. 16.
    Odena, A., Olah, C., Shlens, J.: Conditional image synthesis with auxiliary classifier GANs. arXiv preprint arXiv:1610.09585 (2016)
  17. 17.
    Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434 (2015)
  18. 18.
    Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic back propagation and approximate inference in deep generative models. arXiv:1401.4082 (2014)
  19. 19.
    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, pp. 2234–2242 (2016)Google Scholar
  20. 20.
    Sharma, A., Grau, O., Fritz, M.: VConv-DAE: deep volumetric shape learning without object labels. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9915, pp. 236–250. Springer, Cham (2016). Scholar
  21. 21.
    Smith, E.J., Meger, D.: Improved adversarial systems for 3d object generation and reconstruction. In: Proceedings of the Annual Conference on Robot Learning (2017)Google Scholar
  22. 22.
    Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. In: Advances in Neural Information Processing Systems, pp. 3483–3491 (2015)Google Scholar
  23. 23.
    Sung, M., Kim, V.G., Angst, R., Guibas, L.: Data-driven structural priors for shape completion. ACM Trans. Graph. (TOG) 34(6), 175 (2015)CrossRefGoogle Scholar
  24. 24.
    Wu, J., Zhang, C., Xue, T., Freeman, B., Tenenbaum, J.: Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In: Advances in Neural Information Processing Systems, pp. 82–90 (2016)Google Scholar
  25. 25.
    Wu, Z., Song, S., Khosla, A., Tang, X., Xiao, J.: 3d ShapeNets for 2.5 d object recognition and next-best-view prediction. ArXiv e-prints 2 (2014)Google Scholar
  26. 26.
    Wu, Z., et al.: 3d ShapeNets: a deep representation for volumetric shapes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Yu-Chieh Chen
    • 1
  • Daniel Stanley Tan
    • 1
  • Wen-Huang Cheng
    • 2
  • Kai-Lung Hua
    • 1
    Email author
  1. 1.Department of CSIENational Taiwan University of Science and TechnologyTaipeiTaiwan
  2. 2.Department of EENational Chiao Tung UniversityHsinchuTaiwan

Personalised recommendations