Advertisement

The Visual Computer

, Volume 35, Issue 6–8, pp 1157–1169 | Cite as

Stylistic scene enhancement GAN: mixed stylistic enhancement generation for 3D indoor scenes

  • Suiyun Zhang
  • Zhizhong Han
  • Yu-Kun Lai
  • Matthias Zwicker
  • Hui ZhangEmail author
Original Article
  • 182 Downloads

Abstract

In this paper, we present stylistic scene enhancement GAN, SSE-GAN, a conditional Wasserstein GAN-based approach to automatic generation of mixed stylistic enhancements for 3D indoor scenes. An enhancement indicates factors that can influence the style of an indoor scene such as furniture colors and occurrence of small objects. To facilitate network training, we propose a novel enhancement feature encoding method, which represents an enhancement by a multi-one-hot vector, and effectively accommodates different enhancement factors. A Gumbel-Softmax module is introduced in the generator network to enable the generation of high fidelity enhancement features that can better confuse the discriminator. Experiments show that our approach is superior to the other baseline methods and successfully models the relationship between the style distribution and scene enhancements. Thus, although only trained with a dataset of room images in single styles, the trained generator can generate mixed stylistic enhancements by specifying multiple styles as the condition. Our approach is the first to apply a Gumbel-Softmax module in conditional Wasserstein GANs, as well as the first to explore the application of GAN-based models in the scene enhancement field.

Keywords

Scene enhancement 3D indoor scenes Interior design Conditional generative adversarial nets Gumbel-Softmax Multi-one-hot 

Notes

Acknowledgements

This work was supported by the National Natural Science Foundation of China (61373070), NSF (1813583) and Tsinghua-Kuaishou Institute of Future Media Data.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

References

  1. 1.
    Trimble 3D warehouse (2019). URL https://3dwarehouse.sketchup.com/. Accessed 15 Sept 2018
  2. 2.
    Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., Funkhouser, T.: Semantic scene completion from a single depth image. arXiv preprint arXiv:1611.08974 (2016)
  3. 3.
    Wang, K., Savva, M., Chang, A.X., Ritchie, D.: Deep convolutional priors for indoor scene synthesis. ACM Trans. Graph. (TOG) 37(4), 70:1–70:14 (2018)Google Scholar
  4. 4.
    Chen, G., Li, G., Nie, Y., Xian, C., Mao, A.: Stylistic indoor colour design via Bayesian network. Comput. Graph. 60, 34–45 (2016)CrossRefGoogle Scholar
  5. 5.
    Chen, K., Xu, K., Yu, Y., Wang, T.Y., Hu, S.M.: Magic decorator: automatic material suggestion for indoor digital scenes. ACM Trans. Graph. (TOG) 34(6), 232:1–232:11 (2015)Google Scholar
  6. 6.
    Zhang, S., Han, Z., Martin, R.R., Zhang, H.: Semantic 3D indoor scene enhancement using guide words. Vis. Comput. 33(6–8), 925–935 (2017)CrossRefGoogle Scholar
  7. 7.
    Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  8. 8.
    Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H.: Generative adversarial text to image synthesis. arXiv preprint arXiv:1605.05396 (2016)
  9. 9.
    Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. In: NIPS, 5767–5777 (2017)Google Scholar
  10. 10.
    Chen, X., Li, J., Li, Q., Gao, B., Zou, D., Zhao, Q.: Image2scene: transforming style of 3D room. In: Proceedings of the ACM International Conference on Multimedia, 321–330 (2015)Google Scholar
  11. 11.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NIPS, 2672–2680 (2014)Google Scholar
  12. 12.
    Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)
  13. 13.
    Chen, Y., Lai, Y.K., Liu, Y.J.: CartoonGAN: generative adversarial networks for photo cartoonization. In: IEEE CVPR, 9465–9474 (2018)Google Scholar
  14. 14.
    Wu, H., Zheng, S., Zhang, J., Huang, K.: GP-GAN: Towards realistic high-resolution image blending. arXiv preprint arXiv:1703.07195 (2017)
  15. 15.
    Wu, J., Zhang, C., Xue, T., Freeman, B., Tenenbaum, J.: Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. In: NIPS, pp. 82–90 (2016)Google Scholar
  16. 16.
    Liu, J., Yu, F., Funkhouser, T.: Interactive 3D modeling with a generative adversarial network. In: International Conference on 3D Vision (3DV), 126–134. IEEE (2017)Google Scholar
  17. 17.
    Chen, K., Choy, C.B., Savva, M., Chang, A.X., Funkhouser, T., Savarese, S.: Text2Shape: Generating shapes from natural language by learning joint embeddings. arXiv preprint arXiv:1803.08495 (2018)
  18. 18.
    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan. arXiv preprint arXiv:1701.07875 (2017)
  19. 19.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE CVPR, 5967–5976 (2017)Google Scholar
  20. 20.
    Donahue, C., McAuley, J., Puckette, M.: Adversarial audio synthesis. arXiv preprint arXiv:1802.04208 (2018)
  21. 21.
    Jang, E., Gu, S., Poole, B.: Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 (2016)
  22. 22.
    Maddison, C.J., Mnih, A., Teh, Y.W.: The concrete distribution: a continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 (2016)
  23. 23.
    Camino, R., Hammerschmidt, C., State, R.: Generating multi-categorical samples with generative adversarial networks. arXiv preprint arXiv:1807.01202 (2018)
  24. 24.
    Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., Frey, B.: Adversarial autoencoders. arXiv preprint arXiv:1511.05644 (2015)
  25. 25.
    Gumbel, E.J.: Statistical theory of extreme values and some practical applications. NBS Applied Mathematics Series 33, (1954)Google Scholar
  26. 26.
    Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980 (2014)
  27. 27.
    Xu, Q., Huang, G., Yuan, Y., Guo, C., Sun, Y., Wu, F., Weinberger, K.: An empirical study on evaluation metrics of generative adversarial networks. arXiv preprint arXiv:1806.07755 (2018)
  28. 28.
    Lopez-Paz, D., Oquab, M.: Revisiting classifier two-sample tests. arXiv preprint arXiv:1610.06545 (2016)
  29. 29.
    Bounliphone, W., Belilovsky, E., Blaschko, M.B., Antonoglou, I., Gretton, A.: A test of relative similarity for model selection in generative models. arXiv preprint arXiv:1511.04581 (2015)
  30. 30.
    Nemhauser, G.L., Wolsey, L.A., Fisher, M.L.: An analysis of approximations for maximizing submodular set functions-i. Math. Program. 14(1), 265–294 (1978)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Chen, D.Y., Tian, X.P., Shen, Y.T., Ouhyoung, M.: On visual similarity based 3D model retrieval. Comput. Graph. Forum 22(3), 223–232 (2003)CrossRefGoogle Scholar
  32. 32.
    Sun, J., Ovsjanikov, M., Guibas, L.: A concise and provably informative multi-scale signature based on heat diffusion. Comput. Graph. Forum 28(5), 1383–1392 (2009)CrossRefGoogle Scholar
  33. 33.
    Zhang, Z., Yang, Z., Ma, C., Luo, L., Huth, A., Vouga, E., Huang, Q.: Deep generative modeling for scene synthesis via hybrid representations. arXiv preprint arXiv:1808.02084 (2018)
  34. 34.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  • Suiyun Zhang
    • 1
    • 2
  • Zhizhong Han
    • 3
  • Yu-Kun Lai
    • 4
  • Matthias Zwicker
    • 3
  • Hui Zhang
    • 1
    • 2
    Email author
  1. 1.School of SoftwareTsinghua UniversityBeijingChina
  2. 2.Beijing National Research Center for Information Science and Technology (BNRist)BeijingChina
  3. 3.Department of Computer ScienceUniversity of MarylandCollege ParkUnited States
  4. 4.School of Computer Science and InformaticsCardiff UniversityWalesUnited Kingdom

Personalised recommendations