Abstract
Though generative adversarial networks (GANs) are prominent models to generate realistic and crisp images, they are unstable to train and suffer from the mode collapse problem. The problems of GANs come from approximating the intrinsic discontinuous distribution transform map with continuous DNNs. The recently proposed AE-OT model addresses the discontinuity problem by explicitly computing the discontinuous optimal transform map in the latent space of the autoencoder. Though have no mode collapse, the generated images by AE-OT are blurry. In this paper, we propose the AE-OT-GAN model to utilize the advantages of the both models: generate high quality images and at the same time overcome the mode collapse problems. Specifically, we firstly embed the low dimensional image manifold into the latent space by autoencoder (AE). Then the extended semi-discrete optimal transport (SDOT) map is used to generate new latent codes. Finally, our GAN model is trained to generate high quality images from the latent distribution induced by the extended SDOT map. The distribution transform map from this dataset related latent distribution to the data distribution will be continuous, and thus can be well approximated by the continuous DNNs. Additionally, the paired data between the latent codes and the real images gives us further restriction about the generator and stabilizes the training process. Experiments on simple MNIST dataset and complex datasets like CIFAR10 and CelebA show the advantages of the proposed method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Akash, S., Lazar, V., Chris, R., Gutmann, M.U., Charles, S.: Veegan: reducing mode collapse in gans using implicit variational learning. In: Neural Information Processing Systems (2017)
An, D., Guo, Y., Lei, N., Luo, Z., Yau, S.T., Gu, X.: Ae-ot: a new generative model based on extended semi-discrete optimal transport. In: International Conference on Learning Representations (2020)
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: ICML. pp, 214–223 (2017)
Burda, Y., Grosse, R., Salakhutdinov, R.: Importance weighted autoencoders. In: ICML (2015)
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: Infogan: interpretable representation learning by information maximizing generative adversarial nets. In: Advances in Neural Information Processing Systems (2016)
Dai, B., Wipf, D.: Diagnosing and enhancing VAE models. In: International Conference on Learning Representations (2019)
Dinh, L., Krueger, D., Bengio, Y.: Nice: Non-linear independent components estimation (2014). arXiv preprint arXiv:1410.8516
Dinh, L., Sohl-Dickstein, J., Bengio, S.: Density estimation using real nvp. In: ICLR (2017)
Figalli, A.: Regularity properties of optimal maps between nonconvex domains in the plane. Commun. Partial Diff. Equat. 35(3), 465–479 (2010)
Goodfellow, I.: Nips 2016 tutorial: Generative adversarial networks (2016). arXiv preprint arXiv:1701.00160
Goodfellow, I.J., et al.: Generative adversarial nets. In: Advances in neural information processing systems (2014)
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. In: NIPS, pp. 5769–5779 (2017)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a nash equilibrium. In: Advances in neural information processing systems (2017)
Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of gans for improved quality, stability, and variation. In: ICLR (2018)
Khayatkhoei, M., Singh, M.K., Elgammal, A.: Disconnected manifold learning for generative adversarial networks. In: Advances in Neural Information Processing Systems (2018)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes (2013). arXiv preprint arXiv:1312.6114
Kingma, D.P., Dhariwal, P.: Glow: Generative flow with invertible 1 x 1 convolutions. In: NeurIPS (2018)
Krizhevsky, A.: Learning multiple layers of features from tiny images. Technical report (2009)
Lecun, Y., Chopra, S., Hadsell, R.: A tutorial on energy-based learning. Predict. Struct. Data (2006)
LeCun, Y., Cortes, C.: MNIST handwritten digit database (2010). http://yann.lecun.com/exdb/mnist/
Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
Lee, C.H., Liu, Z., Wu, L., Luo, P.: Maskgan: towards diverse and interactive facial image manipulation (2019). arXiv preprint arXiv:1907.11922
Lin, Z., Khetan, A., Fanti, G., Oh, S.: Pacgan: the power of two samples in generative adversarial networks. In: Advances in Neural Information Processing Systems, pp. 1505–1514 (2018)
Liu, H., Gu, X., Samaras, D.: Wasserstein gan with quadratic transport cost. In: ICCV (2019)
Lucic, M., Kurach, K., Michalski, M., Gelly, S., Bousquet, O.: Are gans created equal? a large-scale study. In: Advances in neural information processing systems, pp. 698–707 (2018)
van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)
Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. In: ICLR (2018)
Nijkamp, E., Hill, M., Zhu, S.C., Wu, Y.N.: On learning non-convergent non-persistent short-run MCMC toward energy-based model (2019). arXiv preprint arXiv:1904.09770
van den Oord, A., Kalchbrenner, N., Espeholt, L., kavukcuoglu, k., Vinyals, O., Graves, A.: Conditional image generation with pixelcnn decoders. In: Advances in Neural Information Processing Systems (2016)
van den Oord, A., Vinyals, O., Kavukcuoglu, K.: Neural discrete representation learning. In: NeurIPS (2017)
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: ICLR (2016)
Razavi, A., Oord, A., Vinyals, O.: Generating diverse high-fidelity images with vq-vae-2. In: Advances in Neural Information Processing Systems (2019)
Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic backpropagation and approximate inference in deep generative models (2014). arXiv preprint arXiv:1401.4082
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. In: Advances in neural information processing systems (2016)
Salimans, T., Zhang, H., Radford, A., Metaxas, D.: Improving GANs using optimal transport. In: International Conference on Learning Representations (2018)
Tenenbaum, J.B., Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2391–232 (2000)
Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders. In: ICLR (2018)
Villani, C.: Optimal Transport: Old and New, vol. 338. Springer Science & Business Media, Berlin (2008)
Wu, J., Huang, Z., Thoma, J., Acharya, D., Gool, L.V.: Wasserstein divergence for gans. In: ECCV (2018)
Xiao, C., Zhong, P., Zheng, C.: Bourgan: Generative networks with metric embeddings. In: NeurIPS (2018)
Xie, J., Lu, Y., Zhu, S., Wu, Y.: Cooperative training of descriptor and generator networks. IEEE Trans. Pattern Anal. Mach. Intell. 42(1), 27–45 (2016)
Xu, L., Skoularidou, M., Cuesta-Infante, A., Veeramachaneni, K.: Modeling tabular data using conditional gan. In: Advances in Neural Information Processing Systems (2019)
Zhang, Z., Luo, P., Loy, C.C., Tang, X.: From facial expression recognition to interpersonal relation prediction. Int. J. Comput. Vis. 126(5), 550–569 (2017). https://doi.org/10.1007/s11263-017-1055-1
Zhu, S., Wu, Y., Mumford, D.: Filters, random fields and maximum entropy (frame): towards a unified theory for texture modeling. Int. J. Comput. Vis. 27(2), 107–126 (1998)
Acknowledgements
The project is partially supported by NSF CMMI-1762287, NSF DMS-1737812 and Ford URP and NSFC (61936002, 61772105, 61720106005).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 35353 KB)
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
An, D., Guo, Y., Zhang, M., Qi, X., Lei, N., Gu, X. (2020). AE-OT-GAN: Training GANs from Data Specific Latent Distribution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12371. Springer, Cham. https://doi.org/10.1007/978-3-030-58574-7_33
Download citation
DOI: https://doi.org/10.1007/978-3-030-58574-7_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58573-0
Online ISBN: 978-3-030-58574-7
eBook Packages: Computer ScienceComputer Science (R0)