Skip to main content
SpringerLink
Log in

Generative Adversarial Networks

  • Chapter
  • First Online:
Machine Learning for Data Science Handbook

  • 1982 Accesses

Abstract

Generative adversarial networks (GANs) are very popular frameworks for generating high-quality data and are immensely used in both the academia and industry in many domains. Arguably, their most substantial impact has been in the area of computer vision, where they achieve state-of-the-art image generation. This chapter gives an introduction to GANs, by discussing their principle mechanism and presenting some of their inherent problems during training and evaluation. We focus on these three issues: (1) mode collapse, (2) vanishing gradients, and (3) generation of low-quality images. We then list some architecture-variant and loss-variant GANs that remedy the above challenges. Lastly, we present two utilization examples of GANs for real-world applications: data augmentation and face images generation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 219.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 279.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2StyleGAN: How to embed images into the StyleGAN latent space? In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4431–4440, 2019.

    Google Scholar 

  2. Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2StyleGAN++: How to edit the embedded images? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.

    Google Scholar 

  3. Rameen Abdal, Peihao Zhu, Niloy J. Mitra, and Peter Wonka. StyleFlow: Attribute-conditioned exploration of StyleGAN-generated images using conditional continuous normalizing flows. ACM Trans. Graph., 40 (3), May 2021.

    Google Scholar 

  4. Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Networks. In ICML, volume 70, pages 214–223, 2017.

    Google Scholar 

  5. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. 7 2016.

    Google Scholar 

  6. David Berthelot, Tom Schumm, and Luke Metz. {BEGAN:} Boundary Equilibrium Generative Adversarial Networks. CoRR, abs/1703.1, 2017.

    Google Scholar 

  7. Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural Photo Editing with Introspective Adversarial Networks. ArXiv, abs/1609.0, 2017.

    Google Scholar 

  8. Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.

    Google Scholar 

  9. Eoin Brophy, Zhengwei Wang, and Tomas E. Ward. Quick and easy time series generation with established image-based GANs. ArXiv, abs/1902.05624, 2019.

    Google Scholar 

  10. Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-Supervised GANs via Auxiliary Rotation Loss. CVPR, pages 12146–12155, 2019.

    Google Scholar 

  11. David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active Learning with Statistical Models. J. Artif. Int. Res., 4 (1): 129–145, 3 1996. ISSN 1076-9757.

    Google Scholar 

  12. Anne Marie Delaney, Eoin Brophy, and Tomas E Ward. Synthesis of Realistic ECG using Generative Adversarial Networks. ArXiv, abs/1909.0, 2019.

    Google Scholar 

  13. Emily L Denton, Soumith Chintala, Arthur Szlam, and Robert Fergus. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks. CoRR, abs/1506.0, 2015.

    Google Scholar 

  14. Gintare Karolina Dziugaite, Daniel M. Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, page 258–267, 2015.

    Google Scholar 

  15. William Fedus, Ian Goodfellow, and Andrew M. Dai. MaskGAN: Better text generation via filling in the _. In International Conference on Learning Representations, 2018.

    Google Scholar 

  16. Yuri Feigin, Hedva Spitzer, and Raja Giryes. Generative adversarial encoder learning, 2020.

    Google Scholar 

  17. Robert M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3 (4): 128–135, 1999.

    Google Scholar 

  18. Rinon Gal, Dana Cohen Hochberg, Amit Bermano, and Daniel Cohen-Or. SWAGAN: A style-based wavelet-driven generative model. ACM Trans. Graph., 40 (4), July 2021a.

    Google Scholar 

  19. Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. StyleGAN-NADA: Clip-guided domain adaptation of image generators, 2021b.

    Google Scholar 

  20. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27, pages 2672–2680, 2014.

    Google Scholar 

  21. Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved Training of Wasserstein GANs. In Advances in Neural Information Processing Systems 30, pages 5767–5777, 2017.

    Google Scholar 

  22. Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 5141–5148, 2018.

    Google Scholar 

  23. Kay Gregor Hartmann, Robin Tibor Schirrmeister, and Tonio Ball. EEG-GAN: Generative adversarial networks for electroencephalograhic (EEG) brain signals. ArXiv, abs/1806.01875, 2018.

    Google Scholar 

  24. Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning, volume 80, pages 1989–1998, 2018.

    Google Scholar 

  25. Shady Abu Hussein, Tom Tirer, and Raja Giryes. Image-adaptive GAN based reconstruction. In AAAI, 2020.

    Google Scholar 

  26. Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML, 2015.

    Google Scholar 

  27. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5967–5976, 2016.

    Google Scholar 

  28. Nikolay Jetchev, Urs Bergmann, and Roland Vollgraf. Texture synthesis with spatial generative adversarial networks. CoRR, abs/1611.08207, 2016.

    Google Scholar 

  29. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.

    Google Scholar 

  30. Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396–4405, 2019.

    Google Scholar 

  31. Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Proc. NeurIPS, 2020a.

    Google Scholar 

  32. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020b.

    Google Scholar 

  33. Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Proc. NeurIPS, 2021.

    Google Scholar 

  34. Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013.

    Google Scholar 

  35. James N Kirkpatrick, Razvan Pascanu, Neil C Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114: 3521–3526, 2017.

    Google Scholar 

  36. Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic DENOYER, et al. Fader networks: Manipulating images by sliding attributes. In Advances in Neural Information Processing Systems, pages 5963–5972, 2017.

    Google Scholar 

  37. Y LeCun, P Haffner, L Bottou, and Yoshua Bengio. Object Recognition with Gradient-Based Learning. In Shape, Contour and Grouping in Computer Vision, 1999.

    Google Scholar 

  38. Christian Ledig, Lucas Theis, Ferenc Huszár, José Antonio Caballero, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 105–114, 2017.

    Google Scholar 

  39. Pauline Luc, Camille Couprie, Soumith Chintala, and Jakob Verbeek. Semantic segmentation using adversarial networks. ArXiv, abs/1611.08408, 2016.

    Google Scholar 

  40. Andrew L Maas. Rectifier Nonlinearities Improve Neural Network Acoustic Models. In ICML, 2013.

    Google Scholar 

  41. Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.

    Google Scholar 

  42. Aimee Mavratzakis, Cornelia Herbert, and Peter Walla. Emotional facial expressions evoke faster orienting responses, but weaker emotional responses at neural and behavioural levels compared to scenes: A simultaneous EEG and facial EMG study. NeuroImage, 124: 931–946, 2016.

    Article  Google Scholar 

  43. Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Which training methods for GANs do actually converge? In International Conference on Machine Learning (ICML), 2018.

    Google Scholar 

  44. Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. In ICLR, 2017.

    Google Scholar 

  45. Mehdi Mirza and Simon Osindero. Conditional Generative Adversarial Nets. ArXiv, abs/1411.1, 2014.

    Google Scholar 

  46. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. In International Conference on Learning Representations, 2018.

    Google Scholar 

  47. Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.

    Google Scholar 

  48. Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, volume 139, pages 8162–8171, 2021.

    Google Scholar 

  49. Augustus Odena. Semi-Supervised Learning with Generative Adversarial Networks. ArXiv, abs/1606.0, 2016.

    Google Scholar 

  50. Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional Image Synthesis with Auxiliary Classifier {GAN}s. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2642–2651, 2017.

    Google Scholar 

  51. George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22 (57): 1–64, 2021.

    Google Scholar 

  52. Sung Woo Park and Junseok Kwon. Sphere Generative Adversarial Network Based on Geometric Moment Matching. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4287–4296, 2019.

    Google Scholar 

  53. Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.

    Google Scholar 

  54. Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. StyleCLIP: Text-driven manipulation of StyleGAN imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2085–2094, October 2021.

    Google Scholar 

  55. Gabriel Peyre and Marco Cuturi. Computational optimal transport: With applications to data science. Foundations and Trendsin Machine Learning, 11 (5-6): 355–607, 2019.

    Google Scholar 

  56. Guo-Jun Qi. Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities. International Journal of Computer Vision, 128: 1118–1140, 2019.

    Article  MathSciNet  MATH  Google Scholar 

  57. Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. CoRR, abs/1511.0, 2015.

    Google Scholar 

  58. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.

    Google Scholar 

  59. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021.

    Google Scholar 

  60. Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a StyleGAN encoder for image-to-image translation. In CVPR, 2021.

    Google Scholar 

  61. Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. SinGAN: Learning a generative model from a single natural image. In IEEE International Conference on Computer Vision (ICCV), 2019.

    Google Scholar 

  62. Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. ArXiv, abs/1606.0, 2016.

    Google Scholar 

  63. Tim Salimans and Durk P Kingma. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. In Advances in Neural Information Processing Systems 29, pages 901–909. Curran Associates, Inc., 2016.

    Google Scholar 

  64. Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in GANs. In CVPR, 2021.

    Google Scholar 

  65. Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. InterfaceGAN: Interpreting the disentangled face representation learned by GANs. TPAMI, 2020.

    Google Scholar 

  66. Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=St1giarCHLP.

  67. Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.

    Google Scholar 

  68. Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for StyleGAN image manipulation. ACM Trans. Graph., 40 (4), 2021.

    Google Scholar 

  69. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, pages 5998–6008, 2017.

    Google Scholar 

  70. T. Wang, M. Liu, J. Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image synthesis and semantic manipulation with conditional GANs. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8798–8807, June 2018. doi: 10.1109/CVPR.2018.00917.

  71. Zhengwei Wang, Qi She, and Tomas E Ward. Generative Adversarial Networks: {A} Survey and Taxonomy. CoRR, abs/1906.0, 2019a.

    Google Scholar 

  72. Zhengwei Wang, Qi She, and Tomas E Ward. Generative Adversarial Networks: {A} Survey and Taxonomy. CoRR, abs/1906.0, 2019b.

    Google Scholar 

  73. Zhengwei Wang, Qi She, Alan F Smeaton, Tomás E Ward, and Graham Healy. Synthetic-Neuroscore: Using a neuro-AI interface for evaluating generative adversarial networks. Neurocomputing, 405: 26–36, 2020.

    Google Scholar 

  74. Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang. GP-GAN: Towards Realistic High-Resolution Image Blending. Proceedings of the 27th ACM International Conference on Multimedia, 2019.

    Google Scholar 

  75. Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey, 2021.

    Google Scholar 

  76. Matthew D Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. In ECCV, 2014.

    Google Scholar 

  77. Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference on Machine Learning, volume 97, pages 7354–7363, 2019.

    Google Scholar 

  78. Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. In Conference on Neural Information Processing Systems (NeurIPS), 2020.

    Google Scholar 

  79. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242–2251, 2017.

    Google Scholar 

Download references

Acknowledgements

We would like to thank Yuval Alaluf, Yotam Nitzan, and Ron Mokady for their helpful comments.

Author information

Authors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Gilad Cohen & Raja Giryes

Authors
  1. Gilad Cohen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Raja Giryes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Raja Giryes .

Editor information

Editors and Affiliations

  1. Department of Software and Information Systems Engineering, Ben-Gurion University of the Negev, Beer-Sheva, Israel

    Lior Rokach

  2. Department of Industrial Engineering, Tel Aviv University, Ramat Aviv, Israel

    Oded Maimon

  3. Department of Industrial Engineering, Tel Aviv University, Tel Aviv, Israel

    Erez Shmueli

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Cohen, G., Giryes, R. (2023). Generative Adversarial Networks. In: Rokach, L., Maimon, O., Shmueli, E. (eds) Machine Learning for Data Science Handbook. Springer, Cham. https://doi.org/10.1007/978-3-031-24628-9_17

Download citation

Publish with us

Policies and ethics

Search

Navigation