Advertisement

Transforming and Projecting Images into Class-Conditional Generative Networks

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12347)

Abstract

We present a method for projecting an input image into the space of a class-conditional generative neural network. We propose a method that optimizes for transformation to counteract the model biases in generative neural networks. Specifically, we demonstrate that one can solve for image translation, scale, and global color transformation, during the projection optimization to address the object-center bias and color bias of a Generative Adversarial Network. This projection process poses a difficult optimization problem, and purely gradient-based optimizations fail to find good solutions. We describe a hybrid optimization strategy that finds good projections by estimating transformations and class parameters. We show the effectiveness of our method on real images and further demonstrate how the corresponding projections lead to better editability of these images. The project page and the code is available at https://minyoungg.github.io/GAN-Transform-and-Project/.

Notes

Acknowledgements

We thank David Bau, Phillip Isola, Lucy Chai, and Erik Härkönen for discussions, and David Bau for encoder training code.

Supplementary material

504434_1_En_2_MOESM1_ESM.pdf (3.6 mb)
Supplementary material 1 (pdf 3691 KB)

References

  1. 1.
    Abdal, R., Qin, Y., Wonka, P.: Image2StyleGAN: how to embed images into the StyleGAN latent space? In: International Conference on Computer Vision (2019)Google Scholar
  2. 2.
    Ankit, R., Li, Y., Bresler, Y.: GAN-based projector for faster recovery with convergence guarantees in linear inverse problems. In: International Conference on Computer Vision (2019)Google Scholar
  3. 3.
    Asim, M., Shamshad, F., Ahmed, A.: Blind image deconvolution using deep generative priors. In: British Machine Vision Conference (2018)Google Scholar
  4. 4.
    Baker, S., Matthews, I.: Lucas-kanade 20 years on: a unifying framework. Int. J. Comput. Vis. 56(3), 221–255 (2004).  https://doi.org/10.1023/B:VISI.0000011205.11775.fdCrossRefGoogle Scholar
  5. 5.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. (TOG) 28, 24 (2009)CrossRefGoogle Scholar
  6. 6.
    Bau, D., et al.: Semantic photo manipulation with a generative image prior. ACM Trans. Graph. (TOG) 38 (2019)Google Scholar
  7. 7.
    Bau, D., et al.: GAN dissection: visualizing and understanding generative adversarial networks. In: International Conference on Learning Representations (2019)Google Scholar
  8. 8.
    Bau, D., et al.: Seeing what a GAN cannot generate. In: International Conference on Computer Vision (2019)Google Scholar
  9. 9.
    Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: International Conference on Learning Representations (2019)Google Scholar
  10. 10.
    Brock, A., Lim, T., Ritchie, J.M., Weston, N.: Neural photo editing with introspective adversarial networks. In: International Conference on Learning Representations (2017)Google Scholar
  11. 11.
    Creswell, A., Bharath, A.A.: Inverting the generator of a generative adversarial network. IEEE Trans. Neural Netw. Learn. Syst. 30(7), 1967–1974 (2018)CrossRefGoogle Scholar
  12. 12.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (2009)Google Scholar
  13. 13.
    Donahue, J., Krähenbühl, P., Darrell, T.: Adversarial feature learning. In: International Conference on Learning Representations (2017)Google Scholar
  14. 14.
    Donahue, J., Simonyan, K.: Large scale adversarial representation learning. In: Advances in Neural Information Processing Systems (2019)Google Scholar
  15. 15.
    Dosovitskiy, A., Brox, T.: Generating images with perceptual similarity metrics based on deep networks. In: Advances in Neural Information Processing Systems (2016)Google Scholar
  16. 16.
    Dosovitskiy, A., Brox, T.: Inverting visual representations with convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  17. 17.
    Dumoulin, V., et al.: Adversarially learned inference. In: International Conference on Learning Representations (2017)Google Scholar
  18. 18.
    Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: ACM SIGGRAPH (2001)Google Scholar
  19. 19.
    Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: International Conference on Computer Vision (1999)Google Scholar
  20. 20.
    Everingham, M., Eslami, S.M.A., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL visual object classes challenge: a retrospective. Int. J. Comput. Vis. 111(1), 98–136 (2015).  https://doi.org/10.1007/s11263-014-0733-5CrossRefGoogle Scholar
  21. 21.
    Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  22. 22.
    Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 2nd edn. Pearson, London (1992)Google Scholar
  23. 23.
    Goodfellow, I.: NIPS 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160 (2016)
  24. 24.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (2014)Google Scholar
  25. 25.
    Gu, J., Shen, Y., Zhou, B.: Image processing using multi-code GAN prior. In: IEEE Conference on Computer Vision and Pattern Recognition (2020)Google Scholar
  26. 26.
    Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evol. Comput. 9, 159–195 (2001)CrossRefGoogle Scholar
  27. 27.
    He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: International Conference on Computer Vision (2017)Google Scholar
  28. 28.
    Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: ACM SIGGRAPH (2001)Google Scholar
  29. 29.
    Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. In: Advances in Neural Information Processing Systems (2015)Google Scholar
  30. 30.
    Jahanian, A., Chai, L., Isola, P.: On the"steerability" of generative adversarial networks. In: International Conference on Learning Representations (2020)Google Scholar
  31. 31.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) European Conference on Computer Vision, vol. 9906, pp. 694–711. Springer, Heidelberg (2016).  https://doi.org/10.1007/978-3-319-46475-6_43
  32. 32.
    Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (2018)Google Scholar
  33. 33.
    Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  34. 34.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2015)Google Scholar
  35. 35.
    Kingma, D.P., Dhariwal, P.: Glow: generative flow with invertible 1x1 convolutions. In: Advances in Neural Information Processing Systems (2018)Google Scholar
  36. 36.
    Krizhevsky, A.: Learning Multiple Layers of Features from Tiny Images. Master’s Thesis, University of Toronto (2009)Google Scholar
  37. 37.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (2012)Google Scholar
  38. 38.
    Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2007)CrossRefGoogle Scholar
  39. 39.
    Lin, T., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) European Conference on Computer Vision, vol. 8693, pp. 740–755. Springer, Heidelberg (2014).  https://doi.org/10.1007/978-3-319-10602-1_48
  40. 40.
    Lipton, Z.C., Tripathi, S.: Precise recovery of latent vectors from generative adversarial networks. ICLR Workshop (2017)Google Scholar
  41. 41.
    Liu, D.C., Nocedal, J.: On the limited memory BFGS method for large scale optimization. Math. Program. 45(1–3), 503–528 (1989).  https://doi.org/10.1007/BF01589116MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Mahendran, A., Vedaldi, A.: Understanding deep image representations by inverting them. In: IEEE Conference on Computer Vision and Pattern Recognition (2015)Google Scholar
  43. 43.
    Olah, C., Mordvintsev, A., Schubert, L.: Feature visualization. Distill 2(11), e7 (2017)CrossRefGoogle Scholar
  44. 44.
    Olah, C., et al.: The building blocks of interpretability. Distill 3(3), e10 (2018)CrossRefGoogle Scholar
  45. 45.
    Oppenheim, A.V., Schafer, R.W., Buck, J.R.: Discrete-Time Signal Processing, 2nd edn. Pearson, London (1999)Google Scholar
  46. 46.
    Perarnau, G., Van De Weijer, J., Raducanu, B., Álvarez, J.M.: Invertible conditional GANs for image editing. In: NIPS 2016 Workshop on Adversarial Training (2016)Google Scholar
  47. 47.
    Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. (TOG) 22(3), 313–318 (2003)CrossRefGoogle Scholar
  48. 48.
    Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: International Conference on Learning Representations (2016)Google Scholar
  49. 49.
    Shah, V., Hegde, C.: Solving linear inverse problems using GAN priors: an algorithm with provable guarantees. In: ICASSP (2018)Google Scholar
  50. 50.
    Shen, Y., Gu, J., Tang, X., Zhou, B.: Interpreting the latent space of GANs for semantic face editing. In: IEEE Conference on Computer Vision and Pattern Recognition (2020)Google Scholar
  51. 51.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)Google Scholar
  52. 52.
    Tiantian, F., Schwing, A.: Co-generation with GANs using AIS based HMC. In: Advances in Neural Information Processing Systems (2019)Google Scholar
  53. 53.
    Wampler, K., Popović, Z.: Optimal gait and form for animal locomotion. ACM Trans. Graph. (TOG) 28, 1–8 (2009)CrossRefGoogle Scholar
  54. 54.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  55. 55.
    Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. arXiv preprint arXiv:1611.05431 (2016)
  56. 56.
    Yeh, R.A., Chen, C., Yian Lim, T., Schwing, A.G., Hasegawa-Johnson, M., Do, M.N.: Semantic image inpainting with deep generative models. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  57. 57.
    Yu, F., Zhang, Y., Song, S., Seff, A., Xiao, J.: LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 (2015)
  58. 58.
    Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) European Conference on Computer Vision, vol. 9907, pp. 649–666 Springer, Heidelberg (2016).  https://doi.org/10.1007/978-3-319-46487-9_40
  59. 59.
    Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep networks as a perceptual metric. In: IEEE Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  60. 60.
    Zhu, J.Y., Krähenbühl, P., Shechtman, E., Efros, A.A.: Generative visual manipulation on the natural image manifold. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) European Conference on Computer Vision, vol. 9909, pp. 597–613. Springer, Heidelberg (2016).  https://doi.org/10.1007/978-3-319-46454-1_36

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.MIT CSAILCambridgeUSA
  2. 2.Adobe ResearchSan FranciscoUSA

Personalised recommendations