Advertisement

Spiral Generative Network for Image Extrapolation

Conference paper
  • 731 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12364)

Abstract

In this paper, motivated by human natural ability to perceive unseen surroundings imaginatively, we propose a novel Spiral Generative Network, SpiralNet, to perform image extrapolation in a spiral manner, which regards extrapolation as an evolution process growing from an input sub-image along a spiral curve to an expanded full image. Our SpiralNet, consisting of ImagineGAN and SliceGAN, disentangles image extrapolation problem into two independent sub-tasks as semantic structure prediction (via ImagineGAN) and contextual detail generation (via SliceGAN), making the whole task more tractable. The design of SliceGAN implicitly harnesses the correlation between generated contents and extrapolating direction, divide-and-conquer while generation-by-parts. Extensive experiments on datasets covering both objects and scenes under different cases show that our method achieves state-of-the-art performance on image extrapolation. We also conduct ablation study to validate efficacy of our design. Our code is available at https://github.com/zhenglab/spiralnet.

Keywords

Image extrapolation GAN cGAN SpiralNet 

Notes

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grants 61771440 and 41776113.

Supplementary material

504475_1_En_41_MOESM1_ESM.pdf (42 mb)
Supplementary material 1 (pdf 43034 KB)

References

  1. 1.
    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: ICML, pp. 214–223 (2017)Google Scholar
  2. 2.
    Avidan, S., Shamir, A.: Seam carving for content-aware image resizing. ACM TOG 26(3), 10 (2007)CrossRefGoogle Scholar
  3. 3.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM TOG 28(3), 24 (2009)CrossRefGoogle Scholar
  4. 4.
    Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018)
  5. 5.
    Cho, W., Choi, S., Park, D.K., Shin, I., Choo, J.: Image-to-image translation via group-wise deep whitening-and-coloring transformation. In: CVPR, pp. 10639–10647 (2019)Google Scholar
  6. 6.
    Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: CVPR, pp. 3213–3223 (2016)Google Scholar
  7. 7.
    Dekel, T., Gan, C., Krishnan, D., Liu, C., Freeman, W.T.: Sparse, smart contours to represent and edit images. In: CVPR, pp. 3511–3520 (2018)Google Scholar
  8. 8.
    Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.A.: What makes Paris look like Paris? ACM TOG 31(4), 101 (2012)CrossRefGoogle Scholar
  9. 9.
    Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: CVPR, pp. 1033–1038 (1999)Google Scholar
  10. 10.
    Fairchild, M.D.: Color Appearance Models. Wiley, Hoboken (2013)CrossRefGoogle Scholar
  11. 11.
    Goodfellow, I., et al.: Generative adversarial nets. In: NIPS, pp. 2672–2680 (2014)Google Scholar
  12. 12.
    Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of Wasserstein GANs. In: NIPS, pp. 5767–5777 (2017)Google Scholar
  13. 13.
    Hanbury, A.: Constructing cylindrical coordinate colour spaces. Pattern Recogn. Lett. 29(4), 494–500 (2008)CrossRefGoogle Scholar
  14. 14.
    Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR, pp. 4700–4708 (2017)Google Scholar
  15. 15.
    Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV, pp. 1501–1510 (2017)Google Scholar
  16. 16.
    Huang, X., Liu, M.Y., Belongie, S., Kautz, J.: Multimodal unsupervised image-to-image translation. In: ECCV, pp. 172–189 (2018)Google Scholar
  17. 17.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR, pp. 1125–1134 (2017)Google Scholar
  18. 18.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: ECCV, pp. 694–711 (2016)Google Scholar
  19. 19.
    Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: ICLR (2018)Google Scholar
  20. 20.
    Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR, pp. 4401–4410 (2019)Google Scholar
  21. 21.
    Kihlstrom, J.F.: The cognitive unconscious. Science 237(4821), 1445–1452 (1987)CrossRefGoogle Scholar
  22. 22.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)Google Scholar
  23. 23.
    Kosslyn, S.M., Ganis, G., Thompson, W.L.: Neural foundations of imagery. Nat. Rev. Neurosci. 2(9), 635 (2001)CrossRefGoogle Scholar
  24. 24.
    Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3D object representations for fine-grained categorization. In: ICCVW, pp. 554–561 (2013)Google Scholar
  25. 25.
    Lee, D., Kim, J., Moon, W.J., Ye, J.C.: CollaGAN: collaborative GAN for missing image data imputation. In: CVPR, pp. 2487–2496 (2019)Google Scholar
  26. 26.
    Li, C., Wand, M.: Precomputed real-time texture synthesis with Markovian generative adversarial networks. In: ECCV, pp. 702–716 (2016)Google Scholar
  27. 27.
    Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: ECCV, pp. 85–100 (2018)Google Scholar
  28. 28.
    MacEvoy, B.: Color Vision. handprint.com (2010)Google Scholar
  29. 29.
    Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Smolley, S.P.: Least squares generative adversarial networks. In: ICCV, pp. 2794–2802 (2017)Google Scholar
  30. 30.
    Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  31. 31.
    Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. In: ICLR (2018)Google Scholar
  32. 32.
    Nazeri, K., Ng, E., Joseph, T., Qureshi, F., Ebrahimi, M.: EdgeConnect: generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212 (2019)
  33. 33.
    Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number of classes. In: ICVGIP, pp. 722–729 (2008)Google Scholar
  34. 34.
    Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: CVPR, pp. 2337–2346 (2019)Google Scholar
  35. 35.
    Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM TOG 22(3), 313–318 (2003)CrossRefGoogle Scholar
  36. 36.
    Pessoa, L., Thompson, E., Noë, A.: Finding out about filling-in: a guide to perceptual completion for visual science and the philosophy of perception. Behav. Brain Sci. 21(6), 723–748 (1998)CrossRefGoogle Scholar
  37. 37.
    Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: ICLR (2016)Google Scholar
  38. 38.
    Sajjadi, M.S., Scholkopf, B., Hirsch, M.: EnhanceNet: single image super-resolution through automated texture synthesis. In: ICCV, pp. 4491–4500 (2017)Google Scholar
  39. 39.
    Sarvaiya, J.N., Patnaik, S., Bombaywala, S.: Image registration by template matching using normalized cross-correlation. In: ICACCTT, pp. 819–822. IEEE (2009)Google Scholar
  40. 40.
    Shan, Q., Curless, B., Furukawa, Y., Hernandez, C., Seitz, S.M.: Photo uncrop. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 16–31. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10599-4_2CrossRefGoogle Scholar
  41. 41.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  42. 42.
    Slossberg, R., Shamai, G., Kimmel, R.: High quality facial surface and texture synthesis via generative adversarial networks. In: ECCV, pp. 498–513 (2018)Google Scholar
  43. 43.
    Teterwak, P., et al.: Boundless: generative adversarial networks for image extension. In: ICCV, pp. 10521–10530 (2019)Google Scholar
  44. 44.
    Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The Caltech-UCSD Birds-200–2011 dataset. Technical report CNS-TR-2011-001, California Institute of Technology (2011)Google Scholar
  45. 45.
    Wang, M., Lai, Y., Liang, Y., Martin, R.R., Hu, S.M.: BiggerPicture: data-driven image extrapolation using graph matching. ACM TOG 33(6), 173 (2014)Google Scholar
  46. 46.
    Wang, N., Li, J., Zhang, L., Du, B.: MUSICAL: multi-scale image contextual attention learning for inpainting. In: IJCAI, pp. 3748–3754 (2019)Google Scholar
  47. 47.
    Wang, R., Zhang, Q., Fu, C.W., Shen, X., Zheng, W.S., Jia, J.: Underexposed photo enhancement using deep illumination estimation. In: CVPR, pp. 6849–6857 (2019)Google Scholar
  48. 48.
    Wang, Y., Tao, X., Shen, X., Jia, J.: Wide-context semantic image extrapolation. In: CVPR, pp. 1399–1408 (2019)Google Scholar
  49. 49.
    Xian, W., et al.: TextureGAN: controlling deep image synthesis with texture patches. In: CVPR, pp. 8456–8465 (2018)Google Scholar
  50. 50.
    Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: CVPR, pp. 5505–5514 (2018)Google Scholar
  51. 51.
    Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: ICCV, pp. 4471–4480 (2019)Google Scholar
  52. 52.
    Zhang, Y., Xiao, J., Hays, J., Tan, P.: FrameBreak: dramatic image extrapolation by guided shift-maps. In: CVPR, pp. 1171–1178 (2013)Google Scholar
  53. 53.
    Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. IEEE TPAMI 40(6), 1452–1464 (2017) CrossRefGoogle Scholar
  54. 54.
    Zhu, J.Y., Krähenbühl, P., Shechtman, E., Efros, A.A.: Generative visual manipulation on the natural image manifold. In: ECCV, pp. 597–613 (2016)Google Scholar
  55. 55.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV, pp. 2223–2232 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Underwater Vision LaboratoryOcean University of ChinaQingdaoChina
  2. 2.Sanya Oceanographic Institution, Ocean University of ChinaSanyaChina

Personalised recommendations