Skip to main content

Learning Graph Neural Networks for Image Style Transfer

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13667))

Included in the following conference series:

Abstract

State-of-the-art parametric and non-parametric style transfer approaches are prone to either distorted local style patterns due to global statistics alignment, or unpleasing artifacts resulting from patch mismatching. In this paper, we study a novel semi-parametric neural style transfer framework that alleviates the deficiency of both parametric and non-parametric stylization. The core idea of our approach is to establish accurate and fine-grained content-style correspondences using graph neural networks (GNNs). To this end, we develop an elaborated GNN model with content and style local patches as the graph vertices. The style transfer procedure is then modeled as the attention-based heterogeneous message passing between the style and content nodes in a learnable manner, leading to adaptive many-to-one style-content correlations at the local patch level. In addition, an elaborated deformable graph convolutional operation is introduced for cross-scale style-content matching. Experimental results demonstrate that the proposed semi-parametric image stylization approach yields encouraging results on the challenging style patterns, preserving both global appearance and exquisite details. Furthermore, by controlling the number of edges at the inference stage, the proposed method also triggers novel functionalities like diversified patch-based stylization with a single model.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. An, J., Huang, S., Song, Y., Dou, D., Liu, W., Luo, J.: ArtFlow: unbiased image style transfer via reversible neural flows. In: CVPR (2021)

    Google Scholar 

  2. Champandard, A.J.: Semantic style transfer and turning two-bit doodles into fine artworks. arXiv preprint arXiv:1603.01768 (2016)

  3. Chen, D., Yuan, L., Liao, J., Yu, N., Hua, G.: StyleBank: an explicit representation for neural image style transfer. In: CVPR (2017)

    Google Scholar 

  4. Chen, D., Yuan, L., Liao, J., Yu, N., Hua, G.: Explicit filterbank learning for neural image style transfer and image processing. TPAMI 43, 2373–2387 (2020)

    Google Scholar 

  5. Chen, H., et al.: Diverse image style transfer via invertible cross-space mapping. In: ICCV (2021)

    Google Scholar 

  6. Chen, T.Q., Schmidt, M.: Fast patch-based style transfer of arbitrary style. In: NeurIPS Workshop on Constructive Machine Learning (2016)

    Google Scholar 

  7. Chen, Z., et al.: DPT: deformable patch-based transformer for visual recognition. In: ACM MM (2021)

    Google Scholar 

  8. Ding, L., Wang, L., Liu, X., Wong, D.F., Tao, D., Tu, Z.: Understanding and improving lexical choice in non-autoregressive translation. In: ICLR (2021)

    Google Scholar 

  9. Ding, L., Wang, L., Tao, D.: Self-attention with cross-lingual position representation. In: ACL (2020)

    Google Scholar 

  10. Ding, L., Wang, L., Wu, D., Tao, D., Tu, Z.: Context-aware cross-attention for non-autoregressive translation. In: COLING (2020)

    Google Scholar 

  11. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: CVPR (2016)

    Google Scholar 

  12. Hamilton, W.L., Ying, R., Leskovec, J.: Inductive representation learning on large graphs. In: NeurIPS (2017)

    Google Scholar 

  13. Hong, K., Jeon, S., Yang, H., Fu, J., Byun, H.: Domain-aware universal style transfer. In: ICCV (2021)

    Google Scholar 

  14. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV (2017)

    Google Scholar 

  15. Huo, J., et al.: Manifold alignment for semantically aligned style transfer. In: ICCV (2021)

    Google Scholar 

  16. Jing, Y., et al.: Dynamic instance normalization for arbitrary style transfer. In: AAAI (2020)

    Google Scholar 

  17. Jing, Y., et al.: Stroke controllable fast style transfer with adaptive receptive fields. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11217, pp. 244–260. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01261-8_15

    Chapter  Google Scholar 

  18. Jing, Y., Yang, Y., Feng, Z., Ye, J., Yu, Y., Song, M.: Neural style transfer: a review. TVCG 26, 3365–3385 (2019)

    Google Scholar 

  19. Jing, Y., Yang, Y., Wang, X., Song, M., Tao, D.: Amalgamating knowledge from heterogeneous graph neural networks. In: CVPR (2021)

    Google Scholar 

  20. Jing, Y., Yang, Y., Wang, X., Song, M., Tao, D.: Meta-aggregator: learning to aggregate for 1-bit graph neural networks. In: ICCV (2021)

    Google Scholar 

  21. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    Chapter  Google Scholar 

  22. Kalischek, N., Wegner, J.D., Schindler, K.: In the light of feature distributions: moment matching for neural style transfer. In: CVPR (2021)

    Google Scholar 

  23. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)

    Google Scholar 

  24. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: ICLR (2017)

    Google Scholar 

  25. Kolkin, N., Salavon, J., Shakhnarovich, G.: Style transfer by relaxed optimal transport and self-similarity. In: CVPR (2019)

    Google Scholar 

  26. Kong, Y., Liu, L., Wang, J., Tao, D.: Adaptive curriculum learning. In: ICCV (2021)

    Google Scholar 

  27. Li, C., Wand, M.: Combining Markov random fields and convolutional neural networks for image synthesis. In: CVPR, pp. 2479–2486 (2016)

    Google Scholar 

  28. Li, Y., Wang, N., Liu, J., Hou, X.: Demystifying neural style transfer. In: IJCAI (2017)

    Google Scholar 

  29. Li, Y., Chen, F., Yang, J., Wang, Z., Lu, X., Yang, M.H.: Diversified texture synthesis with feed-forward networks. In: CVPR (2017)

    Google Scholar 

  30. Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.H.: Universal style transfer via feature transforms. In: NeurIPS (2017)

    Google Scholar 

  31. Liao, J., Yao, Y., Yuan, L., Hua, G., Kang, S.B.: Visual attribute transfer through deep image analogy. TOG 36, 1–15 (2017)

    Google Scholar 

  32. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  33. Liu, H., Yang, Y., Wang, X.: Overcoming catastrophic forgetting in graph neural networks. In: AAAI (2021)

    Google Scholar 

  34. Liu, S., et al.: Paint transformer: feed forward neural painting with stroke prediction. In: ICCV (2021)

    Google Scholar 

  35. Liu, S., et al.: AdaAttN: revisit attention mechanism in arbitrary neural style transfer. In: ICCV (2021)

    Google Scholar 

  36. Liu, X.C., Yang, Y.L., Hall, P.: Learning to warp for style transfer. In: CVPR (2021)

    Google Scholar 

  37. Mechrez, R., Talmi, I., Zelnik-Manor, L.: The contextual loss for image transformation with non-aligned data. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 800–815. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_47

    Chapter  Google Scholar 

  38. Nichol, K.: Painter by numbers (2016). https://www.kaggle.com/c/painter-by-numbers

  39. Ren, S., Zhou, D., He, S., Feng, J., Wang, X.: Shunted self-attention via multi-scale token aggregation. In: CVPR (2022)

    Google Scholar 

  40. Risser, E., Wilmot, P., Barnes, C.: Stable and controllable neural texture synthesis and style transfer using histogram losses. arXiv preprint arXiv:1701.08893 (2017)

  41. Shen, C., Yin, Y., Wang, X., Li, X., Song, J., Song, M.: Training generative adversarial networks in one stage. In: CVPR (2021)

    Google Scholar 

  42. Sheng, L., Shao, J., Lin, Z., Warfield, S., Wang, X.: Avatar-Net: multi-scale zero-shot style transfer by feature decoration. In: CVPR (2018)

    Google Scholar 

  43. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. In: ICLR (2018)

    Google Scholar 

  44. Wang, M., et al.: Deep graph library: towards efficient and scalable deep learning on graphs. In: ICLR Workshop (2019)

    Google Scholar 

  45. Wang, P., Li, Y., Vasconcelos, N.: Rethinking and improving the robustness of image style transfer. In: CVPR (2021)

    Google Scholar 

  46. Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph CNN for learning on point clouds. TOG 38, 1–12 (2019)

    Google Scholar 

  47. Wu, X., Hu, Z., Sheng, L., Xu, D.: StyleFormer: real-time arbitrary style transfer via parametric style composition. In: ICCV (2021)

    Google Scholar 

  48. Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? In: ICLR (2019)

    Google Scholar 

  49. Xu, W., Long, C., Wang, R., Wang, G.: DRB-GAN: a dynamic ResBlock generative adversarial network for artistic style transfer. In: ICCV (2021)

    Google Scholar 

  50. Xu, Y., Zhang, Q., Zhang, J., Tao, D.: ViTAE: vision transformer advanced by exploring intrinsic inductive bias. In: NeurIPS (2021)

    Google Scholar 

  51. Yang, Y., Feng, Z., Song, M., Wang, X.: Factorizable graph convolutional networks. In: NeurIPS (2020)

    Google Scholar 

  52. Yang, Y., Qiu, J., Song, M., Tao, D., Wang, X.: Distilling knowledge from graph convolutional networks. In: CVPR (2020)

    Google Scholar 

  53. Yang, Y., Ren, Z., Li, H., Zhou, C., Wang, X., Hua, G.: Learning dynamics via graph neural networks for human pose estimation and tracking. In: CVPR (2021)

    Google Scholar 

  54. Yang, Y., Wang, X., Song, M., Yuan, J., Tao, D.: SPAGAN: shortest path graph attention network. In: IJCAI (2019)

    Google Scholar 

  55. Ye, J., Jing, Y., Wang, X., Ou, K., Tao, D., Song, M.: Edge-sensitive human cutout with hierarchical granularity and loopy matting guidance. TIP 29, 1177–1191 (2019)

    Google Scholar 

  56. Yu, W., et al.: MetaFormer is actually what you need for vision. In: CVPR (2022)

    Google Scholar 

  57. Zhan, Y., Yu, J., Yu, T., Tao, D.: On exploring undetermined relationships for visual relationship detection. In: CVPR (2019)

    Google Scholar 

  58. Zhan, Y., Yu, J., Yu, T., Tao, D.: Multi-task compositional network for visual relationship detection. IJCV 128, 2146–2165 (2020)

    Google Scholar 

  59. Zhang, H., Dana, K.: Multi-style generative network for real-time transfer. arXiv preprint arXiv:1703.06953 (2017)

  60. Zhang, Q., Xu, Y., Zhang, J., Tao, D.: ViTAEv2: vision transformer advanced by exploring inductive bias for image recognition and beyond. arXiv preprint arXiv:2202.10108 (2022)

  61. Zhang, Q., Xu, Y., Zhang, J., Tao, D.: VSA: learning varied-size window attention in vision transformers. arXiv preprint arXiv:2204.08446 (2022)

  62. Zhao, H., Bian, W., Yuan, B., Tao, D.: Collaborative learning of depth estimation, visual odometry and camera relocalization from monocular videos. In: IJCAI (2020)

    Google Scholar 

  63. Zhou, J., et al.: Graph neural networks: a review of methods and applications. arXiv preprint arXiv:1812.08434 (2018)

  64. Zhou, S., Zhang, J., Zuo, W., Loy, C.C.: Cross-scale internal graph neural network for image super-resolution. In: NeurIPS (2020)

    Google Scholar 

Download references

Acknowledgments

Mr Yongcheng Jing is supported by ARC FL-170100117. Dr Xinchao Wang is supported by AI Singapore (Award No.: AISG2-RP-2021-023) and NUS Faculty Research Committee Grant (WBS: A-0009440-00-00).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xinchao Wang .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 14448 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jing, Y. et al. (2022). Learning Graph Neural Networks for Image Style Transfer. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13667. Springer, Cham. https://doi.org/10.1007/978-3-031-20071-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20071-7_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20070-0

  • Online ISBN: 978-3-031-20071-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics