Skip to main content
SpringerLink
Log in

Style creation: multiple styles transfer with incremental learning and distillation loss

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Neural style transfer aims to transfer style from a style image to a content image by neural learning. A novel style will be brought if one can transfer more than one style to the content image. Because multiple style images can provide different textures and colors for style transfer. In this work, our goal is to train an image generator network to transfer multiple styles to a content image for users to create the satisfactory novel style. Ideally, given a content image and several style images, it is favorable that users can choose different style images according to personal preference to create a novel style because aesthetic preference may differ from user to user. In addition, we found that it is difficult for users to determine which style images can be used to generate a satisfactory novel style. To tackle these challenges, we formulate neural style transfer as an incremental learning process which is essentially trying to make the generator network remember old styles when learning a new style. A perceptual loss is used to train a feed-forward network to learn a new style, while the distillation loss which contain a style difference loss and a texture enhancement loss is designed to remember old styles during training of the network. Experiments demonstrate that during this process, users can drive the stylized result closer to the satisfactory novel style.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data Availability Statement

The datasets generated during the current study are available from the corresponding author on reasonable request.

References

  1. Arbelot B, Vergne R, Hurtut T, Thollot J (2016) Automatic texture guided color transfer and colorization. In: 5th Joint symposium on computational aesthetics, sketch-based interfaces and modeling, and non-photorealistic animation and rendering, expressive 2016, Lisbon, Portugal, May 7-9, 2016, Proceedings, pp 21–32

  2. Castro FM, Marín-Jiménez MJ, Guil N, Schmid C, Alahari K (2018) End-to-end incremental learning, computer vision - ECCV 2018 - 15th European conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XII, pp 241–257

  3. Cermelli F, Mancini M, Bulò SR, Ricci E, Caputo B (2020) Modeling the background for incremental learning in semantic segmentation. In: 2020 IEEE/CVF Conference on computer vision and pattern recognition, CVPR 2020, Seattle, WA, USA, June 13-19, pp 9230–9239

  4. Chen D, Yuan L, Liao J, Yu N, Hua G (2017) StyleBank: an explicit representation for neural image style transfer. In: 2017 IEEE conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, pp 2770–2779

  5. Cheng M-M, Liu X-C, Wang J, Lu S-P, Lai Y-K, Paul L (2020) Rosin: structure-preserving neural style transfer. IEEE Trans Image Process, 909–920

  6. Deng Y, Tang F, Dong W, Sun W, Huang F, Xu C (2020) Arbitrary style transfer via multi-adaptation network, MM ’20: the 28th ACM international conference on multimedia, virtual event / Seattle, WA, USA, October 12-16, pp 2719–2727

  7. Fiser J, Jamriska O, Lukác M, Shechtman E, Asente P, Lu J, Sýkora D (2016) StyLit: illumination-guided example-based stylization of 3D renderings. ACM Trans Graph, 92,1–92,11

  8. Gatys LA, Ecker AS, Bethge M (2015) Texture synthesis using convolutional neural networks. In: Advances in neural information processing systems 28: annual conference on neural information processing systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp 262–270

  9. Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: Conference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, June, 27-30, pp 2414–2423

  10. Gatys LA, Ecker AS, Bethge M, Hertzmann A, Shechtman E (2017) Controlling perceptual factors in neural style transfer. In: Conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, July, 21-26, 2017, pp 3730–3738

  11. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, pp 770–778

  12. Hertzmann A, Jacobs CE, Oliver N, Curless B, Salesin D (2001) Image analogies. In: Proceedings of the 28th annual conference on computer graphics and interactive techniques, SIGGRAPH 2001, Los Angeles, California, USA, August 12-17, pp 327–340

  13. Hinton GE, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network CoRR

  14. Hu Z, Jia J, Liu B, Bu Y, Fu J (2020) Aesthetic-aware image style transfer, MM ’20: the 28th ACM international conference on multimedia, virtual event / Seattle, WA, USA, October 12-16, pp 3320–3329

  15. Huang X, Belongie SJ (2017) Arbitrary style transfer in real-time with adaptive instance normalization. In: IEEE International conference on computer vision, ICCV 2017, Venice, Italy, October 22-29, pp 1510–1519

  16. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: Computer vision - ECCV 2016 - 14th European conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II, pp 694–711

  17. Jing Y, Yang Y, Feng Z, Ye J, Yu Y, Song M Neural style transfer: a review. IEEE Trans Vis Comput Graph, 3365–3385

  18. Kang H, Lee S, Chui CK (2007) Coherent line drawing. In: 5th International symposium on non-photorealistic animation and rendering, NPAR 2007, San Diego, CA, USA, August 4-5, 2007, Proceedings, pp 43–50

  19. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: 3rd International conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, conference track proceedings

  20. LeCun Y, Boser BE, Denker JS, Henderson D, Howard RE, Hubbard WE, Jackel LD Backpropagation applied to handwritten zip code recognition. Neural Comput, 541–551

  21. Li Z, Hoiem D (2018) Learning without forgetting. IEEE Trans Pattern Anal Mach Intell, 2935–2947

  22. Li Y, Fang C, Yang J, Wang Z, Lu X, Yang M-H (2017) Diversified texture synthesis with feed-forward networks. In: 2017 IEEE Conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, pp 266–274

  23. Li Y, Li Z, Ding L, Yang P, Hu Y, Chen W, Gao X (2018) SupportNet: solving catastrophic forgetting in class incremental learning with support data. arXiv:1806.02942

  24. Lin T-Y, Maire M, Belongie SJ, Hays J, Perona P, Ramanan D, Dollár P, Lawrence Zitnick C (2014) Microsoft COCO: common objects in context, computer vision - ECCV 2014 - 13th European conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, pp 740–755

  25. Liu X-C, Cheng M-M, Lai Y-K, Rosin PL (2017) Depth-aware neural style transfer. In: 15th International symposium on non-photorealistic animation and rendering, NPAR@Expressive 2017, Los Angeles, CA, USA, July 29-30, 2017, Proceedings, pp 4:1–4:10

  26. Liu Y, Su Y, Liu A-A, Schiele B, Sun Q (2020) Mnemonics training: multi-class incremental learning without forgetting. In: 2020 IEEE/CVF conference on computer vision and pattern recognition, CVPR 2020, Seattle, WA, USA, June, 13-19 2020, pp 12242–12251

  27. Martín D, Arroyo G, Luzón MV, Isenberg T (2011) Scale-dependent and example-based grayscale stippling. Comput Graph, 160–174

  28. Nguyen A-D, Choi S, Kim W, Lee S (2019) A simple way of multimodal and arbitrary style transfer. In: IEEE International conference on acoustics, speech and signal processing, ICASSP 2019, Brighton, United Kingdom, May 12–17

  29. Pouli T, Reinhard E (2011) Progressive color transfer for images of arbitrary dynamic range. Comput Graph, 67–80

  30. Semmo A, Isenberg T, Döllner J (2017) Neural style transfer: a paradigm shift for image-based artistic rendering?. In: 15th International symposium on non-photorealistic animation and rendering, NPAR@Expressive 2017, Los Angeles, CA, USA, July 29-30, 2017, Proceedings, pp 5:1–5:13

  31. Shmelkov K, Schmid C, Alahari K (2017) Incremental learning of object detectors without catastrophic forgetting. In: IEEE International conference on computer vision, ICCV 2017, Venice, Italy, October 22-29, pp 3420–3429

  32. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: 3rd International conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

  33. Snelgrove X High-resolution multi-scale neural texture synthesis. In: SIGGRAPH Asia 2017 technical briefs, Bangkok, Thailand, November 27 - 30, 2017, pp 13:1–13:4

  34. Wang L, Liu R, Vosoughi S (2020) Salienteye: maximizing engagement while maintaining artistic style on instagram using deep neural networks. In: Proceedings of the 2020 on international conference on multimedia retrieval, ICMR 2020, Dublin, Ireland, June 8-11, pp 331–335

  35. Wu H, Sun Z, Yuan W (2018) Direction-aware neural style transfer. In: 2018 ACM Multimedia conference on multimedia conference, MM 2018, Seoul, Republic of Korea, October 22-26, pp 1163–1171

  36. Wu Y, Chen Y, Wang L, Ye Y, Liu Z, Guo Y, Fu Y (2019) Large scale incremental learning. In: IEEE Conference on computer vision and pattern recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, pp 374–382

Download references

Funding

This work was supported by the National Science Foundation of China (42075139, 42075139, 61272219), Science and Technology Support Program of Jiangsu Province (BE2020082, BE2010072, BE2011058, BY2012190), Postdoctoral Science Foundation of Jiangsu Province (2017M621700), State Key Laboratory of Soil Plant Machinery System Technology (ZZKT2018A09), National High Technology Research and Development Program of China (2007AA01Z334).

Author information

Authors and Affiliations

  1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China

    Chen Ma, Zhengxing Sun & Chengfeng Ruan

Authors
  1. Chen Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhengxing Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chengfeng Ruan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhengxing Sun.

Ethics declarations

Competing interests

The authors declare that there is no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ma, C., Sun, Z. & Ruan, C. Style creation: multiple styles transfer with incremental learning and distillation loss. Multimed Tools Appl 83, 28341–28356 (2024). https://doi.org/10.1007/s11042-023-15532-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15532-5

Keywords

Search

Navigation