Skip to main content
SpringerLink
Log in

Deep photographic style transfer guided by semantic correspondence

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The objective of this paper is to develop an effective photographic style transfer method while preserving the semantic correspondence between the style and content images for both scenery and portrait images. A semantic correspondence guided photographic style transfer algorithm is developed, which is to ensure that the semantic structure of the content image has not been changed while the color of the style images is being migrated. The semantic correspondence is constructed in large scale regions based on image segmentation and also in local scale patches using Nearest-neighbor Field Search in the deep feature domain. Based on the semantic correspondence, a matting optimization is utilized to optimize the style transfer result to ensure the semantic accuracy and transfer faithfulness. The proposed style transfer method is further extended to automatically retrieve the style images from a database to make style transfer more-friendly. The experimental results show that our method could successfully conduct the style transfer while preserving semantic correspondence between diversity of scenes. A user study also shows that our method outperforms state-of-the-art photographic style transfer methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Arbelot B, Hurtut T, Hurtut T, Thollot J (2016) Automatic texture guided color transfer and colorization. Joint symposium on computational aesthetics and sketch based interfaces and modeling and non-photorealistic animation and rendering, pp 21–32

  2. Barnes C, Shechtman E, Finkelstein A, Goldman DB (2009) Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans Graph 28(3):1–11. Proc. of SIGGRAPH

    Article  Google Scholar 

  3. Champandard A J (2016) Semantic style transfer and turning two-bit doodles into fine artworks. arXiv:1603.01768

  4. Dong X, Shen J (2018) Triplet loss in siamese network for object tracking. Proceedings of the European conference on computer vision, pp 459–474

    Chapter  Google Scholar 

  5. Dong X, Shen J, Liu Y, et al. (2019) Quadruplet network with one-shot learning for fast visual object tracking. IEEE Trans Image Process 28(7):3516–3527

    Article  MathSciNet  Google Scholar 

  6. Dong X, Shen J, Wang W, et al. (2018) Hyperparameter optimization for tracking with continuous deep q-learning. Proceedings of the IEEE conference on computer vision and pattern recognition, pp 518–527

  7. Efros A A, Freeman W T (2001) Image quilting for texture synthesis and transfer. Conference on computer graphics and interactive techniques, pp 341–346

  8. Efros A A, Leung T K (2002) Texture synthesis by non-parametric sampling. IEEE international conference on computer vision, pp 1033

  9. Gatys L A, Ecker A S, Bethge M (2015) A neural algorithm of artistic style. arXiv:1508.06576

  10. Hertzmann A, Jacobs C E, Oliver N, Curless B, Salesin D H (2001) Image analogies. Conference on computer graphics and interactive techniques, pp 327–340

  11. Hristova H, Meur O L, Bouatouch K (2015) Style-aware robust color transfer. The workshop on computational aesthetics, pp 67–77

  12. Hwang S J, Kapoor A, Kang S B (2012) Context-based automatic local image enhancement. European conference on computer vision, pp 569–582

  13. Johnson J, Alahi A, Li F (2016) Perceptual losses for real-time style transfer and super-resolution. European conference on computer vision, pp 694–711

    Chapter  Google Scholar 

  14. Kang S B (2017) Visual attribute transfer through deep image analogy. ACM Trans Graph 36(4):120

    Article  Google Scholar 

  15. Kaur P, Zhang H, Dana K J (2017) Photo-realistic facial texture transfer. Computer Science, IEEE international conference on computer vision and pattern recognition

  16. Levin A, Lischinski D, Weiss Y (2008) A closed-form solution to natural image matting. IEEE Trans Pattern Anal Mach Intell 30(2):228–242

    Article  Google Scholar 

  17. Li C, Wand M (2016) Combining markov random fields and convolutional neural networks for image synthesis. IEEE international conference on computer vision and pattern recognition, pp 2479–2486

  18. Lin T Y, Dollar P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. arXiv:1612.03144

  19. Liu J, Yang W, Sun X, Zeng W (2016) Photo stylistic brush: Robust style transfer via superpixel-based bipartite graph. IEEE Transactions on Multimedia pp 1C–1

  20. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. IEEE international conference on computer vision and pattern recognition, pp 3431–3440

  21. Luan F, Paris S, Shechtman E, Bala K (2017) Deep photo style transfer. IEEE international conference on computer vision and pattern recognition, pp 6997–7005

  22. Pitie F, Kokaram A C, Dahyot R (2005) N-dimensional probability density function transfer and its application to color transfer. IEEE international conference on computer vision, pp 1434–1439

  23. Pitie F, Kokaram A C, Dahyot R (2007) Automated colour grading using colour distribution transfer. Comput Vis Image Underst 107(1):123–237

    Article  Google Scholar 

  24. Qin X, Shen J, Mao X, et al. (2015) Structured-patch optimization for dense correspondence. IEEE Trans. Multimedia 17(3):295–306

    Article  Google Scholar 

  25. Rabin J, Ferradans S, Papadakis N (2015) Adaptive color transfer with relaxed optimal transport. IEEE international conference on image processing, pp 4852–4856

  26. Reinhard E, Adhikhmin M, Gooch B, Shirley P (2002) Color transfer between images. IEEE Comput Graph Appl 5:34–41

    Google Scholar 

  27. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. International conference on computing and artificial intelligence, vol 9351, pp 234–241

  28. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  29. Song X, Feng F, Han X, et al. (2018) Neural compatibility modeling with attentive knowledge distillation. The 41st international ACM SIGIR conference on research and development in information retrieval, pp 5–14

  30. Song X, Feng F, Liu J, et al. (2017) Neurostylist: Neural compatibility modeling for clothing matching. Proceedings of the 25th ACM international conference on multimedia, pp 753–761

  31. Wang W, Shen J (2018) Deep visual attention prediction. IEEE Trans Image Process 27(5):2368–2378

    Article  MathSciNet  Google Scholar 

  32. Wang W, Shen J, Ling H (2018) A deep network solution for attention and aesthetics aware photo cropping. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1–1

  33. Wang W, Shen J, Shao L (2017) Video salient object detection via fully convolutional networks. IEEE Transactions on Image Processing, 1–1

  34. Wang W, Shen J, Shao L, et al. (2016) Correspondence driven saliency transfer. IEEE Trans Image Process 25(11):5025–5034

    Article  MathSciNet  Google Scholar 

  35. Wu F, Dong W, Kong Y, Mei X, Paul J C, Zhang X (2013) Content-based colour transfer. Computer graphics forum. Wiley Online Library, pp 190–203

  36. Yang Y, Zhao H, You L, Tu R, Wu X (2017) Semantic portrait color transfer with internet images. Multimed Tools Appl 76(1):523–541

    Article  Google Scholar 

  37. Zhang H, Dana K (2017) Multi-style generative network for real-time transfer. arXiv:1703.06953

Download references

Acknowledgments

The authors wish to acknowledge the financial support from: (i) Chinese Natural Science Foundation under the Grant No. 61602313, 61620106008; (ii) Shenzhen Commission of Scientific Research and Innovations under the Grant No. JCYJ20170302153632883, JCYJ20160422151736824; (iii) Startup Foundation for Advanced Talents, Shenzhen; (iv) The Natural Science Foundation of Guangdong Province No. 2016A030310053, 2017A030310521.

Author information

Authors and Affiliations

  1. College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, 518000, People’s Republic of China

    Xiaoyan Zhang, Xiaole Zhang & Zhijiao Xiao

Authors
  1. Xiaoyan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaole Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhijiao Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhijiao Xiao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, X., Zhang, X. & Xiao, Z. Deep photographic style transfer guided by semantic correspondence. Multimed Tools Appl 78, 34649–34672 (2019). https://doi.org/10.1007/s11042-019-08099-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08099-7

Keywords

Search

Navigation