Abstract
Style transfer is a computer vision task that attempts to transfer the style of an artistic image to a content image. Thanks to the advance in Deep Convolutional Neural Networks, exciting style transfer results has been achieved, but traditional algorithms do not fully understand semantic information. Those algorithms are not aware of which regions in the style image have to be transferred to which regions in the content image. A common failure case is style transfer involving landscape images. After stylization, the textures and colors of the land are often found in incoherent places such as in the river or in the sky. In this work, we investigate semantic style transfer for content images with more than 2 semantic regions. We combine guided Gram matrices with gradient capping and multi-scale representations. Our approach simplifies the parameter tuning problem, improves the style transfer results and is faster than current semantic methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Athalye, A.: Neural style (2015). github.com/anishathalye/neural-style
Castillo, C., De, S., Han, X., Singh, B., Yadav, A., Goldstein, T.: Son of Zorns lemma: targeted style transfer using instance-aware semantic segmentation (2017). arXiv:1701.02357
Champandard, A.J.: Semantic style transfer and turning two-bit doodles into fine artwork (2016). arXiv:1603.01768
Chan, E., Bhargava, R.: Show, divide and neural: weighted style transfer. CS231n: Convolutional Neural Networks for Visual Recognition, Project Reports (2016)
Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style (2017). arXiv:1610.07629
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution (2016). arXiv:1603.081155
Leon, A., Alexander, S., Matthias, B.: Image style transfer using convolutional neural networks (2015). arXiv:1508.06576
Leon, A., Alexander, S., Matthias, B., Hertzmann, A., Shechtman, E.: Supplementary material: controlling perceptual factors in neural style transfer. bethgelab.org (2016)
Leon, A., Alexander, S., Matthias, B., Hertzmann, A., Shechtman, E.: Controlling perceptual factors in neural style transfer (2017). arXiv:1611.07865
Li, C., Wand, M.: Combining markov random fields and convolutional neural networks for image synthesis (2016). arXiv:1601.04589
Lu, M., Zhao, H.L., Yao, A., Xu, F., Chen, Y., Zhang, L.: Decoder network over lightweight reconstructed feature for fast semantic style transfer. In: The IEEE International Conference on Computer Vision (ICCV) (2017)
Mahendran, A., Vedaldi, A.: Multi-style generative network for real-time transfer (2017). arXiv:1412.0035
Novak, R., Nikulin, Y.: Improving the neural algorithm of artistic style (2016). arXiv:1605.04603
Risser, E., Wilmot, P., Barnes, C.: Stable and controllable neural texture synthesis and style transfer using histogram losses (2017). arXiv:1701.08893
Ulyanov, D., Vedaldi, A., Lepitsky, V.: Instance normalization: the missing ingredient for fast stylization (2017). arXiv:1611.07865
Zhang, H., Dana, K.: Understanding deep image representation by inverting them (2017). arXiv:1703.06953
Zhao, H., Rosin, P.L., Lai, Y.: Automatic semantic style transfer using deep convolutional neural networks and soft masks (2017). arXiv:1708.09641
Acknowledgment
This work was supported by NSFC (61671296 and 61521062) and the Shanghai Key Laboratory of Digital Media Processing and Transmissions.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Nicolas, C., Xie, R., Song, L., Zhang, W. (2019). Improving Semantic Style Transfer Using Guided Gram Matrices. In: Zhai, G., Zhou, J., An, P., Yang, X. (eds) Digital TV and Multimedia Communication. IFTC 2018. Communications in Computer and Information Science, vol 1009. Springer, Singapore. https://doi.org/10.1007/978-981-13-8138-6_14
Download citation
DOI: https://doi.org/10.1007/978-981-13-8138-6_14
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-8137-9
Online ISBN: 978-981-13-8138-6
eBook Packages: Computer ScienceComputer Science (R0)