Skip to main content
SpringerLink
Log in

Image Colorization using CycleGAN with semantic and spatial rationality

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The goal of image colorization is to make the generated color images closely approximate the color layout of the real color images. However, most of the existing methods do not consider the semantic and spatial rationality of the generated images, and this could lead to a large difference between the colored image and the real situation. In this research we propose SS-CycleGAN, a novel CycleGAN based solution for automatic image colorization. SS-CycleGAN ensures the rationality of colored images considering three aspects: high-level semantics, detailed semantics, and spatial information of the objects to be colored in the image. We designed a patch discriminator for SS-CycleGAN based on a self-attention mechanism. The self-attention mechanism can guide the patch discriminator to pay attention to spatial structure information and the semantic rationality of colored objects. The loss function of SS-CycleGAN is added with a term for detail loss, which can ensure the consistency of the details of the original image and the generated image. To extract multi-scale features of local areas to capture the spatial information of colored objects, we designed a Multi-scale Cascaded Dilated Convolution (MCDC) module. We trained and tested the proposed SS-CycleGAN on Natura Color Dataset and Flower dataset. The experimental results show that SS-CycleGAN can obtain higher quality colorized images than several state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Anwar S, Tahir M, Li C, Mian A, Khan FS, Muzaffar AW (2020) Image colorization: A survey and dataset. arXiv preprint arXiv:2008.10774

  2. Bothra D, Shetty R, Bhagat S, Patil M (2021) ColorAI -automatic image Colorization using CycleGAN. Int J Sci Res Eng Manag

  3. Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2018) DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Trans Pattern Anal Mach Intell:834–848

  4. Ci Y, Ma X, Wang Z, Li H, Luo Z (2018) User-guided deep anime line art colorization with conditional adversarial networks, In: 26th ACM international conference on Multimedia, 1536–1544

  5. Goodfellow IJ, Pouget-Abadie J, Mirza M et al. (2014) Generative adversarial networks. International Conference on Neural Information Processing Systems (NIPS), 2672–2680

  6. Huang S, Jin X, Jiang Q, Li J, Lee SJ, Wang P, Yao S (2021) A fully-automatic image colorization scheme using improved CycleGAN with skip connections. Multimed Tools Appl 80:1–28

    Article  Google Scholar 

  7. Iizuka S, Simo-Serra E, Ishikawa H (2016) Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans Graph 35:1–11

    Article  Google Scholar 

  8. Irony R, Cohen-Or D, Lischinski D (2005) Colorization by example. Rendering Techniques. 201–210

  9. Isola P, Zhu JY, Zhou T et al Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1125–1134

  10. Larsson G, Maire M, Shakhnarovich G (2007) Learning Representations for Automatic Colorization. European Conference on Computer Vision (ECCV), 577–593

  11. Lee J, Kim E, Lee Y, Kim D, Chang J, Choo J (2020) Reference-based sketch image Colorization using augmented-self reference and dense semantic correspondence. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5800–5809

  12. Luan Q, Wen F, Cohen-Or D, Liang L, Xu Y-Q, Shum H-Y (2007) Natural image colorization. In: Proceedings of the 18th Eurographics Conference on Rendering Techniques

  13. Mehri A, Sappa AD (2019) Colorizing near infrared images through a cyclic adversarial approach of unpaired samples, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 971–979

  14. Miyato T, Kataoka T, Koyama M, Yoshida Y (2018) Spectral normalization for generative adversarial networks. International Conference on Learning Representations

  15. Ozbulak G (2019) Image Colorization by capsule networks, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2150–2158

  16. Qu Y, Wong TT, Heng PA (2006) Manga colorization. ACM Trans Graph (TOG) 25:1214–1220

    Article  Google Scholar 

  17. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention Springer International Publishing, 234–241

  18. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1555, 1–14.https://doi.org/10.48550/arXiv.1409.1556

  19. Su JW, Chu HK, Huang JB (2020) Instance-aware image Colorization, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7965–7974

  20. Suárez PL, Sappa AD, Vintimilla BX (2017) Infrared image Colorization based on a triplet DCGAN architecture, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 212–217

  21. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. The National Conference on Artificial Intelligence (AAAI)

  22. Ulyanov D, Vedaldi A, Lempitsky V (2016). Instance normalization: the missing ingredient for fast stylization. arXiv:1607.08022

  23. URL of Flower Dataset (n.d.) https://www.kaggle.com/alxmamaev/flowers-recognition

  24. Wang P, Patel VM (2018) Generating high quality visible images from SAR images using CNNs, 2018 IEEE Radar Conference (RadarConf18), 0570–0575

  25. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (April 2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  26. Welsh T, Ashikhmin M, Mueller K (2002) Transferring color to greyscale images. ACM Trans Graph 21(3):277–280. https://dl.acm.org/doi/10.1145/566654.566576

  27. Yatziv, Sapiro G (2006) Fast image and video colorization using chrominance blending. IEEE Trans Image Process 15:1120–1129

    Article  Google Scholar 

  28. Zhang R., Isola P., Efros A.A. (2016) Colorful Image Colorization. European Conference Computer Vision (ECCV), 649–666.

  29. Zhang R, Zhu JY, Isola P, Geng X, Lin AS, Yu T, Efros AA (2017) Real-time user-guided image colorization with learned deep priors. ACM Trans Graph (TOG) 36:1–11

    Google Scholar 

  30. Zhu J, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks, 2017 IEEE International Conference on Computer Vision (ICCV), 2242–2251

Download references

Acknowledgements

This research is partially supported by Natural Science Foundation Project of science and Technology Department of Jilin Province under Grant no. 20200201165JC.

Author information

Authors and Affiliations

  1. School of Computer Science, Northeast Electric Power University, Jilin, 132012, China

    Bin Li, Yi Lu & Huixin Xu

  2. Gongqing Institute of Science and Technology, No. 1 Gongqing Road, Gongqing City, 332020, China

    Bin Li

  3. School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh, EH14 4AS, UK

    Wei Pang

Authors
  1. Bin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yi Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Huixin Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bin Li.

Ethics declarations

Conflict of interest

The authors declare that there is no potential conflict (financial or non-financial) of interests regarding the publication of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, B., Lu, Y., Pang, W. et al. Image Colorization using CycleGAN with semantic and spatial rationality. Multimed Tools Appl 82, 21641–21655 (2023). https://doi.org/10.1007/s11042-023-14675-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14675-9

Keywords

Search

Navigation