Skip to main content
SpringerLink
Log in

SPDGAN: a generative adversarial network based on SPD manifold learning for automatic image colorization

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This paper addresses the automatic colorization problem, which converts a grayscale image to a colorized one. Recent deep learning approaches can colorize automatically grayscale images. However, when it comes to different scenes which contain distinct color styles, it is difficult to accurately capture the color characteristics. In this work, we propose a fully automatic colorization approach based on Symmetric Positive Definite (SPD) Manifold Learning with a generative adversarial network (SPDGAN) that improves the quality of the colorization results. Our SPDGAN model establishes an adversarial game between two discriminators and a generator. The latter is based on ResNet architecture with few alterations. Its goal is to generate fake colorized images without losing color information across layers through residual connections. Then, we employ two discriminators from different domains. The first one is devoted to the image pixel domain, while the second one is to the Riemann manifold domain which helps to avoid color misalignment. Extensive experiments are conducted on the Places365 and COCO-stuff databases to test the effect of each component of our SPDGAN. In addition, quantitative and qualitative comparisons with state-of-the-art methods demonstrate the effectiveness of our model by achieving more realistic colorized images with less artifacts visually, and good results of PSNR, SSIM, and FID values.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available in https://github.com/CSAILVision/places365 and https://github.com/nightrome/cocostuff.

References

  1. Levin A, Lischinski D, Weiss Y (2004) Colorization using optimization. In: ACM Transactions on Graphics (tog), vol. 23, pp. 689–694. ACM

  2. Huang YC, Tung YS, Chen JC, Wang SW, Wu JL (2005) An adaptive edge detection based colorization algorithm and its applications. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, pp 351–354

  3. Luan Q, Wen F, Cohen-Or D, Liang L, Xu YQ, Shum HY (2007) Natural image colorization. In: Proceedings of the 18th Eurographics Conference on Rendering Techniques, pp 309–320

  4. Welsh T, Ashikhmin M, Mueller K (2002) Transferring color to greyscale images. In: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, pp 277–280

  5. Chia AY-S, Zhuo S, Gupta RK, Tai Y-W, Cho S-Y, Tan P, Lin S (2011) Semantic colorization with internet images. ACM Trans on Gr (TOG) 30(6):1–8

    Article  Google Scholar 

  6. Liu X, Wan L, Qu Y, Wong TT, Lin S, Leung CS, Heng PA (2008) Intrinsic colorization. In: ACM SIGGRAPH Asia 2008 Papers, pp 1–9

  7. Deshpande A, Rock J, Forsyth D (2015) Learning large-scale automatic image colorization. In: Proceedings of the IEEE International Conference on Computer Vision, pp 567–575

  8. Cheng Z, Yang Q, Sheng B (2015) Deep colorization. In: Proceedings of the IEEE International Conference on Computer Vision, pp 415–423

  9. Zhang R, Isola P, Efros AA (2016) Colorful image colorization. In: European Conference on Computer Vision, pp 649–666 Springer

  10. Larsson G, Maire M, Shakhnarovich G (2016) Learning representations for automatic colorization. In: European Conference on Computer Vision, pp 577–593 Springer

  11. Zhang RY, Zhu JY, Isola P, Geng X, Lin AS, Yu T, Efros AA (2017) Real-time user-guided image colorization with learned deep priors. ACM Trans on Gr 36(4):119

    Article  Google Scholar 

  12. Xia Y, Qu S, Wan S (2018) Scene guided colorization using neural networks. Neural Computing and Applications, 1–14

  13. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134

  14. Nazeri K, Ng E, Ebrahimi M (2018) Image colorization using generative adversarial networks. In: International Conference on Articulated Motion and Deformable Objects, pp. 85–94. Springer

  15. Vitoria P, Raad L, Ballester C (2020) Chromagan: Adversarial picture colorization with semantic class distribution. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2445–2454

  16. Mourchid Y, Donias M, Berthoumieu Y (2020) Dual color-image discriminators adversarial networks for generating artificial-sar colorized images from sentinel-1 images. In: Machine Learning for Earth Observation Workshop (ECML/PKDD 2020)

  17. Mourchid Y, Donias M, Berthoumieu Y (2021) Automatic image colorization based on multi-discriminators generative adversarial networks. In: 2020 28th European Signal Processing Conference (EUSIPCO), pp. 1532–1536 IEEE

  18. Charpiat G, Hofmann M, Schölkopf B (2008)Automatic image colorization via multimodal predictions. In: European Conference on Computer Vision, pp. 126–139 Springer

  19. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034

  20. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y(2014) Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680

  21. Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423

  22. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711. Springer

  23. Balas BJ (2006) Texture synthesis and perception: Using computational models to study texture representations in the human visual system. Vision Res 46(3):299–309

    Article  Google Scholar 

  24. Zhen X, Chakraborty R, Yang L, Singh V (2021) Flow-based generative models for learning manifold to manifold mappings. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 11042–11052

  25. Huang Z, Van Gool L (2017) A riemannian network for spd matrix learning. In: Thirty-First AAAI Conference on Artificial Intelligence

  26. Wang N, Chen G-D, Tian Y (2022) Image colorization algorithm based on deep learning. Symmetry 14(11):2295

    Article  Google Scholar 

  27. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)

  28. Xiao Y, Jiang A, Liu C, Wang M (2019) Single image colorization via modified cyclegan. In: IEEE International Conference on Image Processing (ICIP), pp. 3247–3251

  29. Antic, J (2019) Deoldify. [Online]. Available:https://github.com/jantic/

  30. Li Y, Wang N, Liu J, Hou X (2017) Demystifying neural style transfer. arXiv preprint arXiv:1701.01036

  31. Arsigny V, Fillard P, Pennec X, Ayache N (2007) Geometric means in a novel vector space structure on symmetric positive-definite matrices. SIAM J Matrix Anal Appl 29(1):328–347

    Article  MathSciNet  MATH  Google Scholar 

  32. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456 PMLR

  33. Miyato T, Kataoka T, Koyama M, Yoshida Y (2018) Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957

  34. Ulyanov D, Vedaldi A, Lempitsky V (2016)Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022

  35. Zhou B, Lapedriza A, Khosla A, Oliva A, Torralba A (2017) Places: A 10 million image database for scene recognition. IEEE Trans Pattern Anal Mach Intell 40(6):1452–1464

    Article  Google Scholar 

  36. Caesar H, Uijlings J, Ferrari V (2018) Coco-stuff: Thing and stuff classes in context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1209–1218

  37. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30

  38. Hasler D, Suesstrunk SE (2003) Measuring colorfulness in natural images. In: Human Vision and Electronic Imaging VIII, vol. 5007, pp. 87–95. SPIE

Download references

Funding

The authors did not receive support from any organization for the submitted work. No funding was received to assist with the preparation of this manuscript. No funding was received for conducting this study. No funds, grants, or other support was received.

Author information

Authors and Affiliations

  1. CESI LINEACT, UR 7527, 21800, Dijon, France

    Youssef Mourchid

  2. Universite de Bordeaux, Bordeaux INP, CNRS, IMS, UMR 5218, 33400, Talence, France

    Marc Donias, Yannick Berthoumieu & Mohamed Najim

Authors
  1. Youssef Mourchid
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marc Donias
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yannick Berthoumieu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohamed Najim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Youssef Mourchid.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose. All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript. The authors have no financial or proprietary interests in any material discussed in this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mourchid, Y., Donias, M., Berthoumieu, Y. et al. SPDGAN: a generative adversarial network based on SPD manifold learning for automatic image colorization. Neural Comput & Applic 35, 23581–23597 (2023). https://doi.org/10.1007/s00521-023-08999-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08999-8

Keywords

Search

Navigation