Skip to main content
SpringerLink
Log in

Analysis of Different Losses for Deep Learning Image Colorization

  • Living reference work entry
  • First Online:
Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging

Abstract

Image colorization aims to add color information to a grayscale image in a realistic way. Recent methods mostly rely on deep learning strategies. While learning to automatically colorize an image, one can define well-suited objective functions related to the desired color output. Some of them are based on a specific type of error between the predicted image and ground truth one, while other losses rely on the comparison of perceptual properties. But, is the choice of the objective function that crucial, i.e., does it play an important role in the results? In this chapter, we aim to answer this question by analyzing the impact of the loss function on the estimated colorization results. To that goal, we review the different losses and evaluation metrics that are used in the literature. We then train a baseline network with several of the reviewed objective functions, classic L1 and L2 losses, as well as more complex combinations such as Wasserstein GAN and VGG-based LPIPS loss. Quantitative results show that the models trained with VGG-based LPIPS provide overall slightly better results for most evaluation metrics. Qualitative results exhibit more vivid colors when trained with Wasserstein GAN plus the L2 loss or again with the VGG-based LPIPS. Finally, the convenience of quantitative user studies is also discussed to overcome the difficulty of properly assessing on colorized images, notably for the case of old archive photographs where no ground truth is available.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  • Antic, J.: Deoldify. https://github.com/jantic/DeOldify (2019)

  • Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein Generative Adversarial Networks. In: International Conference on Machine Learning, vol 70, pp. 214–223 (2017)

    Google Scholar 

  • Cao, Y., Zhou, Z., Zhang, W., Yu, Y.: Unsupervised diverse colorization via Generative Adversarial Networks. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 151–166 (2017)

    Google Scholar 

  • Chen, X., Mishra, N., Rohaninejad, M., Abbeel, P.: Pixelsnail: an improved autoregressive generative model. In: International Conference on Machine Learning, pp. 864–872 (2018)

    Google Scholar 

  • Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: IEEE International Conference on Computer Vision, pp. 415–423 (2015)

    Google Scholar 

  • Deshpande, A., Lu, J., Yeh, M.-C., Jin Chong, M., Forsyth, D.: Learning diverse image colorization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6837–6845 (2017)

    Google Scholar 

  • Ding, K., Ma, K., Wang, S., Simoncelli, E.P.: Comparison of full-reference image quality models for optimization of image processing systems. Int. J. Comput. Vis. 129(4), 1258–1281 (2021)

    Article  Google Scholar 

  • Dowson, D., Landau, B.: The Fréchet distance between multivariate normal distributions. J. Multivar. Anal. 12(3), 450–455 (1982)

    Article  MATH  Google Scholar 

  • Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. J. Vis. 16(12), 326 (2016)

    Article  Google Scholar 

  • Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (2014)

    Google Scholar 

  • Guadarrama, S., Dahl, R., Bieber, D., Norouzi, M., Shlens, J., Murphy, K.: Pixcolor: pixel recursive colorization. In: British Machine Vision Conference (2017)

    Google Scholar 

  • Gu, S., Timofte, R., Zhang, R.: Ntire 2019 challenge on image colorization: report. In: Conference on Computer Vision and Pattern Recognition Workshops (2019)

    Google Scholar 

  • Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.: Improved training of Wasserstein GANs. In: Advances in Neural Information Processing Systems, pp. 5769–5779 (2017)

    Google Scholar 

  • He, M., Chen, D., Liao, J., Sander, P.V., Yuan, L.: Deep exemplar-based colorization. ACM Trans. Graph. 37(4), 1–16 (2018)

    Google Scholar 

  • Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  • Ho, J., Kalchbrenner, N., Weissenborn, D., Salimans, T.: Axial attention in multidimensional transformers (2019). arXiv preprint arXiv:1912.12180

    Google Scholar 

  • Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. 35(4), 1–11 (2016)

    Article  Google Scholar 

  • Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

    Google Scholar 

  • Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711 (2016)

    Google Scholar 

  • Kong, G., Tian, H., Duan, X., Long, H.: Adversarial edge-aware image colorization with semantic segmentation. IEEE Access 9, 28194–28203 (2021)

    Article  Google Scholar 

  • Kumar, M., Weissenborn, D., Kalchbrenner, N.: Colorization transformer (2021). arXiv preprint arXiv:2102.04432

    Google Scholar 

  • Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic colorization. In: European Conference on Computer Vision, pp. 577–593 (2016)

    Google Scholar 

  • Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: European Conference on Computer Vision, pp. 740–755 (2014)

    Google Scholar 

  • Lübbe, E.: Colours in the Mind-Colour Systems in Reality: A Formula for Colour Saturation. BoD–Books on Demand, Norderstedt (2010)

    Google Scholar 

  • Mouzon, T., Pierre, F., Berger, M.-O.: Joint CNN and variational model for fully-automatic image colorization. In: Scale Space and Variational Methods in Computer Vision, pp. 535–546 (2019)

    Google Scholar 

  • Nazeri, K., Ng, E., Ebrahimi, M.: Image colorization using Generative Adversarial Networks. In: International Conference on Articulated Motion and Deformable Objects, pp. 85–94 (2018)

    Google Scholar 

  • Oord, A.V.D., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., Kavukcuoglu, K.: Conditional image generation with PixelCNN decoders. In: Advances in Neural Information Processing Systems (2016)

    Google Scholar 

  • Pierre, F., Aujol, J.-F.: Recent approaches for image colorization. In: Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging (2020)

    Google Scholar 

  • Pierre, F., Aujol, J.-F., Bugeau, A., Papadakis, N., Ta, V.-T.: Luminance-chrominance model for image colorization. SIAM J. Imag. Sci. 8(1), 536–563 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  • Pucci, R., Micheloni, C., Martinel, N.: Collaborative image and object level features for image colourisation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2160–2169 (2021)

    Google Scholar 

  • Riba, E., Mishkin, D., Ponsa, D., Rublee, E., Bradski, G.: Kornia: an open source differentiable computer vision library for PyTorch. In: Winter Conference on Applications of Computer Vision, pp. 3674–3683 (2020)

    Google Scholar 

  • Royer, A., Kolesnikov, A., Lampert, C.H.: Probabilistic image colorization. In: British Machine Vision Conference (2017)

    Google Scholar 

  • Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  • Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)

    Google Scholar 

  • Su, J.-W., Chu, H.-K., Huang, J.-B.: Instance-aware image colorization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 7968–7977 (2020)

    Google Scholar 

  • Van Oord, A., Kalchbrenner, N., Kavukcuoglu, K.: Pixel recurrent neural networks. In: International Conference on Machine Learning, pp. 1747–1756 (2016)

    Google Scholar 

  • Vitoria, P., Raad, L., Ballester, C.: ChromaGAN: adversarial picture colorization with semantic class distribution. In: Winter Conference on Applications of Computer Vision, pp. 2445–2454 (2020)

    Google Scholar 

  • Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  • Yoo, S., Bahng, H., Chung, S., Lee, J., Chang, J., Choo, J.: Coloring with limited data: few-shot colorization via memory augmented networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2019)

    Google Scholar 

  • Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: European Conference on Computer Vision, pp. 649–666 (2016)

    Google Scholar 

  • Zhang, R., Zhu, J.-Y., Isola, P., Geng, X., Lin, A.S., Yu, T., Efros, A.A.: Real-time user-guided image colorization with learned deep priors. ACM Trans. Graph. 36, 1–11 (2017)

    Google Scholar 

  • Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)

    Google Scholar 

Download references

Acknowledgements

This study has been carried out with financial support from the French Research Agency through the PostProdLEAP project (ANR-19-CE23-0027-01) and from the EU Horizon 2020 research and innovation programme NoMADS (Marie Skłodowska-Curie grant agreement No 777826). The first and fourth authors acknowledge partial support by MICINN/FEDER UE project, ref. PGC2018-098625-B-I00, and RED2018-102511-T. This chapter was written together with another chapter of the current handbook, called “Influence of Color Spaces for Deep Learning Image Colorization.” All authors have contributed to both chapters.

Author information

Authors and Affiliations

  1. University Pompeu Fabra, Barcelona, Spain

    Coloma Ballester & Patricia Vitoria

  2. LaBRI, CNRS, Bordeaux INP, Université de Bordeaux, Bordeaux, France

    Hernan Carrillo & Michaël Clément

Authors
  1. Coloma Ballester
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hernan Carrillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michaël Clément
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Patricia Vitoria
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michaël Clément .

Editor information

Editors and Affiliations

  1. Department of Mathematical Sciences, The University of Liverpool, Liverpool, UK

    Ke Chen

  2. Dept Appl Math & Theoretical Phys, University of Cambridge, Cambridge, UK

    Carola-Bibiane Schönlieb

  3. Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong

    Xue-Cheng Tai

  4. Department of Applied Mathematics & Stat, Johns Hopkins University, Baltimore, MD, USA

    Laurent Younces

Section Editor information

  1. Department of Mathematical Sciences, University of Liverpool Mathematical Sciences Building, Liverpool, UK

    Ke Chen Ph.D

  2. Department of Applied Mathematics and Statistics, The Johns Hopkins University, Baltimore, MD, USA

    Laurent Younes

  3. Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong, China

    Xuecheng Tai Ph.D

  4. Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK

    Carola-Bibiane Schönlieb Ph.D

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Ballester, C., Carrillo, H., Clément, M., Vitoria, P. (2022). Analysis of Different Losses for Deep Learning Image Colorization. In: Chen, K., Schönlieb, CB., Tai, XC., Younces, L. (eds) Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging. Springer, Cham. https://doi.org/10.1007/978-3-030-03009-4_127-1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-03009-4_127-1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-03009-4

  • Online ISBN: 978-3-030-03009-4

  • eBook Packages: Springer Reference MathematicsReference Module Computer Science and Engineering

Publish with us

Policies and ethics

Search

Navigation