Twin Deep Convolutional Neural Network for Example-Based Image Colorization

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10424)

Abstract

This paper deals with the colorization of grayscale images. Recent papers have shown remarkable results on image colorization utilizing various deep architectures. Unlike previous methods, we perform colorization using a deep architecture and a reference image. Our architecture utilizes two parallel Convolutional Neural Networks which have the same structure. One CNN, which uses the reference image, helps the other CNN in color prediction for the input image. On the other hand, the second CNN, which uses the input image, helps to identify the areas which holds essential information about the color scheme of the scene. Comprehensive experiments and qualitative and quantitative evaluations were conducted on the images of SUN database and on other images. Quantitative evaluations are based on Peak Signal-to-Noise Ratio (PSNR) and on Quaternion Structural Similarity (QSSIM).

Keywords

Image colorization Deep learning Convolutional Neural Network 

Notes

Acknowledgment

The research was supported by the Hungarian Scientific Research Fund (No. OTKA 120499). We are very thankful to Levente Kovács for helping us with professional advices in high-performance computing.

References

  1. 1.
    Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  2. 2.
    Nagy, B., Benedek, C.: 3D CNN based phantom object removing from mobile laser scanning data. In: International Joint Conference on Neural Networks, pp. 4429–4435 (2017)Google Scholar
  3. 3.
    Bochinski, E., Eiselein, V., Sikora, T.: Training a convolutional neural network for multi-class object detection using solely virtual world data. In: IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 278–285 (2016)Google Scholar
  4. 4.
    Lawrence, S., Giles, C.L., Tsoi, A.C., Back, A.D.: Face recognition: a Convolutional Neural Network approach. IEEE Trans. Neural Networks 8(1), 98–113 (1997)CrossRefGoogle Scholar
  5. 5.
    Ciresan, D., Meier, U.: Multi-column deep neural networks for offline handwritten Chinese character classification. In: Proceedings of the International Joint Conference on Neural Networks, pp. 1–6 (2015)Google Scholar
  6. 6.
    Yan, Z., Zhang, H., Wang, B., Paris, S., Yu, Y.: Automatic photo adjustment using deep learning. CoRR, abs/1412.7725 (2014)Google Scholar
  7. 7.
    Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. ACM Trans. Graph. 23(3), 689–694 (2004)CrossRefGoogle Scholar
  8. 8.
    Huang, Y.C., Tung, Y.S., Chen, J.C., Wang, S.W., Wu, J.L.: An adaptive edge detection based colorization algorithm and its applications. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, pp. 351–354 (2005)Google Scholar
  9. 9.
    Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. 15(5), 1120–1129 (2006)CrossRefGoogle Scholar
  10. 10.
    Reinhard, E., Ashikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. IEEE Comput. Graph. Appl. 21(5), 34–41 (2001)CrossRefGoogle Scholar
  11. 11.
    Welsh, T., Ashikhmin, M., Mueller, K.: Transfering color to greyscale images. ACM Trans. Graph. 21(3), 277–280 (2002)CrossRefGoogle Scholar
  12. 12.
    Irony, R., Cohen-Or, D., Lischinski, D.: Colorization by example. In: Eurographics Symposium on Rendering (2005)Google Scholar
  13. 13.
    Charpiat, G., Hofmann, M., Schölkopf, B.: Automatic image colorization via multimodal predictions. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5304, pp. 126–139. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-88690-7_10 CrossRefGoogle Scholar
  14. 14.
    Gupta, R.K., Chia, A.Y.S., Rajan, D., Ng, E.S., Zhiyong, H.: Image colorization using similar images. In: Proceedings of the 20th ACM International Conference on Multimedia, pp. 369–378 (2012)Google Scholar
  15. 15.
    Bugeau, A., Ta, V.T.: Patch-based image colorization. In: Proceedings of the IEEE International Conference on Pattern Recognition, pp. 3058–3061 (2012)Google Scholar
  16. 16.
    Deshpande, A., Rock, J., Forsyth, D.: Learning large-scale automatic image colorization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 567–575 (2015)Google Scholar
  17. 17.
    Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 415–423 (2015)Google Scholar
  18. 18.
    Tola, E., Lepetit, V., Fua, P.: DAISY: an efficient dense descriptor applied to wide-baseline stereo. IEEE Trans. Pattern Anal. Mach. Intell. 32(5), 815–830 (2010)CrossRefGoogle Scholar
  19. 19.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. (TOG) 35(4), 110 (2016)CrossRefGoogle Scholar
  20. 20.
    Varga, D., Szirányi, T.: Fully automatic image colorization based on Convolutional Neural Network. In: International Conference on Pattern Recognition (2016)Google Scholar
  21. 21.
    Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR, abs/1409.1556 (2014)Google Scholar
  22. 22.
    Kolaman, A., Yadid-Pecht, O.: Quaternion structural similarity: a new quality index for color images. IEEE Trans. Image Process. 21(4), 1526–1536 (2012)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). doi: 10.1007/978-3-319-46487-9_40 CrossRefGoogle Scholar
  24. 24.
    Liang, X., Su, Z., Xiao, Y., Guo, J., Luo, X., Deep patch-wise colorization model for grayscale images. SIGGRAPH ASIA 2016 Technical Briefs 13 (2016)Google Scholar
  25. 25.
    He, K., Sun, J., Tang, X.: Guided image filtering. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 1–14. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15549-9_1 CrossRefGoogle Scholar
  26. 26.
    Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 577–593. Springer, Cham (2016). doi: 10.1007/978-3-319-46493-0_35 CrossRefGoogle Scholar
  27. 27.
    Hariharan, B., Arbeláez, P., Girshick, R., Malik, J.: Hypercolumns for object segmentation and fine-grained localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 447–456 (2015)Google Scholar
  28. 28.
    Deshpande, A., Lu, J., Yeh, M.C., Forsyth, D.: Learning Diverse Image Colorization. arXiv preprint arXiv:1612.01958 (2016)
  29. 29.
    Cao, Y., Zhou, Z., Zhang, W., Yu, Y.: Unsupervised Diverse Colorization via Generative Adversarial Networks. arXiv preprint arXiv:1702.06674 (2017)
  30. 30.
    Limmer, M., Lensch, H.: Infrared Colorization Using Deep Convolutional Neural Networks. arXiv preprint arXiv:1604.02245 (2016)
  31. 31.
    Chollet, F.: Keras (2015). https://github.com/fchollet/keras
  32. 32.
    Xiao, J., Hays, J., Ehinger, K., Oliva, A., Torralba, A.: SUN database: large-scale scene recognition from abbey to zoo. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3485–3492 (2010)Google Scholar
  33. 33.
    Kingma, D., Adam, J.B.: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  34. 34.
    Girosi, F., Jones, M., Poggio, T.: Regularization theory and neural networks architectures. Neural Comput. 7(2), 219–269 (1995)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.MTA SZTAKI, Institute for Computer Science and ControlBudapestHungary
  2. 2.Department of Networked Systems and ServicesBudapest University of Technology and EconomicsBudapestHungary
  3. 3.Department of Material Handling and Logistics SystemsBudapest University of Technology and EconomicsBudapestHungary

Personalised recommendations