Learning Representations for Automatic Colorization

  • Gustav LarssonEmail author
  • Michael Maire
  • Gregory Shakhnarovich
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9908)


We develop a fully automatic image colorization system. Our approach leverages recent advances in deep networks, exploiting both low-level and semantic representations. As many scene elements naturally appear according to multimodal color distributions, we train our model to predict per-pixel color histograms. This intermediate output can be used to automatically generate a color image, or further manipulated prior to image formation. On both fully and partially automatic colorization tasks, we outperform existing methods. We also explore colorization as a vehicle for self-supervised visual representation learning.


Color Space Reference Image Color Histogram Deep Convolutional Neural Network Semantic Segmentation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



We thank Ayan Chakrabarti for suggesting lightness-normalized quantile matching and for useful discussions, and Aditya Deshpande and Jason Rock for discussions on their work. We gratefully acknowledge the support of NVIDIA Corporation with the donation of GPUs for this research.

Supplementary material

419976_1_En_35_MOESM1_ESM.pdf (6.8 mb)
Supplementary material 1 (pdf 6918 KB)


  1. 1.
    Bertasius, G., Shi, J., Torresani, L.: Deepedge: a multi-scale bifurcated deep network for top-down contour detection. In: CVPR (2015)Google Scholar
  2. 2.
    Charpiat, G., Bezrukov, I., Altun, Y., Hofmann, M., Schölkopf, B.: Machine learning methods for automatic image colorization. In: Computational Photography: Methods and Applications. CRC Press (2010)Google Scholar
  3. 3.
    Charpiat, G., Hofmann, M., Schölkopf, B.: Automatic image colorization via multimodal predictions. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5304, pp. 126–139. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-88690-7_10 CrossRefGoogle Scholar
  4. 4.
    Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Semantic image segmentation with deep convolutional nets and fully connected CRFs. In: ICLR (2015)Google Scholar
  5. 5.
    Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: ICCV (2015)Google Scholar
  6. 6.
    Chia, A.Y.S., Zhuo, S., Gupta, R.K., Tai, Y.W., Cho, S.Y., Tan, P., Lin, S.: Semantic colorization with internet images. ACM Trans. Graph. (TOG) 30(6) (2011)Google Scholar
  7. 7.
    Deshpande, A., Rock, J., Forsyth, D.: Learning large-scale automatic image colorization. In: ICCV (2015)Google Scholar
  8. 8.
    Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1422–1430 (2015)Google Scholar
  9. 9.
    Donahue, J., Krähenbühl, P., Darrell, T.: Adversarial feature learning (2016). arXiv preprint arXiv:1605.09782
  10. 10.
    Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1915–1929 (2013)CrossRefGoogle Scholar
  11. 11.
    Ganin, Y., Lempitsky, V.S.: N\({}^{\text{4}}\)-fields: neural network nearest neighbor fields for image transforms. In: ACCV (2014)Google Scholar
  12. 12.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS (2010)Google Scholar
  13. 13.
    Gupta, R.K., Chia, A.Y.S., Rajan, D., Ng, E.S., Zhiyong, H.: Image colorization using similar images. In: ACM International Conference on Multimedia (2012)Google Scholar
  14. 14.
    Hariharan, B., an R. Girshick, P.A., Malik, J.: Hypercolumns for object segmentation and fine-grained localization. In: CVPR (2015)Google Scholar
  15. 15.
    Huang, Y.C., Tung, Y.S., Chen, J.C., Wang, S.W., Wu, J.L.: An adaptive edge detection based colorization algorithm and its applications. In: ACM International Conference on Multimedia (2005)Google Scholar
  16. 16.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. (Proc. SIGGRAPH 2016) 35(4) (2016)Google Scholar
  17. 17.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML (2015)Google Scholar
  18. 18.
    Ion, A., Carreira, J., Sminchisescu, C.: Probabilistic joint image segmentation and labeling by figure-ground composition. Int. J. Comput. Vision 107(1), 40–57 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Irony, R., Cohen-Or, D., Lischinski, D.: Colorization by example. In: Eurographics Symposium on Rendering (2005)Google Scholar
  20. 20.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding (2014). arXiv preprint arXiv:1408.5093
  21. 21.
    Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. ACM Trans. Graph. (TOG) 23(3), 689–694 (2004)CrossRefGoogle Scholar
  22. 22.
    Liu, W., Rabinovich, A., Berg, A.C.: Parsenet: looking wider to see better (2015). arXiv preprint arXiv:1506.04579
  23. 23.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)Google Scholar
  24. 24.
    Luan, Q., Wen, F., Cohen-Or, D., Liang, L., Xu, Y.Q., Shum, H.Y.: Natural image colorization. In: Eurographics Conference on Rendering Techniques (2007)Google Scholar
  25. 25.
    Maire, M., Yu, S.X., Perona, P.: Reconstructive sparse code transfer for contour detection and semantic labeling. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 273–287. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-16817-3_18 Google Scholar
  26. 26.
    Morimoto, Y., Taguchi, Y., Naemura, T.: Automatic colorization of grayscale images using multiple images on the web. In: SIGGRAPH: Posters (2009)Google Scholar
  27. 27.
    Mostajabi, M., Yadollahpour, P., Shakhnarovich, G.: Feedforward semantic segmentation with zoom-out features. In: CVPR (2015)Google Scholar
  28. 28.
    Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles (2016). arXiv preprint arXiv:1603.09246 Google Scholar
  29. 29.
    Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: CVPR (2016)Google Scholar
  30. 30.
    Patterson, G., Xu, C., Su, H., Hays, J.: The sun attribute database: beyond categories for deeper scene understanding. Int. J. Comput. Vision 108(1–2), 59–81 (2014)CrossRefGoogle Scholar
  31. 31.
    Qu, Y., Wong, T.T., Heng, P.A.: Manga colorization. ACM Trans. Graph. (TOG) 25(3), 1214–1220 (2006)CrossRefGoogle Scholar
  32. 32.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision (IJCV) 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Sapiro, G.: Inpainting the colors. In: ICIP (2005)Google Scholar
  34. 34.
    Shen, W., Wang, X., Wang, Y., Bai, X., Zhang, Z.: Deepcontour: a deep convolutional feature learned by positive-sharing loss for contour detection. In: CVPR (2015)Google Scholar
  35. 35.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 888–905 (2000)CrossRefGoogle Scholar
  36. 36.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)Google Scholar
  37. 37.
    Sỳkora, D., Buriánek, J., Žára, J.: Unsupervised colorization of black-and-white cartoons. In: International Symposium on Non-Photorealistic Animation and Rendering (2004)Google Scholar
  38. 38.
    Tai, Y.W., Jia, J., Tang, C.K.: Local color transfer via probabilistic segmentation by expectation-maximization. In: CVPR (2005)Google Scholar
  39. 39.
    Tola, E., Lepetit, V., Fua, P.: A fast local descriptor for dense matching. In: CVPR (2008)Google Scholar
  40. 40.
    Tsaftaris, S.A., Casadio, F., Andral, J.L., Katsaggelos, A.K.: A novel visualization tool for art history and conservation: automated colorization of black and white archival photographs of works of art. Stud. Conserv. 59(3), 125–135 (2014)CrossRefGoogle Scholar
  41. 41.
    Welsh, T., Ashikhmin, M., Mueller, K.: Transferring color to greyscale images. ACM Trans. Graph. (TOG) 21(3), 277–280 (2002)CrossRefGoogle Scholar
  42. 42.
    Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: large-scale scene recognition from abbey to zoo. In: CVPR (2010)Google Scholar
  43. 43.
    Xie, S., Tu, Z.: Holistically-nested edge detection. In: ICCV (2015)Google Scholar
  44. 44.
    Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. 15(5), 1120–1129 (2006)CrossRefGoogle Scholar
  45. 45.
    Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: ECCV (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Gustav Larsson
    • 1
    Email author
  • Michael Maire
    • 2
  • Gregory Shakhnarovich
    • 2
  1. 1.University of ChicagoChicagoUSA
  2. 2.Toyota Technological Institute at ChicagoChicagoUSA

Personalised recommendations