Abstract
In this paper, we consider the color-plus-mono dual-camera system and propose an end-to-end convolutional neural network to align and fuse images from it in an efficient and cost-effective way. Our method takes cross-domain and cross-scale images as input, and consequently synthesizes HR colorization results to facilitate the trade-off between spatial-temporal resolution and color depth in the single-camera imaging system. In contrast to the previous colorization methods, ours can adapt to color and monochrome cameras with distinctive spatial-temporal resolutions, rendering the flexibility and robustness in practical applications. The key ingredient of our method is a cross-camera alignment module that generates multi-scale correspondences for cross-domain image alignment. Through extensive experiments on various datasets and multiple settings, we validate the flexibility and effectiveness of our approach. Remarkably, our method consistently achieves substantial improvements, i.e., around 10dB PSNR gain, upon the state-of-the-art methods. Code is at: https://github.com/THU-luvision.
This work is supported in part by the Shenzhen Science and Technology Research and Development Funds (JCYJ20180507183706645), in part by the Provincial Key R &D Program of Zhejiang (Serial No. 2021C01016), and the Shenzhen Key Laboratory of next generation interactive media innovative technolog (Funding No. ZDSYS20210623092001004). The lab website is: http://www.luvision.net.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Brady, D.J., et al.: Multiscale gigapixel photography. Nature 486(7403), 386–389 (2012)
Bruhn, A., Weickert, J., Schnörr, C.: Lucas/kanade meets horn/schunck: combining local and global optic flow methods. Int. J. Comput. Vis. 61(3), 211–231 (2005)
Bugeau, A., et al.: Variational exemplar-based image colorization. IEEE TIP (2013)
Cao, X., Tong, X., Dai, Q., Lin, S.: High resolution multispectral video capture with a hybrid camera system. In: CVPR 2011, pp. 297–304. IEEE (2011)
Cao, Y., Zhou, Z., Zhang, W., Yu, Y.: Unsupervised diverse colorization via generative adversarial networks. In: Ceci, M., Hollmén, J., Todorovski, L., Vens, C., Džeroski, S. (eds.) ECML PKDD 2017. LNCS (LNAI), vol. 10534, pp. 151–166. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-71249-9_10
Chang, H., et al.: Palette-based photo recoloring. ACM Trans, Graph (2015)
Charpiat, G., Hofmann, M., Schölkopf, B.: Automatic image colorization via multimodal predictions. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5304, pp. 126–139. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88690-7_10
Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: ICCV, pp. 415–423 (2015)
Chia, A.Y.S., et al.: Semantic colorization with internet images. In: ACM TOG (2011)
Cossairt, O.S., et al.: Gigapixel computational imaging. In: ICCP. IEEE (2011)
Dong, X., Li, W.: Shoot high-quality color images using dual-lens system with monochrome and color cameras. Neurocomputing 352, 22–32 (2019)
Dosovitskiy, A., et al.: Flownet: learning optical flow with conv networks. In: ICCV (2015)
Goodfellow, I., et al.: Generative adversarial nets. In: NeuIPS, pp. 2672–2680 (2014)
Gupta, R.K., et al.: Image colorization using similar images. In: ACM MM (2012)
HaCohen, Y., Shechtman, E., Goldman, D.B., Lischinski, D.: Non-rigid dense correspondence with applications for image enhancement. ACM TOG 30(4), 1–10 (2011)
He, M., Liao, J., Yuan, L., Sander, P.V.: Neural color transfer between images. arXiv (2017)
He, M., et al.: Deep exemplar-based colorization. In: ACM TOG (2018)
Huang, Y.C., et al.: An adaptive edge detection based colorization algorithm and its applications. In: ACM MM (2005)
Ilg, E., et al.: Flownet 2.0: evolution of optical flow estimation with deep networks. In: CVPR, pp. 2462–2470 (2017)
Ironi, R., et al.: Colorization by example. In: Rendering Techniques. Citeseer (2005)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR, pp. 1125–1134 (2017)
Jancsary, J., Nowozin, S., Sharp, T., Rother, C.: Regression tree fields-an efficient, non-parametric approach to image labeling problems. In: CVPR, pp. 2376–2383. IEEE (2012)
Jeon, H.G., et al.: Stereo matching with color and monochrome cameras in low-light conditions. In: CVPR (2016)
Jin, D., et al.: All-in-depth via cross-baseline light field camera. In: ACM MM (2020)
Levin, A., et al.: Colorization using optimization. In: ACM SIGGRAPH 2004 Papers (2004)
Li, G., et al.: Zoom in to the details of human-centric videos. In: ICIP. IEEE (2020)
Liao, J., et al.: Visual attribute transfer through deep image analogy. arXiv (2017)
Liu, C., et al.: Sift flow: Dense correspondence across scenes and its applications. IEEE TPAMI (2010)
Liu, C., Shan, J., Liu, G.: High resolution array camera (Apr 19 2016), US Patent 9,319,585
Liu, X., et al.: Intrinsic colorization. In: ACM SIGGRAPH Asia 2008 papers, pp. 1–9 (2008)
Lohmann, et al.: Space-bandwidth product of optical signals and systems. JOSA A (1996)
Luan, Q., Wen, F., Cohen-Or, D., Liang, L., Xu, Y.Q., Shum, H.Y.: Natural image colorization. In: Eurographics Conference on Rendering Techniques, pp. 309–320 (2007)
Ma, C., Cao, X., Tong, X., Dai, Q., Lin, S.: Acquisition of high spatial and spectral resolution video with a hybrid camera system. IJCV 110(2), 141–155 (2014)
Mantzel, W., et al.: Shift-and-match fusion of color and mono images (2017), US Patent
Reinhard, E., et al.: Color transfer between images. IEEE Comput. Graph. Appl. 21(5), 34–41 (2001)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. IJCV (2015)
Sharif, S., Jung, Y.J.: Deep color reconstruction for a sparse color sensor. Opt. Express 27(17), 23661–23681 (2019)
Srinivasan, P.P., et al.: Learning to synthesize a 4D RGBD light field from a single image. In: ICCV, pp. 2243–2251 (2017)
Tai, Y.W., Jia, J., Tang, C.K.: Local color transfer via probabilistic segmentation by expectation-maximization. In: CVPR, vol. 1, pp. 747–754. IEEE (2005)
Tai, Y.W., et al.: Image/video deblurring using a hybrid camera. In: CVPR. IEEE (2008)
Tan, Y., et al.: Crossnet++: Cross-scale large-parallax warping for reference-based super-resolution. IEEE TPAMI (2020)
Union, I.T.: Encoding parameters of digital television for studios. CCIR Recommend. (1992)
Vitoria, P., Raad, L., Ballester, C.: Chromagan: An adversarial approach for picture colorization. arXiv preprint arXiv:1907.09837 (2019)
Wang, T.C., et al.: Light field video capture using a learning-based hybrid imaging system. ACM TOG (2017)
Wang, X., et al.: Panda: A gigapixel-level human-centric video dataset. In: CVPR (2020)
Wang, Y., Liu, Y., Heidrich, W., Dai, Q.: The light field attachment: Turning a DSLR into a light field camera using a low budget camera ring. IEEE TVCG 23(10), 2357–2364 (2016)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE TIP 13(4), 600–612 (2004)
Welsh, T., Ashikhmin, M., Mueller, K.: Transferring color to greyscale images. In: Annual Conference on Computer Graphics and Interactive Techniques, pp. 277–280 (2002)
Xiao, C., Han, C., Zhang, Z., others, G., He, S.: Example-based colourization via dense encoding pyramids. In: Computer Graphics Forum. Wiley Online Library (2020)
Xue, T., et al.: Video enhancement with task-oriented flow. In: IJCV (2019)
Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. 15(5), 1120–1129 (2006)
Yoo, S., Bahng, H., Chung, S., Lee, J., Chang, J., Choo, J.: Coloring with limited data: few-shot colorization via memory augmented networks. In: CVPR, pp. 11283–11292 (2019)
Yuan, X., Fang, L., Dai, Q., Brady, D.J., Liu, Y.: Multiscale gigapixel video: a cross resolution image matching and warping approach. In: ICCP, pp. 1–9. IEEE (2017)
Yuan, X., et al.: A modular hierarchical array camera. Science & Applications, Light (2021)
Zhang, J., Zhu, T., Zhang, A., et al.: Multiscale-VR: multiscale gigapixel 3D panoramic videography for virtual reality. In: ICCP, pp. 1–12. IEEE (2020)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR, pp. 586–595 (2018)
Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40
Zhao, Y., Li, G., Wang, Z., Lam, E.Y.: Cross-camera human motion transfer by time series analysis. arXiv preprint arXiv:2109.14174 (2021)
Zhao, Y., et al.: EFENet: reference-based video super-resolution with enhanced flow estimation. In: CICAI, pp. 371–383. Springer (2021). https://doi.org/10.1007/978-3-030-93046-2_32
Zhao, Y., et al.: MANet: improving video denoising with a multi-alignment network. arXiv preprint arXiv:2202.09704 (2022)
Zheng, H., Ji, M., Wang, H., Liu, Y., Fang, L.: CrossNet: an end-to-end reference-based super resolution network using cross-scale warping. In: ECCV, pp. 88–104 (2018)
Zhou, B., et al.: Learning deep features for scene recognition using places database. In: NeuIPS, pp. 487–495 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhao, Y., Zheng, H., Ji, M., Huang, R. (2022). Cross-Camera Deep Colorization. In: Fang, L., Povey, D., Zhai, G., Mei, T., Wang, R. (eds) Artificial Intelligence. CICAI 2022. Lecture Notes in Computer Science(), vol 13604. Springer, Cham. https://doi.org/10.1007/978-3-031-20497-5_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-20497-5_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20496-8
Online ISBN: 978-3-031-20497-5
eBook Packages: Computer ScienceComputer Science (R0)