Advertisement

Low Light Video Enhancement Using Synthetic Data Produced with an Intermediate Domain Mapping

Conference paper
  • 538 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12358)

Abstract

Advances in low-light video RAW-to-RGB translation are opening up the possibility of fast low-light imaging on commodity devices (e.g. smartphone cameras) without the need for a tripod. However, it is challenging to collect the required paired short-long exposure frames to learn a supervised mapping. Current approaches require a specialised rig or the use of static videos with no subject or object motion, resulting in datasets that are limited in size, diversity, and motion. We address the data collection bottleneck for low-light video RAW-to-RGB by proposing a data synthesis mechanism, dubbed SIDGAN, that can generate abundant dynamic video training pairs. SIDGAN maps videos found ‘in the wild’ (e.g. internet videos) into a low-light (short, long exposure) domain. By generating dynamic video data synthetically, we enable a recently proposed state-of-the-art RAW-to-RGB model to attain higher image quality (improved colour, reduced artifacts) and improved temporal consistency, compared to the same model trained with only static real video data.

Supplementary material

504454_1_En_7_MOESM1_ESM.zip (68.8 mb)
Supplementary material 1 (zip 70419 KB)

References

  1. 1.
    Keras: deep learning for humans. https://github.com/keras-team/keras
  2. 2.
    Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous distributed systems. CoRR abs/1603.04467 arXiv:1603.04467 (2016)
  3. 3.
    Arici, T., Dikbas, S., Altunbasak, Y.: A histogram modification framework and its application for image contrast enhancement. IEEE Trans. Image Process. 18(9), 1921–1935 (2009)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Borji, A.: Pros and cons of GAN evaluation measures. CoRR abs/1802.03446 arXiv:1802.03446 (2018)
  5. 5.
    Chen, C., Chen, Q., Do, M.N., Koltun, V.: Seeing motion in the dark. In: The IEEE International Conference on Computer Vision (ICCV), October 2019Google Scholar
  6. 6.
    Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3291–3300 (2018)Google Scholar
  7. 7.
    Chen, S., et al.: Unsupervised image super-resolution with an indirect supervised path. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020Google Scholar
  8. 8.
    Cui, Z., Li, W., Xu, D., Shan, S., Chen, X., Li, X.: Flowing on Riemannian manifold: domain adaptation by shifting covariance. IEEE Trans. Cybern. 44(12), 2264–2273 (2014)CrossRefGoogle Scholar
  9. 9.
    Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2758–2766 (2015)Google Scholar
  10. 10.
    Escolano, C., Costa-jussà, M.R., Fonollosa, J.A.R.: Towards interlingua neural machine translation. CoRR abs/1905.06831 arXiv:1905.06831 (2019)
  11. 11.
    Gaidon, A., Wang, Q., Cabon, Y., Vig, E.: Virtual worlds as proxy for multi-object tracking analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340–4349 (2016)Google Scholar
  12. 12.
    Gecer, B., Bhattarai, B., Kittler, J., Kim, T.K.: Semi-supervised adversarial learning to generate photorealistic face images of new identities from 3D morphable model. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 217–234 (2018)Google Scholar
  13. 13.
    Gong, R., Li, W., Chen, Y., Gool, L.V.: DLOW: Domain flow for adaptation and generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019Google Scholar
  14. 14.
    Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 2672–2680. Curran Associates, Inc. (2014). http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf
  15. 15.
    Gopalan, R., Li, R., Chellappa, R.: Domain adaptation for object recognition: an unsupervised approach. In: 2011 International Conference on Computer Vision, pp. 999–1006. IEEE (2011)Google Scholar
  16. 16.
    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a Nash equilibrium. CoRR abs/1706.08500 arXiv:1706.08500 (2017)
  17. 17.
    Ibrahim, H., Kong, N.S.P.: Brightness preserving dynamic histogram equalization for image contrast enhancement. IEEE Trans. Consum. Electron. 53(4), 1752–1758 (2007)CrossRefGoogle Scholar
  18. 18.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. arxiv (2016)Google Scholar
  19. 19.
    Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Synthetic data and artificial neural networks for natural scene text recognition. In: Workshop on Deep Learning, NIPS (2014)Google Scholar
  20. 20.
    Jiang, H., Zheng, Y.: Learning to see moving objects in the dark. In: The IEEE International Conference on Computer Vision (ICCV), October 2019Google Scholar
  21. 21.
    Jobson, D.J., Rahman, Z.U., Woodell, G.A.: A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans. Image Process. 6(7), 965–976 (1997)CrossRefGoogle Scholar
  22. 22.
    Jobson, D.J., Rahman, Z.U., Woodell, G.A.: Properties and performance of a center/surround retinex. IEEE Trans. Image Process. 6(3), 451–462 (1997)CrossRefGoogle Scholar
  23. 23.
    Kim, M., Park, D., Han, D.K., Ko, H.: A novel approach for denoising and enhancement of extremely low-light video. IEEE Trans. Consum. Electron. 61(1), 72–80 (2015)CrossRefGoogle Scholar
  24. 24.
    Lai, W.S., Huang, J.B., Wang, O., Shechtman, E., Yumer, E., Yang, M.H.: Learning blind video temporal consistency. In: European Conference on Computer Vision (2018)Google Scholar
  25. 25.
    Land, E.H.: The retinex theory of color vision. Sci. Am. 237(6), 108–129 (1977)CrossRefGoogle Scholar
  26. 26.
    Li, Y., Peng, X.: Learning domain adaptive features with unlabeled domain bridges. arXiv preprint arXiv:1912.05004 (2019)
  27. 27.
    Liu, H., Sun, X., Han, H., Cao, W.: Low-light video image enhancement based on multiscale retinex-like algorithm. In: 2016 Chinese Control and Decision Conference (CCDC), pp. 3712–3715. IEEE (2016)Google Scholar
  28. 28.
    Lore, K.G., Akintayo, A., Sarkar, S.: LLNet: a deep autoencoder approach to natural low-light image enhancement. Pattern Recogn. 61, 650–662 (2017)CrossRefGoogle Scholar
  29. 29.
    Lugmayr, A., Danelljan, M., Timofte, R.: Unsupervised learning for real-world super-resolution. arXiv preprint arXiv:1909.09629 (2019)
  30. 30.
    Lv, F., Lu, F., Wu, J., Lim, C.: MBLLEN: Low-light image/video enhancement using CNNs. In: British Machine Vision Conference (BMVC) (2018)Google Scholar
  31. 31.
    Maggioni, M., Boracchi, G., Foi, A., Egiazarian, K.O.: Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms. IEEE Trans. Image Process. 21, 3952–3966 (2012)MathSciNetCrossRefGoogle Scholar
  32. 32.
    McDonagh, S., Klaudiny, M., Bradley, D., Beeler, T., Matthews, I., Mitchell, K.: Synthetic prior design for real-time face tracking. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 639–648. IEEE (2016)Google Scholar
  33. 33.
    Mildenhall, B., Barron, J.T., Chen, J., Sharlet, D., Ng, R., Carroll, R.: Burst denoising with kernel prediction networks. CoRR abs/1712.02327 arXiv:1712.02327 (2017)
  34. 34.
    Nakai, K., Hoshi, Y., Taguchi, A.: Color image contrast enhancement method based on differential intensity/saturation gray-levels histograms. In: 2013 International Symposium on Intelligent Signal Processing and Communication Systems, pp. 445–449. IEEE (2013)Google Scholar
  35. 35.
    Nowruzi, F.E., Kapoor, P., Kolhatkar, D., Hassanat, F.A., Laganiere, R., Rebut, J.: How much real data do we actually need: analyzing object detection performance using synthetic and real data. arXiv preprint arXiv:1907.07061 (2019)
  36. 36.
    Ravuri, S., Vinyals, O.: Seeing is not necessarily believing: limitations of BigGANs for data augmentation (2019)Google Scholar
  37. 37.
    Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: ground truth from computer games. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 102–118. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_7CrossRefGoogle Scholar
  38. 38.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS, vol. 9351, pp. 234–241. Springer, Heidelberg (2015).  https://doi.org/10.1007/978-3-319-24574-4_28, http://lmb.informatik.uni-freiburg.de/Publications/2015/RFB15a, arXiv:1505.04597 [cs.CV]
  39. 39.
    Sankaranarayanan, S., Balaji, Y., Jain, A., Nam Lim, S., Chellappa, R.: Learning from synthetic data: addressing domain shift for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3752–3761 (2018)Google Scholar
  40. 40.
    Schwartz, E., Giryes, R., Bronstein, A.M.: DeepISP: towards learning an end-to-end image processing pipeline. IEEE Trans. Image Process. 28(2), 912–923 (2019)MathSciNetCrossRefGoogle Scholar
  41. 41.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings (2015). arXiv:1409.1556
  42. 42.
    Taigman, Y., Polyak, A., Wolf, L.: Unsupervised cross-domain image generation. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings. OpenReview.net (2017). https://openreview.net/forum?id=Sk2Im59ex
  43. 43.
    Tan, B., Zhang, Y., Pan, S.J., Yang, Q.: Distant domain transfer learning. In: Thirty-first AAAI Conference on Artificial Intelligence (2017)Google Scholar
  44. 44.
    Tao, L., Zhu, C., Song, J., Lu, T., Jia, H., Xie, X.: Low-light image enhancement using CNN and bright channel prior. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 3215–3219. IEEE (2017)Google Scholar
  45. 45.
    Tao, L., Zhu, C., Xiang, G., Li, Y., Jia, H., Xie, X.: LLCNN: a convolutional neural network for low-light image enhancement. In: 2017 IEEE Visual Communications and Image Processing (VCIP), pp. 1–4. IEEE (2017)Google Scholar
  46. 46.
    Tripathy, S., Kannala, J., Rahtu, E.: Learning image-to-image translation using paired and unpaired training samples. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11362, pp. 51–66. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-20890-5_4CrossRefGoogle Scholar
  47. 47.
    Wang, D., Niu, X., Dou, Y.: A piecewise-based contrast enhancement framework for low lighting video. In: Proceedings 2014 IEEE International Conference on Security, Pattern Analysis, and Cybernetics (SPAC), pp. 235–240. IEEE (2014)Google Scholar
  48. 48.
    Xue, T., Chen, B., Wu, J., Wei, D., Freeman, W.T.: Video enhancement with task-oriented flow. CoRR abs/1711.09078 arXiv:1711.09078 (2017)
  49. 49.
    Ying, Z., Li, G., Ren, Y., Wang, R., Wang, W.: A new low-light image enhancement algorithm using camera response model. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 3015–3022 (2017)Google Scholar
  50. 50.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)Google Scholar
  51. 51.
    Zhu, Y., Aoun, M., Krijn, M., Vanschoren, J., Campus, H.T.: Data augmentation using conditional generative adversarial networks for leaf counting in Arabidopsis plants. In: BMVC, p. 324 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Huawei Noah’s Ark LabLondonUK

Personalised recommendations