Advertisement

Deep Recursive HDRI: Inverse Tone Mapping Using Generative Adversarial Networks

  • Siyeong Lee
  • Gwon Hwan An
  • Suk-Ju KangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11206)

Abstract

High dynamic range images contain luminance information of the physical world and provide more realistic experience than conventional low dynamic range images. Because most images have a low dynamic range, recovering the lost dynamic range from a single low dynamic range image is still prevalent. We propose a novel method for restoring the lost dynamic range from a single low dynamic range image through a deep neural network. The proposed method is the first framework to create high dynamic range images based on the estimated multi-exposure stack using the conditional generative adversarial network structure. In this architecture, we train the network by setting an objective function that is a combination of L1 loss and generative adversarial network loss. In addition, this architecture has a simplified structure than the existing networks. In the experimental results, the proposed network generated a multi-exposure stack consisting of realistic images with varying exposure values while avoiding artifacts on public benchmarks, compared with the existing methods. In addition, both the multi-exposure stacks and high dynamic range images estimated by the proposed method are significantly similar to the ground truth than other state-of-the-art algorithms.

Keywords

High dynamic range imaging Inverse tone mapping Image restoration Computational photography Generative adversarial network Deep learning 

Notes

Acknowledgements

This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2018R1D1A1B07048421) and Korea Electric Power Corporation. (Grant number R17XA05-28). We thank Yong Deok Ahn and members of the Sogang Vision and Display Lab. for helpful discussions.

Supplementary material

474176_1_En_37_MOESM1_ESM.pdf (15 mb)
Supplementary material 1 (pdf 15337 KB)

References

  1. 1.
    Endo, Y., Kanamori, Y., Mitani, J.: Deep reverse tone mapping. ACM Trans. Graph. (TOG) 36(6), 177 (2017)CrossRefGoogle Scholar
  2. 2.
    Lee, S., An, G.H., Kang, S.J.: Deep chain HDRI: reconstructing a high dynamic range image from a single low dynamic range image. arXiv preprint arXiv:1801.06277 (2018)
  3. 3.
    Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R.K., Unger, J.: HDR image reconstruction from a single exposure using deep CNNs. ACM Trans. Graph. (TOG) 36(6), 178 (2017)CrossRefGoogle Scholar
  4. 4.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  5. 5.
    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, pp. 2234–2242 (2016)Google Scholar
  6. 6.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networksGoogle Scholar
  7. 7.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint (2017)Google Scholar
  8. 8.
    Kim, T., Cha, M., Kim, H., Lee, J.K., Kim, J.: Learning to discover cross-domain relations with generative adversarial networks. arXiv preprint arXiv:1703.05192 (2017)
  9. 9.
    Maaten, L.V.D., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(Nov), 2579–2605 (2008)zbMATHGoogle Scholar
  10. 10.
    Rempel, A.G.: Ldr2Hdr: on-the-fly reverse tone mapping of legacy video and photographs. ACM Trans. Graph. (TOG) 26, 39 (2007)CrossRefGoogle Scholar
  11. 11.
    Meylan, L., Daly, S., Süsstrunk, S.: The reproduction of specular highlights on high dynamic range displays. In: Color and Imaging Conference, vol. 2006, pp. 333–338. Society for Imaging Science and Technology (2006)Google Scholar
  12. 12.
    Wang, T.H., et al.: Pseudo-multiple-exposure-based tone fusion with local region adjustment. IEEE Trans. Multimed. 17(4), 470–484 (2015)CrossRefGoogle Scholar
  13. 13.
    Hirakawa, K., Simon, P.M.: Single-shot high dynamic range imaging with conventional camera hardware. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1339–1346. IEEE (2011)Google Scholar
  14. 14.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  15. 15.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  16. 16.
    Mao, X.J., Shen, C., Yang, Y.B.: Image restoration using convolutional auto-encoders with symmetric skip connections. arXiv preprint arXiv:1606.08921 (2016)
  17. 17.
    Li, H., Xu, Z., Taylor, G., Goldstein, T.: Visualizing the loss landscape of neural nets. arXiv preprint arXiv:1712.09913 (2017)
  18. 18.
    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein GAN. arXiv preprint arXiv:1701.07875 (2017)
  19. 19.
    Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Smolley, S.P.: Least squares generative adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2813–2821. IEEE (2017)Google Scholar
  20. 20.
    Nowozin, S., Cseke, B., Tomioka, R.: f-GAN: training generative neural samplers using variational divergence minimization. In: Advances in Neural Information Processing Systems, pp. 271–279 (2016)Google Scholar
  21. 21.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, vol. 2, p. 4 (2017)Google Scholar
  22. 22.
    Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  23. 23.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  24. 24.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)Google Scholar
  25. 25.
    Li, C., Wand, M.: Precomputed real-time texture synthesis with markovian generative adversarial networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 702–716. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46487-9_43CrossRefGoogle Scholar
  26. 26.
    Lafferty, J., McCallum, A., Pereira, F.C.: Conditional random fields: probabilistic models for segmenting and labeling sequence data (2001)Google Scholar
  27. 27.
    Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic modelsGoogle Scholar
  28. 28.
    Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2017)CrossRefGoogle Scholar
  29. 29.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  30. 30.
    Nemoto, H., Korshunov, P., Hanhart, P., Ebrahimi, T.: Visual attention in LDR and HDR images. In: 9th International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM), Number EPFL-CONF-203873 (2015)Google Scholar
  31. 31.
    Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: ACM SIGGRAPH 2008 classes, p. 31. ACM (2008)Google Scholar
  32. 32.
    Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic tone reproduction for digital images. ACM Trans. Graph. (TOG) 21(3), 267–276 (2002)CrossRefGoogle Scholar
  33. 33.
    Kim, M.H., Kautz, J.: Consistent tone reproduction. In: Proceedings of the Tenth IASTED International Conference on Computer Graphics and Imaging (CGIM 2008), Innsbruck, Austria, pp. 152–159. IASTED/ACTA Press (2008)Google Scholar
  34. 34.
    Banterle, F., Artusi, A., Debattista, K., Chalmers, A.: Advanced High Dynamic Range Imaging. CRC Press, Boca Raton (2017)CrossRefGoogle Scholar
  35. 35.
    Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph. (TOG) 30, 40 (2011)CrossRefGoogle Scholar
  36. 36.
    Masia, B., Agustin, S., Fleming, R.W., Sorkine, O., Gutierrez, D.: Evaluation of reverse tone mapping through varying exposure conditions. ACM Trans. Graph. (TOG) 28(5), 160 (2009)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Department of Electronic EngineeringSogang UniversitySeoulSouth Korea

Personalised recommendations