Skip to main content
SpringerLink
Log in

Exposure-Aware Dynamic Weighted Learning for Single-Shot HDR Imaging

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13667))

Included in the following conference series:

Abstract

We propose a novel single-shot high dynamic range (HDR) imaging algorithm based on exposure-aware dynamic weighted learning, which reconstructs an HDR image from a spatially varying exposure (SVE) raw image. First, we recover poorly exposed pixels by developing a network that learns local dynamic filters to exploit local neighboring pixels across color channels. Second, we develop another network that combines only valid features in well-exposed regions by learning exposure-aware feature fusion. Third, we synthesize the raw radiance map by adaptively combining the outputs of the two networks that have different characteristics with complementary information. Finally, a full-color HDR image is obtained by interpolating missing color information. Experimental results show that the proposed algorithm significantly outperforms conventional algorithms on various datasets. The source codes and pretrained models are available at https://github.com/viengiaan/EDWL.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The details of the DFN architecture are provided in the supplemental document.

  2. 2.

    The details of the network are provided in the supplemental document.

  3. 3.

    The details of the network architecture is provided in the supplemental document.

  4. 4.

    http://markfairchild.org/HDRPS/HDRthumbs.html.

  5. 5.

    https://www.hdm-stuttgart.de/vmlab/hdm-hdr-2014.

  6. 6.

    https://mmspg.epfl.ch/hdr-eye.

References

  1. Adams, J.E.: Design of practical color filter array interpolation algorithms for digital cameras. In: Proceedings of the SPIE, pp. 117–125, February 1997

    Google Scholar 

  2. Alghamdi, M., Fu, Q., Thabet, A., Heidrich, W.: Transfer deep learning for reconfigurable snapshot HDR imaging using coded masks. Comput. Graph. Forum 40(6), 90–103 (2021)

    Article  Google Scholar 

  3. Aydın, T.O., Mantiuk, R., Seidel, H.P.: Extending quality metrics to full dynamic range images. In: Proceedings of the SPIE, Human Vision and Electronic Imaging XIII, pp. 6806–6810, January 2008

    Google Scholar 

  4. Cho, H., Kim, S.J., Lee, S.: Single-shot high dynamic range imaging using coded electronic shutter. Comput. Graph. Forum 33(7), 329–338 (2014)

    Article  Google Scholar 

  5. Choi, I., Baek, S.H., Kim, M.H.: Reconstructing interlaced high-dynamic-range video using joint learning. IEEE Trans. Image Process. 26(11), 5353–5366 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  6. Çoğalan, U., Akyüz, A.O.: Deep joint deinterlacing and denoising for single shot dual-ISO HDR reconstruction. IEEE Trans. Image Process. 29, 7511–7524 (2020)

    Article  MATH  Google Scholar 

  7. Debevec, P., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: Proceedings of the ACM SIGGRAPH, pp. 369–378, August 1997

    Google Scholar 

  8. Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R.K., Unger, J.: HDR image reconstruction from a single exposure using deep CNNs. ACM Trans. Graph. 36(6), 178:1–178:15 (2017)

    Google Scholar 

  9. Endo, Y., Kanamori, Y., Mitani, J.: Deep reverse tone mapping. ACM Trans. Graph. 36(6), 177:1–177:10 (2017)

    Google Scholar 

  10. Gu, J., Hitomi, Y., Mitsunaga, T., Nayar, S.K.: Coded rolling shutter photography: flexible space-time sampling. In: Proceedings of the ICCP, pp. 1–8, March 2010

    Google Scholar 

  11. Hajisharif, S., Kronander, J., Unger, J.: HDR reconstruction for alternating gain (ISO) sensor readout. In: Eurograph, pp. 25–28, April 2014

    Google Scholar 

  12. Jia, X., De Brabandere, B., Tuytelaars, T., Gool, L.V.: Dynamic filter networks. In: Proceedings NeurIPS, December 2016

    Google Scholar 

  13. Kalantari, N.K., Ramamoorthi, R.: Deep high dynamic range imaging of dynamic scenes. ACM Trans. Graph. 36(4), 144:1–144:12 (2017)

    Google Scholar 

  14. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Proceedings of the ICLR, September 2015

    Google Scholar 

  15. Kronander, J., Gustavson, S., Bonnet, G., Unger, J.: Unified HDR reconstruction from raw CFA data. In: Proceedings of ICCP, pp. 1–9, April 2013

    Google Scholar 

  16. Lee, C., Lam, E.Y.: Computationally efficient truncated nuclear norm minimization for high dynamic range imaging. IEEE Trans. Image Process. 25(9), 4145–4157 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  17. Lee, S., An, G.H., Kang, S.J.: Deep recursive HDRI: Inverse tone mapping using generative adversarial networks. In: Proceedings of the ECCV, pp. 613–628, September 2018

    Google Scholar 

  18. Li, J., Wang, N., Zhang, L., Du, B., Tao, D.: Recurrent feature reasoning for image inpainting. In: Proceedings of the CVPR, pp. 7757–7765, June 2020

    Google Scholar 

  19. Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Proceedings of ECCV, pp. 89–105, September 2018

    Google Scholar 

  20. Liu, Y.L., et al.: Single-image HDR reconstruction by learning to reverse the camera pipeline. In: Proceedings of CVPR, pp. 1651–1660, June 2020

    Google Scholar 

  21. Malvar, H.S., He, L.W., Cutler, R.: High-quality linear interpolation for demosaicing of Bayer-patterned color images. In: Proceedings of ICASSP, pp. 2274–2282, May 2004

    Google Scholar 

  22. Mantiuk, R., Efremov, A., Myszkowski, K., Seidel, H.P.: Backward compatible high dynamic range MPEG video compression. ACM Trans. Graph. 25(3), 713–723 (2006)

    Article  Google Scholar 

  23. Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph. 30(4), 1–14 (2011)

    Google Scholar 

  24. Mantiuk, R., Myszkowski, K., Seidel, H.P.: Lossy compression of high dynamic range images and video. In: Proceedings of SPIE, Human Vision and Electronic Imaging, pp. 6057–6057-10, February 2006

    Google Scholar 

  25. Marnerides, D., Bashford-Rogers, T., Hatchett, J., Debattista, K.: ExpandNet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content. Comput. Graph. Forum 37(2), 37–49 (2018)

    Article  Google Scholar 

  26. Martel, J.N.P., Müller, L.K., Carey, S.J., Dudek, P., Wetzstein, G.: Neural sensors: Learning pixel exposures for HDR imaging and video compressive sensing with programmable sensors. IEEE Trans. Pattern Anal. Mach. Intell. 42(7), 1642–1653 (2020)

    Article  Google Scholar 

  27. Metzler, C.A., Ikoma, H., Peng, Y., Wetzstein, G.: Deep optics for single-shot high-dynamic-range imaging. In: Proceedings of CVPR, pp. 1372–1382, June 2020

    Google Scholar 

  28. Mitsunaga, T., Nayar, S.K.: Radiometric self calibration. In: Proceedings of CVPR, pp. 374–380, August 1999

    Google Scholar 

  29. Narwaria, M., Mantiuk, R., Perreira Da Silva, M., Le Callet, P.: HDR-VDP-2.2: a calibrated method for objective quality prediction of high dynamic range and standard images. J. Electron. Imaging 24(1), 010501 (2015)

    Google Scholar 

  30. Narwaria, M., Da Silva, M.P., Le Callet, P.: HDR-VQM: An objective quality measure for high dynamic range video. Signal Process. Image Commun. 35, 46–60 (2015)

    Google Scholar 

  31. Nayar, S.K., Mitsunaga, T.: High dynamic range imaging: spatially varying pixel exposures. In: Proceedings of the CVPR, pp. 472–479, June 2000

    Google Scholar 

  32. Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings of CVPR, pp. 2536–2544, June 2016

    Google Scholar 

  33. Reinhard, E., Ward, G., Pattanaik, S., Debevec, P., Heidrich, W., Myszkowski, K.: High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting. Morgan Kaufmann Publishers, second edn. (2010)

    Google Scholar 

  34. Robidoux, N., Capel, L.E.G., Seo, D.E., Sharma, A., Ariza, F., Heide, F.: End-to-end high dynamic range camera pipeline optimization. In: Proceedings of CVPR, pp. 6297–6307, June 2021

    Google Scholar 

  35. Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  36. Santos, M.S., Ren, T.I., Kalantari, N.K.: Single image HDR reconstruction using a CNN with masked features and perceptual loss. ACM Trans. Graph. 39(4), 80:1–80:10 (2020)

    Google Scholar 

  37. Sen, P., Aguerrebere, C.: Practical high dynamic range imaging of everyday scenes: photographing the world as we see it with our own eyes. IEEE Signal Process. Mag. 33(5), 36–44 (2016)

    Article  Google Scholar 

  38. Serrano, A., Heide, F., Gutierrez, D., Wetzstein, G., Masia, B.: Convolutional sparse coding for high dynamic range imaging. Comput. Graph. Forum 35(2), 153–163 (2016)

    Article  Google Scholar 

  39. Sharif, S.M.A., Naqvi, R.A., Biswas, M.: Beyond joint demosaicking and denoising: An image processing pipeline for a pixel-bin image sensor. In: Proceedings of the CVPRW, pp. 233–242, June 2021

    Google Scholar 

  40. Suda, T., Tanaka, M., Monno, Y., Okutomi, M.: Deep snapshot HDR imaging using multi-exposure color filter array. In: Proceedings of the ACCV, pp. 353–370, November 2020

    Google Scholar 

  41. Sun, Q., Tseng, E., Fu, Q., Heidrich, W., Heide, F.: Learning rank-1 diffractive optics for single-shot high dynamic range imaging. In: Proceedings of CVPR, pp. 1383–1393, June 2020

    Google Scholar 

  42. Vargas, E., Martel, J.N., Wetzstein, G., Arguello, H.: Time-multiplexed coded aperture imaging: learned coded aperture and pixel exposures for compressive imaging systems. In: Proceedings of ICCV, pp. 2692–2702, October 2021

    Google Scholar 

  43. Vaswani, A., et al.: Attention is all you need. In: Proceedings of NeurIPS, pp. 6000–6010, December 2017

    Google Scholar 

  44. Vien, A.G., Lee, C.: Single-shot high dynamic range imaging via multiscale convolutional neural network. IEEE Access 9, 70369–70381 (2021)

    Article  Google Scholar 

  45. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of CVPR, pp. 7794–7803, June 2018

    Google Scholar 

  46. Wang, Y., Tao, X., Qi, X., Shen, X., Jia, J.: Image inpainting via generative multi-column convolutional neural networks. In: Proceedings of NeurIPS, pp. 329–338, December 2018

    Google Scholar 

  47. Woo, S.M., Ryu, J.H., Kim, J.O.: Ghost-free deep high-dynamic-range imaging using focus pixels for complex motion scenes. IEEE Trans. Image Process. 30, 5001–5016 (2021)

    Article  Google Scholar 

  48. Wu, S., Xu, J., Tai, Y.W., Tang, C.K.: Deep high dynamic range imaging with large foreground motions. In: Proceedings of ECCV, pp. 120–135, September 2018

    Google Scholar 

  49. Xie, C., et al.: Image inpainting with learnable bidirectional attention maps. In: Proceedings of the ICCV, pp. 8857–8866, October 2019

    Google Scholar 

  50. Xu, Y., Liu, Z., Wu, X., Chen, W., Wen, C., Li, Z.: Deep joint demosaicing and high dynamic range imaging within a single shot. IEEE Trans. Circ. Syst. Video Technol. 32(7), 4255–4270 (2022)

    Article  Google Scholar 

  51. Yan, Q., Gong, D., Shi, Q., van den Hengel, A., Shen, C., Reid, I., Zhang, Y.: Attention-guided network for ghost-free high dynamic range imaging. In: Proceedings of CVPR, pp. 1751–1760, June 2019

    Google Scholar 

  52. Yan, Q., et al.: Deep HDR imaging via a non-local network. IEEE Trans. Image Process. 29, 4308–4322 (2020)

    Google Scholar 

  53. Yan, Q., et al.: Deep HDR imaging via a non-local network. IEEE Trans. Image Process. 29, 4308–4322 (2020)

    Google Scholar 

  54. Yang, F., Yang, H., Fu, J., Lu, H., Guo, B.: Learning texture transformer network for image super-resolution. In: Proceedings of the CVPR, pp. 5790–5799, June 2020

    Google Scholar 

  55. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.: Free-form image inpainting with gated convolution. In: Proceedings of the ICCV, pp. 4470–4479, October 2019

    Google Scholar 

  56. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: Proceedings of the CVPR, pp. 5505–5514, June 2018

    Google Scholar 

  57. Zhang, Y., Li, K., Li, K., Zhong, B., Fu, Y.: Residual non-local attention networks for image restoration. In: Proceedings of the ICLR, May 2019

    Google Scholar 

  58. Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2017)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-00011, Video Coding for Machine) and in part by the National Research Foundation of Korea (NRF) grant funded MSIP (No. NRF-2022R1F1A1074402).

Author information

Authors and Affiliations

  1. Department of Multimedia Engineering, Dongguk University, Seoul, Korea

    An Gia Vien & Chul Lee

Authors
  1. An Gia Vien
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chul Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chul Lee .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 18991 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Vien, A.G., Lee, C. (2022). Exposure-Aware Dynamic Weighted Learning for Single-Shot HDR Imaging. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13667. Springer, Cham. https://doi.org/10.1007/978-3-031-20071-7_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20071-7_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20070-0

  • Online ISBN: 978-3-031-20071-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation