Skip to main content
SpringerLink
Log in

Deep snapshot HDR imaging using multi-exposure color filter array

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In this paper, we propose a deep snapshot high dynamic range (HDR) imaging framework that can effectively reconstruct an HDR image from the RAW data captured using a multi-exposure color filter array (ME-CFA), which consists of a mosaic pattern of RGB filters with different exposure levels. To effectively learn the HDR image reconstruction network, we introduce the idea of luminance normalization that simultaneously enables effective loss computation and input data normalization by considering relative local contrasts in the “normalized-by-luminance” HDR domain. This idea enables the network to equally handle the errors in both bright and dark areas regardless of absolute luminance levels, which significantly improves the visual image quality. Experimental results using public HDR image datasets demonstrate that our framework outperforms other snapshot methods and produces high-quality HDR images with fewer visual artifacts, resulting in more than 4dB color peak signal-to-noise ratio improvement in the linear HDR domain.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Notes

  1. https://www.hdrsoft.com.

References

  1. Debevec, P., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: Proceedings of SIGGRAPH, pp. 1–10 (1997)

  2. Kalantari, N.K., Ramamoorthi, R.: Deep high dynamic range imaging of dynamic scenes. ACM Trans. Graph. 36(4), 144 (2017)

    Article  Google Scholar 

  3. Wu, S., Xu, J., Tai, Y.-W., Tang, C.-K.: Deep high dynamic range imaging with large foreground motions. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 121–135 (2018)

  4. Yan, Q., Gong, D., Shi, Q., van den Hengel, A., Shen, C., Reid, I., Zhang, Y.: Attention-guided network for ghost-free high dynamic range imaging. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1751–1760 (2019)

  5. Marnerides, D., Bashford-Rogers, T., Hatchett, J., Debattista, K.: ExpandNet: a deep convolutional neural network for high dynamic range expansion from low dynamic range content. Comput. Graph. Forum 37(2), 37–49 (2018)

    Article  Google Scholar 

  6. Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R.K., Unger, J.: HDR image reconstruction from a single exposure using deep CNNs. ACM Trans. Graph. 36(6), 1–15 (2017)

    Article  Google Scholar 

  7. Lee, S., An, G.H., Kang, S.-J.: Deep chain HDRI: reconstructing a high dynamic range image from a single low dynamic range image. IEEE Access 6, 49913–49924 (2018)

    Article  Google Scholar 

  8. Cho, H., Kim, S.J., Lee, S.: Single-shot high dynamic range imaging using coded electronic shutter. Comput. Graph. Forum 33(7), 329–338 (2014)

    Article  Google Scholar 

  9. Choi, I., Baek, S.-H., Kim, M.H.: Reconstructing interlaced high-dynamic-range video using joint learning. IEEE Trans. Image Process. 26(11), 5353–5366 (2017)

    Article  MathSciNet  Google Scholar 

  10. Narasimhan, S.G., Nayar, S.K.: Enhancing resolution along multiple imaging dimensions using assorted pixels. IEEE Trans. Pattern Anal. Mach. Intell. 27(4), 518–530 (2005)

    Article  Google Scholar 

  11. Nayar, S.K., Mitsunaga, T.: High dynamic range imaging: spatially varying pixel exposures. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  12. Eilertsen, G., Mantiuk, R.K., Unger, J.: Real-time noise-aware tone mapping. ACM Trans. Graph. 34(6), 1–15 (2015)

    Article  Google Scholar 

  13. Suda, T., Tanaka, M., Monno, Y., Okutomi, M.: Deep snapshot HDR imaging using multi-exposure color filter array. In: Proceedings of the Asian Conference on Computer Vision (ACCV)

  14. Ma, K., Duanmu, Z., Yeganeh, H., Wang, Z.: Multi-exposure image fusion by optimizing a structural similarity index. IEEE Trans. Comput. Imaging 4(1), 60–72 (2017)

    Article  MathSciNet  Google Scholar 

  15. Ma, K., Li, H., Yong, H., Wang, Z., Meng, D., Zhang, L.: Robust multi-exposure image fusion: a structural patch decomposition approach. IEEE Trans. Image Process. 26(5), 2519–2532 (2017)

    Article  MathSciNet  Google Scholar 

  16. Mertens, T., Kautz, J., Van Reeth, F.: Exposure fusion: a simple and practical alternative to high dynamic range photography. Comput. Graph. Forum 28(1), 161–171 (2009)

    Article  Google Scholar 

  17. Hasinoff, S.W., Sharlet, D., Geiss, R., Adams, A., Barron, J.T., Kainz, F., Chen, J., Levoy, M.: Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph. 35(6), 1–12 (2016)

    Article  Google Scholar 

  18. Hafner, D., Demetz, O., Weickert, J.: Simultaneous HDR and optic flow computation. In Proceedings of the International Conference on Pattern Recognition (ICPR), pp. 2065–2070 (2014)

  19. Hu, J., Gallo, O., Pulli, K., Sun, X.: HDR deghosting: how to deal with saturation? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1163–1170 (2013)

  20. Sen, P., Kalantari, N.K., Yaesoubi, M., Darabi, S., Goldman, D.B., Shechtman, E.: Robust patch-based HDR reconstruction of dynamic scenes. ACM Trans. Graph. 31(6), 203 (2012)

    Article  Google Scholar 

  21. Lee, C., Li, Y., Monga, V.: Ghost-free high dynamic range imaging via rank minimization. IEEE Signal Process. Lett. 21(9), 1045–1049 (2014)

    Article  Google Scholar 

  22. Oh, T.-H., Lee, J.-Y., Tai, Y.-W., Kweon, I.S.: Robust high dynamic range imaging by rank minimization. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1219–1232 (2014)

    Article  Google Scholar 

  23. Kalantari, N.K., Ramamoorthi, R.: Deep HDR video from sequences with alternating exposures. Comput. Graph. Forum 38(2), 193–205 (2019)

    Article  Google Scholar 

  24. Prabhakar, K.R., Arora, R., Swaminathan, A., Singh, K.P., Babu, R.V.: A fast, scalable, and reliable deghosting method for extreme exposure fusion. In: Proceedings of the IEEE International Conference on Computational Photography (ICCP), pp. 170–177 (2019)

  25. Ram Prabhakar, K., Sai Srikar, V., Venkatesh Babu, R.: DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 4724–4732 (2017)

  26. Yan, Q., Gong, D., Zhang, P., Shi, Q., Sun, J., Reid, I., Zhang, Y.: Multi-scale dense networks for deep high dynamic range imaging. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 41–50 (2019)

  27. Yan, Q., Zhang, L., Liu, Y., Zhu, Y., Sun, J., Shi, Q., Zhang, Y.: Deep HDR imaging via a non-local network. IEEE Trans. Image Process. 29, 4308–4322 (2020)

    Article  Google Scholar 

  28. Niu, Y., Wu, J., Liu, W., Guo, W., Lau, R.W.: HDR-GAN: HDR image reconstruction from multi-exposed LDR images with large motions. IEEE Trans. Image Process. 30, 3885–3896 (2021)

    Article  Google Scholar 

  29. Prabhakar, K.R., Senthil, G., Agrawal, S., Babu, R.V., Gorthi, R.K.S.S.: Labeled from unlabeled: exploiting unlabeled data for few-shot deep HDR deghosting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4875–4885 (2021)

  30. Song, J.W., Park, Y.-I., Kong, K., Kwak, J., Kang, S.-J.: Selective TransHDR: transformer-based selective HDR imaging using ghost region mask. In: Proceedings of the European Conference on Computer Vision (ECCV),pp. 288–304 (2022)

  31. Liu, Z., Wang, Y., Zeng, B., Liu, S.: Ghost-free high dynamic range imaging with context-aware transformer. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 344–360 (2022)

  32. Tursun, O.T., Akyüz, A.O., Erdem, A., Erdem, E.: The state of the art in HDR deghosting: a survey and evaluation. Comput. Graph. Forum 34(2), 683–707 (2015)

    Article  Google Scholar 

  33. Wang, L., Yoon, K.-J.: Deep learning for HDR imaging: state-of-the-art and future trends. arXiv:2110.10394 (2021)

  34. Ogino, Y., Tanaka, M., Shibata, T., Okutomi, M.: Super high dynamic range video. In: Proceedings of the International Conference on Pattern Recognition (ICPR), pp. 4208–4213 (2016)

  35. Tocci, M.D., Kiser, C., Tocci, N., Sen, P.: A versatile HDR video production system. ACM Trans. Graph. 30(4), 1–9 (2011)

    Article  Google Scholar 

  36. Han, J., Zhou, C., Duan, P., Tang, Y., Xu, C., Xu, C., Huang, T., Shi, B.: Neuromorphic camera guided high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),pp. 1730–1739 (2020)

  37. Yang, X., Xu, K., Song, Y., Zhang, Q., Wei, X., Lau, R.W.: Image correction via deep reciprocating HDR transformation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1798–1807 (2018)

  38. Moriwaki, K., Yoshihashi, R., Kawakami, R., You, S., Naemura, T.: Hybrid loss for learning single-image-based HDR reconstruction. arXiv:1812.07134 (2018)

  39. Kim, S.Y., Kim, D.-E., Kim, M.: ITM-CNN: learning the inverse tone mapping from low dynamic range video to high dynamic range displays using convolutional neural networks. In: Proceedings of the Asian Conference on Computer Vision (ACCV), pp. 395–409 (2018)

  40. Santos, M.S., Ren, T.I., Kalantari, N.K.: Single image HDR reconstruction using a CNN with masked features and perceptual loss. ACM Trans. Graph. (TOG) 39(4), 80–1 (2020)

    Article  Google Scholar 

  41. Endo, Y., Kanamori, Y., Mitani, J.: Deep reverse tone mapping. ACM Trans. Graph. 36(6), 177 (2017)

    Article  Google Scholar 

  42. Lee, S., Hwan An, G., Kang, S.-J.: Deep recursive HDRI: inverse tone mapping using generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 613–628 (2018)

  43. Liu, Y.-L., Lai, W.-S., Chen, Y.-S., Kao, Y.-L., Yang, M.-H.,Chuang, Y.-Y., Huang, J.-B.: Single-image HDR reconstruction by learning to reverse the camera pipeline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2020)

  44. Zheng, Z., Ren, W., Cao, X., Wang, T., Jia, X.: Ultra-high-definition image HDR reconstruction via collaborative bilateral learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4449–4458 (2021)

  45. Chen, X., Zhang, Z., Ren, J.S., Tian, L., Qiao, Y., Dong, C.: A new journey from SDRTV to HDRTV. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),pp. 4500–4509 (2021)

  46. Cheng, Z., Wang, T., Li, Y., Song, F., Chen, C., Xiong, Z.: Towards real-world HDRTV reconstruction: a data synthesis-based approach. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 199–216 (2022)

  47. Sun, Q., Tseng, E., Fu, Q., Heidrich, W., Heide, F.: Learning rank-1 diffractive optics for single-shot high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2020)

  48. Metzler, C.A., Ikoma, H., Peng, Y., Wetzstein, G.: Deep optics for single-shot high-dynamic-range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2020)

  49. Gu, J., Hitomi, Y., Mitsunaga, T., Nayar, S.: Coded rolling shutter photography: flexible space-time sampling. In: Proceedings of the IEEE International Conference on Computational Photography (ICCP), pp. 1–8 (2010)

  50. Uda, S., Sakaue, F., Sato, J.: Variable exposure time imaging for obtaining unblurred HDR images. IPSJ Trans. Comput. Vis. Appl. 8(1), 1–7 (2016)

    Google Scholar 

  51. Alghamdi, M., Fu, Q., Thabet, A., Heidrich, W.: Reconfigurable snapshot HDR imaging using coded masks and inception network. In: Proceedings of the Vision, Modeling, and Visualization (VMV), pp. 1–9 (2019)

  52. Nagahara, H., Sonoda, T., Liu, D., Gu, J.: Space-time-brightness sampling using an adaptive pixel-wise coded exposure. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1834–1842 (2018)

  53. Serrano, A., Heide, F., Gutierrez, D., Wetzstein, G., Masia, B.: Convolutional sparse coding for high dynamic range imaging. Comput. Graph. Forum 35(2), 153–163 (2016)

    Article  Google Scholar 

  54. Go, C., Kinoshita, Y., Shiota, S, Kiua, H.: Image fusion for single-shot high dynamic range imaging with spatially varying exposures. In: Proceedings of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 1082–1086 (2018)

  55. Hajisharif, S., Kronander, J., Unger, J.: Adaptive dualISO HDR reconstruction. EURASIP J. Image Video Process. 2015(41), 1–13 (2015)

    Google Scholar 

  56. Heide, F., Steinberger, M., Tsai, Y.-T., Rouf, M., Pająk, D., Reddy, D., Gallo, O., Liu, J., Heidrich, W., Egiazarian, K., Kautz, J., Pulli, K.: FlexISP: a flexible camera image processing framework. ACM Trans. Graph. 33(6), 1–13 (2014)

    Article  Google Scholar 

  57. Aguerrebere, C., Almansa, A., Delon, J., Gousseau, Y., Musé, P.: A Bayesian hyperprior approach for joint image denoising and interpolation, with an application to HDR imaging. IEEE Trans. Comput. Imaging 3(4), 633–646 (2017)

    Article  MathSciNet  Google Scholar 

  58. Aguerrebere, C., Almansa, A., Gousseau, Y., Delon, J., Muse, P.: Single shot high dynamic range imaging using piecewise linear estimators. In: Proceedings of the IEEE International Conference on Computational Photography (ICCP), pp. 1–10 (2014)

  59. An, V.G., Lee, C.: Single-shot high dynamic range imaging via deep convolutional neural network. In: Proceedings of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 1768–1772 (2017)

  60. Rouf, M., Ward, R.K.: High dynamic range imaging with a single exposure-multiplexed image using smooth contour prior. In: Proceedings of the IS &T International Symposium on Electronic Imaging (EI), pp. 440:1–6 (2018)

  61. Cheng, C.-H., Au, O.C., Cheung, N.-M., Liu, C.-H., Yip, K.-Y.: High dynamic range image capturing by spatial varying exposed color filter array with specific demosaicking algorithm. In: Proceedings of the IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PACRIM), pp. 648–653 (2009)

  62. Cogalan, U., Akyüz, A.O.: Deep joint deinterlacing and denoising for single shot dual-ISO HDR reconstruction. IEEE Trans. Image Process. 29, 7511–7524 (2020)

    Article  Google Scholar 

  63. Vien, A.G., Lee, C.: Exposure-aware dynamic weighted learning for single-shot HDR imaging. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 435–452 (2022)

  64. Martel, J.N.P., Müller, L.K., Carey, S.J., Dudek, P., Wetzstein, G.: Neural sensors: learning pixel exposures for HDR imaging and video compressive sensing with programmable sensors. IEEE Trans. Pattern Anal. Mach. Intell. 42(7), 1642–1653 (2020)

    Article  Google Scholar 

  65. Vien, A.G., Lee, C.: Single-shot high dynamic range imaging via multiscale convolutional neural network. IEEE Access 9, 70369–70381 (2021)

    Article  Google Scholar 

  66. Xu, Y., Liu, Z., Wu, X., Chen, W., Wen, C., Li, Z.: Deep joint demosaicing and high dynamic range imaging within a single shot. IEEE Trans. Circuits Syst. Video Technol. 32(7), 4255–4270 (2022)

    Article  Google Scholar 

  67. Bayer, B.E.: Color imaging array, US patent 3971065, (1976)

  68. Cui, K., Jin, Z., Steinbach, E.: Color image demosaicking using a 3-stage convolutional neural network structure. In: Proceedings of the IEEE International Conference on Image Processing (ICIP), pp. 2177–2181 (2018)

  69. Kokkinos, F., Lefkimmiatis, S.: Deep image demosaicking using a cascade of convolutional residual denoising networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 317–333 (2018)

  70. Grossberg, M.D., Nayar, S.K.: What is the space of camera response functions? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (2003)

  71. Kang, H.R.: Computational Color Technology. SPIE Press, Bellingham (2006)

    Book  Google Scholar 

  72. Henz, B., Gastal, E.S., Oliveira, M.M.: Deep joint design of color filter arrays and demosaicing. Compu. Graph. Forum 37(2), 389–399 (2018)

    Article  Google Scholar 

  73. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 234–241 (2015)

  74. Funt, B., Shi, L.: The rehabilitation of MaxRGB. In: Proceedings of the Color and Imaging Conference (CIC), pp. 256–259 (2010)

  75. Froehlich, J., Grandinetti, S., Eberhardt, B., Walter, S., Schilling, A., Brendel, H.: Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays. Proc. SPIE 9023, 279–288 (2014)

    Google Scholar 

  76. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv:1412.6980 (2014)

  77. Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph. 30(4), 1–13 (2011)

    Article  Google Scholar 

  78. Monno, Y., Kiku, D., Tanaka, M., Okutomi, M.: Adaptive residual interpolation for color and multispectral image demosaicking. Sensors 17(12), 2787 (2017)

    Article  Google Scholar 

  79. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, C.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision Workshops (ECCVW), pp. 1–16 (2018)

  80. Fan, Y., Yu, J., Huang, T.S.: Wide-activated deep residual networks based restoration for BPG-compressed images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition workshops (CVPRW), pp. 2621–2624 (2018)

  81. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1132–1140 (2017)

Download references

Author information

Authors and Affiliations

  1. Department of Systems and Control Engineering, School of Engineering, Tokyo Institute of Technology, 2-12-1 Ookayama, Meguro-ku, Tokyo, 152-8550, Japan

    Yutaro Okamoto, Masayuki Tanaka, Yusuke Monno & Masatoshi Okutomi

  2. Advanced Technology Research Institute, Kyocera Corporation, Kanagawa, Japan

    Yutaro Okamoto

Authors
  1. Yutaro Okamoto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Masayuki Tanaka
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yusuke Monno
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Masatoshi Okutomi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yusuke Monno.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 76913 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Okamoto, Y., Tanaka, M., Monno, Y. et al. Deep snapshot HDR imaging using multi-exposure color filter array. Vis Comput 40, 3285–3301 (2024). https://doi.org/10.1007/s00371-023-03032-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03032-4

Keywords

Search

Navigation