Skip to main content

Selective TransHDR: Transformer-Based Selective HDR Imaging Using Ghost Region Mask

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13677))

Included in the following conference series:

Abstract

The primary issue in high dynamic range (HDR) imaging is the removal of ghost artifacts afforded when merging multi-exposure low dynamic range images. In the weakly misaligned region, ghost artifacts can be suppressed using convolutional neural network (CNN)-based methods. However, in highly misaligned regions, it is necessary to extract features from the global region because the necessary information does not exist in the local region. Therefore, the CNN-based methods specialized for local features extraction cannot obtain satisfactory results. To address this issue, we propose a transformer-based selective HDR image reconstruction network that uses a ghost region mask. The proposed method separates a given image into ghost and non-ghost regions, and then, selectively applies either the CNN or the transformer. The proposed selective transformer module divides an entire image into several regions to effectively extract the features of each region for HDR image reconstruction, thereby extracting the whole information required for HDR reconstruction in the ghost regions from the entire image. Extensive experiments conducted on several benchmark datasets demonstrate the superiority of the proposed method over existing state-of-the-art methods in terms of the mitigation of ghost artifacts.

J. W. Song and Y.-I. Park—Equal contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. An, J., Lee, S.H., Kuk, J.G., Cho, N.I.: A multi-exposure image fusion algorithm without ghost effect. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1565–1568. IEEE (2011)

    Google Scholar 

  2. Bogoni, L.: Extending dynamic range of monochrome and color images through fusion. In: Proceedings 15th International Conference on Pattern Recognition, ICPR 2000, vol. 3, pp. 7–12. IEEE (2000)

    Google Scholar 

  3. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13

    Chapter  Google Scholar 

  4. Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: ACM SIGGRAPH 2008 Classes, pp. 1–10 (2008)

    Google Scholar 

  5. Dosovitskiy, A., et al.: An image is worth \(16 \times 16\) words: transformers for image recognition at scale. In: International Conference on Learning Representations (2021)

    Google Scholar 

  6. Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2758–2766 (2015)

    Google Scholar 

  7. Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R.K., Unger, J.: HDR image reconstruction from a single exposure using deep CNNs. ACM Trans. Graph. (TOG) 36(6), 1–15 (2017)

    Article  Google Scholar 

  8. Endo, Y., Kanamori, Y., Mitani, J.: Deep reverse tone mapping. ACM Trans. Graph. (Proceedings of SIGGRAPH ASIA 2017) 36(6) (2017)

    Google Scholar 

  9. Fleet, D., Weiss, Y.: Optical flow estimation. In: Paragios, N., Chen, Y., Faugeras, O. (eds.) Handbook of Mathematical Models in Computer Vision, pp. 237–257. Springer, Boston (2006). https://doi.org/10.1007/0-387-28831-7_15

    Chapter  Google Scholar 

  10. Granados, M., Ajdin, B., Wand, M., Theobalt, C., Seidel, H.P., Lensch, H.P.: On being ‘undigital’ with digital cameras: extending dynamic range by combining differently exposed pictures. In: Proceedings of IS &T, pp. 442–448 (1995)

    Google Scholar 

  11. Granados, M., Ajdin, B., Wand, M., Theobalt, C., Seidel, H.P., Lensch, H.P.: Optimal HDR reconstruction with linear digital cameras. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 215–222. IEEE (2010)

    Google Scholar 

  12. Granados, M., Kim, K.I., Tompkin, J., Theobalt, C.: Automatic noise modeling for ghost-free HDR reconstruction. ACM Trans. Graph. (TOG) 32(6), 1–10 (2013)

    Article  Google Scholar 

  13. Grosch, T., et al.: Fast and robust high dynamic range image generation with camera and object movement. Vision, Modeling and Visualization, RWTH Aachen 277284 (2006)

    Google Scholar 

  14. Heo, Y.S., Lee, K.M., Lee, S.U., Moon, Y., Cha, J.: Ghost-free high dynamic range imaging. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010. LNCS, vol. 6495, pp. 486–500. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19282-1_39

    Chapter  Google Scholar 

  15. Hu, J., Gallo, O., Pulli, K., Sun, X.: HDR deghosting: how to deal with saturation? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1163–1170 (2013)

    Google Scholar 

  16. Jinno, T., Okuda, M.: Motion blur free HDR image acquisition using multiple exposures. In: 2008 15th IEEE International Conference on Image Processing, pp. 1304–1307. IEEE (2008)

    Google Scholar 

  17. Kalantari, N.K., Ramamoorthi, R.: Deep high dynamic range imaging of dynamic scenes. ACM Trans. Graph. 36(4), 144-1 (2017)

    Google Scholar 

  18. Khan, E.A., Akyuz, A.O., Reinhard, E.: Ghost removal in high dynamic range images. In: 2006 International Conference on Image Processing, pp. 2005–2008. IEEE (2006)

    Google Scholar 

  19. Khan, Z., Khanna, M., Raman, S.: FHDR: HDR image reconstruction from a single LDR image using feedback network. In: 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 1–5 (2019). https://doi.org/10.1109/GlobalSIP45357.2019.8969167

  20. Lee, C., Li, Y., Monga, V.: Ghost-free high dynamic range imaging via rank minimization. IEEE Signal Process. Lett. 21(9), 1045–1049 (2014)

    Article  Google Scholar 

  21. Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph. (TOG) 30(4), 1–14 (2011)

    Article  Google Scholar 

  22. Mitsunaga, T., Nayar, S.K.: Radiometric self calibration. In: Proceedings of the 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), vol. 1, pp. 374–380. IEEE (1999)

    Google Scholar 

  23. Moon, Y.S., Tai, Y.M., Cha, J.H., Lee, S.H.: A simple ghost-free exposure fusion for embedded HDR imaging. In: 2012 IEEE International Conference on Consumer Electronics (ICCE), pp. 9–10. IEEE (2012)

    Google Scholar 

  24. Niu, Y., Wu, J., Liu, W., Guo, W., Lau, R.W.: HDR-GAN: HDR image reconstruction from multi-exposed LDR images with large motions. IEEE Trans. Image Process. 30, 3885–3896 (2021)

    Article  Google Scholar 

  25. Prabhakar, K.R., Agrawal, S., Singh, D.K., Ashwath, B., Babu, R.V.: Towards practical and efficient high-resolution HDR deghosting with CNN. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 497–513. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_30

    Chapter  Google Scholar 

  26. Pu, Z., Guo, P., Asif, M.S., Ma, Z.: Robust high dynamic range (HDR) imaging with complex motion and parallax. In: Proceedings of the Asian Conference on Computer Vision (2020)

    Google Scholar 

  27. Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., Myszkowski, K.: High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting. Morgan Kaufmann, Burlington (2010)

    Google Scholar 

  28. Sen, P., Kalantari, N.K., Yaesoubi, M., Darabi, S., Goldman, D.B., Shechtman, E.: Robust patch-based HDR reconstruction of dynamic scenes. ACM Trans. Graph. 31(6), 203-1 (2012)

    Google Scholar 

  29. Srikantha, A., Sidibé, D.: Ghost detection and removal for high dynamic range images: recent advances. Signal Process.: Image Commun. 27(6), 650–662 (2012)

    Google Scholar 

  30. Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

    Google Scholar 

  31. Tursun, O.T., Akyüz, A.O., Erdem, A., Erdem, E.: The state of the art in HDR deghosting: a survey and evaluation. In: Computer Graphics Forum, vol. 34, pp. 683–707. Wiley Online Library (2015)

    Google Scholar 

  32. Tursun, O.T., Akyüz, A.O., Erdem, A., Erdem, E.: An objective deghosting quality metric for HDR images. In: Computer Graphics Forum, vol. 35, pp. 139–152. Wiley Online Library (2016)

    Google Scholar 

  33. Vaswani, A., et al.: Attention is all you need. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017)

    Google Scholar 

  34. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)

    Google Scholar 

  35. Wu, S., Xu, J., Tai, Y.W., Tang, C.K.: Deep high dynamic range imaging with large foreground motions. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 117–132 (2018)

    Google Scholar 

  36. Yan, Q., et al.: Attention-guided network for ghost-free high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1751–1760 (2019)

    Google Scholar 

  37. Yan, Q., et al.: Multi-scale dense networks for deep high dynamic range imaging. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 41–50. IEEE (2019)

    Google Scholar 

  38. Yan, Q., Sun, J., Li, H., Zhu, Y., Zhang, Y.: High dynamic range imaging by sparse representation. Neurocomputing 269, 160–169 (2017)

    Article  Google Scholar 

  39. Yan, Q., et al.: Deep HDR imaging via a non-local network. IEEE Trans. Image Process. 29, 4308–4322 (2020)

    Article  Google Scholar 

  40. Zheng, S., et al.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6881–6890 (2021)

    Google Scholar 

Download references

Acknowledgement

This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2021R1A2C1004208).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jou Won Song .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 16495 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Song, J.W., Park, YI., Kong, K., Kwak, J., Kang, SJ. (2022). Selective TransHDR: Transformer-Based Selective HDR Imaging Using Ghost Region Mask. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13677. Springer, Cham. https://doi.org/10.1007/978-3-031-19790-1_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19790-1_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19789-5

  • Online ISBN: 978-3-031-19790-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics