Abstract
High dynamic range (HDR) deghosting algorithms aim to generate ghost-free HDR images with realistic details. Restricted by the locality of the receptive field, existing CNN-based methods are typically prone to producing ghosting artifacts and intensity distortions in the presence of large motion and severe saturation. In this paper, we propose a novel Context-Aware Vision Transformer (CA-ViT) for ghost-free high dynamic range imaging. The CA-ViT is designed as a dual-branch architecture, which can jointly capture both global and local dependencies. Specifically, the global branch employs a window-based Transformer encoder to model long-range object movements and intensity variations to solve ghosting. For the local branch, we design a local context extractor (LCE) to capture short-range image features and use the channel attention mechanism to select informative local details across the extracted features to complement the global branch. By incorporating the CA-ViT as basic components, we further build the HDR-Transformer, a hierarchical network to reconstruct high-quality ghost-free HDR images. Extensive experiments on three benchmark datasets show that our approach outperforms state-of-the-art methods qualitatively and quantitatively with considerably reduced computational budgets. Codes are available at https://github.com/megvii-research/HDR-Transformer.
Z. Liu and Y. Wang—-Joint First Author.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. IJCV 92(1), 1–31 (2011)
Bogoni, L.: Extending dynamic range of monochrome and color images through fusion. In: Proceedings of the ICPR, pp. 7–12 (2000)
Chen, H., et al.: Pre-trained image processing transformer. In: Proceedings of the CVPR, pp. 12299–12310 (2021)
Chung, H., Cho, N.I.: High dynamic range imaging of dynamic scenes with saturation compensation but without explicit motion compensation. In: Proceedings of the CVPR, pp. 2951–2961 (2022)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Dosovitskiy, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Gallo, O., Gelfandz, N., Chen, W.C., Tico, M., Pulli, K.: Artifact-free high dynamic range imaging. In: Proceedings of the ICCP, pp. 1–7 (2009)
Grosch, T.: Fast and robust high dynamic range image generation with camera and object movement. In: Proceedings of the VMV, pp. 277–284 (2006)
Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge University Press (2003)
Hu, J., Gallo, O., Pulli, K., Sun, X.: Hdr deghosting: How to deal with saturation? In: Proceedings of the CVPR, pp. 1163–1170 (2013)
Jacobs, K., Loscos, C., Ward, G.: Automatic high-dynamic range image generation for dynamic scenes. IEEE Comput. Graphics Appl. 28(2), 84–93 (2008)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Kalantari, N.K., Ramamoorthi, R.: Deep high dynamic range imaging of dynamic scenes. ACM Trans. Graphics 36(4), 144 (2017)
Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High dynamic range video. ACM Trans. Graphics 22(3), 319–325 (2003)
Khan, E.A., Akyuz, A.O., Reinhard, E.: Ghost removal in high dynamic range images. In: Proceedings of the ICIP, pp. 2005–2008 (2006)
Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Swinir: Image restoration using swin transformer. In: Proceedings of the ICCVW, pp. 1833–1844 (2021)
Liu, G., Reda, F.A., Shih, K.J., Wang, T.-C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 89–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_6
Liu, Z., et al.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the ICCV, pp. 10012–10022 (2021)
Liu, Z., et al.: Adnet: Attention-guided deformable convolutional network for high dynamic range imaging. In: Proceedings of the CVPRW, pp. 463–470 (2021)
Ma, K., Duanmu, Z., Zhu, H., Fang, Y., Wang, Z.: Deep guided learning for fast multi-exposure image fusion. IEEE Trans. on Image Processing 29, 2808–2819 (2019)
Ma, K., Li, H., Yong, H., Wang, Z., Meng, D., Zhang, L.: Robust multi-exposure image fusion: a structural patch decomposition approach. IEEE Trans. on Image Processing 26(5), 2519–2532 (2017)
Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graphics 30(4), 1–14 (2011)
Mertens, T., Kautz, J., Van Reeth, F.: Exposure fusion. In: Proceedings of the PG, pp. 382–390 (2007)
Naseer, M., Ranasinghe, K., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Intriguing properties of vision transformers. arXiv preprint arXiv:2105.10497 (2021)
Niu, Y., Wu, J., Liu, W., Guo, W., Lau, R.W.: Hdr-gan: Hdr image reconstruction from multi-exposed ldr images with large motions. IEEE Trans. on Image Processing 30, 3885–3896 (2021)
Oh, T.H., Lee, J.Y., Tai, Y.W., Kweon, I.S.: Robust high dynamic range imaging by rank minimization. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1219–1232 (2014)
Pece, F., Kautz, J.: Bitmap movement detection: Hdr for dynamic scenes. In: Proceedings of the CVMP, pp. 1–8 (2010)
Prabhakar, K.R., Agrawal, S., Singh, D.K., Ashwath, B., Babu, R.V.: Towards practical and efficient high-resolution HDR deghosting with CNN. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 497–513. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_30
Prabhakar, K.R., Arora, R., Swaminathan, A., Singh, K.P., Babu, R.V.: A fast, scalable, and reliable deghosting method for extreme exposure fusion. In: Proceedings of the ICCP, pp. 1–8. IEEE (2019)
Prabhakar, K.R., Senthil, G., Agrawal, S., Babu, R.V., Gorthi, R.K.S.S.: Labeled from unlabeled: Exploiting unlabeled data for few-shot deep hdr deghosting. In: Proceedings of the CVPR, pp. 4875–4885 (2021)
Ram Prabhakar, K., Sai Srikar, V., Venkatesh Babu, R.: Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: Proceedings of the ICCV, pp. 4714–4722 (2017)
Raman, S., Chaudhuri, S.: Reconstruction of high contrast images for dynamic scenes. Vis. Comput. 27(12), 1099–1114 (2011)
Sen, P., Kalantari, N.K., Yaesoubi, M., Darabi, S., Goldman, D.B., Shechtman, E.: Robust patch-based hdr reconstruction of dynamic scenes. ACM Trans. Graphics 31(6), 203 (2012)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Tursun, O.T., Akyüz, A.O., Erdem, A., Erdem, E.: An objective deghosting quality metric for hdr images. In: Proceedings of the CGF, pp. 139–152 (2016)
Vaswani, A., et al.: Attention is all you need. In: Proceedings of the NeurIPS, pp. 5998–6008 (2017)
Wu, S., Xu, J., Tai, Y.-W., Tang, C.-K.: Deep high dynamic range imaging with large foreground motions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 120–135. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_8
Xiao, T., Singh, M., Mintun, E., Darrell, T., Dollár, P., Girshick, R.: Early convolutions help transformers see better. arXiv preprint arXiv:2106.14881 (2021)
Yan, Q., et al.: Attention-guided network for ghost-free high dynamic range imaging. In: Proceedings of the CVPR, pp. 1751–1760 (2019)
Yan, Q., et al.: Deep hdr imaging via a non-local network. IEEE Trans. on Image Processing 29, 4308–4322 (2020)
Zhang, W., Cham, W.K.: Gradient-directed multiexposure composition. IEEE Trans. on Image Processing 21(4), 2318–2323 (2011)
Zimmer, H., Bruhn, A., Weickert, J.: Freehand hdr imaging of moving scenes with simultaneous resolution enhancement. In: Proceedings of the CGF, pp. 405–414 (2011)
Acknowledgement
This work was supported by National Natural Science Foundation of China under grants No. (61872067, 62031009 and 61720106004).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, Z., Wang, Y., Zeng, B., Liu, S. (2022). Ghost-free High Dynamic Range Imaging with Context-Aware Transformer. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13679. Springer, Cham. https://doi.org/10.1007/978-3-031-19800-7_20
Download citation
DOI: https://doi.org/10.1007/978-3-031-19800-7_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-19799-4
Online ISBN: 978-3-031-19800-7
eBook Packages: Computer ScienceComputer Science (R0)