Advertisement

Towards Practical and Efficient High-Resolution HDR Deghosting with CNN

Conference paper
  • 627 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12366)

Abstract

Generating High Dynamic Range (HDR) image in the presence of camera and object motion is a tedious task. If uncorrected, these motions will manifest as ghosting artifacts in the fused HDR image. On one end of the spectrum, there exist methods that generate high-quality results that are computationally demanding and too slow. On the other end, there are few faster methods that produce unsatisfactory results. With ever increasing sensor/display resolution, currently we are very much in need of faster methods that produce high-quality images. In this paper, we present a deep neural network based approach to generate high-quality ghost-free HDR for high-resolution images. Our proposed method is fast and fuses a sequence of three high-resolution images (16-megapixel resolution) in about 10 s. Through experiments and ablations, on different publicly available datasets, we show that the proposed method achieves state-of-the-art performance in terms of accuracy and speed.

Notes

Acknowledgements

This work was supported by a project grant from MeitY (No.4(16)/2019-ITEA), Govt. of India and WIRIN.

Supplementary material

504479_1_En_30_MOESM1_ESM.pdf (39.4 mb)
Supplementary material 1 (pdf 40342 KB)

References

  1. 1.
    Bogoni, L.: Extending dynamic range of monochrome and color images through fusion. In: Proceedings of the 15th International Conference on Pattern Recognition (2000)Google Scholar
  2. 2.
    Chen, J., Adams, A., Wadhwa, N., Hasinoff, S.W.: Bilateral guided upsampling. ACM Trans. Graph. (TOG) 35(6), 203 (2016)CrossRefGoogle Scholar
  3. 3.
    Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: ACM SIGGRAPH 2008 Classes, p. 31. ACM (2008)Google Scholar
  4. 4.
    Eden, A., Uyttendaele, M., Szeliski, R.: Seamless image stitching of scenes with large motions and exposure differences. In: Conference on Computer Vision and Pattern Recognition. IEEE (2006)Google Scholar
  5. 5.
    Gallo, O., Gelfandz, N., Chen, W.C., Tico, M., Pulli, K.: Artifact-free high dynamic range imaging. In: ICCP, pp. 1–7. IEEE (2009)Google Scholar
  6. 6.
    Gallo, O., Troccoli, A., Hu, J., Pulli, K., Kautz, J.: Locally non-rigid registration for mobile HDR photography. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 49–56 (2015)Google Scholar
  7. 7.
    Gharbi, M., Chen, J., Barron, J.T., Hasinoff, S.W., Durand, F.: Deep bilateral learning for real-time image enhancement. ACM Trans. Graph. (TOG) 36(4), 1–12 (2017)CrossRefGoogle Scholar
  8. 8.
    Granados, M., Kim, K.I., Tompkin, J., Theobalt, C.: Automatic noise modeling for ghost-free HDR reconstruction. ACM Trans. Graph. (TOG) 32(6), 201 (2013)CrossRefGoogle Scholar
  9. 9.
    Grosch, T.: Fast and robust high dynamic range image generation with camera and object movement. Vis. Model. Vis. RWTH Aachen 277–284 (2006)Google Scholar
  10. 10.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  11. 11.
    Heo, Y.S., Lee, K.M., Lee, S.U., Moon, Y., Cha, J.: Ghost-free high dynamic range imaging. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010. LNCS, vol. 6495, pp. 486–500. Springer, Heidelberg (2011).  https://doi.org/10.1007/978-3-642-19282-1_39CrossRefGoogle Scholar
  12. 12.
    Hu, J., Gallo, O., Pulli, K.: Exposure stacks of live scenes with hand-held cameras. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7572, pp. 499–512. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33718-5_36CrossRefGoogle Scholar
  13. 13.
    Hu, J., Gallo, O., Pulli, K., Sun, X.: HDR deghosting: How to deal with saturation? In: IEEE Conference on Computer Vision and Pattern Recognition (2013)Google Scholar
  14. 14.
    Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)Google Scholar
  15. 15.
    Kalantari, N.K., Ramamoorthi, R.: Deep high dynamic range imaging of dynamic scenes. ACM Trans. Graph. (Proc. SIGGRAPH 2017) 36(4), 144-1 (2017)Google Scholar
  16. 16.
    Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High dynamic range video. ACM Trans. Graph. (TOG) 22, 319–325 (2003)CrossRefGoogle Scholar
  17. 17.
    Kao, W.C., Hsu, C.C., Chen, L.Y., Kao, C.C., Chen, S.H.: Integrating image fusion and motion stabilization for capturing still images in high dynamic range scenes. IEEE Trans. Consum. Electron. 52(3), 735–741 (2006)CrossRefGoogle Scholar
  18. 18.
    Khan, E.A., Akyiiz, A., Reinhard, E.: Ghost removal in high dynamic range images. In: IEEE International Conference on Image Processing (2006)Google Scholar
  19. 19.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  20. 20.
    Liu, C., et al.: Beyond pixels: exploring new representations and applications for motion analysis. Ph.D. thesis, Massachusetts Institute of Technology (2009)Google Scholar
  21. 21.
    Ma, K., Li, H., Yong, H., Wang, Z., Meng, D., Zhang, L.: Robust multi-exposure image fusion: a structural patch decomposition approach. IEEE Trans. Image Process. 26(5), 2519–2532 (2017)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph. (TOG) 30(4), 40 (2011)CrossRefGoogle Scholar
  23. 23.
    Min, T.H., Park, R.H., Chang, S.: Histogram based ghost removal in high dynamic range images. In: 2009 IEEE International Conference on Multimedia and Expo, pp. 530–533. IEEE (2009)Google Scholar
  24. 24.
    Pece, F., Kautz, J.: Bitmap movement detection: HDR for dynamic scenes. In: 2010 Conference on Visual Media Production, pp. 1–8. IEEE (2010)Google Scholar
  25. 25.
    Prabhakar, K.R., Arora, R., Swaminathan, A., Singh, K.P., Babu, R.V.: A fast, scalable, and reliable deghosting method for extreme exposure fusion. In: 2019 IEEE International Conference on Computational Photography (ICCP), pp. 1–8. IEEE (2019)Google Scholar
  26. 26.
    Prabhakar, K.R., Babu, R.V.: Ghosting-free multi-exposure image fusion in gradient domain. In: IEEE International Conference on Acoustics, Speech and Signal Processing (2016)Google Scholar
  27. 27.
    Prabhakar, K.R., Srikar, V.S., Babu, R.V.: DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: IEEE International Conference on Computer Vision (ICCV), pp. 4724–4732. IEEE (2017)Google Scholar
  28. 28.
    Raman, S., Chaudhuri, S.: Reconstruction of high contrast images for dynamic scenes. Vis. Comput. 27(12), 1099–1114 (2011)CrossRefGoogle Scholar
  29. 29.
    Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., Myszkowski, K.: High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting. Morgan Kaufmann, Burlington (2010)Google Scholar
  30. 30.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  31. 31.
    Schuster, R., Wasenmuller, O., Unger, C., Stricker, D.: SDC-stacked dilated convolution: a unified descriptor network for dense matching tasks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2556–2565 (2019)Google Scholar
  32. 32.
    Sen, P.: Overview of state-of-the-art algorithms for stack-based high-dynamic range (HDR) imaging. Electron. Imaging 2018(5), 1–8 (2018)CrossRefGoogle Scholar
  33. 33.
    Sen, P., Aguerrebere, C.: Practical high dynamic range imaging of everyday scenes: photographing the world as we see it with our own eyes. IEEE Signal Process. Mag. 33(5), 36–44 (2016)CrossRefGoogle Scholar
  34. 34.
    Sen, P., Kalantari, N.K., Yaesoubi, M., Darabi, S., Goldman, D.B., Shechtman, E.: Robust patch-based HDR reconstruction of dynamic scenes. ACM Trans. Graph. 31(6), 203 (2012)CrossRefGoogle Scholar
  35. 35.
    Sidibé, D., Puech, W., Strauss, O.: Ghost detection and removal in high dynamic range images. In: 17th European Signal Processing Conference (2009)Google Scholar
  36. 36.
    Tomaszewska, A., Mantiuk, R.: Image registration for multiexposure high dynamic range image acquisition. In: Proceedings of the International Conference on Computer Graphics, Visualization and Computer Vision (2007)Google Scholar
  37. 37.
    Tursun, O.T., Akyüz, A.O., Erdem, A., Erdem, E.: The state of the art in HDR deghosting: a survey and evaluation. In: Computer Graphics Forum, vol. 34, pp. 683–707. Wiley Online Library (2015)Google Scholar
  38. 38.
    Tursun, O.T., Akyüz, A.O., Erdem, A., Erdem, E.: An objective deghosting quality metric for HDR images. In: Computer Graphics Forum, vol. 35, pp. 139–152. Wiley Online Library (2016)Google Scholar
  39. 39.
    Ward, G.: Fast, robust image registration for compositing high dynamic range photographs from hand-held exposures. J. Graph. Tools 8(2), 17–30 (2003)CrossRefGoogle Scholar
  40. 40.
    Wu, S., Xu, J., Tai, Y.W., Tang, C.K.: Deep high dynamic range imaging with large foreground motions. In: European Conference on Computer Vision, pp. 120–135 (2018)Google Scholar
  41. 41.
    Wu, S., Xie, S., Rahardja, S., Li, Z.: A robust and fast anti-ghosting algorithm for high dynamic range imaging. In: 2010 IEEE International Conference on Image Processing, pp. 397–400. IEEE (2010)Google Scholar
  42. 42.
    Yan, Q., et al.: Attention-guided network for ghost-free high dynamic range imaging. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1751–1760 (2019)Google Scholar
  43. 43.
    Yan, Q., et al.: Multi-scale dense networks for deep high dynamic range imaging. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 41–50. IEEE (2019)Google Scholar
  44. 44.
    Yan, Q., et al.: Deep HDR imaging via a non-local network. IEEE Trans. Image Process. 29, 4308–4322 (2020)CrossRefGoogle Scholar
  45. 45.
    Zhang, W., Cham, W.K.: Reference-guided exposure fusion in dynamic scenes. J. Vis. Commun. Image Represent. 23(3), 467–475 (2012)CrossRefGoogle Scholar
  46. 46.
    Zimmer, H., Bruhn, A., Weickert, J.: Freehand HDR imaging of moving scenes with simultaneous resolution enhancement. In: Computer Graphics Forum, vol. 30, pp. 405–414. Wiley Online Library (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Video Analytics LabIndian Institute of ScienceBangaloreIndia

Personalised recommendations