Advertisement

Deep High Dynamic Range Imaging with Large Foreground Motions

  • Shangzhe WuEmail author
  • Jiarui Xu
  • Yu-Wing Tai
  • Chi-Keung Tang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11206)

Abstract

This paper proposes the first non-flow-based deep framework for high dynamic range (HDR) imaging of dynamic scenes with large-scale foreground motions. In state-of-the-art deep HDR imaging, input images are first aligned using optical flows before merging, which are still error-prone due to occlusion and large motions. In stark contrast to flow-based methods, we formulate HDR imaging as an image translation problem without optical flows. Moreover, our simple translation network can automatically hallucinate plausible HDR details in the presence of total occlusion, saturation and under-exposure, which are otherwise almost impossible to recover by conventional optimization approaches. Our framework can also be extended for different reference images. We performed extensive qualitative and quantitative comparisons to show that our approach produces excellent results where color artifacts and geometric distortions are significantly reduced compared to existing state-of-the-art methods, and is robust across various inputs, including images without radiometric calibration.

Keywords

High dynamic range imaging Computational photography 

Notes

Acknowledgement

This work was supported in part by Tencent Youtu.

References

  1. 1.
    Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1997, pp. 369–378. ACM Press/Addison-Wesley Publishing Co., New York (1997).  https://doi.org/10.1145/258734.258884
  2. 2.
    Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: IEEE ICCV (2015). http://lmb.informatik.uni-freiburg.de/Publications/2015/DFIB15
  3. 3.
    Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R., Unger, J.: HDR image reconstruction from a single exposure using deep CNNs. ACM TOG 36(6), 178 (2017)CrossRefGoogle Scholar
  4. 4.
    Gallo, O., Gelfandz, N., Chen, W.C., Tico, M., Pulli, K.: Artifact-free high dynamic range imaging. In: 2009 IEEE International Conference on Computational Photography (ICCP), pp. 1–7, April 2009.  https://doi.org/10.1109/ICCPHOT.2009.5559003
  5. 5.
    Gallo, O., Sen, P.: Stack-based algorithms for HDR capture and reconstruction. In: Dufaux, F., Callet, P.L., Mantiuk, R.K., Mrak, M. (eds.) High Dynamic Range Video, pp. 85–119. Academic Press (2016).  https://doi.org/10.1016/B978-0-08-100412-8.00003-6Google Scholar
  6. 6.
    Grossberg, M.D., Nayar, S.K.: Determining the camera response from images: what is knowable? IEEE Trans. Pattern Anal. Mach. Intell. 25(11), 1455–1467 (2003).  https://doi.org/10.1109/TPAMI.2003.1240119CrossRefGoogle Scholar
  7. 7.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385
  8. 8.
    Heide, F., et al.: FlexISP: a flexible camera image processing framework. ACM TOG 33(6), 231 (2014)CrossRefGoogle Scholar
  9. 9.
    Heo, Y.S., Lee, K.M., Lee, S.U., Moon, Y., Cha, J.: Ghost-free high dynamic range imaging. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010. LNCS, vol. 6495, pp. 486–500. Springer, Heidelberg (2011).  https://doi.org/10.1007/978-3-642-19282-1_39CrossRefGoogle Scholar
  10. 10.
    Hu, J., Gallo, O., Pulli, K., Sun, X.: HDR deghosting: how to deal with saturation? In: IEEE CVPR (2013)Google Scholar
  11. 11.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE CVPR (2017)Google Scholar
  12. 12.
    Jacobs, K., Loscos, C., Ward, G.: Automatic high-dynamic range image generation for dynamic scenes. IEEE Comput. Graph. Appl. 28(2), 84–93 (2008).  https://doi.org/10.1109/MCG.2008.23CrossRefGoogle Scholar
  13. 13.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution (2016)CrossRefGoogle Scholar
  14. 14.
    Kalantari, N.K., Ramamoorthi, R.: Deep high dynamic range imaging of dynamic scenes. ACM TOG 36(4), 1–14 (2017)CrossRefGoogle Scholar
  15. 15.
    Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High dynamic range video. ACM TOG 22(3), 319–325 (2003).  https://doi.org/10.1145/882262.882270CrossRefGoogle Scholar
  16. 16.
    Khan, E.A., Akyuz, A.O., Reinhard, E.: Ghost removal in high dynamic range images. In: 2006 International Conference on Image Processing, pp. 2005–2008, October 2006.  https://doi.org/10.1109/ICIP.2006.312892
  17. 17.
    Mann, S., Picard, R.W.: On being ‘undigital’ with digital cameras: extending dynamic range by combining differently exposed pictures. In: Proceedings of Imaging Science and Technology, pp. 442–448 (1995)Google Scholar
  18. 18.
    Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM TOG 30(4), 40:1–40:14 (2011).  https://doi.org/10.1145/2010324.1964935CrossRefGoogle Scholar
  19. 19.
    Oh, T.H., Lee, J.Y., Tai, Y.W., Kweon, I.S.: Robust high dynamic range imaging by rank minimization. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1219–1232 (2015).  https://doi.org/10.1109/TPAMI.2014.2361338CrossRefGoogle Scholar
  20. 20.
    Photomatix: Photomatix (2017). https://www.hdrsoft.com
  21. 21.
    Raman, S., Chaudhuri, S.: Reconstruction of high contrast images for dynamic scenes. Vis. Comput. 27(12), 1099–1114 (2011).  https://doi.org/10.1007/s00371-011-0653-0CrossRefGoogle Scholar
  22. 22.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  23. 23.
    Sen, P., Kalantari, N.K., Yaesoubi, M., Darabi, S., Goldman, D.B., Shechtman, E.: Robust patch-based HDR reconstruction of dynamic scenes. ACM TOG 31(6), 203:1–203:11 (2012)CrossRefGoogle Scholar
  24. 24.
    Serrano, A., Heide, F., Gutierrez, D., Wetzstein, G., Masia, B.: Convolutional sparse coding for high dynamic range imaging. Comput. Graph. Forum 35(2), 153–163 (2016)CrossRefGoogle Scholar
  25. 25.
    Tocci, M.D., Kiser, C., Tocci, N., Sen, P.: A versatile HDR video production system. ACM TOG 30(4), 41:1–41:10 (2011).  https://doi.org/10.1145/2010324.1964936CrossRefGoogle Scholar
  26. 26.
    Tomaszewska, A., Mantiuk, R.: Image registration for multi-exposure high dynamic range image acquisition. In: International Conference in Central Europe on Computer Graphics and Visualization, WSCG 2007 (2007). http://wscg.zcu.cz/wscg2007/Papers_2007/full/B13-full.pdf
  27. 27.
    Tursun, O.T., Akyüz, A.O., Erdem, A., Erdem, E.: The state of the art in HDR deghosting: a survey and evaluation. Comput. Graph. Forum 34(2), 683–707 (2015).  https://doi.org/10.1111/cgf.12593CrossRefGoogle Scholar
  28. 28.
    Tursun, O.T., Akyüz, A.O., Erdem, A., Erdem, E.: An objective deghosting quality metric for HDR images. Comput. Graph. Forum 35(2), 139–152 (2016).  https://doi.org/10.1111/cgf.12818CrossRefGoogle Scholar
  29. 29.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE ICCV (2017)Google Scholar
  30. 30.
    Zhu, S., Liu, S., Loy, C.C., Tang, X.: Deep cascaded bi-network for face hallucination. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 614–630. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46454-1_37CrossRefGoogle Scholar
  31. 31.
    Zimmer, H., Bruhn, A., Weickert, J.: Freehand HDR imaging of moving scenes with simultaneous resolution enhancement. Comput. Graph. Forum 30(2), 405–414 (2011).  https://doi.org/10.1111/j.1467-8659.2011.01870.xCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.The Hong Kong University of Science and TechnologyKowloonHong Kong
  2. 2.Tencent YoutuShanghaiChina
  3. 3.University of OxfordOxfordUK

Personalised recommendations