Advertisement

The Visual Computer

, Volume 35, Issue 6–8, pp 997–1011 | Cite as

Deep binocular tone mapping

  • Zhuming Zhang
  • Chu Han
  • Shengfeng He
  • Xueting Liu
  • Haichao Zhu
  • Xinghong Hu
  • Tien-Tsin WongEmail author
Original Article
  • 150 Downloads

Abstract

Binocular tone mapping is studied in the previous works to generate a fusible pair of LDR images in order to convey more visual content than one single LDR image. However, the existing methods are all based on monocular tone mapping operators. It greatly restricts the preservation of local details and global contrast in a binocular LDR pair. In this paper, we proposed the first binocular tone mapping operator to more effectively distribute visual content to an LDR pair, leveraging the great representability and interpretability of deep convolutional neural network. Based on the existing binocular perception models, novel loss functions are also proposed to optimize the output pairs in terms of local details, global contrast, content distribution, and binocular fusibility. Our method is validated with a qualitative and quantitative evaluation, as well as a user study. Statistics show that our method outperforms the state-of-the-art binocular tone mapping frameworks in terms of both visual quality and time performance.

Keywords

Tone mapping Binocular tone mapping Binocular perception Convolutional neural network 

Notes

Acknowledgements

This project is supported by the Research Grants Council of the Hong Kong Special Administrative Region, under RGC General Research Fund (Project No. CUHK 14201017), and Shenzhen Science and Technology Programs (No. JCYJ20160429190300857, No. JCYJ20180507182410327, and No. JCYJ20180507182415428).

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

References

  1. 1.
    Aubry, M., Paris, S., Hasinoff, S.W., Kautz, J., Durand, F.: Fast local laplacian filters: theory and applications. ACM Trans. Graph. (TOG) 33(5), 167 (2014)Google Scholar
  2. 2.
    Banterle, F., Artusi, A., Debattista, K., Chalmers, A.: Advanced High Dynamic Range Imaging. Taylor & Francis, CRC Press (2017)Google Scholar
  3. 3.
    Curtis, D.W., Rule, S.J.: Binocular processing of brightness information: a vector-sum model. J. Exp. Psychol. Human Percept. Perform. 4(1), 132 (1978)Google Scholar
  4. 4.
    Daly, S.J.: Visible differences predictor: an algorithm for the assessment of image fidelity. In: Human Vision, Visual Processing, and Digital Display III, vol. 1666, pp. 2–16. International Society for Optics and Photonics (1992).  https://doi.org/10.1117/12.135952
  5. 5.
    De Weert, C.M., Levelt, W.J.M.: Binocular brightness combinations: additive and nonadditive aspects. Percept. Psychophys. 15(3), 551–562 (1974)Google Scholar
  6. 6.
    Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: ACM SIGGRAPH, pp. 369–378 (1997).  https://doi.org/10.1145/258734.258884
  7. 7.
    Drago, F., Myszkowski, K., Annen, T., Chiba, N.: Adaptive logarithmic mapping for displaying high contrast scenes. In: Computer Graphics Forum, vol. 22, pp. 419–426. Wiley Online Library (2003).  https://doi.org/10.1111/1467-8659.00689
  8. 8.
    Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. In: ACM Transactions on Graphics (TOG), vol. 21, pp. 257–266. ACM (2002).  https://doi.org/10.1145/566570.566574
  9. 9.
    Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R.K., Unger, J.: Hdr image reconstruction from a single exposure using deep cnns. arXiv preprint arXiv:1710.07480 (2017)
  10. 10.
    Endo, Y., Kanamori, Y., Mitani, J.: Deep reverse tone mapping. ACM Trans. Graph. (TOG) 36(6), 177 (2017)Google Scholar
  11. 11.
    Engel, G.: The autocorrelation function and binocular brightness mixing. Vis. Res. 9(9), 1111–1130 (1969)Google Scholar
  12. 12.
    Fairchild, M.D.: The hdr photographic survey. In: Color and Imaging Conference, vol. 2007, pp. 233–238. Society for Imaging Science and Technology (2007)Google Scholar
  13. 13.
    Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. In: ACM Transactions on Graphics (TOG), ACM, vol. 27, p. 67 (2008).  https://doi.org/10.1145/1399504.1360666
  14. 14.
    Feng, M., Loew, M.H.: Video-level binocular tone-mapping framework based on temporal coherency algorithm. In: 2017 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), IEEE, pp. 1–5 (2017)Google Scholar
  15. 15.
    Gharbi, M., Chen, J., Barron, J.T., Hasinoff, S.W., Durand, F.: Deep bilateral learning for real-time image enhancement. ACM Trans. Graph. (TOG) 36(4), 118 (2017)Google Scholar
  16. 16.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014). https://papers.nips.cc/paper/5423-generative-adversarial-nets
  17. 17.
    Hasinoff, S.W., Sharlet, D., Geiss, R., Adams, A., Barron, J.T., Kainz, F., Chen, J., Levoy, M.: Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph. (TOG) 35(6), 192 (2016)Google Scholar
  18. 18.
    Hau Chua, S., Zhang, H., Hammad, M., Zhao, S., Goyal, S., Singh, K.: Colorbless: augmenting visual information for colorblind people with binocular luster effect. ACM Trans. Comput.-Hum. Interact. (TOCHI) 21(6), 32 (2015)Google Scholar
  19. 19.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. (TOG) 35(4), 110 (2016)Google Scholar
  20. 20.
    Jähne, B., Haussecker, H., Geissler, P.: Handbook of computer vision and applications, vol. 2. Citeseer (1999)Google Scholar
  21. 21.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) European Conference on Computer Vision, pp. 694–711. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_43
  22. 22.
    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  23. 23.
    Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  24. 24.
    Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, vol. 2, p. 4 (2017)Google Scholar
  25. 25.
    Legge, G.E.: Binocular contrast summation-II. Quadratic Summ. Vis. Res. 24(4), 385–394 (1984)Google Scholar
  26. 26.
    Legge, G.E., Rubin, G.S.: Binocular interactions in suprathreshold contrast perception. Percept. Psychophys. 30(1), 49–61 (1981)Google Scholar
  27. 27.
    Levelt, W.J.: Binocular brightness averaging and contour information. Br. J. Psychol. 56(1), 1–13 (1965)Google Scholar
  28. 28.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)Google Scholar
  29. 29.
    Maehara, G., Goryo, K.: Binocular, monocular and dichoptic pattern masking. Opt. Rev. 12(2), 76–82 (2005)Google Scholar
  30. 30.
    Meese, T.S., Georgeson, M.A., Baker, D.H.: Interocular masking and summation indicate two stages of divisive contrast gain control. In: Twenty-Eighth European Conference on Visual Perception (2005)Google Scholar
  31. 31.
    Meese, T.S., Georgeson, M.A., Baker, D.H.: Binocular contrast vision at and above threshold. J. Vis. 6(11), 7–7 (2006)Google Scholar
  32. 32.
    Meese, T.S., Hess, R.F.: Low spatial frequencies are suppressively masked across spatial scale, orientation, field position, and eye of origin. J. Vis. 4(10), 2–2 (2004)Google Scholar
  33. 33.
    Nemoto, H., Korshunov, P., Hanhart, P., Ebrahimi, T.: Visual attention in ldr and hdr images. In: 9th International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM), EPFL-CONF-203873 (2015)Google Scholar
  34. 34.
    Paris, S., Hasinoff, S.W., Kautz, J.: Local laplacian filters: edge-aware image processing with a laplacian pyramid. ACM Trans. Graph. (TOG) 30(4), 1–68 (2011)Google Scholar
  35. 35.
    Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch. In: NIPS-W (2017)Google Scholar
  36. 36.
    Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016).  https://doi.org/10.1109/CVPR.2016.278
  37. 37.
    Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., Myszkowski, K.: High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting. Morgan Kaufmann, Burlington (2010)Google Scholar
  38. 38.
    Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic tone reproduction for digital images. ACM Trans. Graph. (TOG) 21(3), 267–276 (2002)Google Scholar
  39. 39.
    Sajjadi, M.S.M., Schölkopf, B., Hirsch, M.: Enhancenet: Single image super-resolution through automated texture synthesis. In: IEEE International Conference on Computer Vision (ICCV) (2017)Google Scholar
  40. 40.
    Schlick, C.: Quantization techniques for visualization of high dynamic range pictures. In: Photorealistic Rendering Techniques, pp. 7–20. Springer (1995)Google Scholar
  41. 41.
    Sendik, O., Cohen-Or, D.: Deep correlations for texture synthesis. ACM Trans. Graph. (TOG) 36(5), 161 (2017).  https://doi.org/10.1145/3015461 Google Scholar
  42. 42.
    Smith, K., Krawczyk, G., Myszkowski, K., Seidel, H.P.: Beyond tone mapping: enhanced depiction of tone mapped hdr images. In: Computer Graphics Forum, vol. 25, pp. 427–438. Wiley Online Library (2006)Google Scholar
  43. 43.
    Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: AAAI, vol. 4, p. 12 (2017). https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/viewPaper/14806
  44. 44.
    Tumblin, J., Rushmeier, H.: Tone reproduction for realistic images. IEEE Comput. Graph. Appl. 13(6), 42–48 (1993).  https://doi.org/10.1109/38.252554 Google Scholar
  45. 45.
    Tumblin, J., Turk, G.: Lcis: A boundary hierarchy for detail-preserving contrast reduction. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 83–90. ACM Press/Addison-Wesley Publishing Co. (1999)Google Scholar
  46. 46.
    von Helmholtz, H., Southall, J.P.C.: Treatise on Physiological Optics, vol. 3. Courier Corporation, North Chelmsford (2005)Google Scholar
  47. 47.
    Wilson, H.R.: Binocular contrast, stereopsis, and rivalry: toward a dynamical synthesis. Vis. Res. 140, 89–95 (2017)Google Scholar
  48. 48.
    Ward, G.: A contrast-based scalefactor for luminance display. Graph. Gems IV, 415–421 (1994)Google Scholar
  49. 49.
    Xie, S., Tu, Z.: Holistically-nested edge detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1395–1403 (2015)Google Scholar
  50. 50.
    Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. arXiv preprint arXiv:1611.09969 (2016)
  51. 51.
    Yang, X., Zhang, L., Wong, T.T., Heng, P.A.: Binocular tone mapping. ACM Trans. Graph. (SIGGRAPH 2012 issue) 31(4), 93:1–93:10 (2012)Google Scholar
  52. 52.
    Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv preprint: arXiv:1511.07122 (2015)
  53. 53.
    Zhang, L., Zhang, L., Mou, X., Zhang, D., et al.: Fsim: a feature similarity index for image quality assessment. IEEE Trans. Image Process. 20(8), 2378–2386 (2011)MathSciNetzbMATHGoogle Scholar
  54. 54.
    Zhang, Z., Hu, X., Liu, X., Wong, T.T.: Binocular tone mapping with improved overall contrast and local details. Comput. Graph. Forum 37(7), 433–442 (2018)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  • Zhuming Zhang
    • 1
    • 2
  • Chu Han
    • 1
    • 2
  • Shengfeng He
    • 3
  • Xueting Liu
    • 4
  • Haichao Zhu
    • 5
  • Xinghong Hu
    • 1
    • 2
  • Tien-Tsin Wong
    • 1
    • 2
    Email author
  1. 1.The Chinese University of Hong KongShatin, NTHong Kong SAR, China
  2. 2.Guangdong Provincial Key Laboratory of Computer Vision and Virtual Reality TechnologyShenzhen Institutes of Advanced Technology, Chinese Academy of SciencesShenzhenChina
  3. 3.South China University of TechnologyGuangzhouChina
  4. 4.Caritas Institute of Higher EducationTseung Kwan O, NTHong Kong, China
  5. 5.Rokid Corporation LtdHangzhouChina

Personalised recommendations