Skip to main content
SpringerLink
Log in

Deep binocular tone mapping

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Binocular tone mapping is studied in the previous works to generate a fusible pair of LDR images in order to convey more visual content than one single LDR image. However, the existing methods are all based on monocular tone mapping operators. It greatly restricts the preservation of local details and global contrast in a binocular LDR pair. In this paper, we proposed the first binocular tone mapping operator to more effectively distribute visual content to an LDR pair, leveraging the great representability and interpretability of deep convolutional neural network. Based on the existing binocular perception models, novel loss functions are also proposed to optimize the output pairs in terms of local details, global contrast, content distribution, and binocular fusibility. Our method is validated with a qualitative and quantitative evaluation, as well as a user study. Statistics show that our method outperforms the state-of-the-art binocular tone mapping frameworks in terms of both visual quality and time performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Aubry, M., Paris, S., Hasinoff, S.W., Kautz, J., Durand, F.: Fast local laplacian filters: theory and applications. ACM Trans. Graph. (TOG) 33(5), 167 (2014)

    Article  Google Scholar 

  2. Banterle, F., Artusi, A., Debattista, K., Chalmers, A.: Advanced High Dynamic Range Imaging. Taylor & Francis, CRC Press (2017)

    Book  Google Scholar 

  3. Curtis, D.W., Rule, S.J.: Binocular processing of brightness information: a vector-sum model. J. Exp. Psychol. Human Percept. Perform. 4(1), 132 (1978)

    Article  Google Scholar 

  4. Daly, S.J.: Visible differences predictor: an algorithm for the assessment of image fidelity. In: Human Vision, Visual Processing, and Digital Display III, vol. 1666, pp. 2–16. International Society for Optics and Photonics (1992). https://doi.org/10.1117/12.135952

  5. De Weert, C.M., Levelt, W.J.M.: Binocular brightness combinations: additive and nonadditive aspects. Percept. Psychophys. 15(3), 551–562 (1974)

    Article  Google Scholar 

  6. Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: ACM SIGGRAPH, pp. 369–378 (1997). https://doi.org/10.1145/258734.258884

  7. Drago, F., Myszkowski, K., Annen, T., Chiba, N.: Adaptive logarithmic mapping for displaying high contrast scenes. In: Computer Graphics Forum, vol. 22, pp. 419–426. Wiley Online Library (2003). https://doi.org/10.1111/1467-8659.00689

  8. Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. In: ACM Transactions on Graphics (TOG), vol. 21, pp. 257–266. ACM (2002). https://doi.org/10.1145/566570.566574

  9. Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R.K., Unger, J.: Hdr image reconstruction from a single exposure using deep cnns. arXiv preprint arXiv:1710.07480 (2017)

  10. Endo, Y., Kanamori, Y., Mitani, J.: Deep reverse tone mapping. ACM Trans. Graph. (TOG) 36(6), 177 (2017)

    Article  Google Scholar 

  11. Engel, G.: The autocorrelation function and binocular brightness mixing. Vis. Res. 9(9), 1111–1130 (1969)

    Article  Google Scholar 

  12. Fairchild, M.D.: The hdr photographic survey. In: Color and Imaging Conference, vol. 2007, pp. 233–238. Society for Imaging Science and Technology (2007)

  13. Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. In: ACM Transactions on Graphics (TOG), ACM, vol. 27, p. 67 (2008). https://doi.org/10.1145/1399504.1360666

  14. Feng, M., Loew, M.H.: Video-level binocular tone-mapping framework based on temporal coherency algorithm. In: 2017 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), IEEE, pp. 1–5 (2017)

  15. Gharbi, M., Chen, J., Barron, J.T., Hasinoff, S.W., Durand, F.: Deep bilateral learning for real-time image enhancement. ACM Trans. Graph. (TOG) 36(4), 118 (2017)

    Article  Google Scholar 

  16. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014). https://papers.nips.cc/paper/5423-generative-adversarial-nets

  17. Hasinoff, S.W., Sharlet, D., Geiss, R., Adams, A., Barron, J.T., Kainz, F., Chen, J., Levoy, M.: Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph. (TOG) 35(6), 192 (2016)

    Article  Google Scholar 

  18. Hau Chua, S., Zhang, H., Hammad, M., Zhao, S., Goyal, S., Singh, K.: Colorbless: augmenting visual information for colorblind people with binocular luster effect. ACM Trans. Comput.-Hum. Interact. (TOCHI) 21(6), 32 (2015)

  19. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. (TOG) 35(4), 110 (2016)

    Article  Google Scholar 

  20. Jähne, B., Haussecker, H., Geissler, P.: Handbook of computer vision and applications, vol. 2. Citeseer (1999)

  21. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) European Conference on Computer Vision, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

  22. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  23. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  24. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, vol. 2, p. 4 (2017)

  25. Legge, G.E.: Binocular contrast summation-II. Quadratic Summ. Vis. Res. 24(4), 385–394 (1984)

    Article  Google Scholar 

  26. Legge, G.E., Rubin, G.S.: Binocular interactions in suprathreshold contrast perception. Percept. Psychophys. 30(1), 49–61 (1981)

    Article  Google Scholar 

  27. Levelt, W.J.: Binocular brightness averaging and contour information. Br. J. Psychol. 56(1), 1–13 (1965)

    Article  Google Scholar 

  28. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

  29. Maehara, G., Goryo, K.: Binocular, monocular and dichoptic pattern masking. Opt. Rev. 12(2), 76–82 (2005)

    Article  Google Scholar 

  30. Meese, T.S., Georgeson, M.A., Baker, D.H.: Interocular masking and summation indicate two stages of divisive contrast gain control. In: Twenty-Eighth European Conference on Visual Perception (2005)

  31. Meese, T.S., Georgeson, M.A., Baker, D.H.: Binocular contrast vision at and above threshold. J. Vis. 6(11), 7–7 (2006)

    Article  Google Scholar 

  32. Meese, T.S., Hess, R.F.: Low spatial frequencies are suppressively masked across spatial scale, orientation, field position, and eye of origin. J. Vis. 4(10), 2–2 (2004)

    Article  Google Scholar 

  33. Nemoto, H., Korshunov, P., Hanhart, P., Ebrahimi, T.: Visual attention in ldr and hdr images. In: 9th International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM), EPFL-CONF-203873 (2015)

  34. Paris, S., Hasinoff, S.W., Kautz, J.: Local laplacian filters: edge-aware image processing with a laplacian pyramid. ACM Trans. Graph. (TOG) 30(4), 1–68 (2011)

    Article  Google Scholar 

  35. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch. In: NIPS-W (2017)

  36. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016). https://doi.org/10.1109/CVPR.2016.278

  37. Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., Myszkowski, K.: High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting. Morgan Kaufmann, Burlington (2010)

    Google Scholar 

  38. Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic tone reproduction for digital images. ACM Trans. Graph. (TOG) 21(3), 267–276 (2002)

    Article  Google Scholar 

  39. Sajjadi, M.S.M., Schölkopf, B., Hirsch, M.: Enhancenet: Single image super-resolution through automated texture synthesis. In: IEEE International Conference on Computer Vision (ICCV) (2017)

  40. Schlick, C.: Quantization techniques for visualization of high dynamic range pictures. In: Photorealistic Rendering Techniques, pp. 7–20. Springer (1995)

  41. Sendik, O., Cohen-Or, D.: Deep correlations for texture synthesis. ACM Trans. Graph. (TOG) 36(5), 161 (2017). https://doi.org/10.1145/3015461

    Article  Google Scholar 

  42. Smith, K., Krawczyk, G., Myszkowski, K., Seidel, H.P.: Beyond tone mapping: enhanced depiction of tone mapped hdr images. In: Computer Graphics Forum, vol. 25, pp. 427–438. Wiley Online Library (2006)

  43. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: AAAI, vol. 4, p. 12 (2017). https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/viewPaper/14806

  44. Tumblin, J., Rushmeier, H.: Tone reproduction for realistic images. IEEE Comput. Graph. Appl. 13(6), 42–48 (1993). https://doi.org/10.1109/38.252554

    Article  Google Scholar 

  45. Tumblin, J., Turk, G.: Lcis: A boundary hierarchy for detail-preserving contrast reduction. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 83–90. ACM Press/Addison-Wesley Publishing Co. (1999)

  46. von Helmholtz, H., Southall, J.P.C.: Treatise on Physiological Optics, vol. 3. Courier Corporation, North Chelmsford (2005)

    Google Scholar 

  47. Wilson, H.R.: Binocular contrast, stereopsis, and rivalry: toward a dynamical synthesis. Vis. Res. 140, 89–95 (2017)

    Article  Google Scholar 

  48. Ward, G.: A contrast-based scalefactor for luminance display. Graph. Gems IV, 415–421 (1994)

    Article  Google Scholar 

  49. Xie, S., Tu, Z.: Holistically-nested edge detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1395–1403 (2015)

  50. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. arXiv preprint arXiv:1611.09969 (2016)

  51. Yang, X., Zhang, L., Wong, T.T., Heng, P.A.: Binocular tone mapping. ACM Trans. Graph. (SIGGRAPH 2012 issue) 31(4), 93:1–93:10 (2012)

  52. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv preprint: arXiv:1511.07122 (2015)

  53. Zhang, L., Zhang, L., Mou, X., Zhang, D., et al.: Fsim: a feature similarity index for image quality assessment. IEEE Trans. Image Process. 20(8), 2378–2386 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  54. Zhang, Z., Hu, X., Liu, X., Wong, T.T.: Binocular tone mapping with improved overall contrast and local details. Comput. Graph. Forum 37(7), 433–442 (2018)

    Article  Google Scholar 

Download references

Acknowledgements

This project is supported by the Research Grants Council of the Hong Kong Special Administrative Region, under RGC General Research Fund (Project No. CUHK 14201017), and Shenzhen Science and Technology Programs (No. JCYJ20160429190300857, No. JCYJ20180507182410327, and No. JCYJ20180507182415428).

Author information

Authors and Affiliations

  1. The Chinese University of Hong Kong, Shatin, NT, Hong Kong SAR, China

    Zhuming Zhang, Chu Han, Xinghong Hu & Tien-Tsin Wong

  2. Guangdong Provincial Key Laboratory of Computer Vision and Virtual Reality Technology, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

    Zhuming Zhang, Chu Han, Xinghong Hu & Tien-Tsin Wong

  3. South China University of Technology, Guangzhou, China

    Shengfeng He

  4. Caritas Institute of Higher Education, Tseung Kwan O, NT, Hong Kong, China

    Xueting Liu

  5. Rokid Corporation Ltd, Hangzhou, China

    Haichao Zhu

Authors
  1. Zhuming Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chu Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shengfeng He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xueting Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haichao Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xinghong Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Tien-Tsin Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tien-Tsin Wong.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Z., Han, C., He, S. et al. Deep binocular tone mapping. Vis Comput 35, 997–1011 (2019). https://doi.org/10.1007/s00371-019-01669-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01669-8

Keywords

Search

Navigation