Skip to main content
SpringerLink
Log in

Unsupervised learning based dual-branch fusion low-light image enhancement

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Distortion-free enhancement on images captured under low-light conditions has always been a challenging problem in computer vision. Although many image enhancement methods have been proposed, most existing algorithms cause artifacts and amplify the noise in the enhanced image, which greatly affect the visual perception of the enhanced image. To address these problems, this paper proposes an unsupervised learning based dual-branch fusion low-light image enhancement algorithm, which can learn the map way of low-light images to normal-light images from unpaired low-light and normal-light datasets. The network is consisted of dual branches, the upper branch is a refinement branch focusing on noise suppression, and the lower branch is a U-Net-like global reconstruction branch based on the attention mechanism for high-quality image generation. The discrimination network adopts the multi-scale discrimination structure of feature pyramid to enhance the global consistency and avoid local overexposure. The loss function is also improved, and a new fidelity cycle consistency loss is introduced to further improve the quality of image texture information recovery. Qualitative and quantitative experimental results show that the proposed method can effectively suppress the generation of artifacts and noise amplification of enhanced images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability

All data generated during this study are included in these published articles:

1) LOL https://doi.org/10.48550/arXiv.1808.04560

2) Brightening https://dl.acm.org/doi/abs/10.1145/2713168.2713194

3) MEF https://ieeexplore.ieee.org/document/7120119/keywords#keywords

4) LIME https://pubmed.ncbi.nlm.nih.gov/28113318/

5) NPE https://ieeexplore.ieee.org/document/6512558

6) DICM https://ieeexplore.ieee.org/document/6615961

References

  1. Abdullah-Al-Wadud M, Kabir MH, Dewan MAA, Chae O (2007) A dynamic histogram equalization for image contrast enhancement. IEEE Trans Consum Electron 53(2):593–600

    Article  Google Scholar 

  2. Bochkovskiy A, Wang C-Y, Liao H-YM (2020) Yolov4: Optimal speed and accuracy of object detection. arXiv:2004.10934

  3. Chaudhary S, Bhardwaj A, Rana P (2022) Image enhancement by linear regression algorithm and sub-histogram equalization. Multimedia Tools Appl

  4. Chen C, Chen Q, Xu J, Koltun V (2018) Learning to see in the dark. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3291–3300

  5. Chen X, Li J, Hua Z (2022) Retinex low-light image enhancement network based on attention mechanism. Multimedia Tools Appl

  6. Dang-Nguyen D. -T., Pasquini C, Conotter V, Boato G (2015) Raise: a raw images dataset for digital image forensics. In: Proceedings of the 6th ACM multimedia systems conference, pp 219–224

  7. Drago F, Myszkowski K, Annen T, Chiba N, Fellner DW (2003) Adaptive logarithmic mapping for displaying high contrast scenes. In: Computer graphics forum

  8. Engin D, Genç A, Kemal Ekenel H (2018) Cycle-dehaze: enhanced cyclegan for single image dehazing. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 825–833

  9. Fang Y, Zhang C, Yang W, Liu J, Guo Z (2018) Blind visual quality assessment for image super-resolution by convolutional neural network. Multimed Tools Appl 77(22):29829–29846

    Article  Google Scholar 

  10. Fang Y, Zhu H, Zeng Y, Ma K, Wang Z (2020) Perceptual quality assessment of smartphone photography. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  11. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv Neural Inf Process Syst:27

  12. Guo C, Li C, Guo J, Loy CC, Hou J, Kwong S, Cong R (2020) Zero-reference deep curve estimation for low-light image enhancement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 1780–1789

  13. Guo X, Li Y, Ling H (2016) Lime: Low-light image enhancement via illumination map estimation. IEEE Trans Image Process 26(2):982–993

    Article  MathSciNet  MATH  Google Scholar 

  14. Hao S, Zhuang F, Guo Y (2018) Low-light image enhancement with a refined illumination map. Multimed Tools Appl 77(22):29639–29650

    Article  Google Scholar 

  15. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7132–7141

  16. Isola P, Zhu J. -Y., Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  17. Jebadass JR, Balasubramaniam P (2022) Low light enhancement algorithm for color images using intuitionistic fuzzy sets with histogram equalization. Multimed Tools Appl 81(6):8093–8106

    Article  Google Scholar 

  18. Jiang Y, Gong X, Liu D, Cheng Y, Fang C, Shen X, Yang J, Zhou P, Wang Z (2021) Enlightengan: deep light enhancement without paired supervision. IEEE Trans Image Process 30:2340–2349

    Article  Google Scholar 

  19. Jin S, Qi N, Zhu Q, Ouyang H (2022) Progressive gan-based transfer network for low-light image enhancement. In: International conference on multimedia modeling, pp 292–304

  20. Kim TH, Lee KM, Schölkopf B., Hirsch M (2017) Online video deblurring via dynamic temporal blending network. In: 2017 IEEE international conference on computer vision (ICCV), pp 4058–4067

  21. Kimmel R, Elad M, Shaked D, Keshet R, Sobel I (2003) A variational framework for retinex. Int J Comput Vision 52(1):7–23

    Article  MATH  Google Scholar 

  22. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv:1412.6980

  23. Land EH (1977) The retinex theory of color vision. Scientific american 237(6):108–129

    Article  MathSciNet  Google Scholar 

  24. Lee C, Lee C, Kim C-S (2013) Contrast enhancement based on layered difference representation of 2d histograms. IEEE Trans Image Process 22 (12):5372–5384

    Article  Google Scholar 

  25. Li C, Guo C, Chen CL (2021) Learning to enhance low-light image via zero-reference deep curve estimation. IEEE Trans Software Eng

  26. Li C, Guo C, Chen CL (2021) Learning to enhance low-light image via zero-reference deep curve estimation. IEEE Trans Patt Anal Mach Intell

  27. Li C, Guo J, Porikli F, Pang Y (2018) Lightennet: a convolutional neural network for weakly illuminated image enhancement. Pattern Recognit Lett 104:15–22

    Article  Google Scholar 

  28. Lin T-Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2117–2125

  29. Liu RW, Guo Y, Lu Y, Chui KT, Gupta BB (2022) Deep network-enabled haze visibility enhancement for visual iot-driven intelligent transportation systems. IEEE Trans Industr Inf:1–1

  30. Liu X, Ma W, Ma X, Wang J (2022) Lae-net: A locally-adaptive embedding network for low-light image enhancement. Pattern Recognit:109039

  31. Liu R, Ma L, Zhang J, Fan X, Luo Z (2021) Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. In: Computer vision and pattern recognition

  32. Loh YP, Chan CS (2019) Getting to know low-light images with the exclusively dark dataset. Comput Vis Image Underst 178:30–42

    Article  Google Scholar 

  33. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3431–3440

  34. Lore KG, Akintayo A, Sarkar S (2017) Llnet: a deep autoencoder approach to natural low-light image enhancement. Pattern Recogn 61:650–662

    Article  Google Scholar 

  35. Lv F, Lu F, Wu J, Lim C (2018) Mbllen: low-light image/video enhancement using cnns. In: BMVC, vol 220, p 4

  36. Ma K, Zeng K, Wang Z (2015) Perceptual quality assessment for multi-exposure image fusion. IEEE Trans Image Process 24(11):3345–3356

    Article  MathSciNet  MATH  Google Scholar 

  37. Meng Y, Kong D, Zhu Z, Zhao Y (2019) From night to day: gans based low quality image enhancement. Neural Process Lett 50(1):799–814

    Article  Google Scholar 

  38. Meylan L, Susstrunk S (2006) High dynamic range image rendering with a retinex-based adaptive filter. IEEE Trans Image Process 15(9):2820–2830

    Article  Google Scholar 

  39. Mirza M, Osindero S (2014) Conditional generative adversarial nets. Comput Sci:2672–2680

  40. Ni Z, Yang W, Wang S, Ma L, Kwong S (2020) Towards unsupervised deep image enhancement with generative adversarial network. IEEE Trans Image Process 29:9140–9151

    Article  MATH  Google Scholar 

  41. Pizer SM, Amburn EP, Austin JD, Cromartie R, Geselowitz A, Greer T, ter Haar Romeny B, Zimmerman JB, Zuiderveld K (1987) Adaptive histogram equalization and its variations. Comput Vision, Graph, Image Process 39 (3):355–368

    Article  Google Scholar 

  42. Rongkai Z, Lanqing G, Siyu H, Bihan W (2021) Rellie: Deep reinforcement learning for customized low-light image enhancement. In: Inproceedings of the 29th ACM international conference on multimedia, pp 2429–2437

  43. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234–241

  44. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Comput Sci

  45. Sun X, Li M, He T, Fan L (2021) Enhance images as you like with unpaired learning. CoRR

  46. Thomas G, Flores-Tapia D, Pistorius S (2011) Histogram specification: a fast and flexible method to process digital images. IEEE Trans Instrum Meas 60(5):1565–1578

    Article  Google Scholar 

  47. Wang P, Wang Z, Lv D, Zhang C, Wang Y (2021) Low illumination color image enhancement based on gabor filtering and retinex theory. Multimed Tools Appl:1–15

  48. Wang S, Zheng J, Hu H-M, Li B (2013) Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Trans Image Process 22(9):3538–3548

    Article  Google Scholar 

  49. Wei C, Wang W, Yang W, Liu J (2018) Deep retinex decomposition for low-light enhancement. arXiv:1808.04560

  50. Yan L, Fu J, Wang C, Ye Z, Chen H, Ling H (2021) Enhanced network optimized generative adversarial network for image enhancement. Multimed Tools Appl:1–19

  51. Yang W, Wang S, Fang Y, Wang Y, Liu J (2020) From fidelity to perceptual quality: a semi-supervised approach for low-light image enhancement. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3063–3072

  52. Zhang R, Isola P, Efros AA, Shechtman E, Wang O (2018) The unreasonable effectiveness of deep features as a perceptual metric. IEEE/CVF Conf Comput Vision Patt Recognit

  53. Zhao M, Zhong S, Fu X, Tang B, Pecht M (2020) Deep residual shrinkage networks for fault diagnosis. IEEE Trans Indust Inf

  54. Zheng C, Shi D, Shi W (2021) Adaptive unfolding total variation network for low-light image enhancement. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 4439–4448

  55. Zhi N, Mao S, Li M (2018) An enhancement algorithm for coal mine low illumination images based on bi-gamma function. Liaoning Gongcheng Jishu Daxue Xuebao (Ziran Kexue Ban)/J Liaoning Techn Univ (Natural Sci Edition) 37(1):191–197

    Google Scholar 

  56. Zhu M, Pan P, Chen W, Yang Y (2020) Eemefn: low-light image enhancement via edge-enhanced multi-exposure fusion network. Proc AAAI Conf Artif Intell 34(7):13106–13113

    Google Scholar 

  57. Zhu J. -Y., Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232

Download references

Acknowledgements

This work was supported by the Natural Science Foundation of China NSFC under Grants 61871445, 61302156; Key R & D Foundation Project of Jiangsu province under Grant BE2016001-4.

Author information

Author notes

    Authors and Affiliations

    1. Engineering Research Center of Wideband Wireless Communication Technology, Ministry of Education, Nanjing University of Posts and Telecommunications, Nanjing, China

      Guang Han, Yu Zhou & Fanyu Zeng

    Authors
    1. Guang Han
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Yu Zhou
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Fanyu Zeng
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Guang Han.

    Ethics declarations

    Conflict of Interests

    We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled, “Unsupervised Learning based Dual-branch Fusion Low-light Image Enhancement”.

    Additional information

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Yu Zhou contributed equally to this work.

    Rights and permissions

    Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Han, G., Zhou, Y. & Zeng, F. Unsupervised learning based dual-branch fusion low-light image enhancement. Multimed Tools Appl 82, 37593–37614 (2023). https://doi.org/10.1007/s11042-023-15147-w

    Download citation

    • Received:

    • Revised:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s11042-023-15147-w

    Keywords

    Search

    Navigation