Skip to main content
SpringerLink
Log in

A two-stage and two-branch generative adversarial network-based underwater image enhancement

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Due to the absorption and scattering of light under the water, various distortions are formed in the underwater images, which seriously limit the underwater visual task. To address this problem, a two-stage and two-branch underwater image enhancement generative adversarial network (TTE-GAN) is proposed. Specifically, considering the characteristics of the underwater images in different frequency domains in the first stage of the generator, guided filtering is used to decompose the raw underwater images, and two branches of low-frequency and high-frequency images are obtained. Since the low-frequency information represents the comprehensive measurement of the intensity of the image, the multi-scale module is used to enhance the comprehensive measurement of the image intensity. Considering that high frequency is a measure of the edge information of an image, a convolutional neural network is constructed for high-frequency information. Furthermore, due to the complexity of the underwater images, a single-step network is challenging to achieve good results. A refinement network is designed to improve the quality of the underwater image, thereby obtaining high-quality underwater images in the second stage of the generator. Finally, comprehensive experiments on three benchmarks demonstrate the validity of the proposed TTE-GAN method both subjectively and objectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Armanini, C., Farman, M., Calisti, M., Giorgio-Serchi, F., Stefanini, C., Renda, F.: Flagellate underwater robotics at macroscale: design, modeling, and characterization. IEEE Tran. Robot. 38(2), 731–747 (2022). https://doi.org/10.1109/TRO.2021.3094051

    Article  Google Scholar 

  2. Pan, X., Shen, Y., Zhang, J.: IoUT based underwater target localization in the presence of time synchronization attacks. IEEE Trans Wirel. Commun. 20(6), 3958–3973 (2021). https://doi.org/10.1109/TWC.2021.3054745

    Article  Google Scholar 

  3. Li, C., Anwar, S., Hou, J., Cong, R., Guo, C., Ren, W.: Underwater image enhancement via medium transmission-guided multi-color space embedding. IEEE Trans. Image. Process. 30, 4985–5000 (2021). https://doi.org/10.1109/TIP.2021.3076367

    Article  Google Scholar 

  4. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013). https://doi.org/10.1109/TPAMI.2012.213

    Article  Google Scholar 

  5. Hummel, R.: Image enhancement by histogram transformation. Comput. Graphics Image Process. 6(2), 184–195 (1977)

    Article  Google Scholar 

  6. S. M. Pizer, R. E. Johnston, J. P. Ericksen, B. C. Yankaskas, and K. E. Müller: Contrast-limited adaptive histogram equalization: speed and effectiveness. In Proc. 1st Conf. Vis. Biomed. Comput., 337–345, May (1990).

  7. Deng, G.: A generalized unsharp masking algorithm. IEEE Trans. Image Process 20(5), 1249–1261 (2011). https://doi.org/10.1109/TIP.2010.2092441

    Article  MathSciNet  MATH  Google Scholar 

  8. Buchsbaum, G.: A spatial processor model for object colour perception. J. Franklin Inst. 310(1), 1–26 (1980). https://doi.org/10.1016/0016-0032(80)90058-7

    Article  Google Scholar 

  9. Liu, Y.-C., Chan, W.-H., Chen, Y.-Q.: Automatic white balance for digital still camera. IEEE Trans. Consum. Electron. 41(3), 460–466 (1995). https://doi.org/10.1109/30.4680455

    Article  Google Scholar 

  10. C. Ancuti, C. O. Ancuti, T. Haber, and P. Bekaert: Enhancing underwater images and videos by fusion. in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 81–88, Jun. 2012. https://doi.org/10.1109/CVPR.2012.6247661

  11. X. Fu, P. Zhuang, Y. Huang, Y. Liao, X.-P. Zhang, and X. Ding: A retinex-based enhancing approach for single underwater image. in Proc. IEEE Int. Conf. Image Process. (ICIP), 4572–4576, Oct. 2014. https://doi.org/10.1109/ICIP.2014.7025927.

  12. Zhang, S., et al.: Underwater image enhancement via extended multi-scale retinex. Neurocomputing 245(5), 1–9 (2017). https://doi.org/10.1016/j.neucom.2017.03.029

    Article  Google Scholar 

  13. Liu, S., Zhang, Y.: Detail-preserving underexposed image enhancement via optimal weighted multi-exposure fusion. IEEE Trans. Consum. Electron. 65(3), 303–311 (2019). https://doi.org/10.1109/TCE.2019.2893644

    Article  Google Scholar 

  14. Liu, K., Liang, Y.: Underwater image enhancement method based on adaptive attenuation-curve prior. Optics Eexpress 29(7), 0321–10345 (2021). https://doi.org/10.1364/OE.413164

    Article  Google Scholar 

  15. G. Singh, N. Jaggi, S. Vasamsetti, H. K. Sardana, S. Kumar and N. Mittal: Underwater image/video enhancement using wavelet based color correction (WBCC) method. IEEE Underwater Technology (UT), 1–5, (2015). https://doi.org/10.1109/UT.2015.7108303

  16. Prabhakar, C.J., Kumar, P.U.: An image based technique for enhancement of underwater images. Int. J. Mach. Intell. 4, 217–224 (2012)

    Google Scholar 

  17. Drews, P., Nascimento, E., Botelho, S., Montenegro Campos, M.F.: Underwater depth estimation and image restoration based on single images. IEEE Comput. Graph. Appl. 36(2), 24–35 (2016). https://doi.org/10.1109/MCG.2016.26

    Article  Google Scholar 

  18. Peng, Y., Cosman, P.: Underwater image restoration based on image blurriness and light absorption. IEEE Trans. Image Process. 26(4), 1579–1594 (2017). https://doi.org/10.1109/TIP.2017.2663846

    Article  MathSciNet  MATH  Google Scholar 

  19. Galdran, A., et al.: Automatic red-channel underwater image restoration. J. Vis. Commun. Image Representation 26, 132–145 (2015). https://doi.org/10.1016/j.jvcir.2014.11.006

    Article  Google Scholar 

  20. Mingye, J., Zhang, D., Wang, X.: Single image dehazing via an improved atmospheric scattering model. Vis. Comput. 33(12), 1613–1625 (2017). https://doi.org/10.1007/s00371-016-1305-1

    Article  Google Scholar 

  21. Zhang, S., He, F., Ren, W., Yao, J.: Joint learning of image detail and transmission map for single image dehazing. Vis. Comput. 36(2), 305–316 (2020). https://doi.org/10.1007/s00371-018-1612-9

    Article  Google Scholar 

  22. Berman, D., Levy, D., Avidan, S., Treibitz, T.: Underwater single image color restoration using haze-lines and a new quantitative dataset. IEEE Trans. Pattern Anal. Mach. Intell. 43(8), 2822–2837 (2021). https://doi.org/10.1109/TPAMI.2020.2977624

    Article  Google Scholar 

  23. Perez, J., Attanasio, A.C., Nechyporenko, N., Sanz, P.J.: A deep learning approach for underwater image enhancement. Biomed. Appl. Based on Nat. Artificial Computing 10338, 183–192 (2017). https://doi.org/10.1007/978-3-319-59773-7_19

    Article  Google Scholar 

  24. Y. Wang, J. Zhang, Y. Cao, and Z. Wang: A deep CNN method for underwater image enhancement. in Proc. IEEE Int. Conf. Image Process. (ICIP), Beijing, China, 1382–1386, Sep. (2017). https://doi.org/10.1109/ICIP.2017.8296508.

  25. Li, C., Anwar, S., Porikli, F.: Underwater scene prior inspired deep underwater image and video enhancement. Pattern Recognit. 98, 107038–107049 (2020). https://doi.org/10.1016/j.patcog.2019.107038

    Article  Google Scholar 

  26. I. Goodfellow et al.: Generative adversarial nets. in Proc. Adv. Neural Info. Process. Syst., Montreal, Canada, 2672–2680, Dec. (2014).

  27. Li, J., Skinner, K.A., Eustice, R.M., Johnson-Roberson, M.: WaterGAN: unsupervised generative network to enable real-time color correction of monocular underwater images. IEEE Robot. Autom. Lett. 3(1), 387–394 (2018). https://doi.org/10.1109/LRA.2017.2730363

    Article  Google Scholar 

  28. C. Fabbri, M. J. Islam, and J. Sattar: Enhancing underwater imagery using generative adversarial networks. in Proc. IEEE Int. Conf. Robot. Autom. 7159–7165, (2018). https://doi.org/10.1109/ICRA.2018.8460552.

  29. J.-Y. Zhu, T. Park, P. Isola, and A.-A. Efros: Unpaired image-to-image translation using cycle-consistent adversarial networks. in Proc. Int. Conf. Comput. Vis., Venice, Italy, 2242–2251, Oct. (2017).

  30. Li, C., Guo, J., Guo, C.: Emerging from water: underwater image color correction based on weakly supervised color transfer. IEEE Signal Process. Lett. 25(3), 323–327 (2018). https://doi.org/10.1109/LSP.2018.2792050

    Article  Google Scholar 

  31. Chen, X., Yu, J., Kong, S., Wu, Z., Fang, X., Wen, L.: Towards real-time advancement of underwater visual quality with GAN. IEEE Trans. Ind. Electron. 66(12), 9350–9359 (2019). https://doi.org/10.1109/TIE.2019.2893840

    Article  Google Scholar 

  32. Guo, Y., Li, H., Zhuang, P.: Underwater image enhancement using a multiscale dense generative adversarial network. IEEE J. Ocean. Eng. 45(3), 862–870 (2020). https://doi.org/10.1109/JOE.2019.2911447

    Article  Google Scholar 

  33. Y. Zhou and K. Yan.: Domain adaptive adversarial learning based on physics model feedback for underwater image enhancement. (2020). https://arxiv.org/abs/2002.09315.

  34. Li, H., Zhuang, P.: DewaterNet: a fusion adversarial real underwater image enhancement network. Signal Process.: Image Commun. 95, 116248 (2021). https://doi.org/10.1016/j.image.2021.116248

    Article  Google Scholar 

  35. P. Isola, J.-Y. Zhu, T. Zhou, and A.-A. Efros: Image-to-image translation with conditional adversarial networks. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Hawaii, USA, 5967–5976, Jul. (2017). https://arxiv.org/abs/1611.07004.

  36. Ronneberger, P. Fischer, and T. Brox: U-Net: Convolutional networks for biomedical image segmentation. in Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York, NY, USA: Springer, 234–241, (2015). https://doi.org/10.1007/978-3-319-24574-4_28

  37. I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville: Improved training of Wasserstein GANs. in Proc. Adv. Neural Inf. Process. Syst., 5767–5777, (2017). https://arxiv.org/abs/1704.00028

  38. X. Mao, Q. Li, H. Xie, R.-Y. Lau, Z. Wang, and S.-P. Smolley: Least squares generative adversarial networks. in Proc. Int. Conf. Comput. Vis., Venice, Italy, 2794–2802, Oct. (2017). https://arxiv.org/abs/1611.04076

  39. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei: ImageNet: A large-scale hierarchical image database. in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 248–255, Jun. (2009). https://doi.org/10.1109/CVPR.2009.5206848

  40. K. Simonyan and A. Zisserman: Very deep convolutional networks for large-scale image recognition. in Proc. Int. Conf. Mach. Learn. (ICML), (2015). https://arxiv.org/abs/1409.1556

  41. H. Zhao et al.: Loss functions for neural networks for image processing. (2015). https://arxiv.org/abs/1511.08861

  42. D. Kingma and J. Ba: Adam: A method for stochastic optimization. (2014). https://arxiv.org/abs/1412.6980

  43. Peng, Lintao, Chunli Zhu, and Liheng Bian (2021): U-shape Transformer for Underwater Image Enhancement. arXiv preprint arXiv:2111 11843. https://doi.org/10.48550/arXiv.2111.11843.

  44. H. Li, J. Li, and W. Wang: A fusion adversarial underwater image enhancement network with a public test dataset. (2019). https://arxiv.org/abs/1906.06819v1.

  45. Li, C., et al.: An underwater image enhancement benchmark dataset and beyond. IEEE Trans. Image Process. 29, 4376–4389 (2020). https://doi.org/10.1109/TIP.2019.2955241

    Article  MATH  Google Scholar 

  46. Yang, M., Sowmya, A.: An underwater color image quality evaluation metric. IEEE Trans. Image Process. 24(12), 6062–6071 (2015). https://doi.org/10.1109/TIP.2015.2491020

    Article  MathSciNet  MATH  Google Scholar 

  47. Panetta, K., Gao, C., Agaian, S.: Human-visual-system-inspired underwater image quality measures. IEEE J. Ocean. Eng. 41(3), 541–551 (2016). https://doi.org/10.1109/JOE.2015.2469915

    Article  Google Scholar 

  48. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell 8(6), 679–698 (1986). https://doi.org/10.1109/TPAMI.1986.4767851

    Article  Google Scholar 

  49. Peng, H., Li, B., Ling, H., Hu, W., Xiong, W., Maybank, S.J.: Salient object detection via structured matrix decomposition. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 818–832 (2017). https://doi.org/10.1109/TPAMI.2016.2562626

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Natural Science Foundation of China under Grant No. 62171243, 61971247, and 61501270, Zhejiang Provincial Natural Science Foundation of China under Grant No. LY22F020020, Education of Zhejiang Province under Grant No. Y201839115, Natural Science Foundation of Ningbo under Grant No.2021J134. It was also sponsored by the K. C. Wong Magna Fund at Ningbo University.

Author information

Authors and Affiliations

  1. School of Mathematics and Statistics, Ningbo University, Ningbo, 315211, Zhejiang, China

    Yong Lai, Haiyong Xu & Lihong Wang

  2. College of Science and Technology, Ningbo University, Ningbo, 315212, China

    Chi Lin & Ting Luo

Authors
  1. Yong Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haiyong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chi Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ting Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lihong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chi Lin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lai, Y., Xu, H., Lin, C. et al. A two-stage and two-branch generative adversarial network-based underwater image enhancement. Vis Comput 39, 4133–4147 (2023). https://doi.org/10.1007/s00371-022-02580-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02580-5

Keywords

Search

Navigation