Skip to main content
SpringerLink
Log in

SFA-GAN: structure–frequency-aware generative adversarial network for underwater image enhancement

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Influenced by light scattering, absorption and water impurities, the quality of underwater image is so poor that significant attention on underwater image enhancement (UIE) has been attracted for producing high-quality visuality. However, both supervised and unsupervised learning-based methods are either more demanding in practical application or easily trap in mapping ambiguity. To solve these issues, we propose a novel unsupervised learning-based underwater image enhancement framework, termed as SFA-GAN. Specifically, we first propose a structure–frequency-aware regularization, which can alleviate both structure and texture loss from spatial and frequency domain, and force SFA-GAN to better enhance underwater images. Afterward, unlike cycle-based underwater image enhancement model with two-sided learning, SFA-GAN adopts one-sided UIE, which is more efficient and faster than the widely used cycle-consistency architectures in training. Extensive experiments on datasets, including: EUVP, UFO-120 and UIEB, demonstrate that the proposed SFA-GAN can achieve state-of-art results on some metrics and produce more clear underwater images without sacrificing model complexity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Availability of data and materials

The data that support the findings of this study are openly available. The codes are available from the corresponding author on reasonable request.

References

  1. Han, J., Shoeiby, M., Malthus, T., Botha, E., Anstee, J., Anwar, S., et al.: Underwater image restoration via contrastive learning and a real-world dataset. arXiv preprint arXiv:210610718.2021

  2. Williams, D.P., Fakiris, E.: Exploiting environmental information for improved underwater target classification in sonar imagery. IEEE Trans. Geosci. Remote Sens. 52(10), 6284–6297 (2014)

    Article  Google Scholar 

  3. Ludeno, G., Capozzoli, L., Rizzo, E., Soldovieri, F., Catapano, I.: A microwave tomography strategy for underwater imaging via ground penetrating radar. Remote Sens. 10(9), 1410 (2018)

    Article  Google Scholar 

  4. Fei, T., Kraus, D., Zoubir, A.M.: Contributions to automatic target recognition systems for underwater mine classification. IEEE Trans. Actions Geosci. Remote Sens. 53(1), 505–518 (2014)

    Article  Google Scholar 

  5. Ancuti, C., Ancuti, C.O., Haber, T., Bekaert, P.: Enhancing underwater images and videos by fusion. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 81–88. IEEE (2012)

  6. Fu, X., Zhuang, P., Huang, Y., Liao, Y., Zhang, X.P., Ding, X.: A retinex-based enhancing approach for single underwater image. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 4572–4576 IEEE (2014)

  7. Zhang, W., Wang, Y., Li, C.: Underwater image enhancement by attenuated color channel correction and detail preserved contrast enhancement. IEEE J. Ocean. Eng. 47(3), 718–735 (2022)

    Article  Google Scholar 

  8. Dong, L., Zhang, W., Xu, W.: Underwater image enhancement via integrated RGB and LAB color models. Signal Process. Image Commun. 104, 116684 (2022)

    Article  Google Scholar 

  9. Wang, Y., Zhang, J., Cao, Y., Wang, Z.: A deep CNN method for underwater image enhancement. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 1382–1386 IEEE (2017)

  10. Li, C., Anwar, S., Porikli, F.: Underwater scene prior inspired deep underwater image and video enhancement. Pattern Recogn. 98, 107038 (2020)

    Article  Google Scholar 

  11. Li, J., Skinner, K.A., Eustice, R.M., JohnsonRoberson, M.: WaterGAN: unsupervised generative network to enable real-time color correction of monocular underwater images. IEEE Robot. Autom. Lett. 3(1), 387–394 (2017)

    Google Scholar 

  12. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired Image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV) (2017)

  13. Park, T., Efros, A.A., Zhang, R., Zhu, J.Y.: Contrastive learning for unpaired image-to-image translation. In: European Conference on Computer Vision, pp. 319–345. Springer (2020)

  14. Li, C., Guo, J., Guo, C.: Emerging from water: underwater image color correction based on weakly supervised color transfer. IEEE Signal Process. Lett. (2017)

  15. Li, C.Y., Cavallaro, A.: Cast-gan: learning to remove colour cast from underwater images. In: 2020 IEEE International Conference on Image Processing (ICIP) (2020)

  16. Xu, Z.Q.J., Zhang, Y., Luo, T., Xiao, Y., Ma, Z.: Frequency principle: fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:190106523.2019

  17. Petit, F., Capelle-Laizé, A.S., Carré, P.: Underwater image enhancement by attenuation inversion with quaternions. In: 2009 IEEE International Conference on Acoustics, Speech and Signal Processing,pp. 1177–1180. IEEE (2009)

  18. Hitam, M.S., Awalludin, E.A., Yussof, W.N.J.H.W., Bachok, Z.: Mixture contrast limited adaptive histogram equalization for underwater image enhancement. In: 2013 International Conference on Computer Applications Technology (ICCAT), pp. 1–5. IEEE (2013)

  19. Ancuti, C.O., Ancuti, C., De Vleeschouwer, C., Bekaert, P.: Color balance and fusion for underwater image enhancement. IEEE Trans. Image Process. 27(1), 379–393 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  20. Perez, J., Attanasio, A.C., Nechyporenko, N., Sanz, P.J.: A deep learning approach for underwater image enhancement. In: International Work-Conference on the Interplay Between Natural and Artificial Computation, pp. 183–192. Springer (2017)

  21. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Imageto-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

  22. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

  23. Fabbri, C., Islam, M.J., Sattar, J.: Enhancing underwater imagery using generative adversarial networks. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7159–7165. IEEE (2018)

  24. Naik, A., Swarnakar, A., Mittal, K.: Shallow UWnet: compressed model for underwater image enhancement. arXiv preprint arXiv:210102073.2021

  25. Chen, Y., Li, G., Jin, C., Liu, S., Li, T.: SSDGAN: measuring the realness in the spatial and spectral domains. arXiv preprint arXiv:201205535.2020

  26. Kim, N., Jang, D., Lee, S., Kim, B., Kim, D.S.: Unsupervised image denoising with frequency domain knowledge. arXiv preprint arXiv:211114362.2021

  27. Zheng, C., Cham, T.J., Cai, J.: The spatially correlative loss for various image translation tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16407–16417 (2021)

  28. Han, J., Shoeiby, M., Malthus, T., Botha, E., Anstee, J., Anwar, S., et al.: Single underwater image restoration by contrastive learning. In: 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pp. 2385–2388. IEEE (2021)

  29. Wang, X., Girshick, R., Gupta, A., He, K.: Nonlocal neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)

  30. Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. In: International Conference on Machine Learning, pp. 7354–7363. PMLR (2019)

  31. Cai, M., Zhang, H., Huang, H., Geng, Q., Li, Y., Huang, G.: Frequency domain image translation: more photo-realistic, better identity-preserving. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13930–13940 (2021)

  32. Durall, R., Keuper, M., Keuper, J.: Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7890–7899 (2020)

  33. Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2794–2802 (2017)

  34. Wang, X., Yu, J.: Learning to cartoonize using white-box cartoon representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8090–8099 (2020)

  35. Wang, P., Li, Y., Vasconcelos, N.: Rethinking and improving the robustness of image style transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 124–133 (2021)

  36. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556.2014

  37. Du, W., Chen, H., Yang, H.: Learning invariant representation for unsupervised image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 14483–14492 (2020)

  38. Bosse, S., Maniry, D., Müller, K.R., Wiegand, T., Samek, W.: Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans. Image Process. 27(1), 206–219 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  39. Yang, M., Sowmya, A.: An underwater color image quality evaluation metric. IEEE Trans. Image Process. 24(12), 6062–6071 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  40. Panetta, K., Gao, C., Agaian, S.: Human-visual system-inspired underwater image quality measures. IEEE J. Ocean. Eng. 41(3), 541–551 (2015)

    Article  Google Scholar 

  41. Huang, P., Wu, J., Porikli, F., Li, C.: Underwater image enhancement with hyper-Laplacian reflectance priors. IEEE Trans. Image Proces. 31, 5442–5455 (2022)

    Article  Google Scholar 

  42. Zhang, W., Zhuang, P., Sun, H.H., Li, G., Kwong, S., Li, C.: Underwater image enhancement via minimal color loss and locally adaptive contrast enhancement. IEEE Trans. Image Process. 31, 3997–4010 (2022)

    Article  Google Scholar 

Download references

Funding

This work is supported by the National Natural Science Foundation of China (Grant No. 62177006, 62077009), and partially supported by the Guangdong Provincial Natural Science Foundation (Grant No. 2214050002868), Guangdong Provincial Department of Education characteristic innovation project (Grant No. 2021KTSCX362), Department of Natural Resources of Guangdong Province (Grant No. GDNRC[2023]47) and Zhuhai Campus of Beijing Normal University ISC.

Author information

Authors and Affiliations

  1. College of Education for the Future, Beijing Normal University, No. 18 Jinfeng Road, Zhuhai, 519087, Guangdong, China

    Yinghui Zhang

  2. School of Statistics, Beijing Normal University, No.18 Jinfeng Road, Zhuhai, 519087, Guangdong, China

    Tingshuai Liu

  3. School of System Science, Beijing Normal University, No.17 Xinjiekouwai Road, Beijing, 100875, China

    Bo Zhao

  4. School of Artificial Intelligence, Beijing Normal University, No.17 Xinjiekouwai Road, Beijing, 100875, China

    Fengxiang Ge

Authors
  1. Yinghui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tingshuai Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bo Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fengxiang Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

YZ provided the idea of this paper and wrote the whole content of this paper. TL conducted data collection, collation and analysis, literature research and organization. BZ provided some suggestions for data analysis. FG supervised the experimental process. All authors reviewed the final manuscript.

Corresponding authors

Correspondence to Yinghui Zhang or Fengxiang Ge.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (DOCX 3025 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Liu, T., Zhao, B. et al. SFA-GAN: structure–frequency-aware generative adversarial network for underwater image enhancement. SIViP 17, 3647–3655 (2023). https://doi.org/10.1007/s11760-023-02591-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02591-2

Keywords

Search

Navigation