Skip to main content
SpringerLink
Log in

VAE-CoGAN: Unpaired image-to-image translation for low-level vision

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Low-level vision problems, such as single image haze removal and single image rain removal, usually restore a clear image from an input image using a paired dataset. However, for many problems, the paired training dataset will not be available. In this paper, we propose an unpaired image-to-image translation method based on coupled generative adversarial networks (CoGAN) called VAE-CoGAN to solve this problem. Different from the basic CoGAN, we propose a shared-latent space and variational autoencoder (VAE) in framework. We use synthetic datasets and the real-world images to evaluate our method. The extensive evaluation and comparison results show that the proposed method can be effectively applied to numerous low-level vision tasks with favorable performance against the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Ren, W., Si, L., Hua, Z., Pan, J., Yang, M.H.: Single image dehazing via multi-scale convolutional neural networks. Springer, Cham (2016)

    Book  Google Scholar 

  2. W. Yang, R. T. Tan, J. Feng, J. Liu, and S. Yan. Deep joint rain detection and removal from a single image. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017

  3. Mehta, A., Sinha, H., Narang, P., Mandal, M.: Hidegan: A hyperspectral-guided image dehazing gan. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020

  4. Jin, X., Chen, Z., Li, W.: Ai-gan: Asynchronous interactive generative adversarial network for single image rain removal. Pattern Recogn. 100, 107143 (2019)

    Article  Google Scholar 

  5. Larsen, Abl., Snderby, Sren Kaae., Larochelle, H., Winther, O.: Autoencoding beyond pixels using a learned similarity metric. JMLR.org, 2015

  6. Bao, J., Dong, C., Fang, W., Li, H., Gang, H.: Cvae-gan: Fine-grained image generation through asymmetric training. In IEEE International Conference on Computer Vision, 2017

  7. Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Bing, X., Bengio, Y.: Generative adversarial nets. MIT Press, 2014

  8. Diederik, P.: Kingma and Max. Welling. Auto-encoding variational bayes. stat 1050, 1 (2014)

    Google Scholar 

  9. Larochelle, H., Murray, I.: The neural autoregressive distribution estimator. J. Mach. Learn. Res. 15, 29–37 (2011)

    Google Scholar 

  10. Anders Boesen Lindbo. Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole. Winther: Autoencoding beyond pixels using a learned similarity metric. In International conference on machine learning, pp. 1558–1566, 2016

  11. D. Engin, Anl. Gen, and Hazm Kemal. Ekenel. Cycle-dehaze: Enhanced cyclegan for single image dehazing. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2018

  12. Ancuti, C., Ancuti, C. O., Vleeschouwer, C. D., Bovik, AC.: Night-time dehazing by fusion. In IEEE International Conference on Image Processing, 2016

  13. He, K., Jian, S., Fellow, I.E.E.E., Tang, X.: Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2011)

    Article  Google Scholar 

  14. Zhu, Q., Mai, J., Shao, L.: A fast single image haze removal algorithm using color attenuation prior. IEEE Trans. Image Process. 24(11), 3522–3533 (2015)

    Article  MATH  MathSciNet  Google Scholar 

  15. Fattal, R.: Single image dehazing. Acm Trans. Graphics 27(3), 1–9 (2008)

    Article  Google Scholar 

  16. Tan, R. T.: Visibility in bad weather from a single image. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 24-26 June 2008, Anchorage, Alaska, USA, 2008

  17. Cai, B., Xu, X., Jia, K., Qing, C., Tao, D.: Dehazenet: An end-to-end system for single image haze removal. IEEE Trans. Image Process. 25(11), 5187–5198 (2016)

    Article  MATH  MathSciNet  Google Scholar 

  18. Zhang, H., Sindagi, Vishwanath., Patel, Vishal M.: Joint transmission map estimation and dehazing using deep networks. IEEE Transactions on Circuits and Systems for Video Technology, 30(7):1975–1986, 2019

  19. Swami, K., Das, S. K.: Candy: Conditional adversarial networks based fully end-to-end system for single image haze removal. In 2018 24th International Conference on Pattern Recognition (2018)

  20. Li, B., Ren, W., Fu, D., Tao, D., Feng, D., Zeng, W., Wang, Z.: Benchmarking single image dehazing and beyond. IEEE Trans. Image Process. 28(1), 492–505 (2018)

    Article  MATH  MathSciNet  Google Scholar 

  21. Zhao, J., Zhang, J., Li, Z., Hwang, J. N., Gao, Y., Fang, Z., Jiang, X., Huang, B.: Dd-cyclegan: Unpaired image dehazing via double-discriminator cycle-consistent generative adversarial network. Engineering Applications of Artificial Intelligence, 82:263–271, 2019

  22. Li, B., Peng, X., Wang, Z., Xu, J., Dan, F.: Aod-net: All-in-one dehazing network. In 2017 IEEE International Conference on Computer Vision, 2017

  23. Liu, M. Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. Advances in neural information processing systems, 30, (2017)

  24. Li, C., Guo, C., Guo, J., Han, P., Fu, H., Cong, R.: Pdr-net: Perception-inspired single image dehazing network with refinement. IEEE Trans. Multimedia 22(3), 704–716 (2020)

    Article  Google Scholar 

  25. Dong, H., Pan, J., Xiang, L., Hu, Z., Zhang, X., Wang, F., Yang, M.: Multi-scale boosted dehazing network with dense feature fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2157–2167, 2020

  26. Liang, Q., Zhu, B., Ngo, C. W.: Pyramid fusion dark channel prior for single image dehazing. arXiv preprint arXiv:2105.10192, 2021

  27. Cho, Y., Malav, R., Pandey, G., Kim, A.: Dehazegan: Simultaneous hazing and dehazing networks using unpaired image-to-image translation. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017

  28. Barnum, P.C., Narasimhan, S., Kanade, T.: Analysis of rain and snow in frequency space. Int. J. Comput. Vision 86(2–3), 256 (2010)

    Article  Google Scholar 

  29. Bossu, J., Hautière, N., Tarel, J.P.: Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int. J. Comput. Vision 93(3), 348–367 (2011)

    Article  Google Scholar 

  30. Garg, K., Nayar, S. K.: Detection and removal of rain from videos. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, 2004

  31. Kim, J.H., Sim, J.Y., Kim, C.S.: Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Trans. Image Process. 24(9), 2658–2670 (2015)

    Article  MATH  MathSciNet  Google Scholar 

  32. Garg, K., Nayar, S. K.: When does a camera see rain? In Tenth IEEE International Conference on Computer Vision Volume 1, 2005

  33. Garg, K., Nayar, S.K.: Vision and rain. Int. J. Comput. Vision 75(1), 3–27 (2007)

    Article  Google Scholar 

  34. Kim, J. H., Lee, C., Sim, J. Y., Kim, C. S.: Single-image deraining using an adaptive nonlocal means filter. In IEEE International Conference on Image Processing, 2014

  35. Pei, S. C., Tsai, Y. T., Lee, C. Y.: Removing rain and snow in a single image using saturation and visibility features. In IEEE International Conference on Multimedia and Expo Workshops, 2014

  36. Li, Y.: Rain streak removal using layer priors. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016

  37. Chen, D., Chen, C., Kang, L.: Visual depth guided color image rain streaks removal using sparse coding. IEEE Trans. Circuits Syst. Video Technol. 24(8), 1430–1455 (2014)

    Article  Google Scholar 

  38. Yu, L., Yong, X., Hui, J.: Removing rain from a single image via discriminative sparse coding. In 2015 IEEE International Conference on Computer Vision, 2015

  39. Fu, X., Huang, J., Zeng, D., Yue, H., Paisley, J.: Removing rain from single images via a deep detail network. In IEEE Conference on Computer Vision and Pattern Recognition, 2017

  40. Lei, Z., Fu, C. W., Lischinski, D., Heng, P. A.: Joint bi-layer optimization for single-image rain streak removal. In IEEE International Conference on Computer Vision, 2017

  41. Eigen, D., Krishnan, D., Fergus, R.: Restoring an image taken through a window covered with dirt or rain. In IEEE International Conference on Computer Vision, 2014

  42. Wei, W., Meng, D., Qian, Z., Xu, Z.: Semi-supervised cnn for single image rain removal. arXiv preprint arXiv:1807.11078, 2018

  43. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016

  44. Shen, L., Yue, Z., Chen, Q., Feng, F., Ma, J.: Deep joint rain and haze removal from single images. In 2018 24th International Conference on Pattern Recognition, 2018

  45. Zhang, H., Sindagi, V., Patel, V. M.: Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology, 2017

  46. Li, R., Cheong, L. F., Tan, R. T.: Single image deraining using scale-aware multi-stage recurrent network. arXiv preprint arXiv:1712.06830, 2017

  47. Si, L., Ren, W., Zhang, J., Yu, J., Guo, X.: Fast single image rain removal via a deep decomposition-composition network. Comput. Vis. Image Underst. 186, 48–57 (2019)

  48. Lin, H., Li, Y., Ding, X., Zeng, W., Huang, Y., Paisley, J.: Rain o’er me: Synthesizing real rain to derain with data distillation. IEEE Transactions on Image Processing, 2019

  49. Denton, E., Chintala, S., Szlam, A., Fergus, R.: Deep generative image models using a laplacian pyramid of adversarial networks. MIT Press, 2015

  50. Huang, X., Li, Y., Poursaeed, O., Hopcroft, J., Belongie, S.: Stacked generative adversarial networks. IEEE Conference on Computer Vision and Pattern Recognition, 2016

  51. Yang, J., Kannan, A., Batra, D., Parikh, D.: Lr-gan: Layered recursive generative adversarial networks for image generation. In 2017 The International Conference on Learning Representations, 2017

  52. Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D N.: Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 5907–5915, 2017

  53. Arjovsky, M., Chintala, S., Bottou, On.: Wasserstein generative adversarial networks. In 2017 International Conference on Machine Learning, 2017

  54. Mao, X., Li, Q., Xie, H., Lau, Ryk., Smolley, S. P.: Least squares generative adversarial networks. In 2017 IEEE International Conference on Computer Vision, 2017

  55. Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders. In 2018 The International Conference on Learning Representations, 2018

  56. He, Z., Patel, Vishal M.: Density-aware single image de-raining using a multi-stream dense network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 695–704, 2018

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 61702322, Grant 61772328, and Grant 61801288.

Author information

Author notes

  1. Xiaoqi Lang, Bo Huang and Xiaoyan Jiang these authors contributed equally to this work.

Authors and Affiliations

  1. Shanghai University of Engineering Science, Street, Shanghai, 201620, China

    Juan Zhang, Xiaoqi Lang, Bo Huang & Xiaoyan Jiang

Authors
  1. Juan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoqi Lang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bo Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoyan Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Lang, X., Huang, B. et al. VAE-CoGAN: Unpaired image-to-image translation for low-level vision. SIViP 17, 1019–1026 (2023). https://doi.org/10.1007/s11760-022-02307-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02307-y

Keywords

Search

Navigation