Skip to main content
SpringerLink
Log in

Bilevel modeling investigated generative adversarial framework for image restoration

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

A Publisher Correction to this article was published on 15 November 2022

This article has been updated

Abstract

Generative adversarial network (GAN), which is developed on the alternative improvement of the generator and discriminator to obtain the optimal network, has become a problem of great interest for computer vision. In image restoration, domain knowledge and heuristic image prior are significant for low-level tasks, while the effectiveness of GAN relies on the empirical designing of networks and the massive training data only, ignoring the intrinsic principle owned by the task. Therefore, the restored results obtained by GAN usually suffer from information loss, and the structure may not be preserved well. To alleviate this issue, we develop a bilevel modeling investigated generative adversarial framework to incorporate task-specific domain knowledge with discriminative prior for adaptive image restoration. In our method, bilevel optimization is investigated to establish our basic iterative mechanism. The lower layer in the paradigm introduces domain knowledge to reveal the task essences in a physics model. At the same time, the upper layer exploits the data-driven discriminative prior to guiding the estimated one. Within the interaction between the lower and upper layers, the upper network realizes the correction of the intermediate results obtained by the lower heuristic prior guidance. We apply it to two low-level image restoration tasks, image deconvolution and single image deraining. Extensive experiments prove that the proposed method performs favorably against the state-of-the-art methods quantitatively and qualitatively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Change history

References

  1. Sonderby, C.K., Caballero, J., Theis, L., Shi, W., Huszár, F.: Amortised map inference for image super-resolution. In: ICLR (2017)

  2. Qian, R., Tan, R.T., Yang, W., Su, J., Liu, J.: Attentive generative adversarial network for raindrop removal from a single image. In: CVPR, pp. 2482–2491 (2018)

  3. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NeurIPS, pp. 2672–2680 (2014)

  4. Denton, E.L., Chintala, S., Fergus, R. et al.: Deep generative image models using a Laplacian pyramid of adversarial networks. In: NIPS (2015)

  5. Zhao, J., Mathieu, M., LeCun, Y.: Energy-based generative adversarial network. arXiv:1609.03126 (2016)

  6. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of gans for improved quality, stability, and variation. arXiv:1710.10196 (2017)

  7. Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. arXiv:1805.08318 (2018)

  8. Mirza, Me., Osindero, S.: Conditional generative adversarial nets. arXiv:1411.1784 (2014)

  9. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434 (2015)

  10. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR, pp. 4401–4410 (2019)

  11. Shaham, T.R., Dekel, T., Michaeli, T.: Singan: Learning a generative model from a single natural image. In: ICCV, pp. 4570–4580 (2019)

  12. Jiang, Z., Li, Z., Yang, S., Fan, X., Liu, R.: Target oriented perceptual adversarial fusion network for underwater image enhancement. IEEE Trans. Circuits Syst. Video Technol. (2022)

  13. Liu, J., Fan, X., Huang, Z., Wu, G., Liu, R., Zhong, W., Luo, Z.: Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. In: CVPR, pp. 5802–5811 (June 2022)

  14. Liu, J., Shang, J., Liu, R., Fan, X.: Attention-guided global-local adversarial learning for detail-preserving multi-exposure image fusion. IEEE Trans. Circuits Syst. Video Technol. (2022)

  15. Liu, R., Jiang, Z., Yang, S., Fan, X.: Twin adversarial contrastive learning for underwater image enhancement and beyond. IEEE Trans. Image Process. 31, 4922–4936 (2022)

    Article  Google Scholar 

  16. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: CVPR, pp. 5505–5514 (2018)

  17. Liu, J., Fan, X., Jiang, J., Liu, R., Luo, Z.: Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion. IEEE Trans. Circuits Syst. Video Technol. 32(1), 105–119 (2021)

    Article  Google Scholar 

  18. Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., Matas, J.: Deblurgan: Blind motion deblurring using conditional adversarial networks. In: CVPR, pp. 8183–8192 (2018)

  19. Dong, Y., Liu, Y., Zhang, H., Chen, S., Qiao, Y.: FD-GAN: generative adversarial networks with fusion-discriminator for single image dehazing. In: AAAI, pp. 10729–10736 (2020)

  20. Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterative regularization method for total variation-based image restoration. Multiscale Model. Simul. 4(2), 460–489 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  21. Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-Laplacian priors. In: NeurIPS, pp. 1033–1041 (2009)

  22. Krishnan, D., Tay, T., Fergus, R.: Blind deconvolution using a normalized sparsity measure. In: CVPR, pp. 233–240. IEEE (2011)

  23. Xu, L., Zheng, S., Jia, J.: Unnatural l0 sparse representation for natural image deblurring. In: CVPR, pp. 1107–1114 (2013)

  24. Schmidt, U., Roth, S.: Shrinkage fields for effective image restoration. In: CVPR, pp. 2774–2781 (2014)

  25. Schmidt, U., Jancsary, J., Nowozin, S., Roth, S.: Cascades of regression tree fields for image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 38(4), 677–689 (2015)

    Article  Google Scholar 

  26. Pan, J., Sun, D., Pfister, H., Yang, M.-H.: Blind image deblurring using dark channel prior. In: CVPR, pp. 1628–1636 (2016)

  27. Fortunato, H.E., Oliveira, M.M.: Fast high-quality non-blind deconvolution using sparse adaptive priors. Vis. Comput. 30(6), 661–671 (2014)

    Article  Google Scholar 

  28. Zhang, J., Pan, J., Lai, W.-S., Lau, R.W.H., Yang, M.-H.: Learning fully convolutional networks for iterative non-blind deconvolution. In: CVPR, pp. 3817–3825 (2017)

  29. Tao, X., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In: CVPR, pp. 8174–8182 (2018)

  30. Zhang, H., Dai, Y., Li, H., Koniusz, P.: Deep stacked hierarchical multi-patch network for image deblurring. In: CVPR, pp. 5978–5986 (2019)

  31. Tran, P., Tran, A.T., Phung, Q., Hoai, M.: Explore image deblurring via encoded blur kernel space. In: CVPR, pp. 11956–11965 (2021)

  32. Pan, X., Zhan, X., Dai, B., Lin, D., Loy, C.C., Luo, P.: Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Trans. Pattern Anal. Mach. Intell. 1 (2021)

  33. Ding, X., Chen, L., Zheng, X., Huang, Y., Zeng, D.: Single image rain and snow removal via guided l0 smoothing filter. Multimed. Tools Appl. 75(5), 2697–2712 (2016)

    Article  Google Scholar 

  34. Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: CVPR, pp. 2736–2744 (Jun. 2016)

  35. Kang, L.-W., Lin, C.-W., Fu, Y.-H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  36. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: ICCV, pp. 3397–3405 (Dec. 2015)

  37. Gu, S., Meng, D., Zuo, W., Zhang, L.: Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: CVPR, pp. 1708–1716 (Jul. 2017)

  38. Zhu, L., Fu, C.-W., Lischinski, D., Heng, P.-A.: Joint bi-layer optimization for single-image rain streak removal. In: ICCV, pp. 2526–2534 (2017)

  39. Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: CVPR, pp 1357–1366 (Jul. 2017)

  40. Fu, X., Huang, J., Huang, Y., Zeng, D., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: CVPR, pp. 1715–1723 (Jul. 2017)

  41. Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: CVPR, pp. 695–704 (Jun. 2018)

  42. Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: ECCV, pp. 254–269 (2018)

  43. Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: a better and simpler baseline. In: CVPR, pp. 3937–3946 (2019)

  44. Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., Lau, R.W.H.: Spatial attentive single-image deraining with a high quality real rain dataset. In: CVPR, pp. 12271–12279 (June 2019)

  45. Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., Ma, J., Jiang, J.: Multi-scale progressive fusion network for single image deraining. In: CVPR, pp. 8346–8355 (2020)

  46. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.-H., Shao, L.: Multi-stage progressive image restoration. In: CVPR, pp. 14821–14831 (2021)

  47. Wang, H., Yue, Z., Xie, Q., Zhao, Q., Zheng, Y., Meng, D.: From rain generation to rain removal. In: CVPR, pp. 14791–14801 (2021)

  48. Kruse, J., Rother, C., Schmidt, U.: Learning to push the limits of efficient FFT-based image deconvolution. In: ICCV, pp. 4586–4594 (2017)

  49. Liu, R., Jiang, Z., Fan, X., Luo, Z.: Knowledge-driven deep unrolling for robust image layer separation. IEEE Trans. Neural Netw. Learn. Syst. 31(5), 1653–1666 (2019)

    Article  Google Scholar 

  50. Liu, R., Cheng, S., He, Y., Fan, X., Lin, Z., Luo, Z.: On the convergence of learning-based iterative methods for nonconvex inverse problems. IEEE Trans. Pattern Anal. Mach. Intell. 42(12), 3027–3039 (2019)

    Article  Google Scholar 

  51. Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  52. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan. arXiv:1701.07875 (2017)

  53. Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: ICCV, pp. 479–486 (2011)

  54. Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: CVPR, vol. 2, pp. 2808–2817 (Jul. 2017)

  55. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV, vol. 2, pp. 416–423 (Jul. 2001)

  56. Levin, A., Weiss, Y., Durand, F., Freeman, W.T.: Understanding and evaluating blind deconvolution algorithms. In: CVPR, pp. 1964–1971 (2009)

  57. Schuler, C.J., Christopher Burger, H., Harmeling, S., Scholkopf, B.: A machine learning approach for non-blind image deconvolution. In: CVPR, pp. 1067–1074 (2013)

  58. Sun, L., Cho, S., Wang, J., Hays, J.: Edge-based blur kernel estimation using patch priors. In: ICCP, pp. 1–8 (2013)

  59. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  60. Deng, L.-J., Huang, T.-Z., Zhao, X.-L., Jiang, T.-X.: A directional global sparse model for single image rain removal. Appl. Math. Model. 59, 662–679 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  61. Wei, W., Meng, D., Zhao, Q., Xu, Z., Wu, Y.: Semi-supervised transfer learning for image rain removal. In: CVPR, pp. 3877–3886 (2019)

  62. Fu, X., Zeng, D., Huang, Y., Zhang, X.-P., Ding, X.: A weighted variational model for simultaneous reflectance and illumination estimation. In: CVPR, pp. 2782–2790 (Jun. 2016)

  63. Li, C., Yin, W., Jiang, H., Zhang, Y.: An efficient augmented Lagrangian method with applications to total variation minimization. Comput. Optim. Appl. 56(3), 507–530 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  64. Hossny, M., Nahavandi, S., Creighton, D.: Comments on’Information measure for performance of image fusion. Electronics letters. 44(18), 1066–1067 (2008)

Download references

Funding

This study was funded by the National Natural Science Foundation of China under Grant (Nos. 61922019 and 61672125).

Author information

Authors and Affiliations

  1. School of Software Technology, Dalian University of Technology, Dalian, China

    Zhiying Jiang & Zengxi Zhang

  2. International School of Information Science and Engineering, Dalian University of Technology, Dalian, China

    Yiyao Yu & Risheng Liu

  3. Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province, Dalian, China

    Zhiying Jiang, Zengxi Zhang & Risheng Liu

Authors
  1. Zhiying Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zengxi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yiyao Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Risheng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Risheng Liu.

Ethics declarations

Conflict of interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled, “Bilevel Modeling Investigated Generative Adversarial Framework for Image Restoration.”

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: figures 2-5 were not correct.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, Z., Zhang, Z., Yu, Y. et al. Bilevel modeling investigated generative adversarial framework for image restoration. Vis Comput 39, 5563–5575 (2023). https://doi.org/10.1007/s00371-022-02681-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02681-1

Keywords

Search

Navigation