Skip to main content
SpringerLink
Log in

Real-World super-resolution under the guidance of optimal transport

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

In the real world, lacking paired training data makes image super-resolution (SR) be a tricky unsupervised task. Existing methods are mainly train models on synthetic datasets and achieve the tradeoff between detail restoration and noise artifact suppression based on a priori knowledge, which indicate it cannot be optimal in both aspects. To solve this problem, we propose OTSR, a single image super-resolution method based on optimal transport theory. OTSR aims to find the optimal solution to the ill-posed SR problem, so that the model can restore high-frequency detail accurately and also suppress noise and artifacts well. Our method consists of three stages: real-world images degradation estimation, LR images generation and model optimization based on quadratic Wasserstein distance. Through the first two stages, the problem of no paired image is solved. In the third stage, under the guidance of optimal transport theory, the optimal mapping from LR to HR image space is learned. Extensive experiments show that our method outperforms the state-of-the-art methods in terms of both detail repair and noise artifact suppression. The source code is available at https://github.com/cognaclee/OTSR.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Adolphs, L., Daneshmand, H., Lucchi, A., Hofmann, T.: Local saddle point optimization: a curvature exploitation approach. In: Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 486–495. PMLR, (2019)

  2. Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1122–1131, (2017)

  3. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: Precup, Doina, Teh, Yee Whye, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 214–223. PMLR, (2017)

  4. Bell-K., Sefi, S., Assaf, I.M.: Blind super-resolution kernel estimation using an internal-gan. CoRR arXiv:1909.06581 (2019)

  5. Chen, J., Chen, J., Chao, H., Yang, M.: Image blind denoising with generative adversarial network based noise modeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3155–3164, (2018)

  6. Deshpande, I., Zhang, Z., Schwing, A.: Generative modeling using the sliced wasserstein distance. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3483–3491, (2018)

  7. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  8. Frogner, C., Zhang, C., Mobahi, H., Araya-Polo, M., Poggio, T.A.: Learning with a wasserstein loss. CoRR arXiv:1506.05439 (2015)

  9. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K.Q. editors. Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc. (2014)

  10. Gu, J., Lu, H., Zuo, W., Dong, C.: Blind super resolution with iterative kernel correction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1604–1613, (2019)

  11. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. CoRR arXiv:1704.00028 (2017)

  12. Haris, M., Shakhnarovich, G., Ukita, N.: Deep back projection networks for super resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1664–1673, (2018)

  13. Ignatov, A., Kobyshev, N., Timofte, R., Vanhoey, K., Gool, L.V.: Dslr-quality photos on mobile devices with deep convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3277–3285 (2017)

  14. Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976, (2017)

  15. Ji, X., Cao, Y., Tai, Y., Wang, C., Li, J., Huang, F.: Real-world super-resolution via kernel estimation and noise injection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, (2020)

  16. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711. Springer International Publishing (2016)

  17. Kanopoulos, N., Vasanthavada, N., Baker, R.L.: Design of an image edge detection filter using the sobel operator. IEEE Journal of Solid-State Circuits 23(2) (1988)

  18. Kantorovich, L.V., Rubinshten, G.S.: On a space of completely additive functions. Vestnik Leningrad Univ 13(7), 52–59 (1958)

    MathSciNet  Google Scholar 

  19. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Yoshua, LeCun, Yann, editors. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015)

  20. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 105–114 (2017)

  21. Lei, N., Kehua, S., Cui, L., Yau, S.-T., Xianfeng David, G.: A geometric view of optimal transportation and generative model. Comput. Aided Geomet. Design 68, 1–21 (2019)

    Article  MathSciNet  Google Scholar 

  22. Lei, N., An, D., Guo, Y., Kehua, S., Liu, S., Luo, Z., Yau, S.-T., Xianfeng, G.: A geometric understanding of deep learning. Engineering 6(3), 361–374 (2020)

    Article  Google Scholar 

  23. Lim, B., Son, S., Kim, H., Nah, S., Lee, KM.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 136-144, (2017)

  24. Liu, H., GU, X., Samaras, D.: A two-step computation of the exact GAN Wasserstein distance. In: Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3159–3168. PMLR, (2018)

  25. Liu, H., Gu, X., Samaras, D.:Asserstein gan with quadratic transport cost. In: Proceedings of the International Conference on Computer Vision

  26. Lugmayr, A., Danelljan, M, Timofte, R.: Unsupervised learning for real-world super-resolution. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3408–3416, (2019)

  27. Luo, Z., H., Yan, L., Shang, W., Liang, T.T.: Unfolding the alternating optimization for blind super resolution. CoRR arXiv:2010.02631 (2020)

  28. Mescheder, L., Geiger, A., Nowozin, S.: Which training methods for GANs do actually converge? In: Dy, Jennifer, Krause, Andreas, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3481–3490. PMLR, (2018)

  29. Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 21(12), 4695–4708 (2012)

    Article  MathSciNet  Google Scholar 

  30. Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind’’ image quality analyzer. IEEE Signal Process. Lett. 20(3), 209–212 (2013)

    Article  Google Scholar 

  31. Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. CoRR arXiv:1802.05957 (2018)

  32. Peyre, G., Cuturi, M.: Computational optimal transport. Found. Trends in Mach. Learn. 11(5–6), 355–607 (2019)

    Article  Google Scholar 

  33. Santambrogio, F.: Optimal transport for applied mathematicians. Progress in Nonlinear Differential Equations and their applications 87 (2015)

  34. Shocher, M. Irani A., Cohen, N.: Zero-shot super-resolution using deep internal learning. In: The Conference on Computer Vision and Pattern Recognition, pp. 3118–3126, (2018)

  35. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Computer Science (2014)

  36. Timofte, R., Agustsson, Ei., Gool, L.V., Yang, Ming-Hsuan, Z., Lei: N.: Challenge on single image super-resolution: Methods and results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1110–1121, (2017)

  37. Wang, Z., Chen, J., Hoi, S.C.H.: Deep learning for image super resolution: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 3365–3387 (2020)

  38. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Loy, C.C.: Esrgan: Enhanced super-resolution generative adversarial networks. In: The European Conference on Computer Vision Workshops (ECCVW), September (2018)

  39. Wang, H.R.S.Z., Bovik, A.C., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  40. Wang, M., Zhenxue Chen, Q.M., Jonathan, W., Jian, M.: Improved face super-resolution generative adversarial networks. Mach. Vis. Appl. 31(4), 1–12 (2020)

    Article  Google Scholar 

  41. Xie, S., Tu, Z.: Holistically nested edge detection. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1395–1403, (2015)

  42. Yann, Brenier: Polar factorization and monotone rearrangement of vector-valued functions. Commun. Pure Appl. Math. 64, 375–417 (1991)

    MathSciNet  MATH  Google Scholar 

  43. Yuan, Y., Liu, S., Zhang, J., Zhang, Y., Dong, C., Lin, L.: Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 701–710, (2018)

  44. Zhang, R., Isola, P., Efros, Alexei A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586–595, (2018)

  45. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: ECCV, pp. 286–301, (2018)

  46. Zhang, K., Zuo, W., Zhang, L.: Deep plug-and-play super-resolution for arbitrary blur kernels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1671–1681 (2019)

  47. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2472–2481, (2018)

  48. Zhang, W., Ma, K., Zhai, G., Yang, X.: Uncertainty-aware blind image quality assessment in the laboratory and wild. IEEE Trans. Image Process. 30, 3474–3486 (2021)

    Article  Google Scholar 

  49. Zhou, R., Susstrunk, S.: Kernel modeling super-resolution on real low resolution images. In: 2019 IEEE International Conference on Computer Vision(ICCV), pp. 2433–2443, (2019)

Download references

Acknowledgements

This research was supported by the National Key R &D Program of China 2021YFA1003003, and the National Natural Science Foundation of China under Grant No. 61936002, 61772105, 61720106005. We are fortunate and thankful for all the advice and guidance we have received during this work.

Author information

Authors and Affiliations

  1. School of Software, Dalian University of Technology, Dalian, 116620, People’s Republic of China

    Zezeng Li

  2. International School of Information and Software, Dalian University of Technology, Dalian, 116620, People’s Republic of China

    Na Lei

  3. Academy for Multidisciplinary Studies, Capital Normal University, Beijing, 100048, People’s Republic of China

    Ji Shi

  4. School of Mathematical Sciences, Capital Normal University, Beijing, 100048, People’s Republic of China

    Hao Xue

Authors
  1. Zezeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Na Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ji Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hao Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Na Lei.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Z., Lei, N., Shi, J. et al. Real-World super-resolution under the guidance of optimal transport. Machine Vision and Applications 33, 48 (2022). https://doi.org/10.1007/s00138-022-01299-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-022-01299-6

Keywords

Search

Navigation