Advertisement

SRFlow: Learning the Super-Resolution Space with Normalizing Flow

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12350)

Abstract

Super-resolution is an ill-posed problem, since it allows for multiple predictions for a given low-resolution image. This fundamental fact is largely ignored by state-of-the-art deep learning based approaches. These methods instead train a deterministic mapping using combinations of reconstruction and adversarial losses. In this work, we therefore propose SRFlow: a normalizing flow based super-resolution method capable of learning the conditional distribution of the output given the low-resolution input. Our model is trained in a principled manner using a single loss, namely the negative log-likelihood. SRFlow therefore directly accounts for the ill-posed nature of the problem, and learns to predict diverse photo-realistic high-resolution images. Moreover, we utilize the strong image posterior learned by SRFlow to design flexible image manipulation techniques, capable of enhancing super-resolved images by, e.g., transferring content from other images. We perform extensive experiments on faces, as well as on super-resolution in general. SRFlow outperforms state-of-the-art GAN-based approaches in terms of both PSNR and perceptual quality metrics, while allowing for diversity through the exploration of the space of super-resolved solutions. Code: git.io/Jfpyu.

Notes

Acknowledgements

This work was supported by the ETH Zürich Fund (OK), a Huawei Technologies Oy (Finland) project, a Google GCP grant, an Amazon AWS grant, and an Nvidia GPU grant.

Supplementary material

504441_1_En_42_MOESM1_ESM.pdf (18.6 mb)
Supplementary material 1 (pdf 19090 KB)

References

  1. 1.
    Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study. In: CVPR Workshops (2017)Google Scholar
  2. 2.
    Ahn, N., Kang, B., Sohn, K.A.: Image super-resolution via progressive cascading residual network. In: CVPR (2018)Google Scholar
  3. 3.
    Ardizzone, L., Lüth, C., Kruse, J., Rother, C., Köthe, U.: Guided image generation with conditional invertible neural networks. CoRR abs/1907.02392 (2019). http://arxiv.org/abs/1907.02392
  4. 4.
    Bahat, Y., Michaeli, T.: Explorable super resolution. arXiv.vol. abs/1912.01839 (2019)
  5. 5.
    Behrmann, J., Grathwohl, W., Chen, R.T.Q., Duvenaud, D., Jacobsen, J.: Invertible residual networks. In: ICML. Proceedings of Machine Learning Research, vol. 97, pp. 573–582. PMLR (2019)Google Scholar
  6. 6.
    Bell-Kligler, S., Shocher, A., Irani, M.: Blind super-resolution kernel estimation using an internal-gan. In: NeurIPS, pp. 284–293 (2019). http://papers.nips.cc/paper/8321-blind-super-resolution-kernel-estimation-using-an-internal-gan
  7. 7.
    Bühler, M.C., Romero, A., Timofte, R.: Deepsee: deep disentangled semantic explorative extreme super-resolution. arXiv preprint arXiv:2004.04433 (2020)
  8. 8.
    Dai, D., Timofte, R., Gool, L.V.: Jointly optimized regressors for image super-resolution. Comput. Graph. Forum 34(2), 95–104 (2015).  https://doi.org/10.1111/cgf.12544CrossRefGoogle Scholar
  9. 9.
    Dinh, L., Krueger, D., Bengio, Y.: NICE: non-linear independent components estimation. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Workshop Track Proceedings (2015)Google Scholar
  10. 10.
    Dinh, L., Sohl-Dickstein, J., Bengio, S.: Density estimation using real NVP. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 24–26 April 2017, Conference Track Proceedings (2017)Google Scholar
  11. 11.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: ECCV, pp. 184–199 (2014).  https://doi.org/10.1007/978-3-319-10593-2_13
  12. 12.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. TPAMI 38(2), 295–307 (2016)CrossRefGoogle Scholar
  13. 13.
    Durkan, C., Bekasov, A., Murray, I., Papamakarios, G.: Neural spline flows. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8–14 December 2019, Vancouver, BC, Canada, pp. 7509–7520 (2019)Google Scholar
  14. 14.
    Goodfellow, I.J., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, 8–13 December 2014, Montreal, Quebec, Canada, pp. 2672–2680 (2014)Google Scholar
  15. 15.
    Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: CVPR (2018)Google Scholar
  16. 16.
    Ignatov, A., et al.: Pirm challenge on perceptual image enhancement on smartphones: report. arXiv preprint arXiv:1810.01641 (2018)
  17. 17.
    Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR, pp. 5967–5976 (2017).  https://doi.org/10.1109/CVPR.2017.632
  18. 18.
    Kim, D., Kim, M., Kwon, G., Kim, D.: Progressive face super-resolution via attention to facial landmark. In: arxiv. vol. abs/1908.08239 (2019)Google Scholar
  19. 19.
    Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: CVPR (2016)Google Scholar
  20. 20.
    Kingma, D.P., Dhariwal, P.: Glow: Generative flow with invertible 1x1 convolutions. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3–8 December 2018, Montréal, Canada, pp. 10236–10245 (2018)Google Scholar
  21. 21.
    Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: CVPR (2017)Google Scholar
  22. 22.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR (2017)Google Scholar
  23. 23.
    Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: CVPR (2017)Google Scholar
  24. 24.
    Liu, R., Liu, Y., Gong, X., Wang, X., Li, H.: Conditional adversarial generative flow for controllable image synthesis. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 7992–8001 (2019)Google Scholar
  25. 25.
    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV), December 2015Google Scholar
  26. 26.
    Lugmayr, A., Danelljan, M., Timofte, R.: Unsupervised learning for real-world super-resolution. In: ICCVW, pp. 3408–3416. IEEE (2019)Google Scholar
  27. 27.
    Lugmayr, A., Danelljan, M., Timofte, R.: Ntire 2020 challenge on real-world image super-resolution: methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020Google Scholar
  28. 28.
    Lugmayr, A., Danelljan, M., Timofte, R., et al.: Aim 2019 challenge on real-world image super-resolution: methods and results. In: ICCV Workshops (2019)Google Scholar
  29. 29.
    Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error. In: ICLR (2016). http://arxiv.org/abs/1511.05440
  30. 30.
    Menon, S., Damian, A., Hu, S., Ravi, N., Rudin, C.: Pulse: self-supervised photo upsampling via latent space exploration of generative models. In: CVPR (2020)Google Scholar
  31. 31.
    Mittal, A., Moorthy, A., Bovik, A.: Referenceless image spatial quality evaluation engine. In: 45th Asilomar Conference on Signals, Systems and Computers, vol. 38, pp. 53–54 (2011)Google Scholar
  32. 32.
    Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind” image quality analyzer. IEEE Signal Process. Lett. 20(3), 209–212 (2013)CrossRefGoogle Scholar
  33. 33.
    Murphy, K.P.: Machine Learning: A Probabilistic Perspective. The MIT Press, Cambridge (2012)zbMATHGoogle Scholar
  34. 34.
    Venkatanath, N., Praneeth, D., Bh, M.C., Channappayya, S.S., Medasani, S.S: Blind image quality evaluation using perception based features. In: NCC, pp. 1–6. IEEE (2015)Google Scholar
  35. 35.
    Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: CVPR, pp. 2536–2544. IEEE Computer Society (2016)Google Scholar
  36. 36.
    Pumarola, A., Popov, S., Moreno-Noguer, F., Ferrari, V.: C-flow: conditional generative flow models for images and 3d point clouds. In: CVPR, pp. 7949–7958 (2020)Google Scholar
  37. 37.
    Rezende, D.J., Mohamed, S.: Variational inference with normalizing flows. In: Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6–11 July 2015, pp. 1530–1538 (2015)Google Scholar
  38. 38.
    Sajjadi, M.S.M., Schölkopf, B., Hirsch, M.: Enhancenet: single image super-resolution through automated texture synthesis. In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, 22–29 October 2017, pp. 4501–4510. IEEE Computer Society (2017).  https://doi.org/10.1109/ICCV.2017.481
  39. 39.
    Shaham, T.R., Dekel, T., Michaeli, T.: Singan: learning a generative model from a single natural image. In: ICCV, pp. 4570–4580 (2019)Google Scholar
  40. 40.
    Shocher, A., Cohen, N., Irani, M.: Zero-shot super-resolution using deep internal learning. In: CVPR (2018)Google Scholar
  41. 41.
    Sun, L., Hays, J.: Super-resolution from internet-scale scene matching. In: ICCP (2012)Google Scholar
  42. 42.
    Timofte, R., et al.: Ntire 2017 challenge on single image super-resolution: methods and results. In: CVPR Workshops (2017)Google Scholar
  43. 43.
    Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 111–126. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-16817-3_8CrossRefGoogle Scholar
  44. 44.
    Timofte, R., Gu, S., Wu, J., Van Gool, L.: Ntire 2018 challenge on single image super-resolution: methods and results. In: CVPR Workshops (2018)Google Scholar
  45. 45.
    Timofte, R., Smet, V.D., Gool, L.V.: Anchored neighborhood regression for fast example-based super-resolution. In: ICCV, pp. 1920–1927 (2013).  https://doi.org/10.1109/ICCV.2013.241
  46. 46.
    Wang, X., et al.: Esrgan: Enhanced super-resolution generative adversarial networks. ECCV (2018)Google Scholar
  47. 47.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  48. 48.
    Winkler, C., Worrall, D.E., Hoogeboom, E., Welling, M.: Learning likelihoods with conditional normalizing flows. arxiv abs/1912.00042 (2019). http://arxiv.org/abs/1912.00042
  49. 49.
    Yang, C., Yang, M.: Fast direct super-resolution by simple functions. In: ICCV, pp. 561–568 (2013).  https://doi.org/10.1109/ICCV.2013.75
  50. 50.
    Yang, G., Huang, X., Hao, Z., Liu, M., Belongie, S.J., Hariharan, B.: Pointflow: 3d point cloud generation with continuous normalizing flows. In: ICCV (2019)Google Scholar
  51. 51.
    Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution as sparse representation of raw image patches. In: CVPR (2008).  https://doi.org/10.1109/CVPR.2008.4587647
  52. 52.
    Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010).  https://doi.org/10.1109/TIP.2010.2050625
  53. 53.
    Yu, X., Porikli, F.: Ultra-resolving face images by discriminative generative networks. In: ECCV, pp. 318–333 (2016).  https://doi.org/10.1007/978-3-319-46454-1_20
  54. 54.
    Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)Google Scholar
  55. 55.
    Zhang, W., Liu, Y., Dong, C., Qiao, Y.: Ranksrgan: generative adversarial networks with ranker for image super-resolution (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Computer Vision LaboratoryETH ZurichZürichSwitzerland

Personalised recommendations