Advertisement

Scale-Recurrent Multi-residual Dense Network for Image Super-Resolution

  • Kuldeep PurohitEmail author
  • Srimanta Mandal
  • A. N. Rajagopalan
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11133)

Abstract

Recent advances in the design of convolutional neural network (CNN) have yielded significant improvements in the performance of image super-resolution (SR). The boost in performance can be attributed to the presence of residual or dense connections within the intermediate layers of these networks. The efficient combination of such connections can reduce the number of parameters drastically while maintaining the restoration quality. In this paper, we propose a scale recurrent SR architecture built upon units containing series of dense connections within a residual block (Residual Dense Blocks (RDBs)) that allow extraction of abundant local features from the image. Our scale recurrent design delivers competitive performance for higher scale factors while being parametrically more efficient as compared to current state-of-the-art approaches. To further improve the performance of our network, we employ multiple residual connections in intermediate layers (referred to as Multi-Residual Dense Blocks), which improves gradient propagation in existing layers. Recent works have discovered that conventional loss functions can guide a network to produce results which have high PSNRs but are perceptually inferior. We mitigate this issue by utilizing a Generative Adversarial Network (GAN) based framework and deep feature (VGG) losses to train our network. We experimentally demonstrate that different weighted combinations of the VGG loss and the adversarial loss enable our network outputs to traverse along the perception-distortion curve. The proposed networks perform favorably against existing methods, both perceptually and objectively (PSNR-based) with fewer parameters.

Keywords

Super-resolution Deep learning Residual networks Dense connections 

References

  1. 1.
    Bevilacqua, M., Roumy, A., Guillemot, C., line Alberi Morel, M.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the British Machine Vision Conference, pp. 135.1-135.10. BMVA Press (2012).  https://doi.org/10.5244/C.26.135
  2. 2.
    Blau, Y., Mechrez, R., Timofte, R., Michaeli, T., Zelnik-Manor, L.: 2018 PIRM Challenge on Perceptual Image Super-resolution. ArXiv e-prints, pp. 1–22, September 2018Google Scholar
  3. 3.
    Blau, Y., Michaeli, T.: The perception-distortion tradeoff. arXiv preprint arXiv:1711.06077 (2017)
  4. 4.
    Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S., Feng, J.: Dual path networks. In: Advances in Neural Information Processing Systems, pp. 4467–4475 (2017)Google Scholar
  5. 5.
    Deng, X.: Enhancing image quality via style transfer for single image super-resolution. IEEE Signal Process. Lett. 25(4), 571–575 (2018)CrossRefGoogle Scholar
  6. 6.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016).  https://doi.org/10.1109/TPAMI.2015.2439281CrossRefGoogle Scholar
  7. 7.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10593-2_13CrossRefGoogle Scholar
  8. 8.
    Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_25CrossRefGoogle Scholar
  9. 9.
    Dong, W., Zhang, L., Shi, G., Li, X.: Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process. 22(4), 1620–1630 (2013).  https://doi.org/10.1109/TIP.2012.2235847MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Dong, W., Zhang, L., Shi, G., Wu, X.: Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Trans. Image Process. 20(7), 1838–1857 (2011).  https://doi.org/10.1109/TIP.2011.2108306MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: IEEE International Conference on Computer Vision (ICCV), pp. 349–356, September 2009.  https://doi.org/10.1109/ICCV.2009.5459271
  12. 12.
    Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680. Curran Associates, Inc. (2014)Google Scholar
  13. 13.
    Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1664–1673, June 2018Google Scholar
  14. 14.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  15. 15.
    Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR, vol. 1, p. 3 (2017)Google Scholar
  16. 16.
    Huang, J., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197–5206, June 2015.  https://doi.org/10.1109/CVPR.2015.7299156
  17. 17.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_43CrossRefGoogle Scholar
  18. 18.
    Kanemura, A., Ichi Maeda, S., Ishii, S.: Superresolution with compound markov random fields via the variational EM algorithm. Neural Networks 22(7), 1025–1034 (2009).  https://doi.org/10.1016/j.neunet.2008.12.005CrossRefzbMATHGoogle Scholar
  19. 19.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654, June 2016.  https://doi.org/10.1109/CVPR.2016.182
  20. 20.
    Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1637–1645, June 2016.  https://doi.org/10.1109/CVPR.2016.181
  21. 21.
    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. CoRR abs/1412.6980, http://arxiv.org/abs/1412.6980 (2014)
  22. 22.
    Lai, W., Huang, J., Ahuja, N., Yang, M.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5835–5843, July 2017.  https://doi.org/10.1109/CVPR.2017.618
  23. 23.
    Lai, W., Huang, J., Ahuja, N., Yang, M.: Fast and accurate image super-resolution with deep laplacian pyramid networks. CoRR abs/1710.01992, http://arxiv.org/abs/1710.01992 (2017)
  24. 24.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 105–114, July 2017.  https://doi.org/10.1109/CVPR.2017.19
  25. 25.
    Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1132–1140, July 2017.  https://doi.org/10.1109/CVPRW.2017.151
  26. 26.
    Ma, C., Yang, C.Y., Yang, X., Yang, M.H.: Learning a no-reference quality metric for single-image super-resolution. Comput. Vis. Image Underst. 158, 1–16 (2017)CrossRefGoogle Scholar
  27. 27.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G., Zisserman, A.: Non-local sparse models for image restoration. In: IEEE 12th International Conference on Computer Vision, pp. 2272–2279, 29 2009–2 October 2009.  https://doi.org/10.1109/ICCV.2009.5459452
  28. 28.
    Mandal, S., Bhavsar, A., Sao, A.K.: Noise adaptive super-resolution from single image via non-local mean and sparse representation. Sign. Process. 132, 134–149 (2017).  https://doi.org/10.1016/j.sigpro.2016.09.017CrossRefGoogle Scholar
  29. 29.
    Marquina, A., Osher, S.J.: Image super-resolution by TV-regularization and bregman iteration. J. Sci. Comput. 37, 367–382 (2008).  https://doi.org/10.1007/s10915-008-9214-8MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 2, pp. 416–423, July 2001.  https://doi.org/10.1109/ICCV.2001.937655
  31. 31.
    Matsui, Y., et al.: Sketch-based manga retrieval using manga109 dataset. Multimedia Tools Appl. 76(20), 21811–21838 (2017).  https://doi.org/10.1007/s11042-016-4020-zCrossRefGoogle Scholar
  32. 32.
    Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind” image quality analyzer. IEEE Sign. Process. Lett. 20(3), 209–212 (2013)CrossRefGoogle Scholar
  33. 33.
    Paszke, A., et al.: Automatic differentiation in PyTorch. In: NIPS-W (2017)Google Scholar
  34. 34.
    Sajjadi, M.S.M., Schlkopf, B., Hirsch, M.: Enhancenet: single image super-resolution through automated texture synthesis. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4501–4510, October 2017.  https://doi.org/10.1109/ICCV.2017.481
  35. 35.
    Soltani, R., Jiang, H.: Higher order recurrent neural networks. arXiv preprint arXiv:1605.00064 (2016)
  36. 36.
    Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2790–2798, July 2017.  https://doi.org/10.1109/CVPR.2017.298
  37. 37.
    Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: a persistent memory network for image restoration. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4549–4557, October 2017.  https://doi.org/10.1109/ICCV.2017.486
  38. 38.
    Timofte, R., Agustsson, E., Gool, L.V., Yang, M., Zhang, L., et al.: Ntire 2017 challenge on single image super-resolution: methods and results. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1110–1121, July 2017. DOI: https://doi.org/10.1109/CVPRW.2017.149
  39. 39.
    Wang, W., Li, X., Yang, J., Lu, T.: Mixed link networks. arXiv preprint arXiv:1802.01808 (2018)
  40. 40.
    Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deep networks for image super-resolution with sparse prior. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 370–378, December 2015.  https://doi.org/10.1109/ICCV.2015.50
  41. 41.
    Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010).  https://doi.org/10.1109/TIP.2010.2050625MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Boissonnat, J.D., et al. (eds.) Curves and Surfaces, pp. 711–730. Springer, Berlin Heidelberg (2012)CrossRefGoogle Scholar
  43. 43.
    Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3262–3271 (June 2018)Google Scholar
  44. 44.
    Zhang, X., Lam, E., Wu, E., Wong, K.: Application of Tikhonov regularization to super-resolution reconstruction of brain MRI images. In: Gao, X., Mller, H., Loomes, M., Comley, R., Luo, S. (eds.) Medical Imaging and Informatics. Lecture Notes in Computer Science, vol. 4987, pp. 51–56. Springer, Berlin Heidelberg (2008).  https://doi.org/10.1007/978-3-540-79490-5_8CrossRefGoogle Scholar
  45. 45.
    Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.IPCV Lab, Department of Electrical EngineeringIIT MadrasChennaiIndia

Personalised recommendations