Skip to main content
SpringerLink
Log in

Optimized highway deep learning network for fast single image super-resolution reconstruction

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

With the success of the deep residual network for image recognition tasks, the residual connection or skip connection has been widely used in deep learning models for various vision tasks, including single image super-resolution (SISR). Most existing SISR approaches pay particular attention to residual learning, while few studies investigate highway connection for SISR. Although skip connection can help to alleviate the vanishing gradient problem and enable fast training of the deep network, it still provides the coarse level of approximation in both forward and backward propagation paths and thus challenging to recover high-frequency details. To address this issue, we propose a novel model for SISR by using highway connection (HNSR), which composes of a nonlinear gating mechanism to further regulate the information. By using the global residual learning and replacing all local residual learning with designed gate unit in highway connection, HNSR has the capability of efficiently learning different hierarchical features and recovering much more details in image reconstruction. The experimental results have validated that HNSR can provide not only improved quality but also less prone to a few common problems during training. Besides, the more robust and efficient model is suitable for implementation in real-time and mobile systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Ha, V.K., Ren, J., Xu, X., et al.: Deep learning based single image super-resolution: a survey. Int. J. Autom. Comput. 16, 413–426 (2019)

    Article  Google Scholar 

  2. Dong, C., Loy, C.C., He, K.M., Tang, X.O.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  3. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

  4. Kim, J., Kwon Lee, J., Mu L.K.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645 (2016)

  5. Tai, Y., Yang, J., Liu, X.M.: Image super-resolution via deep recursive residual network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 2790–2798 (2017)

  6. Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep Laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 2. IEEE, pp. 5835–5843 (2017)

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  8. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

  9. Tai, Y., Yang, J., Liu, X.M., Xu, C.Y.: MemNet: a persistent memory network for image restoration. In: Proceedings of IEEE International Conference on Computer Vision, pp. 4539–4547. IEEE (2017)

  10. Ahn, N., Kang, B., Sohn, K.A.: Fast, accurate, and light-weight super-resolution with cascading residual network. In: Proceedings of the 15th European Conference on Computer Vision, pp. 252–268, Springer (2018)

  11. Zhang, Y.L., Tian, Y.P., Kong, Y., Zhong, B.N., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2472–2481. IEEE (2018)

  12. Shamsolmoali, P., Li, X., Wang, R.: Single image resolution enhancement by efficient dilated densely connected residual network. Sig. Process. Image Commun. 79, 13–23 (2019)

    Article  Google Scholar 

  13. Shamsolmoali, P., Zhang, J., Yang, J.: Image super resolution by dilated dense progressive network. Image Vis. Comput. 88, 9–18 (2019)

    Article  Google Scholar 

  14. Shamsolmoali, P., Zareapoor, M., Wang, R., Jain, D.K., Yang, J.: G-GANISR: gradual generative adversarial network for image super resolution. Neurocomputing 366, 140–153 (2019)

    Article  Google Scholar 

  15. Lim, B., Son, S., Kim, H., Nah, S., Mu, L.K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

  16. Zhang, Y.L., Li, K.P., Li, K., Wang, L.C., Zhong, B.N., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the 15th European Conference on Computer Vision, pp. 286–301. Springer (2018)

  17. Yu, J.H., Fan, Y.C., Yang, J.C., Xu, N., Wang, Z.W., Wang, X.C., Huang, T.: Wide activation for efficient and accurate image super-resolution. arXiv:1808.08718 (2019)

  18. Zhong, Z.S., Shen, T.C., Yang, Y.B., Lin, Z.C., Zhang, C.: Joint sub-bands learning with clique structures for wavelet domain super-resolution. In: Proceedings of the 32nd Conference on Neural Information Processing Systems, pp. 165–175. Curran Associates, Inc. (2018)

  19. Hui, Z., Wang, X.M., Gao, X.B.: Fast and accurate single image super-resolution via information distillation network. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 723–731. IEEE (2018)

  20. Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway networks. arXiv:1505.00387 (2015)

  21. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167 (2015)

  22. LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.R.: Efficient BackProp. In: Montavon, G., Orr, G.B., Müller, K.R. (eds.) Neural Networks: Tricks of the Trade. Lecture Notes in Computer Science, vol. 7700. Springer, Berlin, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_3

    Google Scholar 

  23. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078 (2014)

  24. Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (ELUS). arXiv:1511.07289 (2015)

  25. He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European Conference on Computer Vision, pp. 630–645 (2016)

  26. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.-H., Zhang, L., et al.: Ntire 2017 challenge on single image superresolution: methods and results. In: CVPR 2017 Workshops (2017)

  27. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of British Machine Vision Conference (2012)

  28. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Proceedings of the 7th International Conference on Curves and Surfaces, pp. 711–730. Springer (2010)

  29. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings the 8th IEEE International Conference on Computer Vision. IEEE (2001)

  30. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)

  31. Dong, C., Loy, C.C. , Tang, X.: Accelerating the super-resolution convolutional neural network. In: European Conference on Computer Vision, pp. 391–407. Springer

  32. Choi, J.S., Kim, M.: A deep convolutional neural network with selection units for super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 154–160 (2017)

  33. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  34. Wang, Z., Ren, J., Zhang, D., Sun, M., Jiang, J.: A deep-learning based feature hybrid framework for spatiotemporal saliency detection inside videos. Neurocomputing 287, 68–83 (2018)

    Article  Google Scholar 

  35. Yan, Y., Ren, J., Sun, G., Zhao, H., Han, J., Li, X., Marshall, S., Zhan, J.: Unsupervised image saliency detection with Gestalt-laws guided optimization and visual attention based refinement. Pattern Recogn. 79, 65–78 (2018)

    Article  Google Scholar 

  36. Tschannerl, J., Ren, J., Yuen, P., Sun, G., Zhao, H., Yang, Z., Wang, Z., Marshall, S.: MIMR-DGSA: unsupervised hyperspectral band selection based on information theory and a modified discrete gravitational search algorithm. Inf. Fusion 51, 189–200 (2019)

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like acknowledge the support from the Shanxi Hundred People Plan of China, the ETF Scholarship from the Faculty of Engineering, University of Strathclyde, and the Government Scholarship of Vietnam. The authors would greatly thank our colleagues from the Image Processing Group in Strathclyde University for their valuable suggestions.

Author information

Authors and Affiliations

  1. Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow, G1 1XW, UK

    Viet Khanh Ha, Jinchang Ren, Wenzhi Liao & Sophia Zhao

  2. College of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan, 030600, China

    Jinchang Ren, Xinying Xu & Gaowei Yan

  3. College of Electronics and Information, Xi’an Polytechnic University, Xi’an, 710048, China

    Jie Ren

Authors
  1. Viet Khanh Ha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinchang Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinying Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenzhi Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sophia Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jie Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Gaowei Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinchang Ren.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ha, V.K., Ren, J., Xu, X. et al. Optimized highway deep learning network for fast single image super-resolution reconstruction. J Real-Time Image Proc 17, 1961–1970 (2020). https://doi.org/10.1007/s11554-020-00973-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-020-00973-0

Keywords

Search

Navigation