Skip to main content
SpringerLink
Log in

Clustering-Oriented Multiple Convolutional Neural Networks for Single Image Super-Resolution

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

In contrast to the human visual system (HVS) that applies different processing schemes to visual information of different textural categories, most existing deep learning models for image super-resolution tend to exploit an indiscriminate scheme for processing one whole image. Inspired by the human cognitive mechanism, we propose a multiple convolutional neural network framework trained based on different textural clusters of image local patches. To this end, we commence by grouping patches into K clusters via K-means, which enables each cluster center to encode image priors of a certain texture category. We then train K convolutional neural networks for super-resolution based on the K clusters of patches separately, such that the multiple convolutional neural networks comprehensively capture the patch textural variability. Furthermore, each convolutional neural network characterizes one specific texture category and is used for restoring patches belonging to the cluster. In this way, the texture variation within a whole image is characterized by assigning local patches to their closest cluster centers, and the super-resolution of each local patch is conducted via the convolutional neural network trained by its cluster. Our proposed framework not only exploits the deep learning capability of convolutional neural networks but also adapts them to depict texture diversities for super-resolution. Experimental super-resolution evaluations on benchmark image datasets validate that our framework achieves state-of-the-art performance in terms of peak signal-to-noise ratio and structural similarity. Our multiple convolutional neural network framework provides an enhanced image super-resolution strategy over existing single-mode deep learning models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Yang J, Wright J, Huang T, Ma Y. Image super-resolution as sparse representation of raw image patches. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2008. p. 1–8.

  2. Yang J, Wright J, Huang T S, Ma Y. Image super-resolution via sparse representation. IEEE Trans Image Process 2010;19(11):2861–73.

    Article  PubMed  Google Scholar 

  3. Yang C -Y, Yang M -H. Fast direct super-resolution by simple functions. In: Proceedings of the IEEE international conference on computer vision. 2013. p. 561–568.

  4. Timofte R, Smet V D, Gool L V. Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of the IEEE international conference on computer vision. 2013. p. 1920–1927.

  5. Timofte R, Smet V D, Gool L V. A+: adjusted anchored neighborhood regression for fast super-resolution. In: Proceedings of the Asian conference on computer vision. 2014. p. 111–126.

  6. Yang C -Y, Ma C, Yang M -H. Single-image super-resolution: a benchmark. In: Proceedings of the European conference on computer vision. 2014. p. 372–386.

  7. Sun J, Xu Z, Shum H -Y. Image super-resolution using gradient profile prior. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2008. p. 1–8.

  8. Kim K I, Kwon Y. Single-image super-resolution using sparse regression and natural image prior. IEEE Trans Pattern Anal Mach Intell 2010;32(6):1127–33.

    Article  PubMed  Google Scholar 

  9. Zhang H, Yang J, Zhang Y, Huang T S. Non-local kernel regression for image and video restoration. In: Proceedings of the European conference on computer vision. 2010. p. 566–579.

  10. Chatterjee P, Milanfar P. Clustering-based denoising with locally learned dictionaries. IEEE Trans Image Process 2009;18(7):1438–1451.

    Article  PubMed  Google Scholar 

  11. Wang Q, Tang X, Shum H. Patch based blind image super resolution. In: Proceedings of the IEEE international conference on computer vision. 2005. p. 709–716.

  12. Ni K S, Nguyen T Q. Image superresolution using support vector regression. IEEE Trans Image Process 2007; 16(6):1596–1610.

    Article  PubMed  Google Scholar 

  13. Freedman G, Fattal R. Image and video upscaling from local self-examples. ACM Trans Graph (TOG) 2011; 30(2):12.

    Article  Google Scholar 

  14. Lu X, Yuan Y, Yan P. Image super-resolution via double sparsity regularized manifold learning. IEEE Trans Circ Syst Vid Technol 2013;23(12):2022–33.

    Article  Google Scholar 

  15. Lu X, Yuan Y, Yan P. Alternatively constrained dictionary learning for image superresolution. IEEE Trans Cybern 2014;44(3):366–77.

    Article  PubMed  Google Scholar 

  16. Wang W, Shen J, Shao L. Deep learning for video saliency detection. 2017 arXiv:1702.00871.

  17. Yao X, Han J, Cheng G. Semantic annotation of high-resolution satellite images via weakly supervised learning. IEEE Trans Geosci Remote Sens 2016;54(6):3660–71.

    Article  Google Scholar 

  18. Zhu Q, Du B, Turkbey B, et al. Deeply-supervised CNN for prostate segmentation. In: Proceedings of the international joint conference on neural networks. 2017. p. 178–184.

  19. Zhang D, Han J, Li C. Detection of co-salient objects by looking deep and wide. Int J Comput Vis 2016; 120(2):215–32.

    Article  Google Scholar 

  20. Spratling M W. A hierarchical predictive coding model of object recognition in natural images. Cogn Comput 2017;9(2):151–67.

    Article  CAS  Google Scholar 

  21. Wen G, Hou Z, Li H. Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cogn Comput. 2017;1–14.

  22. Zhang F, Du B, Zhang L, et al. Weakly supervised learning based on coupled convolutional neural networks for aircraft detection. IEEE Trans Geosci Remote Sens 2016;54(9):5553–63.

    Article  Google Scholar 

  23. Jain V, Seung S. Natural image denoising with convolutional networks. In: Proceedings of the advances in neural information processing systems. 2009. p. 769–776.

  24. Burger H C, Schuler C J, Harmeling S. Image denoising: can plain neural networks compete with bm3d? In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2012. p. 2392–2399.

  25. Eigen D, Krishnan D, Fergus R. Restoring an image taken through a window covered with dirt or rain. In: Proceedings of the IEEE international conference on computer vision. 2013. p. 633–640.

  26. Cui Z, Chang H, Shan S, Zhong B, Chen X. Deep network cascade for image super-resolution. In Proceedings of the European conference on computer vision. 2014. p. 49–64.

  27. Dong C, Loy C C, He K, Tang X. Learning a deep convolutional network for image super-resolution. In: Proceedings of the European conference on computer vision. 2014. p. 184–199.

  28. Dong C, Loy C C, He K, Tang X. Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 2016;38(2):295–307.

    Article  PubMed  Google Scholar 

  29. Kim J, Kwon Lee J, Mu Lee K. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. p. 1646–1654.

  30. Kim J, Kwon Lee J, Mu Lee K. Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. p. 1637–1645.

  31. Ho-Phuoc T, Guyader N, Guérin-Dugué A. A functional and statistical bottom-up saliency model to reveal the relative contributions of low-level visual guiding factors. Cogn Comput 2010;2(4):344–59.

    Article  Google Scholar 

  32. Kim S, Kwon S, Kweon I S. A perceptual visual feature extraction method achieved by imitating v1 and v4 of the human visual system. Cogn Comput 2013;5(4):610–628.

    Article  Google Scholar 

  33. Shen J, Hao X, Liang Z. Real-time superpixel segmentation by DBSCAN clustering algorithm. IEEE Trans Image Process 2016;25(12):5933–42.

    Article  Google Scholar 

  34. Glorot X, Bordes A, Bengio Y. Deep sparse rectifier neural networks. In: Proceedings of the international conference on artificial intelligence and statistics. 2011. p. 315–323.

  35. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE 1998;86(11):2278–2324.

    Article  Google Scholar 

  36. Martin D, Fowlkes C, Tal D, Malik J. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the IEEE international conference on computer vision. 2001. p. 416–423.

  37. Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel M L. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the British machine vision conference. 2012. p. 135.1–135.10.

  38. Zeyde R, Elad M, Protter M. On single image scale-up using sparse-representations. In: Proceedings of the international conference on curves and surfaces. 2010. p. 711–730.

  39. Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 2004;13(4):600–12.

    Article  PubMed  Google Scholar 

  40. Chang H, Yeung D -Y, Xiong Y. Super-resolution through neighbor embedding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2004. p. 275– 282.

  41. Dai D, Timofte R, Van Gool Luc L. Jointly optimized regressors for image super-resolution. Comput Graph Forum 2015;34(2):95–104.

    Article  Google Scholar 

  42. Dong C, Loy C C, Tang X. Accelerating the super-resolution convolutional neural network. In Proceedings of the European conference on computer vision. 2016. p. 391–407.

Download references

Funding

This study was funded by the National Natural Science Foundation of China (No. 61671481) and Qingdao Applied Fundamental Research (No. 16-5-1-11-jch), the Fundamental Research Funds for Central Universities, the EPSRC ESR project through ADR Funding Feasibility study on a fully deployable resilient flooding predicting, monitoring and response system and the 2016/2017 University of Exeter Outward Mobility Academic Fellowship.

Author information

Authors and Affiliations

  1. College of Information and Control Engineering, China University of Petroleum (East China), Qingdao, China

    Peng Ren & Wenjian Sun

  2. College of Engineering, Mathematics and Physical Sciences, University of Exeter, Exeter, UK

    Chunbo Luo

  3. Division of Computing Science and Maths, School of Natural Sciences, University of Stirling, Stirling, UK

    Amir Hussain

Authors
  1. Peng Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenjian Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chunbo Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Amir Hussain
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peng Ren.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Conflict of Interests

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ren, P., Sun, W., Luo, C. et al. Clustering-Oriented Multiple Convolutional Neural Networks for Single Image Super-Resolution. Cogn Comput 10, 165–178 (2018). https://doi.org/10.1007/s12559-017-9512-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-017-9512-2

Keywords

Search

Navigation