Advertisement

Cognitive Computation

, Volume 10, Issue 1, pp 165–178 | Cite as

Clustering-Oriented Multiple Convolutional Neural Networks for Single Image Super-Resolution

Article
  • 176 Downloads

Abstract

In contrast to the human visual system (HVS) that applies different processing schemes to visual information of different textural categories, most existing deep learning models for image super-resolution tend to exploit an indiscriminate scheme for processing one whole image. Inspired by the human cognitive mechanism, we propose a multiple convolutional neural network framework trained based on different textural clusters of image local patches. To this end, we commence by grouping patches into K clusters via K-means, which enables each cluster center to encode image priors of a certain texture category. We then train K convolutional neural networks for super-resolution based on the K clusters of patches separately, such that the multiple convolutional neural networks comprehensively capture the patch textural variability. Furthermore, each convolutional neural network characterizes one specific texture category and is used for restoring patches belonging to the cluster. In this way, the texture variation within a whole image is characterized by assigning local patches to their closest cluster centers, and the super-resolution of each local patch is conducted via the convolutional neural network trained by its cluster. Our proposed framework not only exploits the deep learning capability of convolutional neural networks but also adapts them to depict texture diversities for super-resolution. Experimental super-resolution evaluations on benchmark image datasets validate that our framework achieves state-of-the-art performance in terms of peak signal-to-noise ratio and structural similarity. Our multiple convolutional neural network framework provides an enhanced image super-resolution strategy over existing single-mode deep learning models.

Keywords

Clustering Convolutional neural networks Single image super-resolution 

Notes

Funding

This study was funded by the National Natural Science Foundation of China (No. 61671481) and Qingdao Applied Fundamental Research (No. 16-5-1-11-jch), the Fundamental Research Funds for Central Universities, the EPSRC ESR project through ADR Funding Feasibility study on a fully deployable resilient flooding predicting, monitoring and response system and the 2016/2017 University of Exeter Outward Mobility Academic Fellowship.

Compliance with Ethical Standards

Conflict of Interests

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

References

  1. 1.
    Yang J, Wright J, Huang T, Ma Y. Image super-resolution as sparse representation of raw image patches. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2008. p. 1–8.Google Scholar
  2. 2.
    Yang J, Wright J, Huang T S, Ma Y. Image super-resolution via sparse representation. IEEE Trans Image Process 2010;19(11):2861–73.CrossRefPubMedGoogle Scholar
  3. 3.
    Yang C -Y, Yang M -H. Fast direct super-resolution by simple functions. In: Proceedings of the IEEE international conference on computer vision. 2013. p. 561–568.Google Scholar
  4. 4.
    Timofte R, Smet V D, Gool L V. Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of the IEEE international conference on computer vision. 2013. p. 1920–1927.Google Scholar
  5. 5.
    Timofte R, Smet V D, Gool L V. A+: adjusted anchored neighborhood regression for fast super-resolution. In: Proceedings of the Asian conference on computer vision. 2014. p. 111–126.Google Scholar
  6. 6.
    Yang C -Y, Ma C, Yang M -H. Single-image super-resolution: a benchmark. In: Proceedings of the European conference on computer vision. 2014. p. 372–386.Google Scholar
  7. 7.
    Sun J, Xu Z, Shum H -Y. Image super-resolution using gradient profile prior. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2008. p. 1–8.Google Scholar
  8. 8.
    Kim K I, Kwon Y. Single-image super-resolution using sparse regression and natural image prior. IEEE Trans Pattern Anal Mach Intell 2010;32(6):1127–33.CrossRefPubMedGoogle Scholar
  9. 9.
    Zhang H, Yang J, Zhang Y, Huang T S. Non-local kernel regression for image and video restoration. In: Proceedings of the European conference on computer vision. 2010. p. 566–579.Google Scholar
  10. 10.
    Chatterjee P, Milanfar P. Clustering-based denoising with locally learned dictionaries. IEEE Trans Image Process 2009;18(7):1438–1451.CrossRefPubMedGoogle Scholar
  11. 11.
    Wang Q, Tang X, Shum H. Patch based blind image super resolution. In: Proceedings of the IEEE international conference on computer vision. 2005. p. 709–716.Google Scholar
  12. 12.
    Ni K S, Nguyen T Q. Image superresolution using support vector regression. IEEE Trans Image Process 2007; 16(6):1596–1610.CrossRefPubMedGoogle Scholar
  13. 13.
    Freedman G, Fattal R. Image and video upscaling from local self-examples. ACM Trans Graph (TOG) 2011; 30(2):12.CrossRefGoogle Scholar
  14. 14.
    Lu X, Yuan Y, Yan P. Image super-resolution via double sparsity regularized manifold learning. IEEE Trans Circ Syst Vid Technol 2013;23(12):2022–33.CrossRefGoogle Scholar
  15. 15.
    Lu X, Yuan Y, Yan P. Alternatively constrained dictionary learning for image superresolution. IEEE Trans Cybern 2014;44(3):366–77.CrossRefPubMedGoogle Scholar
  16. 16.
    Wang W, Shen J, Shao L. Deep learning for video saliency detection. 2017 arXiv:1702.00871.
  17. 17.
    Yao X, Han J, Cheng G. Semantic annotation of high-resolution satellite images via weakly supervised learning. IEEE Trans Geosci Remote Sens 2016;54(6):3660–71.CrossRefGoogle Scholar
  18. 18.
    Zhu Q, Du B, Turkbey B, et al. Deeply-supervised CNN for prostate segmentation. In: Proceedings of the international joint conference on neural networks. 2017. p. 178–184.Google Scholar
  19. 19.
    Zhang D, Han J, Li C. Detection of co-salient objects by looking deep and wide. Int J Comput Vis 2016; 120(2):215–32.CrossRefGoogle Scholar
  20. 20.
    Spratling M W. A hierarchical predictive coding model of object recognition in natural images. Cogn Comput 2017;9(2):151–67.CrossRefGoogle Scholar
  21. 21.
    Wen G, Hou Z, Li H. Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cogn Comput. 2017;1–14.Google Scholar
  22. 22.
    Zhang F, Du B, Zhang L, et al. Weakly supervised learning based on coupled convolutional neural networks for aircraft detection. IEEE Trans Geosci Remote Sens 2016;54(9):5553–63.CrossRefGoogle Scholar
  23. 23.
    Jain V, Seung S. Natural image denoising with convolutional networks. In: Proceedings of the advances in neural information processing systems. 2009. p. 769–776.Google Scholar
  24. 24.
    Burger H C, Schuler C J, Harmeling S. Image denoising: can plain neural networks compete with bm3d? In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2012. p. 2392–2399.Google Scholar
  25. 25.
    Eigen D, Krishnan D, Fergus R. Restoring an image taken through a window covered with dirt or rain. In: Proceedings of the IEEE international conference on computer vision. 2013. p. 633–640.Google Scholar
  26. 26.
    Cui Z, Chang H, Shan S, Zhong B, Chen X. Deep network cascade for image super-resolution. In Proceedings of the European conference on computer vision. 2014. p. 49–64.Google Scholar
  27. 27.
    Dong C, Loy C C, He K, Tang X. Learning a deep convolutional network for image super-resolution. In: Proceedings of the European conference on computer vision. 2014. p. 184–199.Google Scholar
  28. 28.
    Dong C, Loy C C, He K, Tang X. Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 2016;38(2):295–307.CrossRefPubMedGoogle Scholar
  29. 29.
    Kim J, Kwon Lee J, Mu Lee K. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. p. 1646–1654.Google Scholar
  30. 30.
    Kim J, Kwon Lee J, Mu Lee K. Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. p. 1637–1645.Google Scholar
  31. 31.
    Ho-Phuoc T, Guyader N, Guérin-Dugué A. A functional and statistical bottom-up saliency model to reveal the relative contributions of low-level visual guiding factors. Cogn Comput 2010;2(4):344–59.CrossRefGoogle Scholar
  32. 32.
    Kim S, Kwon S, Kweon I S. A perceptual visual feature extraction method achieved by imitating v1 and v4 of the human visual system. Cogn Comput 2013;5(4):610–628.CrossRefGoogle Scholar
  33. 33.
    Shen J, Hao X, Liang Z. Real-time superpixel segmentation by DBSCAN clustering algorithm. IEEE Trans Image Process 2016;25(12):5933–42.CrossRefGoogle Scholar
  34. 34.
    Glorot X, Bordes A, Bengio Y. Deep sparse rectifier neural networks. In: Proceedings of the international conference on artificial intelligence and statistics. 2011. p. 315–323.Google Scholar
  35. 35.
    LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE 1998;86(11):2278–2324.CrossRefGoogle Scholar
  36. 36.
    Martin D, Fowlkes C, Tal D, Malik J. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the IEEE international conference on computer vision. 2001. p. 416–423.Google Scholar
  37. 37.
    Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel M L. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the British machine vision conference. 2012. p. 135.1–135.10.Google Scholar
  38. 38.
    Zeyde R, Elad M, Protter M. On single image scale-up using sparse-representations. In: Proceedings of the international conference on curves and surfaces. 2010. p. 711–730.Google Scholar
  39. 39.
    Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 2004;13(4):600–12.CrossRefPubMedGoogle Scholar
  40. 40.
    Chang H, Yeung D -Y, Xiong Y. Super-resolution through neighbor embedding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2004. p. 275– 282.Google Scholar
  41. 41.
    Dai D, Timofte R, Van Gool Luc L. Jointly optimized regressors for image super-resolution. Comput Graph Forum 2015;34(2):95–104.CrossRefGoogle Scholar
  42. 42.
    Dong C, Loy C C, Tang X. Accelerating the super-resolution convolutional neural network. In Proceedings of the European conference on computer vision. 2016. p. 391–407.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  • Peng Ren
    • 1
  • Wenjian Sun
    • 1
  • Chunbo Luo
    • 2
  • Amir Hussain
    • 3
  1. 1.College of Information and Control EngineeringChina University of Petroleum (East China)QingdaoChina
  2. 2.College of Engineering, Mathematics and Physical SciencesUniversity of ExeterExeterUK
  3. 3.Division of Computing Science and Maths, School of Natural SciencesUniversity of StirlingStirlingUK

Personalised recommendations