Advertisement

Separable-spectral convolution and inception network for hyperspectral image super-resolution

  • Ke Zheng
  • Lianru GaoEmail author
  • Qiong Ran
  • Ximin Cui
  • Bing Zhang
  • Wenzhi Liao
  • Sen Jia
Original Article
  • 82 Downloads

Abstract

Due to the limitation of the imaging system, it is hard to get Hyperspectral Image (HSI) with very high spatial resolution. Super-Resolution (SR) is a handling missing data technology to restore high-frequency information from the low-resolution image, can be used to solve this problem. Recently, Deep Learning (DL) has achieved great performance in computer vision, including SR. However, most DL-based HSI SR methods neglect the spectral disorder caused by normal 2D convolution. This paper proposes a novel end–end deep learning-based network named Separable-Spectral and Inception Network (SSIN) for HSI SR. In SSIN, the feature extraction module independently extracts features of each band image, and then these features are fused together to further exploit residual image by using feature fusion module. In reconstruction module, a multi-path connection is built to obtain features of different levels to restore high spatial resolution image in a coarse-to-fine manner. Experiments are implemented on two datasets include both indoor and airborne HSIs, and the performances of SSIN are evaluated in different conditions. Experimental results show that adding several separable spectral convolutions and multi-path connection in a deep network can greatly improve the SR performance, and SSIN achieves higher accuracy and better visualization compare with other methods.

Keywords

Deep learning Super-resolution Hyperspectral Image Separable-spectral convolution Multi-path reconstruction 

Notes

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant No. 91638201, No. 61501017, and No. 41722108.

References

  1. 1.
    Li W, Wu G, Zhang F, Du Q (2017) Hyperspectral image classification using deep pixel-pair features. IEEE Trans Geosci Remote Sens 55(2):844–853Google Scholar
  2. 2.
    Yu H, Gao L, Li J et al. (2016) Spectral-spatial hyperspectral image classification using subspace-based support vector machines and adaptive markov random fields. Remote Sens 8(5):355Google Scholar
  3. 3.
    Li W, Du Q, Zhang F, Hu W (2016) Hyperspectral image classification by fusing collaborative and sparse representations. IEEE J Sel Top Appl Earth Obs Remote Sens 9(9):4178–4187Google Scholar
  4. 4.
    Du B, Zhang M, Zhang L, Hu R, Tao D (2017) PLTD: patch-based low-rank tensor decomposition for hyperspectral images. IEEE Trans Multimed 19(1):67–79Google Scholar
  5. 5.
    Li W, Wu G, Du Q (2017) Transferred deep learning for anomaly detection in hyperspectral imagery. IEEE Geosci Remote Sens Lett 14(5):597–601Google Scholar
  6. 6.
    Du B, Zhang Y, Zhang L, Tao D (2016) Beyond the sparsity-based target detector: a hybrid sparsity and statistics based detector for hyperspectral images. IEEE Trans Image Process 25(11):5345–5357MathSciNetGoogle Scholar
  7. 7.
    Zhang H, Zhang L, Shen H (2012) A super-resolution reconstruction algorithm for hyperspectral images. Sig Process 92(9):2082–2096Google Scholar
  8. 8.
    Fernandez-Beltran R, Latorre-Carmona P, Pla F (2017) Single-frame super-resolution in remote sensing: a practical overview. Int J Remote Sens 38(1):314–354Google Scholar
  9. 9.
    Eismann MT, Hardie RC (2005) Hyperspectral resolution enhancement using high-resolution multispectral imagery with arbitrary response functions. IEEE Trans Geosci Remote Sens 43(3):455–465Google Scholar
  10. 10.
    Dong W, Fu F, Shi G et al. (2016) Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans Image Process 25(5):2337–2352MathSciNetGoogle Scholar
  11. 11.
    Akhtar N, Shafait F, Mian A (2015) Bayesian sparse representation for hyperspectral image super resolution. IEEE Conf Comput Vis Pattern Recognit. 2015:3631–3640Google Scholar
  12. 12.
    Akhtar N, Shafait F, Mian A (2014) Sparse spatio-spectral representation for hyperspectral image super-resolution. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) Computer Vision – ECCV 2014. Springer, Cham, pp 63–78Google Scholar
  13. 13.
    Wei H, Liang X, Liu H et al. (2015) Hyperspectral imagery super-resolution by compressive sensing inspired dictionary learning and spatial-spectral regularization. Sensors 15(1):2041Google Scholar
  14. 14.
    Yang J, Wright J, Huang T et al (2008) Image super-resolution as sparse representation of raw image patches. IEEE conf comput vision pattern recogn 2008(1):1–8Google Scholar
  15. 15.
    Yang J, Wright J, Huang T et al. (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873MathSciNetzbMATHGoogle Scholar
  16. 16.
    Li F, Xin L, Guo Y et al. (2017) A framework of mixed sparse representations for remote sensing images. IEEE Trans Geosci Remote Sens PP(99):1–12Google Scholar
  17. 17.
    Timofte R, De V, Gool LV (2013) Anchored neighborhood regression for fast example-based super-resolution. IEEE international conference on computer vision. pp 1920–1927Google Scholar
  18. 18.
    Timofte R, Smet VD, Gool LV (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. Asian Conf Comput Vis 2014(9006):111–126Google Scholar
  19. 19.
    Timofte R, Rothe R, Gool LV (2016) Seven ways to improve example-based single image super resolution. IEEE conference on computer vision and pattern recognition. pp 1865–1873Google Scholar
  20. 20.
    Huang JB, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. IEEE conference on computer vision and pattern recognition. pp 5197–5206Google Scholar
  21. 21.
    Dong C, Chen CL, He K et al. (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307Google Scholar
  22. 22.
    Wang Z, Liu D, Yang J et al. (2016) Deep networks for image super-resolution with sparse prior. IEEE international conference on computer vision. pp 370–378Google Scholar
  23. 23.
    Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. IEEE conference on computer vision and pattern recognition. pp 1646–1654Google Scholar
  24. 24.
    Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. IEEE conference on computer vision and pattern recognition. 1637–1645Google Scholar
  25. 25.
    Lim B, Son S, Kim H et al. (2017) Enhanced deep residual networks for single image super-resolution. IEEE conference on computer vision and pattern recognition. 1132–1140Google Scholar
  26. 26.
    Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(2):2012Google Scholar
  27. 27.
    He K, Zhang X, Ren S et al. (2016) Deep residual learning for image recognition. IEEE conference on computer vision and pattern recognition. 770–778Google Scholar
  28. 28.
    Du B, Xiong W, Wu J et al (2017) Stacked convolutional denoising auto-encoders for feature representation. IEEE Trans Cybern 47(4):1017–1027Google Scholar
  29. 29.
    Ren S, Girshick R, Girshick R et al (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137Google Scholar
  30. 30.
    Redmon J, Divvala S, Girshick R et al. (2016) You only look once: unified, real-time object detection. IEEE conference on computer vision and pattern recognition. 779–788Google Scholar
  31. 31.
    Liu W, Anguelov D, Erhan D et al. (2016) SSD: single shot multibox detector. European conference on computer vision. pp 21–37Google Scholar
  32. 32.
    He K, Gkioxari G, Dollár P et al. (2017) Mask R-CNN. IEEE international conference on computer visionGoogle Scholar
  33. 33.
    Shi W, Caballero J, Huszar F et al. (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. IEEE conference on computer vision and pattern recognition. pp 1874–1883Google Scholar
  34. 34.
    Ledig C, Theis L, Huszar F et al (2016) Photo-realistic single image super-resolution using a generative adversarial network. IEEE conference on computer vision and pattern recognition. pp 105–114Google Scholar
  35. 35.
    Lim B, Son S, Kim H et al (2017) Enhanced deep residual networks for single image super-resolution. IEEE conference on computer vision and pattern recognition workshops. pp 1132–1140Google Scholar
  36. 36.
    Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. IEEE conference on computer vision and pattern recognition. pp 2790–2798Google Scholar
  37. 37.
    Liebel L, Körner M (2016) Single-image super resolution for multispectral remote sensing data using convolutional neural networks. Int Arch Photogramm Remote Sens S XLI-B3. 2016:883–890Google Scholar
  38. 38.
    Li Y, Hu J, Zhao X et al. (2017) Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing 266(29):29–41Google Scholar
  39. 39.
    Hu J, Li Y, Xie W (2017) Hyperspectral image super-resolution by spectral difference learning and spatial error correction. IEEE Geosci Remote Sens Lett 14(10):1825–1829Google Scholar
  40. 40.
    Galliani S, Lanars C, Marmanis D et al. (2017) Learned spectral super-resolution. arXiv preprint arXiv:1703.09470Google Scholar
  41. 41.
    Lei S, Shi Z, Zou Z (2017) Super-Resolution for remote sensing images via local-global combined network. IEEE Geosci Remote Sens Lett PP(99):1–5Google Scholar
  42. 42.
    Yuan Y, Zheng X, Lu X (2017) Hyperspectral image superresolution by transfer learning. IEEE J Sel Top Appl Earth Obs Remote Sensing 10(5):1963–1974Google Scholar
  43. 43.
    Mei S, Yuan X, Ji J et al (2017) Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Remote Sens 9(11):1139Google Scholar
  44. 44.
    Jonas A, Ji QW, Radu T (2017) In defense of shallow learned spectral reconstruction from rgb images. IEEE conference on computer vision and pattern recognition. pp 471–479Google Scholar
  45. 45.
    Chang H, Yeung DY, Xiong Y (2004) Super-resolution through neighbor embedding. IEEE Conf Comput Vis Pattern Recognit. 1:275–282Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.College of Geoscience and Surveying EngineeringChina University of Mining and Technology (Beijing)BeijingChina
  2. 2.Key Laboratory of Digital Earth Science, Institute of Remote Sensing and Digital EarthChinese Academy of SciencesBeijingChina
  3. 3.College of Computer Science and Software Engineering, Computer Vision Research InstituteShenzhen UniversityShenzhenChina
  4. 4.College of Information Science and TechnologyBeijing University of Chemical TechnologyBeijingChina
  5. 5.University of Chinese Academy of SciencesBeijingChina
  6. 6.Department Telecommunications and Information ProcessingGhent UniversityGhentBelgium

Personalised recommendations