Advertisement

Hyperspectral Image Super-Resolution Using Multi-scale Feature Pyramid Network

  • He Sun
  • Zhiwei Zhong
  • Deming Zhai
  • Xianming Liu
  • Junjun JiangEmail author
Conference paper
  • 32 Downloads
Part of the Communications in Computer and Information Science book series (CCIS, volume 1181)

Abstract

Hyperspectral (HS) images are captured with rich spectral information, which have been proved to be useful in many real-world applications, such as earth observation. Due to the limitations of HS cameras, it is difficult to obtain HS images with high-resolution (HR). Recent advances in deep learning (DL) for single image super-resolution (SISR) task provide a powerful tool for restoring high-frequency details from low-resolution (LR) input image. Inspired by this progress, in this paper, we present a novel DL-based model for single HS image super-resolution in which a feature pyramid block is designed to extract multi-scale features of the input HS image. Our method does not need auxiliary inputs which further extends the application scenes. Experiment results show that our method outperforms state-of-the-arts on both objective quality indices and subjective visual results.

Keywords

Hyperspectral image processing Image super-resolution Deep learning 

Notes

Acknowledgements

This work is supported by the National Science Foundation under Grant Nos. 61971165, 61672193, and 61922027, and is also supported by the Fundamental Research Funds for the Central Universities.

References

  1. 1.
    Bioucas-Dias, J.M., et al.: Hyperspectral unmixing overview: geometrical, statistical, and sparse regression-based approaches. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 5, 354–379 (2012)CrossRefGoogle Scholar
  2. 2.
    Haut, J.M., Paoletti, M.E., Plaza, J., Li, J.Y., Plaza, A.J.: Active learning with convolutional neural networks for hyperspectral image classification using a new bayesian approach. IEEE Trans. Geosci. Remote Sens. 56, 6440–6461 (2018)CrossRefGoogle Scholar
  3. 3.
    Camps-Valls, G., Bruzzone, L.: Kernel-based methods for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 43, 1351–1362 (2005)CrossRefGoogle Scholar
  4. 4.
    Nguyen, H.V., Banerjee, A., Chellappa, R.: Tracking via object reflectance using a hyperspectral video camera. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pp. 44–51 (2010)Google Scholar
  5. 5.
    Huete, A.R., Miura, T., Gao, X.: Land cover conversion and degradation analyses through coupled soil-plant biophysical parameters derived from hyperspectral EO-1 hyperion. IEEE Trans. Geosci. Remote Sens. 41, 1268–1276 (2003)CrossRefGoogle Scholar
  6. 6.
    Roberts, D.A., Dennison, P.E., Gardner, M.E., Hetzel, Y., Ustin, S.L., Lee, C.T.: Evaluation of the potential of hyperion for fire danger assessment by comparison to the airborne visible/infrared imaging spectrometer. IEEE Trans. Geosci. Remote Sens. 41, 1297–1310 (2003)CrossRefGoogle Scholar
  7. 7.
    Simões, M., Bioucas-Dias, J.M., Almeida, L.B., Chanussot, J.: A convex formulation for hyperspectral image superresolution via subspace-based regularization. IEEE Trans. Geosci. Remote Sens. 53, 3373–3388 (2015)CrossRefGoogle Scholar
  8. 8.
    Wei, Q., Bioucas-Dias, J.M., Dobigeon, N., Tourneret, J.Y.: Hyperspectral and multispectral image fusion based on a sparse representation. IEEE Trans. Geosci. Remote Sens. 53, 3658–3668 (2014)CrossRefGoogle Scholar
  9. 9.
    Yokoya, N., Yairi, T., Iwasaki, A.: Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Trans. Geosci. Remote Sens. 50, 528–537 (2012)CrossRefGoogle Scholar
  10. 10.
    Zhang, K., Wang, M., Yang, S.Y., Jiao, L.: Spatialspectral-graph-regularized low-rank tensor decomposition for multispectral and hyperspectral image fusion. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 11, 1030–1040 (2018)CrossRefGoogle Scholar
  11. 11.
    Qu, Y., Qi, H., Kwan, C.: Unsupervised sparse Dirichlet-Net for hyperspectral image super-resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2511–2520 (2018)Google Scholar
  12. 12.
    Xie, Q., Zhou, M., Zhao, Q., Meng, D., Zuo, W., Xu, Z.: Multispectral and hyperspectral image fusion by MS/HS fusion net. ArXiv abs/1901.03281 (2019)Google Scholar
  13. 13.
    Loncan, L., et al.: Hyperspectral pansharpening: a review. IEEE Geosci. Remote Sens. Mag. 3, 27–46 (2015)CrossRefGoogle Scholar
  14. 14.
    Akgun, T., Altunbasak, Y., Mersereau, R.M.: Super-resolution reconstruction of hyperspectral images. IEEE Trans. Image Process. 14, 1860–1875 (2005)CrossRefGoogle Scholar
  15. 15.
    Mei, S., Yuan, X., Ji, J., Zhang, Y., Wan, S., Du, Q.: Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Remote Sens. 9, 1139 (2017)CrossRefGoogle Scholar
  16. 16.
    Yuan, Y., Zheng, X., Lu, X.: Hyperspectral image superresolution by transfer learning. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 10(5), 1963–1974 (2017)CrossRefGoogle Scholar
  17. 17.
    Li, Y., Zhang, L., Ding, C., Wei, W., Zhang, Y.: Single hyperspectral image super-resolution with grouped deep recursive residual network. In: 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), pp. 1–4 (2018)Google Scholar
  18. 18.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 295–307 (2014)CrossRefGoogle Scholar
  19. 19.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654 (2015)Google Scholar
  20. 20.
    Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1132–1140 (2017)Google Scholar
  21. 21.
    Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep Laplacian pyramid networks for fast and accurate super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5835–5843 (2017)Google Scholar
  22. 22.
    Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018)Google Scholar
  23. 23.
    Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. ArXiv abs/1807.02758 (2018)Google Scholar
  24. 24.
    Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2017)Google Scholar
  25. 25.
    Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1664–1673 (2018)Google Scholar
  26. 26.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004) CrossRefGoogle Scholar
  27. 27.
    Lin, T.Y., Dollár, P., Girshick, R.B., He, K., Hariharan, B., Belongie, S.J.: Feature pyramid networks for object detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 936–944 (2016)Google Scholar
  28. 28.
    He, K., Gkioxari, G., Dollar, P., Girshick, R.B.: Mask R-CNN. IEEE Trans. Pattern Anal. Mach. Intell. (2018)Google Scholar
  29. 29.
    Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1874–1883 (2016)Google Scholar
  30. 30.
    Yokoya, N., Iwasaki, A.: Airborne hyperspectral data over Chikusei. Technical report SAL-2016-05-27, Space Application Laboratory, University of Tokyo, Japan (May 2016)Google Scholar
  31. 31.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2014)Google Scholar
  32. 32.
    Sidorov, O., Hardeberg, J.Y.: Deep hyperspectral prior: denoising, inpainting, super-resolution. ArXiv abs/1902.00301 (2019)Google Scholar
  33. 33.
    Ulyanov, D., Vedaldi, A., Lempitsky, V.S.: Deep image prior. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9446–9454 (2017)Google Scholar
  34. 34.
    Wald, L.: Data Fusion, Definitions and Architectures - Fusion of Images of Different Spatial Resolutions. Les Presses de l’Ecole des Mines, Paris (2002)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  • He Sun
    • 1
  • Zhiwei Zhong
    • 1
  • Deming Zhai
    • 1
  • Xianming Liu
    • 1
  • Junjun Jiang
    • 1
    Email author
  1. 1.School of Computer Science and TechnologyHarbin Institute of TechnologyHarbinChina

Personalised recommendations