Skip to main content
SpringerLink
Log in

A multi-scale residual capsule network for hyperspectral image classification with small training samples

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Convolutional Neural Network(CNN) has been widely employed in hyperspectral image(HSI) classification. However, CNN cannot attain the relative location relation of spatial information well, hindering the further improvement of classification performance. Capsule Network(CapsNet) has been presented recently and represents features by vectors, which enhances the ability to attain feature space information and identify relative positions, and makes up for the shortcomings of CNN. To further improve the classification performance of HSI using CapsNet under limited labeled samples, this article proposes a multi-scale residual capsule network(MR-CapsNet). The proposed method adopts extended multi-scale convolution blocks to fully extract spectral-spatial features. Subsequently, the features extracted by convolution kernels of different sizes are fused by pointwise convolution. The residual structure is used for splicing with the input data, preventing the problem of vanishing gradients and overfitting. Finally, the fused feature information is classified at the capsule layer through the dynamic routing mechanism. Comparative experiments were carried out on three public datasets of hyperspectral images. The experimental results indicate that the overall classification accuracy of the proposed method has a 4.13%, 2.98%, and 1.43% improvement over the recent DC-CapsNet on three datasets, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Arun PV, Buddhiraju KM, Porwal A (2019) Capsulenet-based spatial–spectral classifier for hyperspectral images. IEEE J Sel Topics Appl Earth Obs Remote Sens 12(6):1849–1865. https://doi.org/10.1109/jstars.2019.2913097

    Article  Google Scholar 

  2. Bandos TV, Bruzzone L, Camps-Valls G (2009) Classification of hyperspectral images with regularized linear discriminant analysis. IEEE Trans Geosci Remote Sens 47(3):862–873. https://doi.org/10.1109/tgrs.2008.2005729

    Article  Google Scholar 

  3. Bazi Y, Melgani F (2010) Gaussian process approach to remote sensing image classification. IEEE Trans Geosci Remote Sens 48(1):186–197. https://doi.org/10.1109/tgrs.2009.2023983

    Article  Google Scholar 

  4. Benediktsson JA, Pesaresi M, Arnason K (2003) Classification and feature extraction for remote sensing images from urban areas based on morphological transformations. IEEE Trans Geosci Remote Sens 41(9):1940–1949. https://doi.org/10.1109/tgrs.2003.814625

    Article  Google Scholar 

  5. Benediktsson JA, Palmason JA, Sveinsson JR (2005) Classification of hyperspectral data from urban areas based on extended morphological profiles. IEEE Trans Geosci Remote Sens 43(3):480–491. https://doi.org/10.1109/tgrs.2004.842478

    Article  Google Scholar 

  6. Bruce LM, Koger CH, Li J (2002) Dimensionality reduction of hyperspectral data using discrete wavelet transform feature extraction. IEEE Trans Geosci Remote Sens 40(10):2331–2338. https://doi.org/10.1109/tgrs.2002.804721

    Article  Google Scholar 

  7. Chakraborty C, Kishor A, Rodrigues JJ (2022) Novel enhanced-grey wolf optimization hybrid machine learning technique for biomedical data computation. Comput Electr Eng 99:107778

    Article  Google Scholar 

  8. Chen Y, Jiang H, Li C, Jia X, Ghamisi P (2016) Deep feature extraction and classification of hyperspectral images based on convolutional neural networks. IEEE Trans Geosci Remote Sens 54(10):6232–6251. https://doi.org/10.1109/tgrs.2016.2584107

    Article  Google Scholar 

  9. Chen Y, Zhu K, Zhu L, He X, Ghamisi P, Benediktsson JA (2019) Automatic design of convolutional neural network for hyperspectral image classification. IEEE Trans Geosci Remote Sens 57(9):7048–7066. https://doi.org/10.1109/tgrs.2019.2910603

    Article  Google Scholar 

  10. Deng F, Pu S, Chen X, Shi Y, Yuan T, Pu S (2018) Hyperspectral image classification with capsule network using limited training samples. Sensors

  11. Ghamisi P, Plaza J, Chen Y, Li J, Plaza AJ (2017) Advanced spectral classifiers for hyperspectral images: a review. IEEE Geosci Remote Sens Mag 5(1):8–32. https://doi.org/10.1109/mgrs.2016.2616418

    Article  Google Scholar 

  12. Govender M, Chetty K, Bulcock H (2009) A review of hyperspectral remote sensing and its application in vegetation and water resource studies. Water SA 33(2). https://doi.org/10.4314/wsa.v33i2.49049

  13. Haut JM, Paoletti M, Plaza J, Plaza A (2016) Cloud implementation of the k-means algorithm for hyperspectral image analysis. J Supercomput 73(1):514–529. https://doi.org/10.1007/s11227-016-1896-3

    Article  Google Scholar 

  14. Haut JM, Paoletti ME, Plaza J, Plaza A (2018) Fast dimensionality reduction and classification of hyperspectral images with extreme learning machines. J Real-Time Image Proc 15(3):439–462. https://doi.org/10.1007/s11554-018-0793-9

    Article  Google Scholar 

  15. Hsieh T-H, Kiang J-F (2020) Comparison of CNN algorithms on hyperspectral image classification in agricultural lands. Sensors 20(6):1734. https://doi.org/10.3390/s20061734

    Article  Google Scholar 

  16. Hu W, Huang Y, Wei L, Zhang F, Li H (2015) Deep convolutional neural networks for hyperspectral image classification. J Sensors 2015:1–12. https://doi.org/10.1155/2015/258619

    Article  Google Scholar 

  17. Jiang X, Liu W, Zhang Y, Liu J, Li S, Lin J (2021) Spectral–spatial hyperspectral image classification using dual-channel capsule networks. IEEE Geosci Remote Sens Lett

  18. Kishor A, Chakraborty C, Jeberson W (2021) Reinforcement learning for medical information processing over heterogeneous networks. Multimed Tools Appl 80(16):23983–24004

    Article  Google Scholar 

  19. Lei R, Zhang C, Du S, Chen W, Zhang X, Zheng H, Huang J, Yu M (2021) A non-local capsule neural network for hyperspectral remote sensing image classification. Remote Sens Lett

  20. Lei R, Zhang C, Liu W, Zhang L, Zhang X, Yang Y, Huang J, Li Z, Zhou Z (2021) Hyperspectral remote sensing image classification using deep convolutional capsule network. IEEE J Sel Topics Appl Earth Obs Remote Sens 14:8297–8315

    Article  Google Scholar 

  21. Lei R, Zhang C, Zhang X, Huang J, Li Z, Liu W, Cui H (2022) Multiscale feature aggregation capsule neural network for hyperspectral remote sensing image classification. Remote Sens 14(7):1652

    Article  Google Scholar 

  22. Li W, Wu G, Zhang F, Du Q (2017) Hyperspectral image classification using deep pixel-pair features. IEEE Trans Geosci Remote Sens 55(2):844–853. https://doi.org/10.1109/tgrs.2016.2616355

    Article  Google Scholar 

  23. Li Y, Zhang H, Shen Q (2017) Spectral–spatial classification of hyperspectral imagery with 3d convolutional neural network. Remote Sens 9(1):67. https://doi.org/10.3390/rs9010067

    Article  Google Scholar 

  24. Li X, Yuan Z, Wang Q (2019) Unsupervised deep noise modeling for hyperspectral image change detection. Remote Sens 11 (3):258. https://doi.org/10.3390/rs11030258

    Article  MathSciNet  Google Scholar 

  25. Li H-C, Wang W-Y, Pan L, Li W, Du Q, Tao R (2020) Robust capsule network based on maximum correntropy criterion for hyperspectral image classification. IEEE J Sel Topics Appl Earth Obs Remote Sens 13:738–751. https://doi.org/10.1109/jstars.2020.2968930

    Article  Google Scholar 

  26. Li R, Zheng S, Duan C, Yang Y, Wang X (2020) Classification of hyperspectral image based on double-branch dual-attention mechanism network. Remote Sens 12(3):582

    Article  Google Scholar 

  27. Liang M, Jiao L, Yang S, Liu F, Hou B, Chen H (2018) Deep multiscale spectral-spatial feature fusion for hyperspectral images classification. IEEE J Sel Topics Appl Earth Obs Remote Sens 11(8):2911–2924. https://doi.org/10.1109/jstars.2018.2836671

    Article  Google Scholar 

  28. Liang H, Li Q (2016) Hyperspectral imagery classification using sparse representations of convolutional neural network features. Remote Sens 8 (2):99. https://doi.org/10.3390/rs8020099

    Article  Google Scholar 

  29. Liu B, Yu X (2021) Patch-free bilateral network for hyperspectral image classification using limited samples. IEEE J Sel Topics Appl Earth Obs Remote Sens 14:10794–10807. https://doi.org/10.1109/jstars.2021.3121334

    Article  Google Scholar 

  30. Liu B, Yu A, Yu X, Wang R, Gao K, Guo W (2021) Deep multiview learning for hyperspectral image classification. IEEE Trans Geosci Remote Sens 59(9):7758–7772. https://doi.org/10.1109/tgrs.2020.3034133

    Article  Google Scholar 

  31. Nalepa J, Antoniak M, Myller M, Lorenzo PR, Marcinkiewicz M (2020) Towards resource-frugal deep convolutional neural networks for hyperspectral image segmentation. Microprocess Microsyst 73:102994. https://doi.org/10.1016/j.micpro.2020.102994

    Article  Google Scholar 

  32. Paoletti ME, Haut JM, Fernandez-Beltran R, Plaza J, Plaza A, Li J, Pla F (2019) Capsule networks for hyperspectral image classification. IEEE Trans Geosc Remote Sens

  33. Paoletti ME, Haut JM, Plaza J, Plaza A (2019) Deep learning classifiers for hyperspectral imaging: a review. ISPRS J Photogramm Remote Sens 158:279–317. https://doi.org/10.1016/j.isprsjprs.2019.09.006

    Article  Google Scholar 

  34. Plaza A, Martinez P, Plaza J, Perez R (2005) Dimensionality reduction and classification of hyperspectral image data using sequences of extended morphological transformations. IEEE Trans Geosci Remote Sens 43 (3):466–479. https://doi.org/10.1109/tgrs.2004.841417

    Article  Google Scholar 

  35. Roy SK, Krishna G, Dubey SR, Chaudhuri BB (2020) HybridSN: exploring 3-d–2-d CNN feature hierarchy for hyperspectral image classification. IEEE Geosci Remote Sens Lett 17(2):277–281. https://doi.org/10.1109/lgrs.2019.2918719

    Article  Google Scholar 

  36. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. arXiv:1710.09829

  37. Salman M, Yuksel SE (2016) Hyperspectral data classification using deep convolutional neural networks. In: 2016 24th Signal processing and communication application conference (SIU). IEEE. https://doi.org/10.1109/siu.2016.7496193

  38. Sellami A, Tabbone S (2022) Deep neural networks-based relevant latent representation learning for hyperspectral image classification. Pattern Recogn 121:108224

    Article  Google Scholar 

  39. Signoroni A, Savardi M, Baronio A, Benini S (2019) Deep learning meets hyperspectral image analysis: a multidisciplinary review. J Imaging 5(5):52. https://doi.org/10.3390/jimaging5050052

    Article  Google Scholar 

  40. Sun G, Zhang X, Jia X, Ren J, Zhang A, Yao Y, Zhao H (2020) Deep fusion of localized spectral features and multi-scale spatial features for effective classification of hyperspectral images. Int J Appl Earth Obs Geoinformation 91:102157. https://doi.org/10.1016/j.jag.2020.102157

    Article  Google Scholar 

  41. Tan X, Xue Z, Yu X, Sun Y, Gao K (2022) Hyperspectral image classification with deep 3d capsule network and markov random field. IET Image Process 16(1):79–91

    Article  Google Scholar 

  42. Tarabalka Y, Fauvel M, Chanussot J, Benediktsson JA (2010) SVM- And MRF-based method for accurate classification of hyperspectral images. IEEE Geosci Remote Sens Lett 7(4):736–740. https://doi.org/10.1109/lgrs.2010.2047711

    Article  Google Scholar 

  43. Teke M, Deveci HS, Haliloğlu O, Gürbüz SZ, Sakarya U (2013) A short survey of hyperspectral remote sensing applications in agriculture. In: 2013 6th International conference on recent advances in space technologies (RAST). IEEE, pp 171–176

  44. Wang X, Tan K, Chen Y (2018) Capsnet and triple-GANs towards hyperspectral classification. In: 2018 5th International workshop on earth observation and remote sensing applications (EORSA). IEEE. https://doi.org/10.1109/eorsa.2018.8598574

  45. Wang J, Siying G, Runhu H, Linhao L, Zhang X, Jiao L (2021) Dual-channel capsule generation adversarial network for hyperspectral image classification. IEEE Trans Geosci Remote Sens

  46. Wang X, Tan K, Du Q, Chen Y, Du P (2019) Caps-tripleGAN: GAN-assisted CapsNet for hyperspectral image classification. IEEE Trans Geosci Remote Sens 57(9):7232–7245. https://doi.org/10.1109/tgrs.2019.2912468

    Article  Google Scholar 

  47. Xia J, Ghamisi P, Yokoya N, Iwasaki A (2018) Random forest ensembles and extended multiextinction profiles for hyperspectral image classification. IEEE Trans Geosci Remote Sens 56(1):202–216. https://doi.org/10.1109/tgrs.2017.2744662

    Article  Google Scholar 

  48. Xu Y, Zhang L, Du B, Zhang F (2018) Spectral-spatial unified networks for hyperspectral image classification. IEEE Trans Geosci Remote Sens:1–17. https://doi.org/10.1109/tgrs.2018.2827407

  49. Xu Q, Wang D, Luo B (2021) Faster multiscale capsule network with octave convolution for hyperspectral image classification. IEEE Geosci Remote Sens Lett 18(2):361–365. https://doi.org/10.1109/lgrs.2020.2970079

    Article  Google Scholar 

  50. Yang S, Shi Z (2016) Hyperspectral image target detection improvement based on total variation. IEEE Trans Image Process 25(5):2249–2258. https://doi.org/10.1109/tip.2016.2545248

    Article  MathSciNet  MATH  Google Scholar 

  51. Yin J, Li S, Zhu H, Luo X (2019) Hyperspectral image classification using capsnet with well-initialized shallow layers. IEEE Geosci Remote Sens Lett

  52. Yu S, Jia S, Xu C (2017) Convolutional neural networks for hyperspectral image classification. Neurocomputing 219:88–98. https://doi.org/10.1016/j.neucom.2016.09.010

    Article  Google Scholar 

  53. Zhang B, Wu D, Zhang L, Jiao Q, Li Q (2011) Application of hyperspectral remote sensing for environment monitoring in mining areas. Environ Earth Sci 65(3):649–658. https://doi.org/10.1007/s12665-011-1112-y

    Article  Google Scholar 

  54. Zhang H, Meng L, Wei X, Tang X, Tang X, Wang X, Jin B, Yao W (2019) 1d-convolutional capsule network for hyperspectral image classification. arXiv: Comput Vision Pattern Recognit

  55. Zheng Z, Zhong Y, Ma A, Zhang L (2020) FPGA: Fast Patch-free global learning framework for fully end-to-end hyperspectral image classification. IEEE Trans Geosci Remote Sens 58(8):5612–5626. https://doi.org/10.1109/tgrs.2020.2967821

    Article  Google Scholar 

  56. Zhu K, Chen Y, Ghamisi P, Jia X, Benediktsson JA (2019) Deep convolutional capsule network for hyperspectral image spectral and spectral-spatial classification. Remote Sens 11(3):223. https://doi.org/10.3390/rs11030223

    Article  Google Scholar 

  57. Zhuravel YN, Fedoseev AA (2013) The features of hyperspectral remote sensing data processing under environment monitoring tasks solution. Comput Opt 37(4):471–476. https://doi.org/10.18287/0134-2452-2013-37-4-471-476

    Article  Google Scholar 

Download references

Funding

This paper was supported by the Open Fund of Hubei Key Laboratory of Intelligent Geo Information Processing (Grant No. ZRIGIP-201801).

Author information

Authors and Affiliations

  1. School of Computer Science, China University of Geosciences, Wuhan, 430078, China

    Meilin Shi, Xilong Zeng & Jiansi Ren

  2. Hubei Key Laboratory of Intelligent Geo-Information Processing, China University of Geosciences, Wuhan, 430078, China

    Jiansi Ren

  3. School of Geography and Planning, Sun Yat-Sen University, Guangzhou, 510006, China

    Yichang Shi

Authors
  1. Meilin Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xilong Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiansi Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yichang Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All the authors made significant contributions to the work. Meilin Shi, Xilong Zeng, and Jiansi Ren designed the research, analyzed the results, and accomplished the validation work. Yichang Shi provided advice for the revision of the paper. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Jiansi Ren.

Ethics declarations

Conflict of Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shi, M., Zeng, X., Ren, J. et al. A multi-scale residual capsule network for hyperspectral image classification with small training samples. Multimed Tools Appl 82, 40473–40501 (2023). https://doi.org/10.1007/s11042-023-15017-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15017-5

Keywords

Search

Navigation