Skip to main content
Log in

Recent advances in object tracking using hyperspectral videos: a survey

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Short-Term Single-Object (STSO) tracking using Hyperspectral Videos (HSVs), which has become a hotspot recently, is a challenging task. Hyperspectral Object Tracking (HOT) makes full use of spatial and spectral information during the tracking process. In HOT, multiple features, including deep network features, have been combined with correlation filter methods, which increases time-consuming efficiency and tracking accuracy. However, redundant spectral information needs to be obtained in an effective way. In addition, there is currently no detailed investigation of HOT algorithms. Therefore, this survey studies the development of HOT algorithms in recent years. Specifically, several HSVs are listed, an investigation of HOT algorithms is conducted, and components of HOT are described in detail. Furthermore, several popular HOT algorithms, including our previous work BS-SiamPRN and AD-SiamRPN, are compared quantitatively and qualitatively. Finally, the research status of HOT is summarized, and future work has been described, which lays the foundation for future HOT or STSO referring to HSVs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Data Availibility Statement

Data are available from the corresponding author upon reasonable request.

References

  1. José Manuel A, Hamid B, Saioa E (2015) Hyperspectral image analysis. A tutorial. Anal Chim Acta 896:34–51

    Google Scholar 

  2. Anderson GL, Carruthers RI, Shaokui G, Peng G (2005) Cover: monitoring of invasive Tamarix distribution and effects of biological control with airborne hyperspectral remote sensing. Int J Remote Sens 26(12):2487–2489

    Google Scholar 

  3. Banerjee A, Burlina P, Broadwater J (2009) Hyperspectral video for illumination-invariant tracking. In 009 first workshop on hyperspectral image and signal processing: evolution in remote sensing. IEEE, pp 1–4

  4. Bolme DS, Beveridge JR, Draper BA, Lui YM (2010) Visual object tracking using adaptive correlation filters. In 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, pp 2544–2550

  5. Barquero G, Hupont I, Tena CF (2021) Rank-based verification for long-term face tracking in crowded scenes. arXiv:2107.13273

  6. Bottou L (2012) Stochastic gradient descent tricks. In Neural networks: tricks of the trade: second edition. Berlin, Heidelberg, pp 421–436

  7. Bertinetto L, Valmadre J, Henriques JF, Vedaldi A, Torr PH (2016) Fully-convolutional siamese networks for object tracking. In European conference on computer vision. Springer, pp 850–865

  8. Chang CI (2016) Real-time progressive hyperspectral image processing. Springer

    Google Scholar 

  9. Chen L, Zhao Y, Yao J, Chen J, Li N, Chan JC, Kong SG (2021) Object tracking in hyperspectral-oriented video with fast spatial-spectral features. Remote Sensing 13(10):1922

  10. Danelljan M, Hager G, Shahbaz Khan F, Felsberg M (2015) Learning spatially regularized correlation filters for visual tracking. In Proceedings of the IEEE international conference on computer vision, pp 4310–4318

  11. Du D, Qi Y, Yu H, Yang Y, Duan K, Li G, Zhang W, Huang Q, Tian Q (2018) The unmanned aerial vehicle benchmark: object detection and tracking. In Proceedings of the European conference on computer vision (ECCV), pp 370–386

  12. Danelljan M, Shahbaz Khan F, Felsberg M, Van de Weijer J (2014) Adaptive color attributes for real-time visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1090–1097

  13. Guo Q, Feng W, Zhou C, Huang R, Wan L, Wang S (2017) Learning dynamic siamese network for visual object tracking. In Proceedings of the IEEE international conference on computer vision, pp 1763–1771

  14. Ghafir I, Prenosil V, Svoboda J, Hammoudeh M (2016) A survey on network security monitoring systems. In 2016 IEEE 4th international conference on future internet of things and cloud workshops (FiCloudW). IEEE, pp 77–82

  15. Guo D, Wang J, Cui Y, Wang Z, Chen S (2020) SiamCAR: siamese fully convolutional classification and regression for visual tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6269–6277

  16. Ghamisi P, Yokoya N, Li J, Liao W, Liu S, Plaza J, Rasti B, Plaza A (2017) Advances in hyperspectral image and signal processing: a comprehensive overview of the state of the art. IEEE Geosci Remote Sens Magazine 5(4):37–78

    Google Scholar 

  17. Henriques JF, Caseiro R, Martins P, Batista J (2014) High-speed tracking with kernelized correlation filters. IEEE Trans Pattern Anal Mach Intell 37(3):583–96

    Google Scholar 

  18. Hou Z, Li W, Zhou J, Tao R (2022) Spatial-spectral weighted and regularized tensor sparse correlation filter for object tracking in hyperspectral videos. IEEE Trans Geosci Remote Sens 60:1–2

    Google Scholar 

  19. He YJ, Li M, Zhang J, Yao JP (2015) Infrared target tracking via weighted correlation filter. Infrared Phys Technol 73:103–14

    Google Scholar 

  20. Huang L, Zhao X, Huang K (2021) Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Trans Pattern Anal Mach Intell 43(5):1562–77

  21. Kruse FA, Boardman JW, Huntington JF (2003) Comparison of airborne hyperspectral data and EO-1 Hyperion for mineral mapping. IEEE Trans Geosci Remote Sens 41(6):1388–400

    Google Scholar 

  22. Kong J, Ding Y, Jiang M, Li S (2020) Collaborative model tracking with robust occlusion handling. IET Image Proc 14(9):1701–9

    Google Scholar 

  23. Kiani Galoogahi H, Fagg A, Lucey S (2017) Learning background-aware correlation filters for visual tracking. In Proceedings of the IEEE international conference on computer vision, pp 1135–1143

  24. Kittler J, Hojjatoleslami SA (1998) A weighted combination of classifiers employing shared and distinct representations. In Proceedings. 1998 IEEE computer society conference on computer vision and pattern recognition (Cat. No. 98CB36231). IEEE, pp 924–929

  25. Kumar A, Kim H, Hancke GP (2012) Environmental monitoring systems: a review. IEEE Sens J 13(4):1329–39

    Google Scholar 

  26. Landgrebe D (2002) Hyperspectral image data analysis. IEEE Signal Process Mag 1:17–28

    Google Scholar 

  27. Li P, Chen B, Ouyang W, Wang D, Yang X, Lu H (2019) GradNet: gradient-guided network for visual object tracking. In Proceedings of the IEEE/CVF international conference on computer vision, pp 6162–6171

  28. Liu H, Li B (2020) Target tracker with masked discriminative correlation filter. IET Image Proc 10:2227–34

    Google Scholar 

  29. Li X, Liu Q, Fan N, He Z, Wang H (2019) Hierarchical spatial-aware siamese network for thermal infrared object tracking. Knowl-Based Syst 166:71–81

    Google Scholar 

  30. Liu Q, Lu X, He Z, Zhang C, Chen WS (2017) Deep convolutional neural networks for thermal infrared object tracking. Knowl-Based Syst 134:189–98

    Google Scholar 

  31. Lu X, Li J, He Z, Wang W, Wang H (2019) Distracter-aware tracking via correlation filter. Neurocomputing 348:134–44

    Google Scholar 

  32. Luo W, Li X, Li W, Hu W (2011) Robust visual tracking via transfer learning. In 2011 18th IEEE international conference on image processing. IEEE, pp 485–488

  33. Lei J, Liu P, Xie W, Gao L, Li Y, Du Q (2022) Spatial-spectral cross-correlation embedded dual-transfer network for object tracking using hyperspectral videos. Remote Sens 14(15):3512

    Google Scholar 

  34. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: common objects in context. In computer vision-ECCV 2014: 13th European conference, proceedings, part V 13. Springer International Publishing, Zurich, pp 740–755

  35. Lu S, Shimizu Y, Ishii J, Washitani I, Omasa K (2011) Identification of invasive vegetation using hyperspectral imagery in the shore of the Kinu River, Japan. J Agri Meteor 67(2):85–8

    Google Scholar 

  36. Li F, Tian C, Zuo W, Zhang L, Yang MH (2018) Learning spatial-temporal regularized correlation filters for visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4904–913

  37. Liu Z, Wang X, Shu M, Li G, Sun C, Liu Z, Zhong Y (2021) An anchor-free Siamese target tracking network for hyperspectral video. In 2021 11th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–5

  38. Liu Z, Wang X, Zhong Y, Shu M, Sun C (2022) SiamHYPER: learning a hyperspectral object tracker from an RGB-based tracker. IEEE Trans Image Process 31:7116–29

    Google Scholar 

  39. Li W, Wu G, Zhang F, Du Q (2017) Hyperspectral image classification using deep pixel-pair features. IEEE Trans Geosci Remote Sens 55(2):844–53

    Google Scholar 

  40. Li Z, Xiong F, Zhou J, Wang J, Lu J, Qian Y (2020) BAE-Net: a band attention aware ensemble network for hyperspectral object tracking. In 2020 IEEE international conference on image processing (ICIP). IEEE, pp 2106–2110

  41. Li S, Yeung DY (2017) Visual object tracking for unmanned aerial vehicles: a benchmark and new motion models. In Proceedings of the AAAI conference on artificial intelligence, vol 31 no 1

  42. Li B, Yan J, Wu W, Zhu Z, Hu X (2018) High performance visual tracking with Siamese region proposal network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8971–8980

  43. Li Z, Ye X, Xiong F, Lu J, Zhou J, Qian Y (2021) Spectral-spatial-temporal attention network for hyperspectral tracking. In2021 11th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–5

  44. Lan X, Yang Z, Zhang W, Yuen PC (2021) Spatial-temporal regularized multi-modality correlation filters for tracking with re-detection. ACM Trans Multimed Comput Commun Appl (TOMM) 17(2):1–6

  45. Liu Z, Zhong Y, Wang X, Shu M, Zhang L (2021) Unsupervised deep hyperspectral video target tracking and high spectral-spatial-temporal resolution \((H^3)\) benchmark dataset. IEEE Trans Geosci Remote Sens 60:1–4

  46. McDonald TL (2003) Review of environmental monitoring methods: survey designs. Environ Monit Assess 85:277–92

    Google Scholar 

  47. Moorthy S, Choi JY, Joo YH (2020) Gaussian-response correlation filter for robust visual object tracking. Neurocomputing 411:78–90

    Google Scholar 

  48. Marchal S, Jiang X, State R, Engel T (2014) A big data architecture for large scale security monitoring. In 2014 IEEE international congress on big data. IEEE, pp 56–63

  49. Martinez AM, Kak AC (2001) Pca versus lda. IEEE Trans Pattern Anal Mach Intell 23(2):228–33

    Google Scholar 

  50. Nam H, Han B (2016) Learning multi-domain convolutional neural networks for visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4293–4302

  51. Oliveira LM, Rodrigues JJ (2011) Wireless sensor networks: a survey on environmental monitoring. J Commun 6(2):143–51

    Google Scholar 

  52. Plaza A, Benediktsson JA, Boardman JW, Brazile J, Bruzzone L, Camps-Valls G, Chanussot J, Fauvel M, Gamba P, Gualtieri A, Marconcini M (2009) Recent advances in techniques for hyperspectral image processing. Remote Sens Environ 113:S110-22

    Google Scholar 

  53. Qian K, Chen P, Zhao D (2023) GOMT: multispectral video tracking based on genetic optimization and multi-features integration. IET Image Proc 5:1578–89

    Google Scholar 

  54. Qian K, Zhou J, Xiong F, Zhou H, Du J (2018) Object tracking in hyperspectral videos with convolutional features and kernelized correlation filter. In international conference on smart multimedia. Springer, pp 308–319

  55. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC (2015) Imagenet large scale visual recognition challenge. Int J Comput Vision 115:211–52

    MathSciNet  Google Scholar 

  56. Shen M, Gan H, Ning C, Hua Y, Zhang T (2022) TransCS: a transformer-based hybrid architecture for image compressed sensing. IEEE Trans Image Process 31:6991–7005

    Google Scholar 

  57. Su N, Liu H, Zhao C, Yan Y, Wang J, He J (2022) A transformer-based three-branch Siamese network for hyperspectral object tracking. In 2022 12th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–5

  58. Sanders C, Smith J (2013) Applied network security monitoring: collection, detection, and analysis. Elsevier

    Google Scholar 

  59. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  60. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30

  61. Li W, Hou Z, Zhou J, Tao R (2023) SiamBAG: band attention grouping-based Siamese object tracking network for hyperspectral videos. IEEE Trans Geosci Remote Sens

  62. Liu Y, Zhang Y, Wang Y, Mei S (2023) BiTSRS: a Bi-decoder transformer segmentor for high-spatial-resolution remote sensing images. Remote Sens 15(3):840

    Google Scholar 

  63. Wang S, Qian K, Shen J, Ma H, Chen P (2023) AD-SiamRPN: anti-deformation object tracking via an improved Siamese region proposal network on hyperspectral videos. Remote Sens 15(7):1731

    Google Scholar 

  64. Wang Y, Liu Y, Ma M, Mei S (2023) A spectral; spatial transformer fusion method for hyperspectral video tracking. Remote Sens 15(7):1735

  65. Tao R, Gavves E, Smeulders AW (2016) Siamese instance search for tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1420–1429

  66. Tu B, Kuang W, Zhao G, He D, Liao Z, Ma W (2019) Hyperspectral image classification by combining local binary pattern and joint sparse representation. Int J Remote Sens 40(24):9484–500

    Google Scholar 

  67. Uzkent B, Hoffman MJ, Vodacek A (2015) Spectral validation of measurements in a vehicle tracking DDDAS. Procedia Comput Sci 51:2493–502

    Google Scholar 

  68. Uzkent B, Hoffman MJ, Vodacek A (2016) Real-time vehicle tracking in aerial video using hyperspectral features. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 36–44

  69. Uzkent B, Rangnekar A, Hoffman M (2017) Aerial vehicle tracking by adaptive fusion of hyperspectral likelihood maps. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 39–48

  70. Uzkent B, Rangnekar A, Hoffman MJ (2018) Tracking in aerial hyperspectral videos using deep kernelized correlation filters. IEEE Trans Geosci Remote Sens 57(1):449–61

    Google Scholar 

  71. Valmadre J, Bertinetto L, Henriques J, Vedaldi A, Torr PH (2017) End-to-end representation learning for correlation filter based tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2805–2813

  72. Voigtlaender P, Luiten J, Torr PH, Leibe B (2020) Siam r-cnn: visual tracking by re-detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6578–6588

  73. Van Nguyen H, Banerjee A, Chellappa R (2010) Tracking via object reflectance using a hyperspectral video camera. In 2010 IEEE computer society conference on computer vision and pattern recognition-workshops. IEEE, pp 44–51

  74. Vasile M, Walker L, Dunphy RD, Zabalza J, Murray P, Marshall S, Savitski V (2022) Intelligent characterisation of space objects with hyperspectral imaging. Acta Astronaut 203:510–34

    Google Scholar 

  75. Wei B, Chen H, Ding Q, Luo H (2022) SiamOAN: Siamese object-aware network for real-time target tracking. Neurocomputing 471:161–74

    Google Scholar 

  76. Wang S, Jia D, Weng X (2018) Deep reinforcement learning for autonomous driving. arXiv:1811.11329

  77. Wu Y, Lim J, Yang MH (2013) Online object tracking: a benchmark. In Proceedings of the IEEE conference on computer vision and pattern recognition 2013, pp 2411–2418

  78. Wang Y, Liu Y, Zhang G, Su Y, Zhang S, Mei S (2022) Spectral-spatial-aware transformer fusion network for hyperspectral object tracking. In2022 12th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–5

  79. Wang L, Ouyang W, Wang X, Lu H (2015) Visual tracking with fully convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp 3119–3127

  80. Wang S, Qian K, Chen P (2022) BS-SiamRPN: hyperspectral video tracking based on band selection and the Siamese region proposal network. In 2022 12th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–8

  81. Wang N, Yeung DY (2013) Learning a deep compact image representation for visual tracking. Adv Neural Inf Process Syst 26

  82. Wang N, Zhou W, Tian Q, Hong R, Wang M, Li H (2018) Multi-cue correlation filters for robust visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4844–4853

  83. Xiu C, Chai Z (2019) Target tracking based on the cognitive associative network. IET Image Proc 3:498–505

    Google Scholar 

  84. Xijun L, Jun L (2009) An adaptive band selection algorithm for dimension reduction of hyperspectral images. In 2009 international conference on image analysis and signal processing. IEEE, pp 114–118

  85. Xiong F, Zhou J, Qian Y (2020) Material based object tracking in hyperspectral videos. IEEE Trans Image Process 29:3719–33

    Google Scholar 

  86. Yurtsever E, Lambert J, Carballo A, Takeda K (2020) A survey of autonomous driving: common practices and emerging technologies. IEEE Access 8:58443–69

    Google Scholar 

  87. Yu H, Li G, Zhang W, Huang Q, Du D, Tian Q, Sebe N (2020) The unmanned aerial vehicle benchmark: object detection, tracking and baseline. Int J Comput Vision 128:1141–59

    Google Scholar 

  88. Yin Z, Porikli F, Collins RT (2008) Likelihood map fusion for visual object tracking. In 2008 IEEE workshop on applications of computer vision. IEEE, pp 1–7

  89. Yang J, Zhang D, Frangi AF, Yang JY (2004) Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Trans Pattern Anal Mach Intell 26(1):131–7

    Google Scholar 

  90. Zhao H, Bruzzone L, Guan R, Zhou F, Yang C (2021) Spectral-spatial genetic algorithm-based unsupervised band selection for hyperspectral image classification. IEEE Trans Geosci Remote Sens 59(11):9616–32

    Google Scholar 

  91. Zhao D, Cao J, Zhu X, Zhang Z, Arun PV, Guo Y, Qian K, Zhang L, Zhou H, Hu J (2022) Hyperspectral video target tracking based on deep edge convolution feature and improved context filter. Remote Sens 14(24):6219

    Google Scholar 

  92. Zhao C, Liu H, Su N, Wang L, Yan Y (2022) RANet: a reliability-guided aggregation network for hyperspectral and RGB fusion tracking. Remote Sens 14(12):2765

    Google Scholar 

  93. Zhang Y, Li X, Wang F, Wei B, Li L (2022) A fast hyperspectral object tracking method based on channel selection strategy. In 2022 12th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–5

  94. Zhang Z, Peng H, Fu J, Li B, Hu W (2020) Ocean: object-aware anchor-free tracking. In European conference on computer vision. Springer, pp 771–787

  95. Zhang Z, Qian K, Du J, Zhou H (2021) Multi-features integration based hyperspectral videos tracker. In 2021 11th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–5

  96. Zhang T, Quan S, Yang Z, Guo W, Zhang Z, Gan H (2022) A two-stage method for ship detection using PolSAR image. IEEE Trans Geosci Remote Sens 60:1–8

    Google Scholar 

  97. Zhang L, Suganthan PN (2017) Robust visual tracking via co-trained kernelized correlation filters. Pattern Recogn 69:82–93

    Google Scholar 

  98. Zhu Z, Wang Q, Li B, Wu W, Yan J, Hu W (2018) Distractor-aware Siamese networks for visual object tracking. In Proceedings of the European conference on computer vision (ECCV), pp 101–117

  99. Zhang K, Zhang L, Liu Q, Zhang D, Yang MH (2014) Fast visual tracking via dense spatio-temporal context learning. In European conference on computer vision. Springer, pp 127–141

  100. Zhang Z, Zhu X, Zhao D, Arun PV, Zhou H, Qian K, Hu J (2022) Hyperspectral video target tracking based on deep features with spectral matching reduction and adaptive scale 3d hog features. Remote Sens 14(23)

Download references

Acknowledgements

We are grateful to the workers who have made contributions to hyperspectral video tracking for providing us with novel tracking methods and accurate experimental results. This work is partially supported by the Fundamental Research Funds for the Central Universities (JUSRP121072), Jiangsu Engineering Research Center of Digital Twinning Technology for Key Equipment in Petrochemical Process (DTEC202202), the International Science and Technology Cooperation Project of Jiangsu Province (BZ2020069), and the Major Program of University Natural Science Research of Jiangsu Province (21KJA520001).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Kun Qian.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qian, K., Shen, J., Wang, S. et al. Recent advances in object tracking using hyperspectral videos: a survey. Multimed Tools Appl 83, 56155–56181 (2024). https://doi.org/10.1007/s11042-023-17758-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-17758-9

Keywords

Navigation