Abstract
Short-Term Single-Object (STSO) tracking using Hyperspectral Videos (HSVs), which has become a hotspot recently, is a challenging task. Hyperspectral Object Tracking (HOT) makes full use of spatial and spectral information during the tracking process. In HOT, multiple features, including deep network features, have been combined with correlation filter methods, which increases time-consuming efficiency and tracking accuracy. However, redundant spectral information needs to be obtained in an effective way. In addition, there is currently no detailed investigation of HOT algorithms. Therefore, this survey studies the development of HOT algorithms in recent years. Specifically, several HSVs are listed, an investigation of HOT algorithms is conducted, and components of HOT are described in detail. Furthermore, several popular HOT algorithms, including our previous work BS-SiamPRN and AD-SiamRPN, are compared quantitatively and qualitatively. Finally, the research status of HOT is summarized, and future work has been described, which lays the foundation for future HOT or STSO referring to HSVs.
Similar content being viewed by others
Data Availibility Statement
Data are available from the corresponding author upon reasonable request.
References
José Manuel A, Hamid B, Saioa E (2015) Hyperspectral image analysis. A tutorial. Anal Chim Acta 896:34–51
Anderson GL, Carruthers RI, Shaokui G, Peng G (2005) Cover: monitoring of invasive Tamarix distribution and effects of biological control with airborne hyperspectral remote sensing. Int J Remote Sens 26(12):2487–2489
Banerjee A, Burlina P, Broadwater J (2009) Hyperspectral video for illumination-invariant tracking. In 009 first workshop on hyperspectral image and signal processing: evolution in remote sensing. IEEE, pp 1–4
Bolme DS, Beveridge JR, Draper BA, Lui YM (2010) Visual object tracking using adaptive correlation filters. In 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, pp 2544–2550
Barquero G, Hupont I, Tena CF (2021) Rank-based verification for long-term face tracking in crowded scenes. arXiv:2107.13273
Bottou L (2012) Stochastic gradient descent tricks. In Neural networks: tricks of the trade: second edition. Berlin, Heidelberg, pp 421–436
Bertinetto L, Valmadre J, Henriques JF, Vedaldi A, Torr PH (2016) Fully-convolutional siamese networks for object tracking. In European conference on computer vision. Springer, pp 850–865
Chang CI (2016) Real-time progressive hyperspectral image processing. Springer
Chen L, Zhao Y, Yao J, Chen J, Li N, Chan JC, Kong SG (2021) Object tracking in hyperspectral-oriented video with fast spatial-spectral features. Remote Sensing 13(10):1922
Danelljan M, Hager G, Shahbaz Khan F, Felsberg M (2015) Learning spatially regularized correlation filters for visual tracking. In Proceedings of the IEEE international conference on computer vision, pp 4310–4318
Du D, Qi Y, Yu H, Yang Y, Duan K, Li G, Zhang W, Huang Q, Tian Q (2018) The unmanned aerial vehicle benchmark: object detection and tracking. In Proceedings of the European conference on computer vision (ECCV), pp 370–386
Danelljan M, Shahbaz Khan F, Felsberg M, Van de Weijer J (2014) Adaptive color attributes for real-time visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1090–1097
Guo Q, Feng W, Zhou C, Huang R, Wan L, Wang S (2017) Learning dynamic siamese network for visual object tracking. In Proceedings of the IEEE international conference on computer vision, pp 1763–1771
Ghafir I, Prenosil V, Svoboda J, Hammoudeh M (2016) A survey on network security monitoring systems. In 2016 IEEE 4th international conference on future internet of things and cloud workshops (FiCloudW). IEEE, pp 77–82
Guo D, Wang J, Cui Y, Wang Z, Chen S (2020) SiamCAR: siamese fully convolutional classification and regression for visual tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6269–6277
Ghamisi P, Yokoya N, Li J, Liao W, Liu S, Plaza J, Rasti B, Plaza A (2017) Advances in hyperspectral image and signal processing: a comprehensive overview of the state of the art. IEEE Geosci Remote Sens Magazine 5(4):37–78
Henriques JF, Caseiro R, Martins P, Batista J (2014) High-speed tracking with kernelized correlation filters. IEEE Trans Pattern Anal Mach Intell 37(3):583–96
Hou Z, Li W, Zhou J, Tao R (2022) Spatial-spectral weighted and regularized tensor sparse correlation filter for object tracking in hyperspectral videos. IEEE Trans Geosci Remote Sens 60:1–2
He YJ, Li M, Zhang J, Yao JP (2015) Infrared target tracking via weighted correlation filter. Infrared Phys Technol 73:103–14
Huang L, Zhao X, Huang K (2021) Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Trans Pattern Anal Mach Intell 43(5):1562–77
Kruse FA, Boardman JW, Huntington JF (2003) Comparison of airborne hyperspectral data and EO-1 Hyperion for mineral mapping. IEEE Trans Geosci Remote Sens 41(6):1388–400
Kong J, Ding Y, Jiang M, Li S (2020) Collaborative model tracking with robust occlusion handling. IET Image Proc 14(9):1701–9
Kiani Galoogahi H, Fagg A, Lucey S (2017) Learning background-aware correlation filters for visual tracking. In Proceedings of the IEEE international conference on computer vision, pp 1135–1143
Kittler J, Hojjatoleslami SA (1998) A weighted combination of classifiers employing shared and distinct representations. In Proceedings. 1998 IEEE computer society conference on computer vision and pattern recognition (Cat. No. 98CB36231). IEEE, pp 924–929
Kumar A, Kim H, Hancke GP (2012) Environmental monitoring systems: a review. IEEE Sens J 13(4):1329–39
Landgrebe D (2002) Hyperspectral image data analysis. IEEE Signal Process Mag 1:17–28
Li P, Chen B, Ouyang W, Wang D, Yang X, Lu H (2019) GradNet: gradient-guided network for visual object tracking. In Proceedings of the IEEE/CVF international conference on computer vision, pp 6162–6171
Liu H, Li B (2020) Target tracker with masked discriminative correlation filter. IET Image Proc 10:2227–34
Li X, Liu Q, Fan N, He Z, Wang H (2019) Hierarchical spatial-aware siamese network for thermal infrared object tracking. Knowl-Based Syst 166:71–81
Liu Q, Lu X, He Z, Zhang C, Chen WS (2017) Deep convolutional neural networks for thermal infrared object tracking. Knowl-Based Syst 134:189–98
Lu X, Li J, He Z, Wang W, Wang H (2019) Distracter-aware tracking via correlation filter. Neurocomputing 348:134–44
Luo W, Li X, Li W, Hu W (2011) Robust visual tracking via transfer learning. In 2011 18th IEEE international conference on image processing. IEEE, pp 485–488
Lei J, Liu P, Xie W, Gao L, Li Y, Du Q (2022) Spatial-spectral cross-correlation embedded dual-transfer network for object tracking using hyperspectral videos. Remote Sens 14(15):3512
Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: common objects in context. In computer vision-ECCV 2014: 13th European conference, proceedings, part V 13. Springer International Publishing, Zurich, pp 740–755
Lu S, Shimizu Y, Ishii J, Washitani I, Omasa K (2011) Identification of invasive vegetation using hyperspectral imagery in the shore of the Kinu River, Japan. J Agri Meteor 67(2):85–8
Li F, Tian C, Zuo W, Zhang L, Yang MH (2018) Learning spatial-temporal regularized correlation filters for visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4904–913
Liu Z, Wang X, Shu M, Li G, Sun C, Liu Z, Zhong Y (2021) An anchor-free Siamese target tracking network for hyperspectral video. In 2021 11th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–5
Liu Z, Wang X, Zhong Y, Shu M, Sun C (2022) SiamHYPER: learning a hyperspectral object tracker from an RGB-based tracker. IEEE Trans Image Process 31:7116–29
Li W, Wu G, Zhang F, Du Q (2017) Hyperspectral image classification using deep pixel-pair features. IEEE Trans Geosci Remote Sens 55(2):844–53
Li Z, Xiong F, Zhou J, Wang J, Lu J, Qian Y (2020) BAE-Net: a band attention aware ensemble network for hyperspectral object tracking. In 2020 IEEE international conference on image processing (ICIP). IEEE, pp 2106–2110
Li S, Yeung DY (2017) Visual object tracking for unmanned aerial vehicles: a benchmark and new motion models. In Proceedings of the AAAI conference on artificial intelligence, vol 31 no 1
Li B, Yan J, Wu W, Zhu Z, Hu X (2018) High performance visual tracking with Siamese region proposal network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8971–8980
Li Z, Ye X, Xiong F, Lu J, Zhou J, Qian Y (2021) Spectral-spatial-temporal attention network for hyperspectral tracking. In2021 11th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–5
Lan X, Yang Z, Zhang W, Yuen PC (2021) Spatial-temporal regularized multi-modality correlation filters for tracking with re-detection. ACM Trans Multimed Comput Commun Appl (TOMM) 17(2):1–6
Liu Z, Zhong Y, Wang X, Shu M, Zhang L (2021) Unsupervised deep hyperspectral video target tracking and high spectral-spatial-temporal resolution \((H^3)\) benchmark dataset. IEEE Trans Geosci Remote Sens 60:1–4
McDonald TL (2003) Review of environmental monitoring methods: survey designs. Environ Monit Assess 85:277–92
Moorthy S, Choi JY, Joo YH (2020) Gaussian-response correlation filter for robust visual object tracking. Neurocomputing 411:78–90
Marchal S, Jiang X, State R, Engel T (2014) A big data architecture for large scale security monitoring. In 2014 IEEE international congress on big data. IEEE, pp 56–63
Martinez AM, Kak AC (2001) Pca versus lda. IEEE Trans Pattern Anal Mach Intell 23(2):228–33
Nam H, Han B (2016) Learning multi-domain convolutional neural networks for visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4293–4302
Oliveira LM, Rodrigues JJ (2011) Wireless sensor networks: a survey on environmental monitoring. J Commun 6(2):143–51
Plaza A, Benediktsson JA, Boardman JW, Brazile J, Bruzzone L, Camps-Valls G, Chanussot J, Fauvel M, Gamba P, Gualtieri A, Marconcini M (2009) Recent advances in techniques for hyperspectral image processing. Remote Sens Environ 113:S110-22
Qian K, Chen P, Zhao D (2023) GOMT: multispectral video tracking based on genetic optimization and multi-features integration. IET Image Proc 5:1578–89
Qian K, Zhou J, Xiong F, Zhou H, Du J (2018) Object tracking in hyperspectral videos with convolutional features and kernelized correlation filter. In international conference on smart multimedia. Springer, pp 308–319
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC (2015) Imagenet large scale visual recognition challenge. Int J Comput Vision 115:211–52
Shen M, Gan H, Ning C, Hua Y, Zhang T (2022) TransCS: a transformer-based hybrid architecture for image compressed sensing. IEEE Trans Image Process 31:6991–7005
Su N, Liu H, Zhao C, Yan Y, Wang J, He J (2022) A transformer-based three-branch Siamese network for hyperspectral object tracking. In 2022 12th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–5
Sanders C, Smith J (2013) Applied network security monitoring: collection, detection, and analysis. Elsevier
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30
Li W, Hou Z, Zhou J, Tao R (2023) SiamBAG: band attention grouping-based Siamese object tracking network for hyperspectral videos. IEEE Trans Geosci Remote Sens
Liu Y, Zhang Y, Wang Y, Mei S (2023) BiTSRS: a Bi-decoder transformer segmentor for high-spatial-resolution remote sensing images. Remote Sens 15(3):840
Wang S, Qian K, Shen J, Ma H, Chen P (2023) AD-SiamRPN: anti-deformation object tracking via an improved Siamese region proposal network on hyperspectral videos. Remote Sens 15(7):1731
Wang Y, Liu Y, Ma M, Mei S (2023) A spectral; spatial transformer fusion method for hyperspectral video tracking. Remote Sens 15(7):1735
Tao R, Gavves E, Smeulders AW (2016) Siamese instance search for tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1420–1429
Tu B, Kuang W, Zhao G, He D, Liao Z, Ma W (2019) Hyperspectral image classification by combining local binary pattern and joint sparse representation. Int J Remote Sens 40(24):9484–500
Uzkent B, Hoffman MJ, Vodacek A (2015) Spectral validation of measurements in a vehicle tracking DDDAS. Procedia Comput Sci 51:2493–502
Uzkent B, Hoffman MJ, Vodacek A (2016) Real-time vehicle tracking in aerial video using hyperspectral features. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 36–44
Uzkent B, Rangnekar A, Hoffman M (2017) Aerial vehicle tracking by adaptive fusion of hyperspectral likelihood maps. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 39–48
Uzkent B, Rangnekar A, Hoffman MJ (2018) Tracking in aerial hyperspectral videos using deep kernelized correlation filters. IEEE Trans Geosci Remote Sens 57(1):449–61
Valmadre J, Bertinetto L, Henriques J, Vedaldi A, Torr PH (2017) End-to-end representation learning for correlation filter based tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2805–2813
Voigtlaender P, Luiten J, Torr PH, Leibe B (2020) Siam r-cnn: visual tracking by re-detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6578–6588
Van Nguyen H, Banerjee A, Chellappa R (2010) Tracking via object reflectance using a hyperspectral video camera. In 2010 IEEE computer society conference on computer vision and pattern recognition-workshops. IEEE, pp 44–51
Vasile M, Walker L, Dunphy RD, Zabalza J, Murray P, Marshall S, Savitski V (2022) Intelligent characterisation of space objects with hyperspectral imaging. Acta Astronaut 203:510–34
Wei B, Chen H, Ding Q, Luo H (2022) SiamOAN: Siamese object-aware network for real-time target tracking. Neurocomputing 471:161–74
Wang S, Jia D, Weng X (2018) Deep reinforcement learning for autonomous driving. arXiv:1811.11329
Wu Y, Lim J, Yang MH (2013) Online object tracking: a benchmark. In Proceedings of the IEEE conference on computer vision and pattern recognition 2013, pp 2411–2418
Wang Y, Liu Y, Zhang G, Su Y, Zhang S, Mei S (2022) Spectral-spatial-aware transformer fusion network for hyperspectral object tracking. In2022 12th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–5
Wang L, Ouyang W, Wang X, Lu H (2015) Visual tracking with fully convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp 3119–3127
Wang S, Qian K, Chen P (2022) BS-SiamRPN: hyperspectral video tracking based on band selection and the Siamese region proposal network. In 2022 12th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–8
Wang N, Yeung DY (2013) Learning a deep compact image representation for visual tracking. Adv Neural Inf Process Syst 26
Wang N, Zhou W, Tian Q, Hong R, Wang M, Li H (2018) Multi-cue correlation filters for robust visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4844–4853
Xiu C, Chai Z (2019) Target tracking based on the cognitive associative network. IET Image Proc 3:498–505
Xijun L, Jun L (2009) An adaptive band selection algorithm for dimension reduction of hyperspectral images. In 2009 international conference on image analysis and signal processing. IEEE, pp 114–118
Xiong F, Zhou J, Qian Y (2020) Material based object tracking in hyperspectral videos. IEEE Trans Image Process 29:3719–33
Yurtsever E, Lambert J, Carballo A, Takeda K (2020) A survey of autonomous driving: common practices and emerging technologies. IEEE Access 8:58443–69
Yu H, Li G, Zhang W, Huang Q, Du D, Tian Q, Sebe N (2020) The unmanned aerial vehicle benchmark: object detection, tracking and baseline. Int J Comput Vision 128:1141–59
Yin Z, Porikli F, Collins RT (2008) Likelihood map fusion for visual object tracking. In 2008 IEEE workshop on applications of computer vision. IEEE, pp 1–7
Yang J, Zhang D, Frangi AF, Yang JY (2004) Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Trans Pattern Anal Mach Intell 26(1):131–7
Zhao H, Bruzzone L, Guan R, Zhou F, Yang C (2021) Spectral-spatial genetic algorithm-based unsupervised band selection for hyperspectral image classification. IEEE Trans Geosci Remote Sens 59(11):9616–32
Zhao D, Cao J, Zhu X, Zhang Z, Arun PV, Guo Y, Qian K, Zhang L, Zhou H, Hu J (2022) Hyperspectral video target tracking based on deep edge convolution feature and improved context filter. Remote Sens 14(24):6219
Zhao C, Liu H, Su N, Wang L, Yan Y (2022) RANet: a reliability-guided aggregation network for hyperspectral and RGB fusion tracking. Remote Sens 14(12):2765
Zhang Y, Li X, Wang F, Wei B, Li L (2022) A fast hyperspectral object tracking method based on channel selection strategy. In 2022 12th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–5
Zhang Z, Peng H, Fu J, Li B, Hu W (2020) Ocean: object-aware anchor-free tracking. In European conference on computer vision. Springer, pp 771–787
Zhang Z, Qian K, Du J, Zhou H (2021) Multi-features integration based hyperspectral videos tracker. In 2021 11th workshop on hyperspectral imaging and signal processing: evolution in remote sensing (WHISPERS). IEEE, pp 1–5
Zhang T, Quan S, Yang Z, Guo W, Zhang Z, Gan H (2022) A two-stage method for ship detection using PolSAR image. IEEE Trans Geosci Remote Sens 60:1–8
Zhang L, Suganthan PN (2017) Robust visual tracking via co-trained kernelized correlation filters. Pattern Recogn 69:82–93
Zhu Z, Wang Q, Li B, Wu W, Yan J, Hu W (2018) Distractor-aware Siamese networks for visual object tracking. In Proceedings of the European conference on computer vision (ECCV), pp 101–117
Zhang K, Zhang L, Liu Q, Zhang D, Yang MH (2014) Fast visual tracking via dense spatio-temporal context learning. In European conference on computer vision. Springer, pp 127–141
Zhang Z, Zhu X, Zhao D, Arun PV, Zhou H, Qian K, Hu J (2022) Hyperspectral video target tracking based on deep features with spectral matching reduction and adaptive scale 3d hog features. Remote Sens 14(23)
Acknowledgements
We are grateful to the workers who have made contributions to hyperspectral video tracking for providing us with novel tracking methods and accurate experimental results. This work is partially supported by the Fundamental Research Funds for the Central Universities (JUSRP121072), Jiangsu Engineering Research Center of Digital Twinning Technology for Key Equipment in Petrochemical Process (DTEC202202), the International Science and Technology Cooperation Project of Jiangsu Province (BZ2020069), and the Major Program of University Natural Science Research of Jiangsu Province (21KJA520001).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Qian, K., Shen, J., Wang, S. et al. Recent advances in object tracking using hyperspectral videos: a survey. Multimed Tools Appl 83, 56155–56181 (2024). https://doi.org/10.1007/s11042-023-17758-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-17758-9