Skip to main content
SpringerLink
Log in

Learning background-aware and spatial-temporal regularized correlation filters for visual tracking

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In visual tracking, correlation Filters (CFs) have attracted increasing research attention and achieved superior performance. However, owing to the larger search area, more background information is introduced to the shifted samples, meaning that tracking errors are prone to appear in the detection stage. Accordingly, in this work, firstly, hand-crafted features and deep features extracted from pre-trained convolutional networks are combined to improve the representation ability of object appearance. For deep features, we use two different VGG networks for extraction. Secondly, in an attempt to solve the problem of the object background of the traditional CF model not being modeled over time, and owing to the lack of spatial-temporal information of the image, we propose a new background-aware and spatial-temporal regularized correlation filters model (BSTCF) that introduces the background constraint and spatial-temporal regularization. The proposed BSTCF can effectively model not only the background but also variations in the background over time. Finally, we transform the objective function of BSTCF into an unconstrained Augmented Lagrange multiplier formular to promote convergence to the global optimum solution. Moreover, we adopt the alternating direction multiplier method (ADMM) to produce three sub-problems with closed-form solution, then propose a corresponding algorithm. Based on the above, we construct an intelligent tracking system and carry out extensive experiments to test its performance on OTB-2013, OTB-2015, TC128, UAV123, and VOT2016 public datasets. The experimental results demonstrate that the tracking algorithm achieves superior performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Zhang J, Yuan T, He Y, Wang J (2022) A background-aware correlation filter with adaptive saliency-aware regularization for visual tracking. Neural Comput Appl 34(8):6359–5376. https://doi.org/10.1007/s00521-021-06771-4

    Article  Google Scholar 

  2. Li C, Sun X, Wang X, Zhang L, Tang J (2017) Grayscale-thermal object tracking via multitask laplacian sparse representation. IEEE Trans Syst Man Cybern: Syst 47(4):673–681

    Article  Google Scholar 

  3. Zhang B, Li Z, Cao X, Ye Q, Chen C, Shen L, Perina A, Jill R (2017) Output constraint transfer for kernelized correlation filter in tracking. IEEE Trans Syst Man Cybern: Syst 47(4):693–703. https://doi.org/10.1109/TSMC.2016.2629509

    Article  Google Scholar 

  4. Zhou Z, Zhang W, Zhao J (2019) Robust visual tracking using discriminative sparse collaborative map. Int J Mach Learn Cyber 10(11):3201–3212

    Article  Google Scholar 

  5. Bruni V, Vitulano D (2014) An improvement of kernel-based object tracking based on human perception. IEEE Trans Syst Man Cybern: Syst 44(11):1474–1485

    Article  Google Scholar 

  6. Zhang J, Sun J, Wang J, Li Z, Chen X (2022) An object tracking framework with recapture based on correlation filters and siamese networks. Comput Electr Eng 98:107730. https://doi.org/10.1016/j.compeleceng.2022.107730

    Article  Google Scholar 

  7. Danelljan M, Hager G, Shahbaz KF, Felsberg M (2015) Learning spatially regularized correlation filters for visual tracking. In: Proceedings of the IEEE international conference on computer vision, pp 4310–4318

  8. Cheng X, Zhang Y, Cui J, Zhou L (2016) Object tracking via temporal consistency dictionary learning. IEEE Trans Systems Man Cybern: Syst 47(4):628–638

    Article  Google Scholar 

  9. Li F, Tian C, Zuo W, Zhang L, Yang M (2018) Learning spatial-temporal regularized correlation filters for visual tracking. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4904–4913

  10. Ma C, Huang J, Yang X, Yang M (2015) Hierarchical convolutional features for visual tracking. In: Proceedings of the IEEE international conference on computer vision, pp 3074–3082

  11. Zhang J, Sun J, Wang J, Yue X (2021) Visual object tracking based on residual network and cascaded correlation filters. J Ambient Intell Humanized Comput 12(8):8427–8440

    Article  Google Scholar 

  12. Zhang J, Feng W, Yuan T, Wang J, Sangaiah A (2022) Scstcf: spatial-channel selection and temporal regularized correlation filters for visual tracking. Appl Soft Comput 118:108485. https://doi.org/10.1016/j.asoc.2022.108485

    Article  Google Scholar 

  13. Danelljan M, Robinson A, Khan FS, Felsberg M (2016) Beyond correlation filters: learning continuous convolution operators for visual tracking. In: European conference on computer vision. Springer, pp 472–488

  14. Henriques JF, Caseiro R, Martins P, Batista J (2014) High-speed tracking with kernelized correlation filters. IEEE Trans Pattern Anal Mach Intell 37(3):583–596

    Article  Google Scholar 

  15. Galoogahi HK, Fagg A, Lucey S (2017) Learning background-aware correlation filters for visual tracking. In: 2017 IEEE international conference on computer vision (ICCV), pp 1144–1152. https://doi.org/10.1109/ICCV.2017.129

  16. Wu Y, Lim J, Yang M (2013) Online object tracking: a benchmark. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2411–2418

  17. Wu Y, Lim J, Yang M (2015) Object tracking benchmark. IEEE Trans Pattern Anal Mach Intell 37:1–1. https://doi.org/10.1109/TPAMI.2014.2388226

    Article  Google Scholar 

  18. Liang P, Blasch E, Ling H (2015) Encoding color information for visual tracking: algorithms and benchmark. IEEE Trans Image Process 24(12):5630–5644

    Article  MathSciNet  MATH  Google Scholar 

  19. Mueller M, Smith N, Ghanem B (2016) A benchmark and simulator for uav tracking. In: European conference on computer vision. Springer, pp 445–461

  20. Kristan M et al (2016) The visual object tracking vot2016 challenge results. In: Hua G, Jégou H (eds) Computer vision – ECCV 2016 workshops. Springer, pp 777–823

  21. Zhang J, Jin X, Sun J, Wang J, Sangaiah AK (2020) Spatial and semantic convolutional features for robust visual object tracking. Multimed Tools Appl 79(21):15095–15115

    Article  Google Scholar 

  22. Danelljan M, Häger G, Khan FS, Felsberg M (2015) Convolutional features for correlation filter based visual tracking. In: 2015 IEEE International conference on computer vision workshop (ICCVW), pp 621–629. https://doi.org/10.1109/ICCVW.2015.84https://doi.org/10.1109/ICCVW.2015.84

  23. Danelljan M, Bhat G, Khan FS, Felsberg M (2017) Eco: efficient convolution operators for tracking. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 6931–6939. https://doi.org/10.1109/CVPR.2017.733

  24. Nam H, Han B (2016) Learning multi-domain convolutional neural networks for visual tracking. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4293–4302

  25. Bertinetto L, Valmadre J, Henriques JF, Vedaldi A, Torr PHS (2016) Fully-convolutional siamese networks for object tracking. In: European conference on computer vision. Springer, pp 850–865

  26. Guo Q, Feng W, Zhou C, Huang R, Wan L, Wang S (2017) Learning dynamic siamese network for visual object tracking. In: 2017 IEEE international conference on computer vision (ICCV), pp 1781–1789. https://doi.org/10.1109/ICCV.2017.196

  27. Li B, Yan J, Wu W, Zhu Z, Hu X (2018) High performance visual tracking with siamese region proposal network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8971–8980

  28. Zhang J, Huang B, Ye Z, Kuang L, Ning X (2021) Siamese anchor-free object tracking with multiscale spatial attentions. Scientific Reports 11(1):22908. https://doi.org/10.1038/s41598-021-02095-4

    Article  Google Scholar 

  29. Valmadre J, Bertinetto L, Henriques J, Vedaldi A, Torr PHS (2017) End-to-end representation learning for correlation filter based tracking. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 5000–5008. https://doi.org/10.1109/CVPR.2017.531

  30. Bolme DS, Beveridge JR, Draper BA, Lui YM (2010) Visual object tracking using adaptive correlation filters. In: 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, pp 2544–2550

  31. Henriques JF, Caseiro R, Martins P, Batista J (2012) Exploiting the circulant structure of tracking-by-detection with kernels. In: European conference on computer vision. Springer, pp 702–715

  32. Bertinetto L, Valmadre J, Golodetz S, Miksik O, Torr PHS (2016) Staple: complementary learners for real-time tracking. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1401–1409

  33. Danelljan M, Häger G, Khan F, Felsberg M (2014) Accurate scale estimation for robust visual tracking. In: British machine vision conference, Nottingham. Bmva Press, 1-5 September, 2014

  34. Li Y, Zhu J (2014) A scale adaptive kernel correlation filter tracker with feature integration. In: European conference on computer vision. Springer, pp 254–265

  35. Bhat G, Johnander J, Danelljan M, Khan FS, Felsberg M (2018) Unveiling the power of deep tracking. In: Computer vision – ECCV 2018, pp 493–509

  36. Boyd S, Parikh N, Chu E (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Now Publishers Inc

  37. Wang N, Zhou W, Song Y, Ma J, Liu W, Liu H (2021) Unsupervised deep representation learning for real-time tracking. Int J Comput Vis 129(2):400–418. https://doi.org/10.1007/s11263-020-01357-4

    Article  Google Scholar 

  38. Li P, Chen B, Ouyang W, Wang D, Yang X, Lu H (2019) GradNet: gradient-guided network for visual object tracking. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 6161–6170. https://doi.org/10.1109/ICCV.2019.00626

  39. Zhang T, Xu C, Yang M (2017) Multi-task correlation particle filter for robust object tracking. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 4819–4827. https://doi.org/10.1109/CVPR.2017.512

  40. Feng S, Hu K, Fan E, Zhao L, Wu C (2021) Kalman filter for spatial-temporal regularized correlation filters. IEEE Trans Image Process 30:3263–3278. https://doi.org/10.1109/TIP.2021.3060164

    Article  MathSciNet  Google Scholar 

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China under Grant 61972056, the Basic Research Fund of Zhongye Changtian International Engineering Co., Ltd. under Grant 2020JCYJ07, the Postgraduate Training Innovation Base Construction Project of Hunan Province under Grant 2019-248-51, the “Double First-class” International Cooperation and Development Scientific Research Project of CSUST under Grant 2019IC34, the Enterprise-University Joint Postgraduate Scientific Research Innovation Fund of Hunan Province under Grant QL20210205, and the Postgraduate Scientific Research Innovation Fund of CSUST under Grant CX2021SS70.

Author information

Authors and Affiliations

  1. Hunan Provincial Key Laboratory of Intelligent Processing of Big Data on Transportation, School of Computer and Communication Engineering, Changsha University of Science and Technology, Changsha, 410114, China

    Jianming Zhang, Yaoqi He, Wenjun Feng & Jin Wang

  2. Department of Mathematics and Computer Science, Northeastern State University, Tahlequah, OK 74464, USA

    Neal N. Xiong

Authors
  1. Jianming Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yaoqi He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenjun Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Neal N. Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianming Zhang.

Ethics declarations

Conflict of Interests

The authors declare no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., He, Y., Feng, W. et al. Learning background-aware and spatial-temporal regularized correlation filters for visual tracking. Appl Intell 53, 7697–7712 (2023). https://doi.org/10.1007/s10489-022-03868-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03868-8

Keywords

Search

Navigation