Skip to main content
SpringerLink
Log in

Color-saliency-aware correlation filters with approximate affine transform for visual tracking

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Aspects like deformation and occlusion are still the challenge cases which will result in failures of visual tracking. Many existing correlation filters (CFs) try to fuse the color information to improve the performance but ignore the sensitivity of color information to the background interference. For this case, we propose a color-saliency-aware correlation filter which exploits the color statistics as the model of image boundary connectivity cues. The proposed method limits the drift of correlation filter because of the saliency proposal. In addition, the bounding boxes of CFs absorb too much background information which can easily lead to the tracking failure. To solve this problem, we also present a decoupled-Fourier-Mellin (DFM) transform which is related to the independence of scale variations in log-polar coordinates. In addition to the rotation angle, the proposed DFM can also gain the scale factors of both horizontal and vertical directions, and the larger search space (5-DoF) is closer to the upper bound of object masks. Ultimately, multiple popular benchmarks demonstrate the superiority of our tracker. Compared with the current advanced CFs, our method achieves better performance, which is of great significance for continuous tasks requiring high DoF information, such as manipulator visual servo.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Abbass, M.Y., Kwon, K.C., Kim, N., Abdelwahab, S.A., El-Samie, F.E., Khalaf, A.A.: Efficient object tracking using hierarchical convolutional features model and correlation filters. Vis. Comput. 37(4), 831–842 (2021). https://doi.org/10.1007/s00371-020-01833-5

    Article  Google Scholar 

  2. Bertinetto, L., Valmadre, J., Golodetz, S., Miksik, O., Torr, P.: Staple: complementary learners for real-time tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1401–1409. arxiv:1512.01355 (2015)

  3. Bertinetto, L., Valmadre, J., Henriques, J.F., Vedaldi, A., Torr, P.H.S.: Fully-convolutional siamese networks for object tracking. In: Hua, G., Jégou, H. (eds.) European Conference on Computer Vision, pp. 850–865. Springer, Cham (2016)

    Google Scholar 

  4. Bolme, D.S., Beveridge, J.R., Draper, B.A., Lui, Y.M.: Visual object tracking using adaptive correlation filters. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2544–2550 (2010). https://doi.org/10.1109/CVPR.2010.5539960

  5. Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., Lu, H.: Transformer tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8126–8135. arxiv:2103.15436 (2021)

  6. Chen, Z., Hong, Z., Tao, D.: An experimental survey on correlation filter-based tracking. Computer Science 53(6025), 68–83 (2015)

    Google Scholar 

  7. Dai, K., Wang, D., Lu, H., Sun, C., Li, J.: Visual tracking via adaptive spatially-regularized correlation filters. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4670–4679 (2019)

  8. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 886–893 (2005). https://doi.org/10.1109/CVPR.2005.177

  9. Danelljan, M., Häger, G., Khan, F.S., Felsberg, M.: Accurate scale estimation for robust visual tracking. In: BMVC, pp. 1–5 (2014a). https://doi.org/10.5244/c.28.65

  10. Danelljan, M., Khan, F.S., Felsberg, M., Van De Weijer, J.: Supplementary material adaptive color attributes for real-time visual tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1090–1097 (2014b)

  11. Danelljan, M., Hager, G., Khan, F.S., Felsberg, M.: Learning spatially regularized correlation filters for visual tracking. In: Proceedings of the IEEE International Conference on Computer Vision 2015 Inter, pp. 4310–4318 (2015a). https://doi.org/10.1109/ICCV.2015.490

  12. Danelljan, M., Hager, G., Shahbaz Khan, F., Felsberg, M.: Convolutional features for correlation filter based visual tracking. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops (2015b)

  13. Danelljan, M., Robinson, A., Shahbaz Khan, F., Felsberg, M.: Beyond correlation filters: learning continuous convolution operators for visual tracking. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) European Conference on Computer Vision, pp. 472–488. Springer, Cham (2016)

    Google Scholar 

  14. Danelljan, M., Robinson, A., Shahbaz Khan, F., Felsberg, M.: Beyond correlation filters: learning continuous convolution operators for visual tracking. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) European Conference on Computer Vision, pp. 472–488. Springer, Cham (2016)

    Google Scholar 

  15. Danelljan, M., Bhat, G., Khan, F.S., Felsberg, M.: ECO: efficient convolution operators for tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6638–6646. arxiv:1611.09224 (2017a)

  16. Danelljan, M., Häger, G., Khan, F.S., Felsberg, M.: Discriminative scale space tracking. IEEE Trans. Pattern Anal. Mach. Intell. 39(8), 1561–1575 (2017). https://doi.org/10.1109/TPAMI.2016.2609928

    Article  Google Scholar 

  17. Elayaperumal, D., Joo, Y.H.: Robust visual object tracking using context-based spatial variation via multi-feature fusion. Inf. Sci. 577, 467–482 (2021). https://doi.org/10.1016/j.ins.2021.06.084

    Article  MathSciNet  Google Scholar 

  18. Feng, W., Han, R., Guo, Q., Zhu, J., Wang, S.: Dynamic saliency-aware regularization for correlation filter-based object tracking. IEEE Trans. Image Process. 28(7), 3232–3245 (2019). https://doi.org/10.1109/TIP.2019.2895411

    Article  MathSciNet  MATH  Google Scholar 

  19. Fu, C., Lin, F., Li, Y., Chen, G.: Correlation filter-based visual tracking for UAV with online multi-feature learning. Remote Sens. 11(5), 1–23 (2019). https://doi.org/10.3390/rs11050549

    Article  Google Scholar 

  20. Fu, C., Xu, J., Lin, F., Guo, F., Liu, T., Zhang, Z.: Object saliency-aware dual regularized correlation filter for real-time aerial tracking. IEEE Trans. Geosci. Remote Sens. 58(12), 8940–8951 (2020). https://doi.org/10.1109/TGRS.2020.2992301

    Article  Google Scholar 

  21. Galoogahi, H.K., Sim, T., Lucey, S.: Correlation filters with limited boundaries. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 4630–4638 (2015). https://doi.org/10.1109/CVPR.2015.7299094

  22. Galoogahi, H.K., Fagg, A., Lucey, S.: Learning background-aware correlation filters for visual tracking. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1135–1143 (2017)

  23. Gao, L., Liu, B., Fu, P., Xu, M., Li, J.: Visual tracking via dynamic saliency discriminative correlation filter. Appl. Intell. (2021). https://doi.org/10.1007/s10489-021-02260-2

    Article  Google Scholar 

  24. Gupta, D.K., Arya, D., Gavves, E.: Rotation equivariant siamese networks for tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12362–12371. arxiv:2012.13078 (2021)

  25. Han, Z., Wang, P., Ye, Q.: Adaptive discriminative deep correlation filter for visual object tracking. IEEE Trans. Circuits Syst. Video Technol. 30(1), 155–166 (2020). https://doi.org/10.1109/TCSVT.2018.2888492

    Article  Google Scholar 

  26. Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: Exploiting the circulant structure of tracking-by-detection with kernels. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) European Conference on Computer Vision, pp. 702–715. Springer, Berlin (2012)

    Google Scholar 

  27. Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 583–596 (2015). https://doi.org/10.1109/TPAMI.2014.2345390

    Article  Google Scholar 

  28. Huang, D., Luo, L., Wen, M., Chen, Z., Zhang, C.: Enable scale and aspect ratio adaptability in visual tracking with detection proposals. In: Proceedings of British Machine Vision Conference, British Machine Vision Association and Society for Pattern Recognition, pp. 185.1–185.12 (2015). https://doi.org/10.5244/c.29.185

  29. Huang, D., Luo, L., Chen, Z., Wen, M., Zhang, C.: Applying detection proposals to visual tracking for scale and aspect ratio adaptability. Int. J. Comput. Vision 122(3), 524–541 (2017). https://doi.org/10.1007/s11263-016-0974-6

    Article  Google Scholar 

  30. Huang, Z., Fu, C., Li, Y., Lin, F., Lu, P.: Learning aberrance repressed correlation filters for real-time UAV tracking. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2891–2900 (2019). https://doi.org/10.1109/ICCV.2019.00298

  31. Islam, M.M., Hu, G., Liu, Q., Dan, W., Lyu, C.: Correlation filter based moving object tracking with scale adaptation and online re-detection. IEEE Access 6, 75244–75258 (2018). https://doi.org/10.1109/ACCESS.2018.2883650

    Article  Google Scholar 

  32. Kristan, M., Leonardis, A., et al.: The visual object tracking vot2016 challenge results. In: European Conference on Computer Vision (2016)

  33. Kristan, M., Leonardis, A., et al.: The visual object tracking vot2017 challenge results. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops (2017)

  34. Kristan, M., Leonardis, A., et al.: The sixth visual object tracking vot2018 challenge results. In: European Conference on Computer Vision (2018)

  35. Kristan, M., Matas, J., Leonardis, A., et al.: The seventh visual object tracking vot2019 challenge results. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops (2019)

  36. Li, F., Tian, C., Zuo, W., Zhang, L., Yang, M.H.: Learning spatial-temporal regularized correlation filters for visual tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4904–4913 (2018). arxiv:1803.08679

  37. Li, Y., Liu, G.: Learning a scale-and-rotation correlation filter for robust visual tracking. In: Proceedings of International Conference on Image Processing, ICIP, pp. 454–458. IEEE Computer Society (2016). https://doi.org/10.1109/ICIP.2016.7532398

  38. Li, Y., Zhu, J.: A scale adaptive kernel correlation filter tracker with feature integration. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) Computer Vision—ECCV 2014 Workshops, pp. 254–265. Springer, Cham (2015)

    Chapter  Google Scholar 

  39. Li, Y., Zhu, J., Hoi, S.C.H.: Reliable patch trackers: robust visual tracking by exploiting reliable patches. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 353–361 (2015)

  40. Li, Y., Zhu, J., Hoi, S.C.H., Song, W., Wang, Z., Liu, H.: Robust estimation of similarity transformation for visual object tracking. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8666–8673 (2019). www.aaai.org

  41. Li, Y., Fu, C., DIng, F., Huang, Z., Lu G.: AutoTrack: towards high-performance visual tracking for UAV with automatic spatio-temporal regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11920–11929 (2020). https://doi.org/10.1109/CVPR42600.2020.01194

  42. Liang, P., Blasch, E., Ling, H.: Encoding color information for visual tracking: algorithms and benchmark. IEEE Trans. Image Process. 24(12), 5630–5644 (2015). https://doi.org/10.1109/TIP.2015.2482905

    Article  MathSciNet  MATH  Google Scholar 

  43. Liang, P., Wu, Y., Lu, H., Wang, L., Liao, C., Ling, H.: Planar object tracking in the wild: a benchmark. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 651–658. Institute of Electrical and Electronics Engineers Inc. (2018). https://doi.org/10.1109/ICRA.2018.8461037

  44. Lin, F., Fu, C., He, Y., Guo, F., Tang, Q.: Learning temporary block-based bidirectional incongruity-aware correlation filters for efficient UAV object tracking. IEEE Trans. Circuits Syst. Video Technol. 31(6), 2160–2174 (2021). https://doi.org/10.1109/TCSVT.2020.3023440

    Article  Google Scholar 

  45. Lu, X., Ma, C., Ni, B., Yang, X.: Adaptive region proposal with channel regularization for robust object tracking. IEEE Trans. Circuits Syst. Video Technol. 31(4), 1268–1282 (2021). https://doi.org/10.1109/TCSVT.2019.2944654

    Article  Google Scholar 

  46. Lukezic, A., Vojir, T., Cehovin Zajc, L., Matas, J.J., Kristan, M., Lukežič, A., Vojíř, T., Zajc, L., Matas, J.J., Kristan M.: Discriminative correlation filter with channel and spatial reliability. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6309–6318 (2017)

  47. Meng, Y., Deng, Z., Zhao, K., Xu, Y., Liu, H.: Hierarchical correlation siamese network for real-time object tracking. Appl. Intell. 51(6), 3202–3211 (2021). https://doi.org/10.1007/s10489-020-01992-x

    Article  Google Scholar 

  48. Miao, Q., Xu, C., Li, F., Zuo, W., Meng, Z.: Delayed rectification of discriminative correlation filters for visual tracking. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02401-9

    Article  Google Scholar 

  49. Montero, A.S., Lang, J., Laganière, R.: Scalable kernel correlation filter with sparse feature integration. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 587–594. Institute of Electrical and Electronics Engineers Inc. (2016). https://doi.org/10.1109/ICCVW.2015.80

  50. Mueller, M., Smith, N., Ghanem, B.: A benchmark and simulator for UAV tracking. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) European Conference on Computer Vision, pp. 445–461. Springer, Cham (2016)

    Google Scholar 

  51. Nam, H., Han, B.: Learning multi-domain convolutional neural networks for visual tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  52. Possegger, H., Mauthner, T.: In defense of color-based model-free tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2113–2120 (2015)

  53. Ross, D.A., Lim, J., Lin, R.S., Yang, M.H.: Incremental learning for robust visual tracking. Int. J. Comput. Vis. 77(1), 125–141 (2008). https://doi.org/10.1007/s11263-007-0075-7

    Article  Google Scholar 

  54. Smeulders, A.W., Chu, D.M., Cucchiara, R., Calderara, S., Dehghan, A., Shah, M.: Visual tracking: an experimental survey. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1442–1468 (2014). https://doi.org/10.1109/TPAMI.2013.230

    Article  Google Scholar 

  55. Touil, D.E., Terki, N., Medouakh, S.: Learning spatially correlation filters based on convolutional features via PSO algorithm and two combined color spaces for visual tracking. Appl. Intell. 48(9), 2837–2846 (2018). https://doi.org/10.1007/s10489-017-1120-z

    Article  Google Scholar 

  56. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, vol. 1, pp. I–I (2001). https://doi.org/10.1109/CVPR.2001.990517

  57. Wang, M., Liu, Y., Huang, Z.: Large margin object tracking with circulant feature maps. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4021–4029 (2017)

  58. Wang, Q., Zhang, L., Bertinetto, L., Hu, W., Torr, P.H.S.: Fast online object tracking and segmentation: a unifying approach. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1328–1338 (2019). arxiv:1812.05050

  59. Wei, Y., Wen, F., Zhu, W., Sun, J.: Geodesic saliency using background priors. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) European Conference on Computer Vision, pp. 29–42. Springer, Berlin (2012)

    Google Scholar 

  60. Wu, Y., Lim, J., Yang, M.H.: Online object tracking: a benchmark. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2411–2418 (2013)

  61. Xu, L., Gao, M., Li, Q., Zou, G., Pan, J., Jiang, J.: Visual tracking for UAV using adaptive spatio-temporal regularized correlation filters. Appl. Intell. (2021). https://doi.org/10.1007/s10489-021-02825-1

    Article  Google Scholar 

  62. Xue, X., Li, Y., Dong, H., Shen, Q.: Robust correlation tracking for UAV videos via feature fusion and saliency proposals. Remote Sens. (2018). https://doi.org/10.3390/rs10101644

    Article  Google Scholar 

  63. Zhang, J., Ma, S., Sclaroff, S.: MEEM: robust tracking via multiple experts using entropy minimization. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) European Conference on Computer Vision, pp. 188–203. Springer, Cham (2014)

    Google Scholar 

  64. Zhang, J., Sclaroff, S., Lin, Z., Shen, X., Price, B., Ech, R.M.: Minimum barrier salient object detection at 80 FPS. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1404–1412 (2015a)

  65. Zhang, J., Liu, Y., Liu, H., Wang, J., Zhang, Y.: Distractor-aware visual tracking using hierarchical correlation filters adaptive selection. Appl. Intell. (2021). https://doi.org/10.1007/s10489-021-02694-8

    Article  Google Scholar 

  66. Zhang, M., Xing, J., Gao, J., Shi, X., Wang, Q., Hu, W.: Joint scale-spatial correlation tracking with adaptive rotation estimation. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 32–40 (2015b)

  67. Zhao, D., Xiao, L., Fu, H., Wu, T., Xu, X., Dai, B.: Augmenting cascaded correlation filters with spatial-temporal saliency for visual tracking. Inf. Sci. 470, 78–93 (2019). https://doi.org/10.1016/j.ins.2018.08.053

    Article  Google Scholar 

  68. Zhao, X., Pang, Y., Zhang, L., Lu, H., Zhang, L.: Suppress and balance: a simple gated network for salient object detection. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) European Conference on Computer Vision, pp. 35–51. Springer, Cham (2020)

    Google Scholar 

  69. Zheng, G., Fu, C., Ye, J., Lin, F., Ding, F.: Mutation sensitive correlation filter for real-time UAV tracking with adaptive hybrid label. In: IEEE International Conference on Robotics and Automation (ICRA). arxiv:2106.08073 (2021)

  70. Zitnick, C.L., Dollár, P.: Edge boxes: locating object proposals from edges. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) European Conference on Computer Vision, pp. 391–405. Springer, Cham (2014)

    Google Scholar 

  71. Zokai, S., Wolberg, G.: Image registration using log-polar mappings for recovery of large-scale similarity and projective transformations. IEEE Trans. Image Process. 14(10), 1422–1434 (2005). https://doi.org/10.1109/TIP.2005.854501

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The project is supported by NKRDPC (No. 2018YFA0704603), National Natural Science Foundation of China (Nos. 51975098 and U1937602), LiaoNing Revitalization Talents Program (Nos. XLYC1907006, XLYCYSZX1901), Science and Technology Innovation Fund of Dalian (No. 2019CT01) and the Fundamental Research Funds for the Central Universities. The authors wish to thank the anonymous reviewers for their comments which led to improvements of this paper.

Author information

Authors and Affiliations

  1. Key Laboratory for Precision and Non-traditional Machining Technology of the Ministry of Education, School of Mechanical Engineering, Dalian University of Technology, Dalian, China

    Jianwei Ma, Qi Lv, Huiteng Yan, Tao Ye, Yabin Shen & Hechen Sun

Authors
  1. Jianwei Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huiteng Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tao Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yabin Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hechen Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianwei Ma.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ma, J., Lv, Q., Yan, H. et al. Color-saliency-aware correlation filters with approximate affine transform for visual tracking. Vis Comput 39, 4065–4086 (2023). https://doi.org/10.1007/s00371-022-02573-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02573-4

Keywords

Search

Navigation