Skip to main content
SpringerLink
Log in

Multispectral particle filter tracking using adaptive decision-based fusion of visible and thermal sequences

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

One of the main challenges of detection and tracking of objects in video monitoring is the lighting conditions of the scene under surveillance and its variations. Due to the availability of various visible cameras and beyond visible spectrum sensors at low costs, a solution for this challenge is the fusion of different sensors data, especially visible and thermal sensors. However, the main problem is, proposing an efficient fusion of various sensors and determining the reliability of their data to make the most use of the sensor’s information. For this purpose, this paper presents a multispectral particle filter tracking based on an adaptive fusion of visible and thermal data. In the proposed fusion method, a confidence measure is assigned to each sensor’s data. To evaluate the confidence of the sensor’s data in each frame, in the proposed method, two criteria consisting of local mean luminance in the visible spectrum and local contrast in the thermal spectrum are calculated. These criteria are then employed to determine the confidence of each sensor using a fuzzy inference system (FIS). Also, a second confidence measure for the visible spectrum is defined based on the variation of local mean luminance in successive frames. The total confidence of visible and thermal spectrum is then employed for the adaptive fusion of data for particle filter tracking. The proposed method was evaluated quantitatively and qualitatively through several simulations performed on the thermal and visible benchmark video sequences, which include variable and low light conditions. The results were promising to improve the tracking in challenging scenes and demonstrated the effectiveness of the proposed fusion algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Aherne FJ, Thacker NA, Rockett PI (1998) The Bhattacharyya metric as an absolute similarity measure for frequency coded data. Kybernetika 34(4):363–368

    MathSciNet  MATH  Google Scholar 

  2. Airouche M, Bentabet L, Zelmat M, Gao G (2012) Pedestrian tracking using color, thermal and location cue measurements: a DSmT-based framework. Mach Vis Appl:1–12

  3. Alldieck T, Bahnsen CH, Moeslund TB (2016) Context-aware fusion of RGB and thermal imagery for traffic monitoring. Sensors 16(11)

  4. Alsulami F, Ahmadi K, Salari E (2016) Detection and tracking of dim objects in infrared (IR) images using Support Vector Machine. In: 2016 IEEE International Conference on Electro Information Technology (EIT), pp 0357–0361

  5. Bhat G, Johnander J, Danelljan M, Shahbaz Khan F (2018) Felsberg M Unveiling the power of deep tracking. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 483–498

  6. Čehovin L, Leonardis A, Kristan M (2016) Visual object tracking performance measures revisited. IEEE Trans Image Process 25(3):1261–1274

    MathSciNet  MATH  Google Scholar 

  7. Chen SY (2012) Kalman filter for robot vision: a survey. IEEE Trans Ind Electron 59(11):4409–4420

    Google Scholar 

  8. Chen S-J, Chen S-M (2005) A prioritized information fusion method for handling fuzzy decision-making problems. Appl Intell 22(3):219–232

    MATH  Google Scholar 

  9. Comaniciu D, Ramesh V (2000) Real-time tracking of non-rigid objects using mean shift. In: IEEE conferance on computer vision and pattern Recognition.CVPR, IEEE

  10. Conaire CÓ, O’Connor NE, Smeaton A (2008) Thermo-visual feature fusion for object tracking using multiple spatiogram trackers. Mach Vis Appl 19(5–6):483–494

    Google Scholar 

  11. Cui J, Liu Y, Xu Y, Zhao H, Zha H (2013) Tracking generic human motion via fusion of low-and high-dimensional approaches. IEEE transactions on systems, man, and cybernetics: systems 43(4):996–1002

    Google Scholar 

  12. Dai Y, Liu B (2016) Robust video object tracking via Bayesian model averaging-based feature fusion. Opt Eng 55(8)

  13. Dalal N (2005) Triggs B Histograms of oriented gradients for human detection. In: Computer Vision and Pattern Recognition. CVPR 2005. IEEE computer society conference on, 2005. IEEE, pp 886–893

  14. Danelljan M, Robinson A, Khan FS, Felsberg M (2016) Beyond correlation filters: Learning continuous convolution operators for visual tracking. In: Leibe B, Matas J, Sebe N, Welling M (eds) European Conference on Computer Vision. Springer International Publishing, Cham, pp 472–488

    Google Scholar 

  15. Daneshyar MA, Nahvi M (2015) Improvement of moving objects tracking via modified particle distribution in particle filter algorithm. In: Pattern Recognition and Image Analysis (IPRIA), 2015 2nd International Conference on, IEEE, pp 1–6

  16. Daneshyar SA, Nahvi M (2017) Moving objects tracking based on improved particle filter algorithm by elimination of unimportant particles. Optik-International Journal for Light and Electron Optics 138:455–469

    Google Scholar 

  17. Davis JW, Sharma V (2005) Fusion-based background-subtraction using contour saliency. In: Computer Vision and Pattern Recognition-Workshops, 2005. CVPR Workshops. IEEE Computer Society Conference on. IEEE, pp 11–11

  18. Davis JW, Sharma V (2007) Background-subtraction using contour-based fusion of thermal and visible imagery. Comput Vis Image Underst 106(2–3):162–182

    Google Scholar 

  19. Denman S, Fookes C, Sridharan S, Ryan D (2010) Multi-Modal Object Tracking using Dynamic Performance Metrics. In: 2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance. pp 286–293.

  20. Doucet SGA, Andrieu C (2000) On sequential Montecarlo sampling methods for. Bayesian filtering Statistics and computing 10(3):197–208

    Google Scholar 

  21. Doucet A, Godsill S, Andrieu C (2000) On sequential Monte Carlo sampling methods for Bayesian filtering. Stat Comput 10(3):197–208

    Google Scholar 

  22. Eitel A, Springenberg JT, Spinello L, Riedmiller M, Burgard W (2015) Multimodal deep learning for robust RGB-D object recognition. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 681–687

  23. Exner D, Bruns E, Kurz D, Grundhöfer A, Bimber O (2010) Fast and robust CAMShift tracking. In: Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE computer society conference on, IEEE, pp 9–16

  24. Faouzi N-EE, Leung H, Kurian A (2011) Data fusion in intelligent transportation systems: Progress and challenges – a survey. Information Fusion 12(1):4–10

    Google Scholar 

  25. Fendri E, Boukhriss RR, Hammami M (2017) Fusion of thermal infrared and visible spectra for robust moving object detection. Pattern Anal Applic 20(4):907–926

    MathSciNet  Google Scholar 

  26. Fernández Caballero A, López Bonal MT, Serrano Cuerda J, Novais P (2014) Selection of a visible-light vs. Thermal infrared sensor in dynamic environments based on confidence measures. Applied Sciences 4(3):331–350

    Google Scholar 

  27. Gade R, Moeslund TB (2014) Thermal cameras and applications: a survey. Mach Vis Appl 25(1):245–262

    Google Scholar 

  28. Gade R, Moeslund TB (2014) Thermal tracking of sports players. Sensors 14(8):13679–13691

    Google Scholar 

  29. Gundogdu E, Koc A, Solmaz B, Hammoud RI (2016) Aydin Alatan a evaluation of feature channels for correlation-filter-based visual object tracking in infrared spectrum. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 24–32

  30. Gundogdu E, Ozkan H, Seckin Demir H, Ergezer H, Akagunduz E (2015) Kubilay Pakin S comparison of infrared and visible imagery for object tracking: toward trackers with superior ir performance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 1–9

  31. Hanbury A, Serra J (2003) A 3D-polar coordinate colour representation suitable for image analysis. Technical Report, Vienna University of Technology, Vienna, PRIP-TR-77

  32. Hwang S, Park J, Kim N, Choi Y, So Kweon I (2015) Multispectral pedestrian detection: benchmark dataset and baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1037–1045

  33. INO’s Video Analytics Dataset. http://www.inoca/en/technologies/video-analytics-dataset/

  34. Kaewtrakulpong P, Bowden R (2001) An improved adaptive background mixture model for realtime tracking with shadow detection. In: Video-based surveillance systems Boston, springer, pp 135–144

  35. Kang JK, Hong HG, Park KR (2017) Pedestrian detection based on adaptive selection of visible light or far-infrared light camera image by fuzzy inference system and convolutional neural network-based verification. Sensors 17(7):1598

    Google Scholar 

  36. Kim D-H, Kim H-K, Lee S-J, Park W-J, Ko S-J (2014) Kernel-based structural binary pattern tracking. IEEE Transactions on Circuits and Systems for Video Technology 24(8):1288–1300

    Google Scholar 

  37. Kristan M (2016) The thermal infrared visual object tracking VOT-TIR2016 challenge results. In: Proc. Eur. Conf. Comput. Vis.(ECCV), pp 824–849

  38. Kristan ALM, Matas J, Felsberg M, Pflugfelder R, Čehovin L, T. Vojir, G. Häger, Lukezîč A, Fernandez G et al (2016) The visual object tracking vot2016 challenge results. In: Computer Vision, European Conferance, In Proc

  39. Kristan M, Matas J, Leonardis A, Felsberg M, Cehovin L, Fernández G, Hager VT et al (2016) The visual object tracking vot2016 challenge results. In: ECCV Workshop, vol 6. p 8

  40. Kumar P, Mittal A, Kumar P (2010) Addressing uncertainty in multi-modal fusion for improved object detection in dynamic environment. Information Fusion 11(4):311–324

    Google Scholar 

  41. Lee D, Choi S (2011) Multisensor fusion-based object detection and tracking using active shape model. In: Digital Information Management (ICDIM), 2011 Sixth International Conference on. IEEE, pp 108–114

  42. Lewis JJ, O’Callaghan RJ, Nikolov SG, Bull DR, Canagarajah N (2007) Pixel-and region-based image fusion with complex wavelets. Information fusion 8(2):119–130

    Google Scholar 

  43. Liu Y, Cui J, Zhao H, Zha H (2012) Fusion of low-and high-dimensional approaches by trackers sampling for generic human motion tracking. In: Proceedings of the 21st international conference on pattern recognition (ICPR2012), IEEE, pp 898–901

  44. Mamdani EH, Assilian S (1975) An experiment in linguistic synthesis with a fuzzy logic controller. International Journal of Man-Machine Studies 7(1):1–13

    MATH  Google Scholar 

  45. Nam H, Baek M, Han B (2016) Modeling and propagating cnns in a tree structure for visual tracking. arXiv preprint arXiv:160807242

  46. Nummiaro K, Koller-Meier E, Van Gool L (2003) An adaptive color-based particle filter. Image Vis Comput 21(1):99–110

    MATH  Google Scholar 

  47. O'Conaire C, O'Connor NE, Cooke E, Smeaton AF (2006) Comparison of fusion methods for thermo-visual surveillance tracking. In: Information Fusion, 2006 9th International Conference on, IEEE, pp 1–7

  48. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    MATH  Google Scholar 

  49. Qian X, Han L, Cheng Y (2014) An object tracking method based on local matting for night fusion image. Infrared Phys Technol 67:455–461

    Google Scholar 

  50. Qian X, Han L, Zhang Y, Ding M(2017) An object tracking method using deep learning and adaptive particle filter for night fusion image. In: 2017 International Conference on Progress in Informatics and Computing (PIC), IEEE, pp 138–142

  51. Qian X, Wang Y, Han L (2016) An object tracking method based on guided filter for night fusion image. Infrared Phys Technol 74:38–43

    Google Scholar 

  52. Sajid H, Cheung S-CS (2017) Universal multimode background subtraction. IEEE Trans Image Process 26(7):3249–3260

    MathSciNet  MATH  Google Scholar 

  53. Särkkä S (2013) Particle filtering. In: Bayesian Filtering and Smoothing. Cambridge University Press, pp 116–128

  54. Schnelle SR, Chan AL (2011) Enhanced target tracking through infrared-visible image fusion. In: Information Fusion (FUSION), 2011 Proceedings of the 14th International Conference on. IEEE, pp 1–8

  55. Smeulders AW, Chu DM, Cucchiara R, Calderara S, Dehghan A, Shah M (2014) Visual tracking: an experimental survey. IEEE Trans Pattern Anal Mach Intell 36(7):1442–1468

    Google Scholar 

  56. Stauffer C, Grimson WEL (1999) Adaptive background mixture models for real-time tracking. In: Conferance on Computer Vision and Pattern Recognition, IEEE, pp 246–252

  57. St-Laurent L, Maldague X, Prévost D (2007) Combination of color and thermal sensors for enhanced object detection. In: Information Fusion, 2007 10th International Conference on. IEEE, pp 1–8

  58. Talha M, Stolkin R (2014) Particle filter tracking of camouflaged targets by adaptive fusion of thermal and visible spectra camera data. IEEE Sensors J 14(1):159–166

    Google Scholar 

  59. TrainingImageLabeler (2014). URL: http:www//ukmathworks.com/help/vision/ref/trainingimagelabeler-apphtml

  60. Tsakanikas V, Dagiuklas T (2018) Video surveillance systems-current status and future trends. Comput Electr Eng 70:736–753

    Google Scholar 

  61. Van Broekhoven E, De Baets B (2006) Fast and accurate center of gravity defuzzification of fuzzy system outputs defined on trapezoidal fuzzy partitions. Fuzzy Sets Syst 157(7):904–918

    MathSciNet  MATH  Google Scholar 

  62. Wagner J, Fischer V, Herman M, Behnke S (2016) Multispectral pedestrian detection using deep fusion convolutional neural networks. In: 24th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), pp 509–514

  63. Wang L-X (1999) A course in fuzzy systems. Prentice-Hall press, USA

    Google Scholar 

  64. Wang X, Hou Z, Yu W, Jin Z, Zha Y, Qin X (2017) Online scale adaptive visual tracking based on multilayer convolutional features. IEEE transactions on cybernetics 99:1–13

    Google Scholar 

  65. Xia Y, Qu S, Cai L (2013) Multi-object tracking based on thermal-visible video sequence fusion. In: Proceedings of the 32nd Chinese Control Conference, 2013. pp 3685–3690

  66. Yin F, Makris D, Velastin SA (2007) Performance evaluation of object tracking algorithms. In: IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, Rio De Janeiro, Brazil. Citeseer, p 25

  67. Yuan Y, Lu X, Chen X (2015) Multi-spectral pedestrian detection. Signal Process 110:94–100

    Google Scholar 

  68. Zhao G-P, Bo Y-M, Yin M-F (2012) An object tracking method based on infrared and visible dual-channel video. Dianzi Yu Xinxi Xuebao(Journal of Electronics and Information Technology) 34(3):529–534

    Google Scholar 

  69. Zhao G-P, Bo Y-M, Yin M-F (2012) An object tracking method based on infrared and visible dual-channel video. Dianzi Yu Xinxi Xuebao (Journal Of Electronics and Information Technology) 34:529–534

    Google Scholar 

  70. Zheng Y (2011) Image fusion and its applications. InTech

  71. Zhou Y, Mayyas A, Qattawi A, Omar M (2010) Feature-level and pixel-level fusion routines when coupled to infrared night-vision tracking scheme. Infrared Phys Technol 53(1):43–49

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, University of Guilan, Rasht, Iran

    Negin Pourmomtaz & Manoochehr Nahvi

Authors
  1. Negin Pourmomtaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manoochehr Nahvi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manoochehr Nahvi.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pourmomtaz, N., Nahvi, M. Multispectral particle filter tracking using adaptive decision-based fusion of visible and thermal sequences. Multimed Tools Appl 79, 18405–18434 (2020). https://doi.org/10.1007/s11042-020-08640-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08640-z

Keywords

Search

Navigation