Skip to main content
SpringerLink
Log in

Research of multi-object detection and tracking using machine learning based on knowledge for video surveillance system

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Recently, as the risk of crime and accidents increases, interest in security and surveillance of individuals and the public is increasing rapidly, and video surveillance system technology is continuously developing. Reliable object detection in the system is the basis of all elements using image information and it is used in various applications using the information, so accurate object detection and tracking are needed. Therefore, we propose a system for analyzing images with a knowledge-based deep learning technology for multi-object recognition and tracking enhancement. Algorithms for recognizing objects using existing convolution neural network (CNN) classifiers have a problem that it is difficult to process in real time because the processing time is increased when there are a lot of objects to be classified in the image. Therefore, we propose an algorithm that combines optical flow while maintaining the recognition performance through a knowledge-based CNN. An optical flow-based tracker can forecast the position of objects in the next frame based on the position of objects in the current frame. A CNN-based detector can detect the position of objects through a knowledge-based mining method between the two images. CNN-based detectors also carry out mining method on current frame information. This detector can select more capacity features based on the background to more accurately forecast the location of the tracked targets and targets. The fusion of the tracker and detector compensates for accumulated errors that can occur in the tracker and for drift from the detector. The experimental results show that the proposed algorithm combining CNN and optical flow can detect and trace multiple objects in a video stream, and can carry out robust detection and tracing even in a complex environment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Collins RT, Lipton AJ, Kanade T, Fujiyoshi H, Duggins D, Tsin Y, Tolliver D, Enomoto N, Hasegawa O, Burt P, Wixson L (2000) A system for video surveillance and monitoring. The Robotics Institute, Carnegie Mellon University, Pittsburgh, pp 1–68

    Google Scholar 

  2. Ahn H, Lee Y (2016) Performance analysis of object recognition and tracking for the use of surveillance system. J Ambient Intell Humaniz Comput 7(5):673–679

    Article  Google Scholar 

  3. Valera M, Velastin SA (2005) Intelligent distributed surveillance systems: a review, IEE Proc Vision Image Signal Process, vol 152. No. 2. IET, pp 192–204

  4. Wu Y, Jongwoo L, Ming-Hsuan Y (2015) Object tracking benchmark. IEEE Trans Pattern Anal Mach Intell 37(9):1834–1848

    Article  Google Scholar 

  5. Yi W, Lim JW, Yang MH (2013) Online object tracking: a benchmark, In: CVPR, pp 2411–2418

  6. Yilmaz A, Javed O, Shah M (2006) Object tracking: a survey. ACM Comput Surv (CSUR) 38(4):13

    Article  Google Scholar 

  7. Kang S, Chung K, Lee J (2014) Development of head detection and tracking systems for visual surveillance. Pers Ubiquit Comput 18(3):515–522

    Article  Google Scholar 

  8. Alostaz A, Hamed B (2016) Optimized automated tracking of a moving object with a robotic eye system. Control Intell Syst 44(1)

  9. Comaniciu D, Ramesh V, Meer P (2003) Kernel-based object tracking. IEEE Trans Pattern Anal Mach Intell 25(5):564–577

    Article  Google Scholar 

  10. Allen JG, Xu RY, Jin JS (2004) Object tracking using camshift algorithm and multiple quantized feature spaces. In: Proceedings of the Pan-Sydney area workshop on Visual information processing. Australian Computer Society, Inc., Kent Town, pp 3–7

    Google Scholar 

  11. Ahn H, Shin I (2018) Study on a robust object tracking algorithm based on improved SURF method with CamShift. Journal of the Korea Society of Computer and Information 23(1):41–48

    Google Scholar 

  12. Grabner H, Matas J, Gool LJV, Cattin PC (2010) Tracking the invisible: learning where the object might be, CVPR 2010, pp 1285–1292

  13. Babenko B, Yang M, Belongie S (2011) Robust object tracking with online multiple instance learning. IEEE Trans Pattern Anal Mach Intell 33(8):1619–1632

    Article  Google Scholar 

  14. Babenko B, Yang M, Belongie S (2009) Visual tracking with online multiple instance learning, In ComputerVision and pattern recognition, 2009. CVPR 2009. IEEE conference on, pp 983–990

  15. Kalal Z, Mikolajczyk K, Matas J (2012) Tracking-learning-detection. IEEE Trans Pattern Anal Mach Intell 34(7):1409–1422

    Article  Google Scholar 

  16. Chen Y, Yang X, Zhong B, Pan S, Chen D, Zhang H (2016) CNNTracker: online discriminative object tracking via deep convolutional neural network. Appl Soft Comput 38:1088–1098

    Article  Google Scholar 

  17. Zhang K, Zhang L, Yang M (2012) Real-time compressive tracking. In: European conference on computer vision. Springer, Berlin, pp 864–877

    Google Scholar 

  18. Takala V, Pietikäinen M (2007) Multi-object tracking using color, texture and motion, CVPR 2007, pp 1–7

  19. Jepson AD, Fleet DJ, El-Maraghi TF (2003) Robust online appearance models for visual tracking. IEEE Trans Pattern Anal Mach Intell 25(10):1296–1311

    Article  Google Scholar 

  20. Elgammal A, Duraiswami R, Harwood D, Davis LS (2002) Background and foreground modeling using nonparametric kernel density estimation for visual surveillance. Proc IEEE 90(7):1151–1163

    Article  Google Scholar 

  21. Stauffer C, Grimson WEL (1999) Adaptive background mixture models for real-time tracking, In computer vision and pattern recognition, IEEE Computer Society Conference on, vol 2, pp 2246

  22. Gómez-Romero J, Serrano MA, Patricio MA, García J, Molina JM (2012) Context-based scene recognition from visual data in smart homes: an information fusion approach. Pers Ubiquit Comput 16(7):835–857

    Article  Google Scholar 

  23. Grabner H, Grabner M, Bischof H (2006) Real-time tracking via on-line boosting. In: Bmvc, vol 1, No. 5, pp 6

  24. Grabner H, Bischof H (2006) On-line boosting and vision, CVPR (1), pp 260–267

  25. Viola P, Jones MJ (2001) Rapid object detection using a boosted cascade of simple features, CVPR 2001, vol 1, pp 1–1

  26. Han G, Shen J, Liu L, Qian A, Shu L (2016) TGM-COT: energy-efficient continuous object tracking scheme with two-layer grid model in wireless sensor networks. Pers Ubiquit Comput 20(3):349–359

    Article  Google Scholar 

  27. Z S, Yu X, Sui Y, Zhao S, Zhang L (2015) Object tracking with multi-view support vector machines. IEEE Trans Multimedia 17(3):265–278

    Google Scholar 

  28. Avidan S (2004) Support vector tracking. IEEE Trans Pattern Anal Mach Intell 26(8):1064–1072

    Article  Google Scholar 

  29. Bai Y, Tang M (2012) Robust tracking via weakly supervised ranking SVM, CVPR 2012, pp 1854–1861

  30. Cortes C, Vapnik VN (1995) Support-vector networks. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  31. Luo L (2018) Network text sentiment analysis method combining LDA text representation and GRU-CNN. Pers Ubiquit Comput (1–8)

  32. Steinkrau D, Simard PY, Buck I (2005) Using GPUs for machine learning algorithms. ICDAR 2005:1115–1119

    Google Scholar 

  33. Chellapilla K, Kumar SP, Simard P (2006) High performance convolutional neural networks for document processing, Tenth international workshop on Frontiers in handwriting recognition

  34. Senst T, Eiselein V, Sikora T (2012) Robust local optical flow for feature tracking. IEEE Trans Circuits Syst Video Technol 22(9):1377

    Article  Google Scholar 

  35. Barron JL, Fleet DJ, Beauchemin SS (1994) Performance of optical flow techniques. Int J Comput Vis 12(1):43–77

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Energy IT Engineering, Far East University, Eumseong, Chungbuk, Republic of Korea

    Hyochang Ahn & Han-Jin Cho

Authors
  1. Hyochang Ahn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Han-Jin Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Han-Jin Cho.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahn, H., Cho, HJ. Research of multi-object detection and tracking using machine learning based on knowledge for video surveillance system. Pers Ubiquit Comput 26, 385–394 (2022). https://doi.org/10.1007/s00779-019-01296-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-019-01296-z

Keywords

Search

Navigation