Skip to main content
SpringerLink
Log in

An end to end trained hybrid CNN model for multi-object tracking

  • 1220: Visual and Sensory Data Processing for Real Time Intelligent Surveillance System
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

A robust MOT (multi-object tracking) is very crucial for computer vision applications such as crowd density estimation and autonomous vehicles. Most of the existing mot approaches perform object tracking in a two task manner such as motion estimation and Re-identification but these approaches pose some drawbacks like the model is not end-to-end trained, the Re-Id required lots of identity switches thus incurred computational overhead and the performance further degrades in complex crowd scenarios. To overcome such drawbacks we are motivated to design an end-to-end trained DNN for MOT. The proposed model utilizes a matching technique that utilizes the relative scale between the boundary boxes and relative position calculates the relative distance between the objects for MOT. To solve the problems, we proposed a matching technique that poses two subtasks to efficiently scale up a single shot DNN tracking approach for an indefinite number of objects in the video frames. The proposed method uses a relative scale and relative position to matching between the detected and targeted objects. The achieved state-of-the-art results of the tasks allow to obtain high accuracy of tracking with detection and surpasses existing state-of-the-art methods by a huge margin on various public datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Bae SH, Yoon KJ (2018) Confidence-based data association and discriminative deep appearance learning for robust online multi-object tracking. IEEE Trans Pattern Anal Mach Intell 40(3):595–610

    Article  Google Scholar 

  2. Bajracharya M, Moghaddam B, Howard A, Brennan S, Matthies LH (2009) A fast stereo-based system for detecting and tracking pedestrians from a moving vehicle. Int J Rob Res 28(11–12):1466–1485

    Article  Google Scholar 

  3. Bergmann P, Meinhardt T, Leal-Taixe L (2019) Tracking without bells and whistles. Proc. IEEE Int. Conf. Comput. Vis., vol. 2019-Octob, no. October, pp. 941–951

  4. Bernardin K, Stiefelhagen R (2008) Evaluating Multiple Object Tracking Performance : The CLEAR MOT Metrics vol. 2008, 2008

  5. Bewley A, Ge Z, Ott L, Ramos F, Upcroft B (2016) Simple online and realtime tracking. Proc. - Int. Conf. Image Process. ICIP, vol. 2016-Augus, pp. 3464–3468

  6. Butler DJ,Wulff J, Stanley GB, Black MJ (2012) For Optical Flow Evaluation,” pp. 611–625

  7. Chen L, Ai H, Shang C, Zhuang Z, Bai B (2018) Online multi-object tracking with convolutional neural networks. Proc. - Int. Conf. Image Process. ICIP, vol. 2017-Septe, pp. 645–649

  8. Choi W, Savarese S (2010) Multiple target tracking in world coordinate with single, minimally calibrated camera, pp 553–567

  9. Chu P, Fan H, Tan CC, Ling H (2019) Online multi-object tracking with instance-aware tracker and dynamic model refreshment. Proc. - 2019 IEEE Winter Conf. Appl. Comput. Vision, WACV 2019, pp. 161–170

  10. Dash PP, Mishra SK, Sethi J (2012) Kernel based object tracking using color histogram technique 2 fundamentals of object tracking in video sequences. 2(4):28–35

  11. Dicle C, Camps OI, Sznaier M (2013) The way they move: Tracking multiple targets with similar appearance. Proc. IEEE Int. Conf. Comput. Vis., pp 2304–2311

  12. Fang K, Xiang Y, Li X, Savarese S (2018) Recurrent Autoregressive Networks for Online Multi-object Tracking. Proc. - 2018 IEEE Winter Conf. Appl. Comput. Vision, WACV 2018, vol. 2018-Janua, pp. 466–475

  13. Feng W, Hu Z, Wu W, Yan J, Ouyang W (2019) Multi-object tracking with multiple cues and switcher-aware classification. arXiv, no. c

  14. Fragkiadaki K, Zhang W, Zhang G, Shi J (2012) Two-granularity tracking: mediating trajectory and detection graphs for tracking under occlusions. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol 7576 LNCS, no. PART 5, pp 552–565

  15. Fulkerson B, Vedaldi A, Soatto S (2008) Localizing objects with smart dictionaries. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol 5302 LNCS, no PART 1, pp 179–192

  16. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol 2016-Decem, pp 770–778

  17. Ilg E, Mayer N, Saikia T, Keuper M, Dosovitskiy A, Brox T FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

  18. Kalman RE (1960) A new approach to linear filtering and prediction problems. J Fluids Eng Trans ASME 82(1):35–45

    Article  MathSciNet  Google Scholar 

  19. Kampker A, Sefati M, Abdul Rachman AS, Kreisköther K, Campoy P (2018) Towards multi-object detection and tracking in urban scenario under uncertainties. VEHITS 2018 - Proc 4th Int Conf Veh Technol Intell Transp Syst vol 2018, no. Vehits 2018, pp 156–167. https://doi.org/10.5220/0006706101560167

  20. Keuper M, Tang S, Andres B, Brox T, Schiele B (2018) Motion segmentation & multiple object tracking by correlation co-clustering. IEEE Trans Pattern Anal Mach Intell 8828(c):1–13

  21. Kim C, Li F, Rehg JM (2018) Multi-object tracking with neural gating using bilinear LSTM, vol 11212 LNCS. Springer International Publishing

  22. Kuhn HW (2005) The Hungarian method for the assignment problem. Nav Res Logist 52(1):7–21

    Article  Google Scholar 

  23. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  24. Mahmoudi N (2018) Multi-target tracking using CNN-based features : CNNMTT. https://doi.org/10.1007/s11042-018-6467-6

  25. Milan A, Schindler K, Roth S (2016) Multi-target tracking by discrete-continuous energy minimization. IEEE Trans Pattern Anal Mach Intell 38(10):2054–2068

    Article  Google Scholar 

  26. Milan A, Rezatofighi SH, Dick A, Reid I, Schindler K (2017) Online multi-target tracking using recurrent neural networks. 31st AAAI Conf. Artif Intell AAAI 2017, pp 4225–4232

  27. Mitzel D (2012) Taking mobile multi-object tracking to the next level. Eccv

  28. Pang B, Li Y, Zhang Y, Li M, Lu C (2020) Tubetk: Adopting tubes to track multi-object in a one-step training model. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit, pp 6307–6317

  29. Pellegrini S, Ess A, Schindler K, Van Gool L (2009) You ’ ll never walk alone : modeling social behavior for multi-target tracking, no. Iccv, pp 261–268. https://doi.org/10.1109/ICCV.2009.5459260

  30. Peng J et al (2020) Chained-Tracker: Chaining paired attentive regression results for end-to-end joint multiple-object detection and tracking

  31. Qin Z, Shelton CR (2016) Social grouping for multi-target tracking and head pose estimation in video. IEEE Trans Pattern Anal Mach Intell 38(10):2082–2095

    Article  Google Scholar 

  32. Roth M, Bauml M, Nevatia R, Stiefelhagen R (2012) Robust multi-pose face tracking by multi-stage tracklet association. Proc. - Int. Conf. Pattern Recognit., pp 1012–1016

  33. Sadeghian A, Alahi A,Savarese S (2017) Tracking the Untrackable: Learning to Track Multiple Cues with Long-Term Dependencies. Proc IEEE Int Conf Comput Vis, vol 2017-Octob, pp 300–311

  34. Sanchez-Matilla R, Poiesi F, Cavallaro A (2016) Online multi-target tracking with strong and weak detections. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol 9914 LNCS, pp 84–99

  35. Sun SJ, Akhtar N, Song HS, Mian A, Shah M (2018) Deep affinity network for multiple object tracking. arXiv 13(9):1–15

  36. Wang Z, Zheng L, Liu Y, Wang S (2019) Towards real-time multi-object tracking

  37. Wojke N, Bewley A, Paulus D (2018) Simple online and realtime tracking with a deep association metric. Proc - Int Conf Image Process ICIP, vol 2017-September, pp 3645–3649

  38. Wu Z, Thangali A, Sclaroff S, Betke M (1927) Coupling detection and data association for multiple object tracking. https://doi.org/10.1109/CVPR.2012.6247896

  39. Wu B, Wan A, Yue X, Keutzer K (2018) SqueezeSeg: convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3D LiDAR Point Cloud. Proc - IEEE Int Conf Robot Autom, pp 1887–1893

  40. Wu H, Hu Y, Wang K, Li H, Nie L, Cheng H (2019) Instance-aware representation learning and association for online multi-person tracking. Pattern Recognit 94:25–34

    Article  Google Scholar 

  41. Xiang Y, Alahi A, Savarese S (2015) Learning to track: Online multi-object tracking by decision making. Proc IEEE Int Conf Comput Vis, vol. 2015 Inter, pp 4705–4713. https://doi.org/10.1109/ICCV.2015.534

  42. Xie C, Xiang Y, Harchaoui Z, Fox D Object discovery in videos as foreground motion clustering

  43. Xu J, Cao Y, Zhang Z, Hu H (2019) Spatial-temporal relation networks for multi-object tracking. Proc IEEE Int Conf Comput Vis 2019:3987–3997

  44. Yang B, Nevatia R (2012) An online learned CRF model for multi-target tracking. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp 2034–2041. https://doi.org/10.1109/CVPR.2012.6247907

  45. Yang M, Wu Y, Jia Y (2017) A hybrid data association framework for robust online multi-object tracking. IEEE Trans Image Process 26(12):5667–5679

    Article  MathSciNet  MATH  Google Scholar 

  46. Yoon YC, Boragule A, Song YM, Yoon K, Jeon M (2018) Online multi-object tracking with historical appearance matching and scene adaptive detection filtering

  47. Yu F, Li W, Li Q, Liu Y, Shi X, Yan J (2016) POI: Multiple object tracking with high performance detection and appearance feature. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol 9914 LNCS, pp 36–42

  48. Zhang J et al (2020) Multiple object tracking by flowing and fusing

  49. Zhang Y, Wang C, Wang X, Zeng W, Liu W (2020) FairMOT: on the fairness of detection and re-identification in multiple object tracking, pp 1–13

  50. Zhao Q, Road WX (2018) MULTI-OBJECT TRACKING USING ONLINE METRIC LEARNING WITH LONG SHORT-TERM MEMORY Xingyu Wan Jinjun Wang Zhifeng Kong Xi’an Jiaotong University Institute of Artificial Intelligence and Robotics No . 1 Tanhu 2nd Road, Canglong island, Jiangxia district 2018 25th IEEE Int. Conf. Image Process, pp 788–792. https://doi.org/10.1109/ICIP.2018.8451174

  51. Zhou Z, Xing J, Zhang M, Hu W (2018) Online multi-target tracking with tensor-based high-order graph matching. Proc Int Conf Pattern Recognit 2018:1809–1814. https://doi.org/10.1109/ICPR.2018.8545450

  52. Zhou X, Koltun V, Krähenbühl P, Austin UT, Labs I (2020) Tracking objects as points

  53. Zhu J, Yang H, Liu N, Kim M, Zhang W, Yang MH (2018) Online multi-object tracking with dual matching attention networks. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol 11209 LNCS, pp 379–396

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology, Banaras Hindu University, Uttar Pradesh, Varanasi, 221005, India

    Divya Singh & Rajeev Srivastava

Authors
  1. Divya Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rajeev Srivastava
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Divya Singh.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Singh, D., Srivastava, R. An end to end trained hybrid CNN model for multi-object tracking. Multimed Tools Appl 81, 42209–42221 (2022). https://doi.org/10.1007/s11042-021-11463-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11463-1

Keyword

Search

Navigation