Unsupervised classification of erroneous video object trajectories

Abstract

The paper proposes a method to detect failures in object tracking. Detection is done with the help of two types of errors, namely jump and stop errors. Jump errors occur when an abrupt change in object’s motion is observed, whereas stop errors occur when a moving object remains stationary for longer duration at any point. In our framework, moving objects are first tracked using well-known trackers and their trajectories are obtained. Discrepancies between trajectories are measured. We have shown that the proposed method can be reliable for detection of tracking failures. This can help to find error-free trajectories that are essential in various computer vision tasks. We have also shown that the tracking performance can be further improved while processing the output trajectories without much knowledge about the underlying tracking algorithms. The effect of tracking failure is investigated to identify erroneous trajectories. It has been observed that when a tracker fails, velocity profile of the moving object usually changes significantly. Based on this hypothesis, erroneous trajectories are detected and a set of error-free trajectories are marked and grouped. Two recently proposed tracking algorithms, namely real-time compressive tracker (CT) and real-time L1-tracker (L1APG), have been used to track the objects. We have tested our framework on five publicly available datasets containing more than 300 trajectories. Our experiments reveal that average classification rate of erroneous trajectories can be as high as 80.4% when objects are tracked using L1APG tracker. Accuracy can be as high as 81.2% when applied on trajectories obtained using CT tracker. Average accuracy of tracking increases significantly (19.2% with respect to L1APG tracker and 24.8% with respect to CT tracker) when the decision is taken using a fused framework.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

References

  1. Adam A, Rivlin E, Shimshoni I (2006) Robust fragments-based tracking using the integral histogram. In: Proceedings of the 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06). IEEE, vol 1, pp 798–805

  2. Ahmed A, Dogra D, Kar S, Kim B, Hill P, Bhaskar H (2016) Localization of region of interest in surveillance scene. Multimed Tools Appl. doi:10.1007/s11042-016-3762-y

    Google Scholar 

  3. Ali S, Shah M (2007) A lagrangian particle dynamics approach for crowd flow segmentation and stability analysis. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 1–6

  4. Andriyenko A, Schindler K (2011) Multi-target tracking by continuous energy minimization. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 1265–1272

  5. Babenko YMH B, Belongie S (2011) Robust object tracking with online multiple instance learning. IEEE Trans Pattern Anal Mach Intell 33(8):1619–1632

    Article  Google Scholar 

  6. Bae SH, Yoon KJ (2014) Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1218–1225

  7. Bailer C, Pagani A, Stricker D (2014) A superior tracking approach: Building a strong tracker through fusion. In: Computer vision–ECCV 2014. Springer, pp 170–185

  8. Bao C, Wu Y, Ling H, Ji H (2012) Real time robust l1 tracker using accelerated proximal gradient approach. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 1830–1837

  9. Benfold B, Reid I (2011) Stable multi-target tracking in real-time surveillance video. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 3457–3464

  10. Bernardin K, Stiefelhagen R (2008) Evaluating multiple object tracking performance: the clear mot metrics. J Image Video Process 2008:1

    Article  Google Scholar 

  11. Bibby C, Reid I (2008) Robust real-time visual tracking using pixel-wise posteriors. In: Computer vision–ECCV 2008. Springer, pp 831–844

  12. Birchfield S (2007) Klt: An implementation of the Kanade–Lucas–Tomasi feature tracker

  13. Comaniciu D, Ramesh V, Meer P (2000) Real-time tracking of non-rigid objects using mean shift. Proc IEEE Comput Soc Conf Comput Vis Pattern Recogn 2:142–149

    Google Scholar 

  14. Comaniciu D, Ramesh V, Meer P (2003) Kernel-based object tracking. IEEE Trans Pattern Anal Mach Intell 25(5):564–577

    Article  Google Scholar 

  15. Dehghan A, Idrees H, Zamir A, Shah M (2014) Automatic detection and tracking of pedestrians in videos with various crowd densities. In: Proceedings of the pedestrian and evacuation dynamics, pp 3–19

  16. Doğan S, Temiz M, Külür S (2010) Real time speed estimation of moving vehicles from side view images from an uncalibrated video camera. Sensors 10(5):4805–4824

    Article  Google Scholar 

  17. Dogra D, Ahmed A, Bhaskar H (2015a) Interest area localization using trajectory analysis in surveillance scenes. In: Proceedings of the 10th international conference on computer vision theory and applications, pp 478–485

  18. Dogra D, Reddy R, Subramanyam K, Ahmed A, Bhaskar H (2015b) Scene representation and anomalous activity detection using weighted region association graph. In: Proceedings of the 10th international conference on computer vision theory and applications, pp 104–112

  19. Dogra D, Ahmed A, Bhaskar H (2016) Smart video summarization using mealy machine-based trajectory modelling for surveillance applications. Multimed Tools Appl 75:6373–6401

    Article  Google Scholar 

  20. Doyle D, Jennings A, Black J (2014) Optical flow background estimation for real-time pan/tilt camera object tracking. Measurement 48:195–207

    Article  Google Scholar 

  21. Ekin A, Tekalp A, Mehrotra R (2003) Automatic soccer video analysis and summarization. IEEE Trans Image Process 12(7):796–807

    Article  Google Scholar 

  22. Gall J, Yao A, Razavi N, Gool LV, Lempitsky V (2011) Hough forests for object detection, tracking, and action recognition. IEEE Trans Pattern Anal Mach Intell 33(11):2188–2202

    Article  Google Scholar 

  23. Hua Y, Alahari K, Schmid C (2015) Online object tracking with proposal selection. In: Proceedings of the IEEE international conference on computer vision, pp 3092–3100

  24. Jurie F, Dhome M (2001) Real time tracking of 3d objects with occultations. In: Proceedings of international conference on image processing. IEEE, vol 1, pp 413–416

  25. Kaempchen N, Buehler M, Dietmayer K (2005) Feature-level fusion for free-form object tracking using laser scanner and video. In: Intelligent vehicles symposium, 2005. Proceedings. IEEE, pp 453–458

  26. Khan M, Valstar M, Pridmore T (2013) A multiple motion model tracker handling occlusion and rapid motion variation. In: British machine vision workshop

  27. Kim S, Kwak S, Feyereisl J, Han B (2012) Online multi-target tracking by large margin structured learning. In: Asian conference on computer vision. Springer, pp 98–111

  28. Lalonde M, Beaulieu M, Gagnon L (2001) Fast and robust optic disc detection using pyramidal decomposition and hausdorff-based template matching. IEEE Trans Med Imaging 20(11):1193–1200

    Article  Google Scholar 

  29. Lan X, Ma A, Yuen P (2014) Multi-cue visual tracking using robust feature-level fusion based on joint sparse representation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1194–1201

  30. Leibe B, Schindler K, Gool L (2007) Coupled detection and trajectory estimation for multi-object tracking. In: IEEE 11th international conference on computer vision (ICCV), IEEE, pp 1–8

  31. Li F, Cao T, Song Z, Wang W (2015) Object tracking based on locality-constrained linear coding joint sparse representation appearance model. In: IEEE 16th international conference on communication technology (ICCT). IEEE, pp 892–896

  32. Li X, Hu W, Shen C, Zhang Z, Dick A, Hengel A (2013) A survey of appearance models in visual object tracking. ACM Trans Intell Syst Technol (TIST) 4(4):58

    Google Scholar 

  33. Li Y, Huang C, Nevatia R (2009) Learning to associate: hybrid boosted multi-target tracker for crowded scene. In: IEEE conference on computer vision and pattern recognition, 2009 (CVPR, 2009). IEEE, pp 2953–2960

  34. Long C, Hua G, Kapoor A (2015) A joint gaussian process model for active visual recognition with expertise estimation in crowdsourcing. Int J Comput Vis. doi:10.1007/s11263-015-0834-9

    Google Scholar 

  35. Ma L, Lu J, Feng J, Zhou J (2015) Multiple feature fusion via weighted entropy for visual tracking. In: Proceedings of the IEEE international conference on computer vision, pp 3128–3136

  36. Mariano V, Min J, Park J, Kasturi R, Mihalcik D, Li H, Doermann D, Drayer T (2002) Performance evaluation of object detection algorithms. In: Proceedings of 16th international conference on pattern recognition, 2002. IEEE, vol 3, pp 965–969

  37. Martín R, Martínez J (2014) Evaluation of bounding box level fusion of single target video object trackers. In: Hybrid artificial intelligence systems. Springer, pp 200–210

  38. MS Temiz SK, Dogan S (2012) Real time speed estimation from monocular video. In: International archives of the photogrammetry, remote sensing and spatial information sciences, pp 427–432

  39. Nummiaro K, Koller-Meier E, Gool LV (2002) Object tracking with an adaptive color-based particle filter. In: Pattern recognition. Springer, pp 353–360

  40. Ogiela L, Ogiela MR (2015) Comparison of cognitive information systems supporting management tasks. In: 2015 international conference on intelligent networking and collaborative systems (INCOS). IEEE, pp 49–56

  41. Papadourakis V, Argyros A (2010) Multiple objects tracking in the presence of long-term occlusions. Comput Vis Image Underst 114(7):835–846

    Article  Google Scholar 

  42. Pettersen S, Johansen D, Oth (2014) Soccer video and player position dataset. In: Proceedings of the 5th ACM multimedia systems conference, pp 18–23

  43. Power D, Sharda R, Burstein F (2015) Decision support systems. Wiley Online Library, London

    Book  Google Scholar 

  44. Ristic B, Arulampalam S, Gordon N (2004) Beyond the kalman filter. IEEE Aerosp Electron Syst Mag 19(7):37–38

    Article  MATH  Google Scholar 

  45. Rousseeuw P (1987) Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math 20:53–65

    Article  MATH  Google Scholar 

  46. Saini R, Ahmed A, Dogra DP, Roy PP (2017a) Classification of object trajectories represented by high-level features using unsupervised learning. In: Proceedings of international conference on computer vision and image processing. Springer, pp 273–284

  47. Saini R, Ahmed A, Dogra DP, Roy PP (2017b) Surveillance scene segmentation based on trajectory classification using supervised learning. In: Proceedings of international conference on computer vision and image processing. Springer, pp 261–271

  48. Schoepflin T, Dailey D (2003) Dynamic camera calibration of roadside traffic management cameras for vehicle speed estimation. IEEE Trans Intell Transp Syst 4(2):90–98

    Article  Google Scholar 

  49. Shah M, Javed O, Shafique K (2007) Automated visual surveillance in realistic scenarios. IEEE Multimed 14(1):30–39

    Article  Google Scholar 

  50. Shao J, Loy C, Wang X (2014) Scene-independent group profiling in crowd. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 2227–2234

  51. Shin J, Kim S, Kang S, Lee S, Paik J, Abidi B, Abidi M (2005) Optical flow-based real-time object tracking using non-prior training active feature model. Real-Time Imaging 11(3):204–218

    Article  Google Scholar 

  52. Smeulders A, Chu D, Cucchiara R, Calderara S, Dehghan A, Shah M (2014) Visual tracking: an experimental survey. IEEE Trans Pattern Anal Mach Intell 36(7):1442–1468

    Article  Google Scholar 

  53. Stauffer C, Grimson W (2000) Learning patterns of activity using real-time tracking. IEEE Trans Pattern Anal Mach Intell 22(8):747–757

    Article  Google Scholar 

  54. Takala V, Pietikainen M (2007) Multi-object tracking using color, texture and motion. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 1–7

  55. Tang F, Brennan S, Zhao Q, Tao H (2007) Co-tracking using semi-supervised support vector machines. In: IEEE 11th international conference on computer vision (ICCV). IEEE, pp 1–8

  56. Wang X, Ma K, Ng G, Grimson W (2011) Trajectory analysis and semantic region modeling using nonparametric hierarchical bayesian models. Int J Comput Vis 95(3):287–312

    Article  Google Scholar 

  57. Wu H, Sankaranarayanan A, Chellappa R (2010) Online empirical evaluation of tracking algorithms. IEEE Trans Pattern Anal Mach Intell 32(8):1443–1458

    Article  Google Scholar 

  58. Wu Y, Lim J, Yang M (2015) Object tracking benchmark. IEEE Trans Pattern Anal Mach Intell 37(9):1834–1848

    Article  Google Scholar 

  59. Xiang Y, Alahi A, Savarese S (2015) Learning to track: online multi-object tracking by decision making. In: Proc IEEE Int Conf Comput Vis, pp 4705–4713

  60. Yilmaz A (2007) Object tracking by asymmetric kernel mean shift with automatic scale and orientation selection. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 1–6

  61. Yilmaz A, Li X, Shah M (2004) Contour-based object tracking with occlusion handling in video acquired using mobile cameras. IEEE Trans Pattern Anal Mach Intell 26(11):1531–1536

    Article  Google Scholar 

  62. Yilmaz A, Javed O, Shah M (2006) Object tracking: a survey. ACM Comput Surv 38(4):13

    Article  Google Scholar 

  63. Zhang K, Zhang L, Yang M (2012) Real-time compressive tracking. In: Proceedings of the European conference on computer vision, pp 864–877

  64. Zhong Y, Jain A, Dubuisson-Jolly M (2000) Object tracking using deformable templates. IEEE Trans Pattern Anal Mach Intell 22(5):544–549

    Article  Google Scholar 

  65. Zhou B, Wang X, Tang X (2012) Understanding collective crowd behaviors: learning a mixture model of dynamic pedestrian-agents. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 2871–2878

  66. Zhou B, Tang X, Zhang H, Wang X (2014a) Measuring crowd collectiveness. IEEE Trans Pattern Anal Mach Intell 36(8):1586–1599

    Article  Google Scholar 

  67. Zhou B, Tang X, Wang X (2015) Learning collective crowd behaviors with dynamic pedestrian-agents. Int J Comput Vis 111(1):50–68

    Article  Google Scholar 

  68. Zhou H, Yuan Y, Shi C (2009) Object tracking using sift features and mean shift. Comput Vis Image Underst 113(3):345–352

    Article  Google Scholar 

  69. Zhou Y, Rao C, Lu Q, Bai X, Liu W (2011) Multiple feature fusion for object tracking. In: Intelligent science and intelligent data engineering. Springer, pp 145–152

  70. Zhou Z, Wu D, Peng X, Zhu Z, Luo K (2014b) Object tracking based on camshift with multi-feature fusion. J Softw 9(1):147–153

    Google Scholar 

  71. Zivkovic Z, Krose B (2004) An em-like algorithm for color-histogram-based object tracking. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition (CVPR). IEEE, vol 1, pp I–798

Download references

Acknowledgements

This study has not been funded by any agency.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Sk. Arif Ahmed.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest regarding the publication of this paper.

Ethical approval

This article does not contain any studies on human participants or animals.

Informed consent

Consent was obtained from all volunteers/individual participants included in the study.

Additional information

Communicated by V. Loia.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ahmed, S.A., Dogra, D.P., Kar, S. et al. Unsupervised classification of erroneous video object trajectories. Soft Comput 22, 4703–4721 (2018). https://doi.org/10.1007/s00500-017-2656-x

Download citation

Keywords

  • Object tracking
  • Tracker fusion
  • Unsupervised tracking