Abstract
Occasions such as stalled vehicles or crashes led by abnormal trajectories should be instantly identified and then dealt with quickly by the city traffic management system for the sake of road safety. However, a fast and accurate automatic detection system based on machine learning in general meets with great challenges from the shortage of recorded accident data, resulting in low detection accuracy. Many existing studies implement a two-level detection approach: stalled vehicles are detected at the stationary level, while abnormal trajectories are detected at the mobile level. This paper proposes a novel triple-layer framework to distribute these two levels to three parallel layers for maximum efficiency. A straightforward background extraction algorithm is applied at the beginning of this framework for motion-stationary distribution. Layer 1 implements a lightweight optical-flow-based feature extraction algorithm to convert the mobile visual features to learnable data. With a clustering algorithm that learns the common trajectories in an unsupervised manner, abnormal trajectories are detected in Layer 2. Simultaneously, in Layer 3, a custom-trained object detection algorithm is applied to detect the stall/crashed vehicles. The computational efficiency is improved and the detection accuracy is boosted. Experiments conducted on Nvidia AI City Challenge Dataset demonstrate the effectiveness of our LRATD (Lightweight Real-Time Abnormal Trajectory Detection framework) in terms of \(104\%\) gain in detection speed compared to the fastest entry, while achieving 0.935 S4-Score, only \(2.1\%\) less than the current state-of-the-art method. Overall, the performance of LRATD opens the possibility of its real-life application.
Similar content being viewed by others
References
Abdallah A, Gouiffès M, Lacassagne L (2016) A modular system for global and local abnormal event detection and categorization in videos. Mach Vis Appl 27:463–481. https://doi.org/10.1007/s00138-016-0752-z
Aboah A, Shoman M, Mandal V, Davami S, Adu-Gyamfi Y, Sharma A (2021) A vision-based system for traffic anomaly detection using deep learning and decision trees. CoRR arXiv:2104.06856
Adam A, Rivlin E, Shimshoni I, Reinitz D (2008) Robust real-time unusual event detection using multiple fixed-location monitors. IEEE Trans Pattern Anal Mach Intell 30(3):555–560. https://doi.org/10.1109/TPAMI.2007.70825
Al-Nawashi M, Al-hazaimeh O, Saraee M (2017) A novel framework for intelligent surveillance system based on abnormal human activity detection in academic environments. Neural Comput Appl. https://doi.org/10.1007/s00521-016-2363-z
Al-Smadi M, Abdulrahim K, Seman K, Abdul Salam R (2020) A new spatio-temporal background-foreground bimodal for motion segmentation and detection in urban traffic scenes. Neural Comput Appl 32:1–17. https://doi.org/10.1007/s00521-019-04458-5
Andrade E, Blunsden S, Fisher R (2005) Characterisation of optical flow anomalies in pedestrian traffic. In: The IEE international symposium on imaging for crime detection and prevention, 2005. ICDP 2005., 73–78 https://doi.org/10.1049/ic:20050073
Bai S, He Z, Lei Y, Wu W, Zhu C, Sun M, Yan J (2019) Traffic anomaly detection via perspective map based on spatial-temporal information matrix. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops
Bansod S, Nandedkar A (2017) Detection and localization of anomalies from videos based on optical flow magnitude and direction. https://doi.org/10.2991/iccasp-16.2017.68
Bewley A, Ge Z, Ott L, Ramos F, Upcroft B (2016) Simple online and realtime tracking. CoRR arXiv:1602.00763
Biradar KM, Gupta A, Mandal M, Vipparthi SK (2019) Challenges in time-stamp aware anomaly detection in traffic videos. CoRR arXiv:1906.04574
Buch N, Velastin SA, Orwell J (2011) A review of computer vision techniques for the analysis of urban traffic. IEEE Trans Intell Transp Syst 12(3):920–939. https://doi.org/10.1109/TITS.2011.2119372
Cai Z, Vasconcelos N (2017) Cascade R-CNN: delving into high quality object detection. CoRR arXiv:1712.00726
Chang MC, Chiang CK, Tsai CM, Chang YK, Chiang HL, Wang YA, Chang SY, Li YL, Tsai MS, Tseng HY (2020) Ai city challenge 2020—computer vision for smart transportation applications. In: 2020 IEEE/CVF Conference on computer vision and pattern recognition workshops (CVPRW), pp 2638–2647. https://doi.org/10.1109/CVPRW50498.2020.00318
Chang MC, Wei J, Zhu ZA, Chen YM, Hu CS, Jiang MX, Chiang CK (2019) Ai city challenge 2019—city-scale video analytics for smart transportation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops
Chang MC, Wei Y, Song N, Lyu S (2018) Video analytics in smart transportation for the aic’18 challenge. In: 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 61–617. https://doi.org/10.1109/CVPRW.2018.00016
Chen J, Ding G, Yang Y, Han W, Xu K, Gao T, Zhang Z, Ouyang W, Cai H, Chen Z (2021) Dual-modality vehicle anomaly detection via bilateral trajectory tracing
Chong YS, Tay YH (2017) Abnormal event detection in videos using spatiotemporal autoencoder. CoRR arXiv:1701.01546
Comaniciu D, Meer P (2002) Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 24(5):603–619. https://doi.org/10.1109/34.1000236
Doshi K, Yilmaz Y (2020) Fast unsupervised anomaly detection in traffic videos. In: 2020 IEEE/CVF Conference on computer vision and pattern recognition workshops (CVPRW), pp 2658–2664. https://doi.org/10.1109/CVPRW50498.2020.00320
Doshi K, Yilmaz Y (2021) An efficient approach for anomaly detection in traffic videos. CoRR arXiv:2104.09758
Feng W, Ji D, Wang Y, Chang S, Ren H, Gan W (2018) Challenges on large scale surveillance video analysis, pp 69–697. https://doi.org/10.1109/CVPRW.2018.00017
Giannakeris P, Kaltsa V, Avgerinakis K, Briassouli A, Vrochidis S, Kompatsiaris I (2018) Speed estimation and abnormality detection from surveillance cameras. In: 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 93–936. https://doi.org/10.1109/CVPRW.2018.00020
Girshick RB (2015) Fast R-CNN. CoRR arXiv:1504.08083
Hassaballah M, Awad A (2020) Deep learning in computer vision: principles and application. CRC Press, Boca Raton. https://doi.org/10.1201/9781351003827
Hassaballah M, Kenk MA, Muhammad K, Minaee S (2021) Vehicle detection and tracking in adverse weather using a deep learning framework. IEEE Trans Intell Transp Syst 22:4230–4242
He K, Gkioxari G, Dollár P, Girshick RB (2017) Mask R-CNN. CoRR arXiv:1703.06870
Hospedales T, Gong S, Xiang T (2012) Video behaviour mining using a dynamic topic model. IJCV 98:303–323. https://doi.org/10.1007/s11263-011-0510-7
Huang J, Rathod V, Sun C, Zhu M, Korattikara A, Fathi A, Fischer I, Wojna Z, Song Y, Guadarrama S, Murphy K (2016) Speed/accuracy trade-offs for modern convolutional object detectors. CoRR arXiv:1611.10012
Jeong H, Yoo Y, Yi K, Choi J (2014) Two-stage online inference model for traffic pattern analysis and anomaly detection. Mach Vis Appl 25:1501–1517. https://doi.org/10.1007/s00138-014-0629-y
Jin X, Han J (2010) K-means clustering. Springer, Boston, pp 563–564. https://doi.org/10.1007/978-0-387-30164-8_425
Jocher G, Stoken A, Borovec J, NanoCode012 ChristopherSTAN Changyu, L, Laughing tkianai Hogan, A, lorenzomammana yxNONG, AlexWang1900 Diaconu L, Marc wanghaoyang0106, ml5ah Doug, Ingham F, Frederik Guilhen Hatovix Poznanski J, Fang J, Yu L, changyu98 Wang M, Gupta N, Akhtar O, PetrDvoracek Rai P (2020) ultralytics/yolov5: v3.1—bug fixes and performance improvements. https://doi.org/10.5281/zenodo.4154370
Kaltsa V, Briassouli A, Kompatsiaris I, Strintzis MG (2014) Swarm-based motion features for anomaly detection in crowds. In: 2014 IEEE international conference on image processing (ICIP), pp 2353–2357. https://doi.org/10.1109/ICIP.2014.7025477
Kuettel D, Breitenstein MD, Van Gool L, Ferrari V (2010) What’s going on? discovering spatio-temporal dependencies in dynamic scenes. In: 2010 IEEE computer society conference on computer vision and pattern recognition, pp 1951–1958. https://doi.org/10.1109/CVPR.2010.5539869
Kumaran SK, Dogra DP, Roy PP (2019) Anomaly detection in road traffic using visual surveillance: a survey. CoRR arXiv:1901.08292
Lee DG, Suk HI, Park SK, Lee SW (2015) Motion influence map for unusual human activity detection and localization in crowded scenes. IEEE Trans Circuits Syst Video Technol 25(10):1612–1623. https://doi.org/10.1109/TCSVT.2015.2395752
Lee S, Kim HG, Ro YM (2018) STAN: spatio-temporal adversarial networks for abnormal event detection. CoRR arXiv:1804.08381
Li Y, Wu J, Bai X, Yang X, Tan X, Li G, Wen S, Zhang H, Ding E (2020) Multi-granularity tracking with modularlized components for unsupervised vehicles anomaly detection. In: 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 2501–2510. https://doi.org/10.1109/CVPRW50498.2020.00301
Lin T, Goyal P, Girshick RB, He K, Dollár P (2017) Focal loss for dense object detection. CoRR arXiv:1708.02002
Lin T, Maire M, Belongie SJ, Bourdev LD, Girshick RB, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft COCO: common objects in context. CoRR arXiv:1405.0312
Lin W, Zhou Y, Xu H, Yan J, Xu M, Wu J, Liu Z (2017) A tube-and-droplet-based approach for representing and analyzing motion trajectories. IEEE Trans Pattern Anal Mach Intell 39(8):1489–1503. https://doi.org/10.1109/TPAMI.2016.2608884
Liu W, Anguelov D, Erhan D, Szegedy C, Reed SE, Fu C, Berg AC (2015) SSD: single shot multibox detector. CoRR arXiv:1512.02325
Liu W, Luo W, Lian D, Gao S (2017) Future frame prediction for anomaly detection—a new baseline. CoRR arXiv:1712.09867
Loy CC, Xiang T, Gong S (2011) Stream-based active unusual event detection. In: Kimmel R, Klette R, Sugimoto A (eds) Computer Vision—ACCV 2010. Springer, Berlin, pp 161–175
Lu C, Shi J, Jia J (2013) Abnormal event detection at 150 fps in matlab. In: 2013 IEEE International Conference on Computer Vision, pp 2720–2727. https://doi.org/10.1109/ICCV.2013.338
Lucas B, Kanade T (1981) An iterative image registration technique with an application to stereo vision (ijcai)
Luo W, Liu W, Gao S (2017) A revisit of sparse coding based anomaly detection in stacked rnn framework. In: 2017 IEEE International conference on computer vision (ICCV), pp 341–349. https://doi.org/10.1109/ICCV.2017.45
Ma X, Wu J, Xue S, Yang J, Sheng QZ, Xiong H (2021) A comprehensive survey on graph anomaly detection with deep learning. CoRR arXiv:2106.07178
Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. In: 2010 IEEE computer society conference on computer vision and pattern recognition, pp 1975–1981. https://doi.org/10.1109/CVPR.2010.5539872
Mahalakshi V, Anumol KP, Kokiladeepa M (2016) Abnormal visual events detection using optical flow orientation histogram
Mao T, Zhang W, He H, Lin Y, Kale V, Stein A, Kostic Z (2018) Aic2018 report: Traffic surveillance research. In: 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 85–857. https://doi.org/10.1109/CVPRW.2018.00019
Medel JR, Savakis AE (2016) Anomaly detection in video using predictive convolutional long short-term memory networks. CoRR arXiv:1612.00390
Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. In: 2009 IEEE conference on computer vision and pattern recognition, pp 935–942. https://doi.org/10.1109/CVPR.2009.5206641
Naphade M, Tang Z, Chang MC, Anastasiu DC, Sharma A, Chellappa R, Wang S, Chakraborty P, Huang T, Hwang JN, Lyu S (2019) The 2019 ai city challenge. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops
Nguyen KT, Hoang TH, Tran MT, Le TN, Bui NM, Do TL, Vo-Ho VK, Luong QA, Tran MK, Nguyen TA, Truong TD, Nguyen VT, Do MN (2019) Vehicle re-identification with learned representation and spatial verification and abnormality detection with multi-adaptive vehicle detectors for traffic video analysis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops
Pang G, Yan C, Shen C, van den Hengel A, Bai X (2020) Self-trained deep ordinal regression for end-to-end video anomaly detection. CoRR arXiv:2003.06780
Pathak D, Sharang A, Mukerjee A (2015) Anomaly localization in topic-based analysis of surveillance videos. In: 2015 IEEE winter conference on applications of computer vision, pp 389–395. https://doi.org/10.1109/WACV.2015.58
Peri N, Khorramshahi P, Rambhatla SS, Shenoy V, Rawat S, Chen JC, Chellappa R (2020) Towards real-time systems for vehicle re-identification, multi-camera tracking, and anomaly detection. In: 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 2648–2657. https://doi.org/10.1109/CVPRW50498.2020.00319
Ranjith R, Athanesious JJ, Vaidehi V (2015) Anomaly detection using dbscan clustering technique for traffic video surveillance. In: 2015 Seventh international conference on advanced computing (ICoAC), pp 1–6. https://doi.org/10.1109/ICoAC.2015.7562795
Reddy V, Sanderson C, Lovell BC (2011) Improved anomaly detection in crowded scenes via cell-based analysis of foreground speed, size and texture. In: CVPR 2011 workshops, pp 55–61. https://doi.org/10.1109/CVPRW.2011.5981799
Redmon J, Divvala SK, Girshick RB, Farhadi A (2015) You only look once: Unified, real-time object detection. CoRR arXiv:1506.02640
Redmon J, Farhadi A (2018) Yolov3: an incremental improvement. CoRR arXiv:1804.02767
Ren S, He K, Girshick RB, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. CoRR arXiv:1506.01497
Sandhan T, Sethi A, Srivastava T, Choi JY (2013) Unsupervised learning approach for abnormal event detection in surveillance video by revealing infrequent patterns. In: 2013 28th International conference on image and vision computing New Zealand (IVCNZ 2013), pp 494–499. https://doi.org/10.1109/IVCNZ.2013.6727064
Shine L, Vaishnav MA, Jiji CV (2020) Fractional data distillation model for anomaly detection in traffic videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops
Shine L, Edison A, JCV (2019) A comparative study of faster r-cnn models for anomaly detection in 2019 ai city challenge. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556
Sultani W, Chen C, Shah M (2018) Real-world anomaly detection in surveillance videos. CoRR arXiv:1801.04264
Szegedy C, Liu W, Jia Y, Sermanet P, Reed SE, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2014) Going deeper with convolutions. CoRR arXiv:1409.4842
Tang Z, Naphade M, Liu M, Yang X, Birchfield S, Wang S, Kumar R, Anastasiu DC, Hwang J (2019) Cityflow: A city-scale benchmark for multi-target multi-camera vehicle tracking and re-identification. CoRR arXiv:1903.09254
Tran MT, Nguyen TV, Hoang TH, Le TN, Nguyen KT, Dinh DT, Nguyen TA, Nguyen HD, Hoang XN, Nguyen TT, Vo-Ho VK, Do TL, Nguyen L, Le MQ, Nguyen-Dinh HP, Pham TT, Nguyen XV, Nguyen ER, Tran QC, Tran H, Dao H, Tran MK, Nguyen QT, Nguyen TP, Vu-Le TA, Diep GH, Do MN (2020) itask—intelligent traffic analysis software kit. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops
Ullah H, Ullah M (2019) A hybrid social influence model for pedestrian motion segmentation. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3527-9
Varadarajan J, Emonet R, Odobez JM (2013) A sequential topic model for mining recurrent activities from long term video logs. Int J Comput Vis. https://doi.org/10.1007/s11263-012-0596-6
Wang G, Wang Y, Zhang H, Gu R, Hwang J (2018) Exploit the connectivity: multi-object tracking with trackletnet. CoRR arXiv:1811.07258
Wang G, Yuan X, Zheng A, Hsu HM, Hwang JN (2019) Anomaly candidate identification and starting time estimation of vehicles from traffic videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops
Wang H, Wu J, Hu W, Wu X (2019) Detecting and assessing anomalous evolutionary behaviors of nodes in evolving social networks. ACM Trans Knowl Discov Data. https://doi.org/10.1145/3299886
Wang J, Fu W, Lu H, Ma S (2014) Bilayer sparse topic model for scene analysis in imbalanced surveillance videos. IEEE Trans Image Process 23(12):5198–5208. https://doi.org/10.1109/TIP.2014.2363408
Wang J, Xia L, Hu X, Xiao Y (2019) Abnormal event detection with semi-supervised sparse topic model. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3417-1
Wang X, Ma X, Grimson W (2009) Unsupervised activity perception in crowded and complicated scenes using hierarchical bayesian models. IEEE Trans Pattern Anal Mach Intell 31(3):539–555. https://doi.org/10.1109/TPAMI.2008.87
Wei J, Zhao J, Zhao Y, Zhao Z (2018) Unsupervised anomaly detection for traffic surveillance based on background modeling. In: 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 129–1297. https://doi.org/10.1109/CVPRW.2018.00025
Wen J, Lai Z, Ming Z, Wong WK, Zhong Z (2017) Directional gaussian model for automatic speeding event detection. IEEE Trans Inf Forensics Secur 12(10):2292–2307. https://doi.org/10.1109/TIFS.2017.2705623
Wen L, Du D, Cai Z, Lei Z, Chang M, Qi H, Lim J, Yang M, Lyu S (2015) DETRAC: A new benchmark and protocol for multi-object tracking. CoRR arXiv:1511.04136
Wojke N, Bewley A, Paulus D (2017) Simple online and realtime tracking with a deep association metric. CoRR arXiv:1703.07402
Wu J, Wang X, Xiao X, Wang Y (2021) Box-level tube tracking and refinement for vehicles anomaly detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops, pp 4112–4118
Wu S, Moore BE, Shah M (2010) Chaotic invariants of lagrangian particle trajectories for anomaly detection in crowded scenes. In: 2010 IEEE computer society conference on computer vision and pattern recognition, pp 2054–2060. https://doi.org/10.1109/CVPR.2010.5539882
Wu S, Moore BE, Shah M (2010) Chaotic invariants of lagrangian particle trajectories for anomaly detection in crowded scenes. In: 2010 IEEE computer society conference on computer vision and pattern recognition, pp 2054–2060. https://doi.org/10.1109/CVPR.2010.5539882
Xie S, Zhang X, Cai J (2019) Video crowd detection and abnormal behavior model detection based on machine learning method. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3692-x
Xu Y, Ouyang X, Cheng Y, Yu S, Xiong L, Ng CC, Pranata S, Shen S, Xing J (2018) Dual-mode vehicle motion pattern learning for high performance road traffic anomaly detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) workshops
Zhao Y, Wu W, He Y, Li Y, Tan X, Chen S (2021) Good practices and A strong baseline for traffic anomaly detection. CoRR arXiv:2105.03827
Zhong H, Shi J, Visontai M (2004) Detecting unusual activity in video. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004., 2, II–II . https://doi.org/10.1109/CVPR.2004.1315249
Zivkovic Z (2004) Improved adaptive gaussian mixture model for background subtraction. In: Proceedings of the 17th international conference on pattern recognition, 2004. ICPR 2004., 2, 28–31 2. https://doi.org/10.1109/ICPR.2004.1333992
Zivkovic Z, van der Heijden F (2006) Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recognit Lett 27(1636734P):773–780. https://doi.org/10.1016/j.patrec.2005.11.005
Acknowledgements
This work was supported in part by National Key Research and Development Project No.2019YFC1511003, National Natural Science Foundation of China (NSFC) No.61803004, Aeronautical Science Foundation of China No.20161375002 through grants for our project.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Supplementary file 1 (avi 0 KB)
Supplementary file 2 (avi 0 KB)
Supplementary file 3 (avi 0 KB)
Supplementary file 4 (mp4 0 KB)
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhang, C., Ren, K. LRATD: a lightweight real-time abnormal trajectory detection approach for road traffic surveillance. Neural Comput & Applic 34, 22417–22434 (2022). https://doi.org/10.1007/s00521-022-07626-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-022-07626-2