Abstract
Analyzing a vehicle’s abnormal behavior in surveillance videos is a challenging field, mainly due to the wide variety of anomaly cases and the complexity of surveillance videos. In this study, a novel intelligent vehicle behavior analysis framework based on a digital twin is proposed. First, detecting vehicles based on deep learning is implemented, and Kalman filtering and feature matching are used to track vehicles. Subsequently, the tracked vehicle is mapped to a digital-twin virtual scene developed in the Unity game engine, and each vehicle’s behavior is tested according to the customized detection conditions set up in the scene. The stored behavior data can be used to reconstruct the scene again in Unity for a secondary analysis. The experimental results using real videos from traffic cameras illustrate that the detection rate of the proposed framework is close to that of the state-of-the-art abnormal event detection systems. In addition, the implementation and analysis process show the usability, generalization, and effectiveness of the proposed framework.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
JIANG F, WU Y, KATSAGGELOS A K. Abnormal event detection from surveillance video by dynamic hierarchical clustering [C]//2007 IEEE International Conference on Image Processing. San Antonio, TX: IEEE, 2007: V145–V148.
WANG J Q, LI H P. Intelligent analysis of vehicle illegal behavior based on deep learning algorithm [J]. Journal of Shanghai Ship and Shipping Research Institute, 2019, 42(4): 49–54 (in Chinese).
FULLER A, FAN Z, DAY C, et al. Digital twin: Enabling technologies, challenges and open research [J]. IEEE Access, 2020, 8: 108952–108971.
KUMAR S A P, MADHUMATHI R, CHELLIAH P R, et al. A novel digital twin-centric approach for driver intention prediction and traffic congestion avoidance [J]. Journal of Reliable Intelligent Environments, 2018, 4(4): 199–209.
HAM Y, KIM J. Participatory sensing and digital twin City: Updating virtual City models for enhanced risk-informed decision-making [J]. Journal of Management in Engineering, 2020, 36(3): 04020005.
JOCHER G, NISHIMURA K, MINEEVA T, et al. YOLOv5 (2020) [EB/OL]. (2020-07-10) [2021-01-30]. https://github.com/ultralytics/YOLOv5.
WANG C Y, MARK LIAO H Y, WU Y H, et al. CSPNet: A new backbone that can enhance learning capability of CNN [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Seattle, WA: IEEE, 2020: 1571–1580.
LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI: IEEE, 2017: 936–944.
WANG W H, XIE E Z, SONG X G, et al. Efficient and accurate arbitrary-shaped text detection with pixel aggregation network [C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 8439–8448.
LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: Common objects in context [C]//Computer Vision-ECCV 2014. Cham: Springer, 2014: 740–755.
WEN L Y, DU D W, CAI Z W, et al. UA-DETRAC: A new benchmark and protocol for multi-object detection and tracking [J]. Computer Vision and Image Understanding, 2020, 193: 102907.
PUNN N S, SONBHADRA S K, AGARWAL S, et al. Monitoring COVID-19 social distancing with person detection and tracking via fine-tuned YOLO v3 and Deepsort techniques [EB/OL]. [2021-01-30]. https://arxiv.org/pdf/2005.01385.pdf.
MIKHAIL E M, BETHEL J S, MCGLONE J C. Introduction to modern photogrammetry [M]. New York: John Wiley, 2001.
HUSSEIN A, GARCÍA F, OLAVERRI-MONREAL C. ROS and unity based framework for intelligent vehicles control and simulation [C]//2018 IEEE International Conference on Vehicular Electronics and Safety (ICVES). Madrid: IEEE, 2018: 1–6.
ZHAO C H, LI Y W, SHI X, et al. Recognition of abnormal vehicle behaviors using combined features [C]//19th COTA International Conference of Transportation Professionals. Reston, VA: ASCE, 2019: 2180–2186.
CHEN Z J, WU C Z, HUANG Z, et al. Dangerous driving behavior detection using video-extracted vehicle trajectory histograms [J]. Journal of Intelligent Transportation Systems, 2017, 21(5): 409–421.
DOSHI K, YILMAZ Y. Fast unsupervised anomaly detection in traffic videos [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Seattle, WA: IEEE, 2020: 2658–2664.
GARCIA-AUNON P, ROLDÁN J J, BARRIENTOS A. Monitoring traffic in future cities with aerial swarms: Developing and optimizing a behavior-based surveillance algorithm [J]. Cognitive Systems Research, 2019, 54: 273–286.
WAHYONO, JO K H. Cumulative dual foreground differences for illegally parked vehicles detection [J]. IEEE Transactions on Industrial Informatics, 2017, 13(5): 2464–2473.
XIE X M, WANG C Y, CHEN S, et al. Real-time illegal parking detection system based on deep learning [C]//2017 International Conference on Deep Learning Technologies-ICDLT’17. New York: ACM, 2017: 23–27.
CAI Y F, WANG H, CHEN X B, et al. Trajectory-based anomalous behaviour detection for intelligent traffic surveillance [J]. IET Intelligent Transport Systems, 2015, 9(8): 810–816.
WANG C, MUSAEV A, SHEINIDASHTEGOL P, et al. Towards detection of abnormal vehicle behavior using traffic cameras [C]//2019 International Conference on Big Data. Cham: Springer, 2019: 125–136.
WANG G A, YUAN X Y, ZHENG A T, et al. Anomaly candidate identification and starting time estimation of vehicles from traffic videos [C]//AI City Challenge Workshop IEEE/CVF Computer Vision and Pattern Recognition (CVPR) Conference. Long Beach, CA: IEEE, 2019: 382–390.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Li, L., Hu, Z. & Yang, X. Intelligent Analysis of Abnormal Vehicle Behavior Based on a Digital Twin. J. Shanghai Jiaotong Univ. (Sci.) 26, 587–597 (2021). https://doi.org/10.1007/s12204-021-2348-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12204-021-2348-7