This paper presents a deep learning-based framework to detect and localize the pedestrians’ anomaly behaviors in videos captured at the grade crossing. A skeleton detection and tracking algorithm are employed to capture the key point trajectories of body movements of the pedestrians. A deep recurrent neural network is applied to learn the normal patterns of pedestrians’ movements using dynamics skeleton trajectories features. An anomaly behaviors detection and localization algorithm are developed by analyzing each pedestrian’s reconstructed trajectories. In the experiments, a video dataset involving normal pedestrian behaviors is established by collecting data at multiple grade crossing spots with different camera angles. Then the proposed framework is trained on the dataset to learn the regularity patterns of normal pedestrians and localize the anomaly behaviors during the testing phase. To the best of our knowledge, it is the first attempt to analyze pedestrians’ behavior at a grade crossing. The experimental results show that the proposed framework can detect and localize the anomaly behaviors, such as squatting down, lingering, and other behaviors that may cause safety issues at the grade crossing. Our study also points out the direction for further improvement of the present development to meet the need for real-world applications.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
FRA (2019) Highway-rail crossing handbook - Third Edition. https://safety.fhwa.dot.gov/hsip/xings/com_roaduser/fhwasa18040/fhwasa18040v2.pdf
FRA (2018) National strategy to prevent trespassing on railroad property. https://www.fra.dot.gov/eLib/Details/L19817
FRA (2021) Highway/rail grade crossing incidents. https://railroads.dot.gov/accident-and-incident-reporting/highwayrail-grade-crossing-incidents/highwayrail-grade-crossing
Pang G, Shen C, Cao L, Hengel A (2020) Deep learning for anomaly detection: a review. ACM Comput Surv Mar 54(2):1–38
Zhou JT, Du J, Zhu H, Peng X, Liu Y, Goh R (2019) Anomalynet: an anomaly detection network for video surveillance. IEEE Trans Inf Forensics Secur Oct 14(10):2537–2550
Andrews J, Tanay T, Morton EJ, Griffin LD (2016) Transfer representation-learning for anomaly detection. Proc. Int. Conf., Machine learning (ICML), July, New York, NY
Ionescu RT, Khan FS, Georgescu M, Shao L(2019) Object-centric auto-encoders and dummy anomalies for abnormal event detection in video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 7834–7843
Yu W, Cheng W, Aggarwal CC, Zhang K, Chen H, Wang W (2018) Netwalk: a flexible deep embedding approach for anomaly detection in dynamic networks. In: Proceedings of 24th ACM SIGKDD International conference on Knowledge Discovery and Data Mining (KDD), London, UK , 2672–2681
Sabokrou M, Khalooei M, Fathy M, Adeli E(2018) Adversarially learned one-class classifier for novelty detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, Utah, USA, 3379–3389 June 2018
Nguyen T, Meunier J (2019) Anomaly Detection in Video Sequence with Appearance-Motion Correspondence. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV), Seoul, Korea, 1273–1283, Oct. 2019
Zhang C, Song D, . Chen Y, Feng X, Lumezanu C, Cheng W, Ni J, Zong B, Chen H, Chawla NV (2019) A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. In: Proceedings of the AAAI conference on artificial intelligence, Jan., Honolulu, Hawaii, USA
Chong YS, Tay YH (2017) Abnormal event detection in videos using spatiotemporal autoencoder. Adv Neural Netw - ISNN 2017(10262):189–196
Zhao Y, Deng B, Shen C, Liu Y, Lu H, Hua X (2017) Spatio-temporal autoencoder for video anomaly detection. In: Proceedings of the 25th ACM international conference on Multimedia, Mountain View, CA, USA , 1933–1941
Malhotra P, Ramakrishnan A, Anand G, Vig L, Agarwal P, Shroff G (2016) LSTM-based encoder-decoder for multi-sensor anomaly detection. International conference of machine learning (ICML) Anomaly detection Workshop, New York, NY
Liu W, Luo W, Lian D, Gao S(2018) Future frame prediction for anomaly detection - a new baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, Utah, 6536–6545
Pang G, Yan C, Shen C, Hengel A, Bai X(2020) Self-trained deep ordinal regression for end-to-end video anomaly detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 12173–12182
Park H, Noh J, Ham B(2020) Learning memory-guided normality for anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14372–14381
. Gupta A, Johnson J, Li F, Savarese S, . Alahi A(2018) Social GAN: socially acceptable trajectories with generative adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, Utah, 2255–2264
Du Y, Fu Y, Wang L (2016) Representation learning of temporal dynamics for skeleton-based action recognition. IEEE Trans Image Process Oct 25(7):3010–3022
Fragkiadaki K, Levine S, Felsen P, Malik J(2015) Recurrent network models for human dynamics. In: Proceedings of the IEEE International conference on computer vision (ICCV), Santiago, Chile, 4346–4354
Villegas R, Yang J, Zou Y, Sohn S, Lin X, Lee H(2017) Learning to generate long-term future via hierarchical prediction. International conference on machine Learning (ICML), Sydney, Australia, 3560–3569
Bera A, Manocha D(2018) Interactive surveillance technologies for dense crowds. In: Proceedings of the Association for the Advances of Artificial Intelligence. (AAAI), Arlington, Virginia, USA
Piergiovanni A, Ryoo MS(2019) Representation flow for action recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), Long Beach, CA 9945–9953
Morais R, Le V, Tran T, Saha B, Mansour M, Venkatesh S(2019) Learning regularity in skeleton trajectories for anomaly detection in videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition(CVPR), Long Beach, CA, 11996–12004
Pishchulin L, Andriluka M, Schiele B (2014) Fine-grained activity recognition with holistic and pose based features. Pattern Recogn 8753:678–689
Su C, Li J, Zhang S, Xing J, Gao W, Tian Q(2017) Pose-driven deep convolutional model for person re-identification. In: Proceedings of the IEEE international conference on computer vision (ICCV), Venice, Italy, 3980–3989 Oct 2017
Fang H, . Lu G, Fang X, Xie J, Tai Y, Lu C(2018) Weakly and semi supervised human body part parsing via pose-guided knowledge transfer. In: Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), Salt Lake City, Utah, USA 70–78, June 2018
Li J, Wang C, Zhu H, Mao Y, Fang H, Lu C(2019) CrowdPose: efficient crowded scenes pose estimation and a new benchmark. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), Long Beach, CA, USA, 10855–10864, June 2019
Fang H, Xie S, Tai Y, Lu C(2017) RMPE: regional multi-person pose estimation. InProceedings of the IEEE international conference on computer vision (ICCV), Venice, Italy, 4321–4331 Oct 2017
Cao Z, Simon T, Wei S, Sheikh Y(2017) Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, HI, USA, 7291–7299 July 2017
Xiu Y, Li J, Wang H, Fang Y, Lu C (2018) Pose flow: efficient online pose tracking. In: Proceedings of British Machine Vision Conference (BMVC), Newcastle, UK, Sep. 2018
Simon T, Joo H, Matthews I, Sheikh Y (2017) Hand keypoint detection in single images using multiview bootstrapping. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 1145–1153 July 2017
Gao Y, Glowacka D (2016) Deep gate recurrent neural network. In: Proceedings of The 8th Asian Conference on Machine Learning, Hamilton, New Zealand, 350–365 Nov. 2016
Zhang Z, Trivedi C, Liu X (2018) Automated detection of grade-crossing-trespassing near misses based on computer vision analysis of surveillance video data. Saf Sci Dec 110:276–285
Zhang J, Yang K, Rainer S (2021) ISSAFE: improving semantic segmentation in accidents by fusing event-based data. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 1132–1139 Oct. 2021
Jiang Z, Guo F, Qian Y, Wang Y (2022) A deep learning-assisted mathematical model for decongestion time prediction at railroad grade crossings. Neural Comput Appl Oct 34:4715–4732
Zhao Y, Wu W, He Y, Li Y, Tan X, Chen S(2021) Good practices and a strong baseline for traffic anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual, 3993–4001 June 2021
Doshi K, Yilmaz Y(2020) Fast unsupervised anomaly detection in traffic videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPR), Virtual, 624–625 June 2020
Gasparini R, Pini S, Borghi G, Scaglione G, . Calderara S, Fedeli E, Cucchiara R (2020) Anomaly detection for vision-based railway inspection. In: Proceedings of European Dependable Computing Conference, Munich, Germany, Sep. 2020
UCSD Anomaly detection dataset. http://www.svcl.ucsd.edu/projects/anomaly/dataset.html
ShanghaiTech campus dataset (Anomaly detection). https://svip-lab.github.io/dataset/campus_dataset.html
This research is partially funded by the Federal Railroad Administration (FRA), Contract No. 693JJ620C000021. Dr. Shala Blue, Mr. Francesco Bedini, Mr. Michael Jones, and Dr. Starr Kidda from FRA have provided essential guidance and insight. The City of Columbia, especially the Columbia Fire Department, Department of Transportation, and 911 Dispatching Center; and CSX have provided tremendous help. The opinions expressed in this article are solely those of the authors and do not represent the opinions of the funding agencies. Mr. Thomas Johnson and Mr. Tianqi Huang made significant contribution to imagery data generation.
Conflict of interest
The authors declare that they have no conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Corresponding author: Yi Wang. Note: This work was performed at University of South Carolina when Zhuocheng Jiang worked there as a postdoc.
About this article
Cite this article
Jiang, Z., Song, G., Qian, Y. et al. A deep learning framework for detecting and localizing abnormal pedestrian behaviors at grade crossings. Neural Comput & Applic 34, 22099–22113 (2022). https://doi.org/10.1007/s00521-022-07660-0