Skip to main content
SpringerLink
Log in

A deep learning framework for detecting and localizing abnormal pedestrian behaviors at grade crossings

Neural Computing and Applications volume 34, pages 22099–22113 (2022)Cite this article

Abstract

This paper presents a deep learning-based framework to detect and localize the pedestrians’ anomaly behaviors in videos captured at the grade crossing. A skeleton detection and tracking algorithm are employed to capture the key point trajectories of body movements of the pedestrians. A deep recurrent neural network is applied to learn the normal patterns of pedestrians’ movements using dynamics skeleton trajectories features. An anomaly behaviors detection and localization algorithm are developed by analyzing each pedestrian’s reconstructed trajectories. In the experiments, a video dataset involving normal pedestrian behaviors is established by collecting data at multiple grade crossing spots with different camera angles. Then the proposed framework is trained on the dataset to learn the regularity patterns of normal pedestrians and localize the anomaly behaviors during the testing phase. To the best of our knowledge, it is the first attempt to analyze pedestrians’ behavior at a grade crossing. The experimental results show that the proposed framework can detect and localize the anomaly behaviors, such as squatting down, lingering, and other behaviors that may cause safety issues at the grade crossing. Our study also points out the direction for further improvement of the present development to meet the need for real-world applications.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

References

  1. FRA (2019) Highway-rail crossing handbook - Third Edition. https://safety.fhwa.dot.gov/hsip/xings/com_roaduser/fhwasa18040/fhwasa18040v2.pdf

  2. FRA (2018) National strategy to prevent trespassing on railroad property. https://www.fra.dot.gov/eLib/Details/L19817

  3. FRA (2021) Highway/rail grade crossing incidents. https://railroads.dot.gov/accident-and-incident-reporting/highwayrail-grade-crossing-incidents/highwayrail-grade-crossing

  4. Pang G, Shen C, Cao L, Hengel A (2020) Deep learning for anomaly detection: a review. ACM Comput Surv Mar 54(2):1–38

    Google Scholar 

  5. Zhou JT, Du J, Zhu H, Peng X, Liu Y, Goh R (2019) Anomalynet: an anomaly detection network for video surveillance. IEEE Trans Inf Forensics Secur Oct 14(10):2537–2550

    Article  Google Scholar 

  6. Andrews J, Tanay T, Morton EJ, Griffin LD (2016) Transfer representation-learning for anomaly detection. Proc. Int. Conf., Machine learning (ICML), July, New York, NY

    Google Scholar 

  7. Ionescu RT, Khan FS, Georgescu M, Shao L(2019) Object-centric auto-encoders and dummy anomalies for abnormal event detection in video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 7834–7843

  8. Yu W, Cheng W, Aggarwal CC, Zhang K, Chen H, Wang W (2018) Netwalk: a flexible deep embedding approach for anomaly detection in dynamic networks. In: Proceedings of 24th ACM SIGKDD International conference on Knowledge Discovery and Data Mining (KDD), London, UK , 2672–2681

  9. Sabokrou M, Khalooei M, Fathy M, Adeli E(2018) Adversarially learned one-class classifier for novelty detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, Utah, USA, 3379–3389 June 2018

  10. Nguyen T, Meunier J (2019) Anomaly Detection in Video Sequence with Appearance-Motion Correspondence. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV), Seoul, Korea, 1273–1283, Oct. 2019

  11. Zhang C, Song D, . Chen Y, Feng X, Lumezanu C, Cheng W, Ni J, Zong B, Chen H, Chawla NV (2019) A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. In: Proceedings of the AAAI conference on artificial intelligence, Jan., Honolulu, Hawaii, USA

  12. Chong YS, Tay YH (2017) Abnormal event detection in videos using spatiotemporal autoencoder. Adv Neural Netw - ISNN 2017(10262):189–196

    Google Scholar 

  13. Zhao Y, Deng B, Shen C, Liu Y, Lu H, Hua X (2017) Spatio-temporal autoencoder for video anomaly detection. In: Proceedings of the 25th ACM international conference on Multimedia, Mountain View, CA, USA , 1933–1941

  14. Malhotra P, Ramakrishnan A, Anand G, Vig L, Agarwal P, Shroff G (2016) LSTM-based encoder-decoder for multi-sensor anomaly detection. International conference of machine learning (ICML) Anomaly detection Workshop, New York, NY

  15. Liu W, Luo W, Lian D, Gao S(2018) Future frame prediction for anomaly detection - a new baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, Utah, 6536–6545

  16. Pang G, Yan C, Shen C, Hengel A, Bai X(2020) Self-trained deep ordinal regression for end-to-end video anomaly detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 12173–12182

  17. Park H, Noh J, Ham B(2020) Learning memory-guided normality for anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14372–14381

  18. . Gupta A, Johnson J, Li F, Savarese S, . Alahi A(2018) Social GAN: socially acceptable trajectories with generative adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, Utah, 2255–2264

  19. Du Y, Fu Y, Wang L (2016) Representation learning of temporal dynamics for skeleton-based action recognition. IEEE Trans Image Process Oct 25(7):3010–3022

    Article  MathSciNet  MATH  Google Scholar 

  20. Fragkiadaki K, Levine S, Felsen P, Malik J(2015) Recurrent network models for human dynamics. In: Proceedings of the IEEE International conference on computer vision (ICCV), Santiago, Chile, 4346–4354

  21. Villegas R, Yang J, Zou Y, Sohn S, Lin X, Lee H(2017) Learning to generate long-term future via hierarchical prediction. International conference on machine Learning (ICML), Sydney, Australia, 3560–3569

  22. Bera A, Manocha D(2018) Interactive surveillance technologies for dense crowds. In: Proceedings of the Association for the Advances of Artificial Intelligence. (AAAI), Arlington, Virginia, USA

  23. Piergiovanni A, Ryoo MS(2019) Representation flow for action recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), Long Beach, CA 9945–9953

  24. Morais R, Le V, Tran T, Saha B, Mansour M, Venkatesh S(2019) Learning regularity in skeleton trajectories for anomaly detection in videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition(CVPR), Long Beach, CA, 11996–12004

  25. Pishchulin L, Andriluka M, Schiele B (2014) Fine-grained activity recognition with holistic and pose based features. Pattern Recogn 8753:678–689

    MathSciNet  Google Scholar 

  26. Su C, Li J, Zhang S, Xing J, Gao W, Tian Q(2017) Pose-driven deep convolutional model for person re-identification. In: Proceedings of the IEEE international conference on computer vision (ICCV), Venice, Italy, 3980–3989 Oct 2017

  27. Fang H, . Lu G, Fang X, Xie J, Tai Y, Lu C(2018) Weakly and semi supervised human body part parsing via pose-guided knowledge transfer. In: Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), Salt Lake City, Utah, USA 70–78, June 2018

  28. Li J, Wang C, Zhu H, Mao Y, Fang H, Lu C(2019) CrowdPose: efficient crowded scenes pose estimation and a new benchmark. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), Long Beach, CA, USA, 10855–10864, June 2019

  29. Fang H, Xie S, Tai Y, Lu C(2017) RMPE: regional multi-person pose estimation. InProceedings of the IEEE international conference on computer vision (ICCV), Venice, Italy, 4321–4331 Oct 2017

  30. Cao Z, Simon T, Wei S, Sheikh Y(2017) Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, HI, USA, 7291–7299 July 2017

  31. Xiu Y, Li J, Wang H, Fang Y, Lu C (2018) Pose flow: efficient online pose tracking. In: Proceedings of British Machine Vision Conference (BMVC), Newcastle, UK, Sep. 2018

  32. Simon T, Joo H, Matthews I, Sheikh Y (2017) Hand keypoint detection in single images using multiview bootstrapping. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 1145–1153 July 2017

  33. Gao Y, Glowacka D (2016) Deep gate recurrent neural network. In: Proceedings of The 8th Asian Conference on Machine Learning, Hamilton, New Zealand, 350–365 Nov. 2016

  34. Zhang Z, Trivedi C, Liu X (2018) Automated detection of grade-crossing-trespassing near misses based on computer vision analysis of surveillance video data. Saf Sci Dec 110:276–285

    Article  Google Scholar 

  35. Zhang J, Yang K, Rainer S (2021) ISSAFE: improving semantic segmentation in accidents by fusing event-based data. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 1132–1139 Oct. 2021

  36. Jiang Z, Guo F, Qian Y, Wang Y (2022) A deep learning-assisted mathematical model for decongestion time prediction at railroad grade crossings. Neural Comput Appl Oct 34:4715–4732

    Article  Google Scholar 

  37. Zhao Y, Wu W, He Y, Li Y, Tan X, Chen S(2021) Good practices and a strong baseline for traffic anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual, 3993–4001 June 2021

  38. Doshi K, Yilmaz Y(2020) Fast unsupervised anomaly detection in traffic videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPR), Virtual, 624–625 June 2020

  39. Gasparini R, Pini S, Borghi G, Scaglione G, . Calderara S, Fedeli E, Cucchiara R (2020) Anomaly detection for vision-based railway inspection. In: Proceedings of European Dependable Computing Conference, Munich, Germany, Sep. 2020

  40. UCSD Anomaly detection dataset. http://www.svcl.ucsd.edu/projects/anomaly/dataset.html

  41. ShanghaiTech campus dataset (Anomaly detection). https://svip-lab.github.io/dataset/campus_dataset.html

Download references

Acknowledgements

This research is partially funded by the Federal Railroad Administration (FRA), Contract No. 693JJ620C000021. Dr. Shala Blue, Mr. Francesco Bedini, Mr. Michael Jones, and Dr. Starr Kidda from FRA have provided essential guidance and insight. The City of Columbia, especially the Columbia Fire Department, Department of Transportation, and 911 Dispatching Center; and CSX have provided tremendous help. The opinions expressed in this article are solely those of the authors and do not represent the opinions of the funding agencies. Mr. Thomas Johnson and Mr. Tianqi Huang made significant contribution to imagery data generation.

Author information

Authors and Affiliations

  1. NEC Laboratories America, Inc., 4 Independence Way, Princeton, NJ, 08540, USA

    Zhuocheng Jiang

  2. Department of Mechanical Engineering, University of South Carolina, Columbia, SC, 29208, USA

    Zhuocheng Jiang, Ge Song & Yi Wang

  3. Department of Civil and Environment Engineering, University of South Carolina, Columbia, SC, 29208, USA

    Yu Qian

Authors
  1. Zhuocheng Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ge Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yi Wang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Corresponding author: Yi Wang. Note: This work was performed at University of South Carolina when Zhuocheng Jiang worked there as a postdoc.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, Z., Song, G., Qian, Y. et al. A deep learning framework for detecting and localizing abnormal pedestrian behaviors at grade crossings. Neural Comput & Applic 34, 22099–22113 (2022). https://doi.org/10.1007/s00521-022-07660-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07660-0

Keywords

search

Navigation