Skip to main content
SpringerLink
Log in

Real-time and accurate abnormal behavior detection in videos

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Abnormal crowd behavior detection is a hot research topic in the field of computer vision. In order to solve the problems of high computational cost and the imbalance between positive and negative samples, we propose an efficient algorithm that can detect and locate anomalies in videos. In order to solve the problem of less negative samples, the algorithm uses the spatiotemporal autoencoder to identify and extract the negative samples (contain abnormal behaviors) in the dataset in an unsupervised learning method. On this basis, a spatiotemporal convolutional neural network (CNN) is constructed with simple structure and low computational complexity. The supervised training method is used to train the spatiotemporal CNN with positive and negative samples to generate the detection model. Experiments are conducted on the UCSD and UMN datasets. The experiment results show that the proposed algorithm can detect and locate abnormal behaviors in real time (using only CPU), and the accuracy of the algorithm exceeds those of the existing algorithms at both the pixel level and frame level.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Popoola, O.P., Wang, K.: Video-based abnormal human behavior recognition—a review. IEEE Trans. Syst. Man Cybern. C Cybern. 42(6), 865–878 (2012)

    Article  Google Scholar 

  2. Yong, S.C., Yong, H.T.: Abnormal event detection in videos using spatiotemporal autoencoder. In: IEEE Computer Vision and Pattern Recognition. pp. 189–196 (2017)

  3. Ravanbakhsh, M., Nabi, M., Sangineto, E., et al.: Abnormal event detection in videos using generative adversarial nets. In: 2017 IEEE International Conference on Image Processing (ICIP). IEEE, pp. 1577–1581

  4. Akcay, S., Atapour-Abarghouei, A., Breckon, T. P.: Ganomaly: semi-supervised anomaly detection via adversarial training. In: Springer Asian Conference on Computer Vision, pp. 622–637 (2018)

  5. Sabokrou, M., Fathy, M., Moayed, Z., et al.: Fast and accurate detection and localization of abnormal behavior in crowded scenes. Mach. Vis. Appl. 28(8), 965–985 (2017)

    Article  Google Scholar 

  6. Marsden, M., Mcguinness, K., Little, S., et al.: Holistic features for real-time crowd behaviour anomaly detection. In: IEEE International Conference on Image Processing, pp. 918–922 (2016)

  7. Wu, S., Moore, B.E., Shah, M.: Chaotic invariants of lagrangian particle trajectories for anomaly detection in crowded scenes. In: IEEE Computer Vision and Pattern Recognition, pp. 2054–2060 (2010)

  8. Basharat, A., Gritai, A., Shah, M.: Learning object motion patterns for anomaly detection and improved object detection. In: IEEE Computer Vision and Pattern Recognition, pp. 1–8 (2008)

  9. Hospedales, T., Gong, S., Xiang, T.: A markov clustering topic model for mining behaviour in video. In: IEEE International Conference on Computer Vision, pp. 1165–1172 (2009)

  10. Kratz, L., Nishino, K.: Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In: IEEE Computer Vision and Pattern Recognition, pp. 1446–1453 (2009)

  11. Zhao, B., Li, F.F., Xing, E.P.: Online detection of unusual events in videos via dynamic sparse coding. In: IEEE Computer Vision and Pattern Recognition, pp. 3313–3320 (2011)

  12. Schlegl, T., Seeböck, P., Waldstein, S. M., et al.: Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In: Springer International Conference on Information Processing in Medical Imaging, pp. 146–157 (2017)

  13. Rasheed, N., Khan, S.A., Khalid, A.: Tracking and abnormal behavior detection in video surveillance using optical flow and neural networks. In: IEEE International Conference on Advanced Information NETWORKING and Applications Workshops, pp. 61–66 (2014)

  14. Zhou, S., Shen, W., Zeng, D., et al.: Unusual event detection in crowded scenes by trajectory analysis. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1300–1304 (2015)

  15. Li, C., Han, Z., Ye, Q., et al.: Abnormal behavior detection via sparse reconstruction analysis of trajectory. In: IEEE Sixth International Conference on Image and Graphics, pp. 807–810 (2011)

  16. Piciarelli, C., Micheloni, C., Foresti, G.L.: Trajectory-based anomalous event detection. IEEE Trans. Circuits Syst. Video Technol. 18(11), 1544–1554 (2008)

    Article  Google Scholar 

  17. Mo, X., Monga, V., Bala, R., et al.: Adaptive sparse representations for video anomaly detection. IEEE Trans. Circuits Syst. Video Technol. 24(4), 631–645 (2014)

    Article  Google Scholar 

  18. Anjum, N., Cavallaro, A.: Multifeature object trajectory clustering for video analysis. IEEE Trans. Circuits Syst. Video Technol. 18(11), 1555–1564 (2008)

    Article  Google Scholar 

  19. Kratz, L., Nishino, K.: Tracking pedestrians using local spatio-temporal motion patterns in extremely crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. 34(5), 987–1002 (2012)

    Article  Google Scholar 

  20. Marques, J.S., Jorge, P.M., Abrantes, A.J., et al.: Tracking groups of pedestrians in video sequences. In: IEEE Computer Vision and Pattern Recognition Workshop, p. 101 (2008)

  21. Kim, J., Grauman, K.: Observe locally, infer globally: A space-time MRF for detecting abnormal activities with incremental updates. In: IEEE Computer Vision and Pattern Recognition, pp. 2921–2928 (2009)

  22. Mehran, R., Oyama, A., Shah, M.: Abnormal crowd behavior detection using social force model. In: IEEE Computer Vision and Pattern Recognition, pp. 935–942 (2009)

  23. Reddy, V., Sanderson, C., Lovell, B.C.: Improved anomaly detection in crowded scenes via cell-based analysis of foreground speed, size and texture. In: IEEE Computer Vision and Pattern Recognition Workshops, pp. 55–61 (2013)

  24. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Vision and Pattern Recognition, pp. 886–893 (2005)

  25. Xiao, T., Zhang, C., Zha, H., et al.: Anomaly detection via local coordinate factorization and spatio-temporal pyramid. In: Computer Vision ACCV, pp. 66–82 (2014)

  26. Laptev, I., Marszalek, M., Schmid, C., et al.: Learning realistic human actions from movies. In: IEEE Computer Vision and Pattern Recognition, pp. 1–8 (2008)

  27. Cong, Y., Yuan, J., Tang, Y.: Video anomaly search in crowded scenes via spatio-temporal motion context. IEEE Trans. Inform. Forensics Security 8(10), 1590–1599 (2013)

    Article  Google Scholar 

  28. Cong, Y., Yuan, J., Liu, J.: Sparse reconstruction cost for abnormal event detection. IEEE Comput. Vis. Pattern Recognit. 32(14), 3449–3456 (2011)

    Google Scholar 

  29. Yang, C., Yuan, J., Liu, J.: Abnormal event detection in crowded scenes using sparse representation. Pattern Recogn. 46(7), 1851–1864 (2013)

    Article  Google Scholar 

  30. Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 18–32 (2013)

    Google Scholar 

  31. Lu, C., Shi, J., Jia, J.: Abnormal event detection at 150 fps in matlab. In: IEEE International Conference on Computer Vision, pp. 2720–2727 (2014)

  32. Zhou, S., Shen, W., Zeng, D., et al.: Spatial-temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Process. Image Commun. 47, 358–368 (2016)

    Article  Google Scholar 

  33. Hasan, M., Choi, J., Neumann, J., et al.: Learning temporal regularity in video sequences. In: EEE Computer Vision and Pattern Recognition, pp. 733–742 (2016)

  34. Medel, J.R.: Anomaly detection using predictive convolutional long short-term memory units. Available http://scholarworks.rit.edu/theses/9319 (2016)

  35. Mahadevan, V., Li, W., Bhalodia, V., et al.: Anomaly detection in crowded scenes. In: IEEE Computer Vision and Pattern Recognition, pp. 1975–1981 (2010)

  36. Raghavendra, R., Bue, A.D., Cristani, M.: Unusual crowd activity dataset of University of Minnesota. Available http://mha.cs.umn.edu/Movies/Crowd-Activity-All.avi (2006)

  37. Adam, A., Rivlin, E., Shimshoni, I., et al.: Robust real-time unusual event detection using multiple fixed-location monitors. IEEE Trans. Pattern Anal. Mach. Intell. 30(3), 555–560 (2008)

    Article  Google Scholar 

  38. Ravanbakhsh, M., Nabi, M., Mousavi, H., et al.: Plug-and-play cnn for crowd motion analysis: An application in abnormal event detection. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, pp. 1689–1698

  39. Lee, D.G., Suk, H.I., Park, S.K., et al.: Motion influence map for unusual human activity detection and localization in crowded scenes. IEEE Trans. Circuits Syst. Video Technol. 25(10), 1612–1623 (2015)

    Article  Google Scholar 

  40. Turchini, F., Seidenari, L., Bimbo, A.D.: Convex polytope ensembles for spatio-temporal anomaly detection. In: International Conference on Image Analysis and Processing, pp. 174–184 (2017)

  41. Sabokrou, M., Fayyaz, M., Fathy, M., Klette, R.: Deep-cascade: cascading 3d deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans. Image Process. 26(4), 1992–2004 (2017)

    Article  MathSciNet  Google Scholar 

  42. Sabokrou, M., Fayyaz, M., Fathy, M., et al.: Deep-anomaly: fully convolutional neural network for fast anomaly detection in crowded scenes. Comput. Vis. Image Understanding 172, 88–97 (2018)

    Article  Google Scholar 

  43. Sabokrou, M., Khalooei, M., Fathy, M., Adeli, E.: Adversarially learned one-class classifier for novelty detection. In: IEEE Computer Vision and Pattern Recognition, pp. 3379–3388 (2018)

  44. Sabokrou, M., Pourreza, M., Fayyaz, M., et al.: Avid: Adversarial visual irregularity detection. In: Springer Asian Conference on Computer Vision, pp. 488–505 (2018)

  45. Sabokrou, M., Khalooei, M., Adeli, E.: Self-supervised representation learning via neighborhood-relational encoding. In: IEEE International Conference on Computer Vision, pp. 8010–8019 (2019)

  46. Sabokrou, M., Fathy, M., Hoseini, M.: Video anomaly detection and localisation based on the sparsity and reconstruction error of auto-encoder. Electron. Lett. 52(13), 1122–1124 (2016)

    Article  Google Scholar 

  47. Mousavi, H., Nabi, M., Kiani, H., et al.: Crowd motion monitoring using tracklet-based commotion measure. In: IEEE International Conference on Image Processing, pp. 2354–2358 (2015)

  48. Boiman, O., Irani, M.: Detecting irregularities in images and in video. Int. J. Comput. Vis. 74(1), 17–31 (2007)

    Article  Google Scholar 

  49. Roshtkhari, M.J., Levine, M.D.: An on-line, real-time learning method for detecting anomalies in videos using spatio-temporal compositions. Comput. Vis. Image Understanding 117(10), 1436–1452 (2013)

    Article  Google Scholar 

  50. Hassner, T., Itcher, Y., Kliper-Gross, O.: Violent flows: real-time detection of violent crowd behavior. In: IEEE Computer Vision and Pattern Recognition Workshops, pp. 1–6 (2012)

  51. Kaltsa, V., Briassouli, A., Kompatsiaris, I., et al.: Swarm intelligence for detecting interesting events in crowded environments. IEEE Trans. Image Process. 24(7), 2153 (2015)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

This study was funded by Natural Science Foundation of Beijing Municipality (No. L192036) and National Natural Science Foundation of China (No. 61701029) and Basic Research Foundation of Beijing Institute of Technology (No. 20170542008) and Industry-University-Research Innovation Foundation of the Science and Technology Development Center of the Ministry of Education (No. 2018A02012).

Author information

Authors and Affiliations

  1. School of Information and Electronics, Beijing Institute of Technology, Beijing, China

    Zheyi Fan, Jianyuan Yin, Yu Song & Zhiwen Liu

Authors
  1. Zheyi Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianyuan Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhiwen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zheyi Fan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, Z., Yin, J., Song, Y. et al. Real-time and accurate abnormal behavior detection in videos. Machine Vision and Applications 31, 72 (2020). https://doi.org/10.1007/s00138-020-01111-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-020-01111-3

Keywords

Search

Navigation