Skip to main content
SpringerLink
Log in

Spatio-temporal based video anomaly detection using deep neural networks

  • Original Research
  • Published:
International Journal of Information Technology Aims and scope Submit manuscript

Abstract

Anomaly detection is the identification of unexpected events. The state–of-the-art algorithms try to reduce the reconstruction errors of training data, but there is still no guarantee that the reconstruction error will be smaller in the case of any event that is not usual. In this article, a framework is suggested to address the issues with anomaly detection. U-Network can capture spatial information, while long short-term memory (LSTM) can process temporal information well. The proposed method is a combination of U-Net and ConvLSTM to handle spatial information and temporal motion. To enhance the quality of the frames, bilateral filtering has been introduced, which will increase the range of the peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM), the two performance measures used. Using PSNR and SSIM, a weighted regular score function is derived to classify the frames based on the scores. Finally, results are compared with those of the state-of-the-art method to prove the superiority of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Nayak R, Pati UC, Das SK (2021) A comprehensive review on deep learning-based methods for video anomaly detection. Image Vision Comp. 106:104078

    Article  Google Scholar 

  2. Leyva R, Sanchez V, Li C-T (2017) Video anomaly detection with compact feature sets for online performance. IEEE Trans Image Process 26(7):3463–3478

    Article  MathSciNet  MATH  Google Scholar 

  3. Feng Y, Yuan Y, Xiaoqiang Lu (2017) Learning deep event models for crowd anomaly detection. Neurocomputing 219:548–556

    Article  Google Scholar 

  4. Xu D, Ricci E, Yan Y, Song J, Sebe N (2015) Learning deep representations of appearance and motion for anomalous event detection. ArXiv preprint arXiv. 33(1510):01553

    Google Scholar 

  5. Ribeiro M, Lazzaretti AE, Lopes HSo (2018) A study of deep convolutional auto-encoders for anomaly detection in videos. Pattern Recogn Lett 105:13–22

    Article  Google Scholar 

  6. Zhao, Yiru, Bing Deng, Chen Shen, Yao Liu, Hongtao Lu, and Xian-Sheng Hua. 2017 Spatio-temporal autoencoder for video anomaly detection. In Proceedings of the 25th ACM international conference on Multimedia. 1933–1941

  7. Liu, Wen, Weixin Luo, Dongze Lian, and Shenghua Gao. 2018 Future frame prediction for anomaly detection–a new baseline. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6536–6545.

  8. Zyuzin V, Sergey P, Mukhtarov A, Chumarnaya T, Solovyova O, Bobkova A, Myasnikov V (2018) Identification of the left ventricle endocardial border on two-dimensional ultrasound images using the convolutional neural network Unet. Symp Biomed Eng. 55:76–78

    Google Scholar 

  9. Donahue, Jeffrey, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015 Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2625–2634.

  10. Nanni L, Ghidoni S, Brahnam S (2017) Handcrafted vs. non-handcrafted features for computer vision classification. Pattern Recogn 71:158–172

    Article  Google Scholar 

  11. Jiang F, Yuan J, Tsaftaris SA, Katsaggelos AK (2011) Anomalous video event detection using spatiotemporal context. Comput Vis Image Underst 115(3):323–333

    Article  Google Scholar 

  12. Tung F, Zelek JS, Clausi DA (2011) Goal-based trajectory analysis for unusual behaviour detection in intelligent surveillance. Image Vis Comput 29(4):230–240

    Article  Google Scholar 

  13. Jiang F, Ying Wu, Katsaggelos AK (2009) A dynamic hierarchical clustering method for trajectory-based unusual video event detection. IEEE Trans Image Process 18(4):907–913

    Article  MathSciNet  MATH  Google Scholar 

  14. Piciarelli C, Foresti GL (2006) On-line trajectory clustering for anomalous events detection. Patt Recogn Lett 27:1835–1842

    Article  Google Scholar 

  15. Zhou, Yue, Shuicheng Yan, and Thomas S. Huang. 2007 Detecting anomaly in videos from trajectory similarity analysis. In 2007 IEEE International Conference on Multimedia and Expo. 1087–1090.

  16. Kim, Jaechul, and Kristen Grauman. 2009 Observe locally, infer globally: a space-time MRF for detecting abnormal activities with incremental updates. In 2009 IEEE conference on computer vision and pattern recognition. 2921–2928.

  17. Adam A, Rivlin E, Shimshoni I, Reinitz D (2008) Robust real-time unusual event detection using multiple fixed-location monitors. IEEE Trans Pattern Anal Mach Intell 30(3):555–560

    Article  Google Scholar 

  18. Kratz, Louis, and Ko Nishino. 2009 Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In 2009 IEEE conference on computer vision and pattern recognition. 1446–1453

  19. Amraee S, Vafaei A, Jamshidi K, Adibi P (2018) Anomaly detection and localization in crowded scenes using connected component analysis. Multimed Tools Appl 77(12):14767–14782

    Article  Google Scholar 

  20. Ullah H, Altamimi AB, Uzair M, Ullah M (2018) Anomalous entities detection and localization in pedestrian flows. Neurocomputing 290:74–86

    Article  Google Scholar 

  21. Cong, Yang, Junsong Yuan, and Ji Liu. 2011 Sparse reconstruction cost for abnormal event detection. In CVPR 2011. 3449–3456

  22. Kim H, Lee S, Kim Y, Lee S, Lee D, Jinsun Ju, Myung H (2016) Weighted joint-based human behavior recognition algorithm using only depth information for low-cost intelligent video-surveillance system. Expert Syst Appl 45:131–141

    Article  Google Scholar 

  23. Bhatnagar, Shobhit, Deepanway Ghosal, and Maheshkumar H. Kolekar. 2017 Classification of fashion article images using convolutional neural networks." In 2017 Fourth International Conference on Image Information Processing (ICIIP). 1–6

  24. Zhou S, Shen W, Zeng D, Fang M, Wei Y, Zhang Z (2016) Spatial–temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Proc Image Commun 47:358–368

    Article  Google Scholar 

  25. Luo, Weixin, Wen Liu, and Shenghua Gao. 2017 Remembering history with convolutional lstm for anomaly detection. In 2017 IEEE International Conference on Multimedia and Expo (ICME). 439–444.

  26. Ravanbakhsh, Mahdyar, Moin Nabi, Enver Sangineto, Lucio Marcenaro, Carlo Regazzoni, and Nicu Sebe. 2017 Abnormal event detection in videos using generative adversarial nets. In 2017 IEEE international conference on image processing (ICIP). 1577–1581.

  27. Mahadevan, Vijay, Weixin Li, Viral Bhalodia, and Nuno Vasconcelos. 2010 Anomaly detection in crowded scenes. In 2010 IEEE computer society conference on computer vision and pattern recognition. 1975–1981

  28. Akcay S, Atapour-Abarghouei A, Breckon TP (2018) Ganomaly: Semi-supervised anomaly detection via adversarial training. In Asian conference on computer vision, Springer, Cham

    Google Scholar 

  29. Chakraborty, Pranamesh, Anuj Sharma, and Chinmay Hegde. 2018 Freeway traffic incident detection from cameras: A semi-supervised learning approach. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC). 1840–1845

  30. Hasan, Mahmudul, Jonghyun Choi, Jan Neumann, Amit K. Roy-Chowdhury, and Larry S. Davis. 2016 Learning temporal regularity in video sequences. In Proceedings of the IEEE conference on computer vision and pattern recognition. 733–742

  31. Medel JR, Savakis A (2016) Anomaly detection in video using predictive convolutional long short-term memory networks. ArXiv preprint ArXiv. 44:1612–00390

    Google Scholar 

  32. Mathieu M, Couprie Ce, LeCun Y (2015) Deep multi-scale video prediction beyond mean square error. ArXiv preprint arXiv. 20(1511):05440

    Google Scholar 

  33. Gavaskar RG, Chaudhury KN (2018) Fast adaptive bilateral filtering. IEEE Trans Image Process 28(2):779–790

    Article  MathSciNet  MATH  Google Scholar 

  34. Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. 2015 U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, Cham.

  35. Chetty G, Yamin M, White M (2022) A low resource 3D U-Net based deep learning model for medical image analysis. Int J Inf Technol 14(1):95–103

    Google Scholar 

  36. Patraucean V, Handa A, Cipolla R (2015) Spatio-temporal video autoencoder with differentiable memory. ArXiv preprint arXiv. 65(1511):06309

    Google Scholar 

  37. Li Y, Cai Y, Liu J, Lang S, Zhang X (2019) Spatio-temporal unity networking for video anomaly detection. IEEE Access 7:172425–172432

    Article  Google Scholar 

  38. Wang L, Xiong Y, Zhe Wang Y, Qiao DL, Tang X, Van Gool L (2018) Temporal segment networks for action recognition in videos. IEEE transactions pattern anal mach intell. 41:2740–2755

    Article  Google Scholar 

  39. Muttoo SK, Kumar S (2009) Data hiding in JPEG images. Inter J Inform Technol 1:13–16

    Google Scholar 

  40. Mai Z-Y, Yang C-L, Xie S-L (2005) A novel rate-distortion optimization based on structural similarity in color image encoder. Inter J Inform Technol 11:7

    Google Scholar 

  41. Lu, Cewu, Jianping Shi, and Jiaya Jia. 2013 Abnormal event detection at 150 fps in matlab. In Proceedings of the IEEE international conference on computer vision. 2720–2727.

  42. Shaikh TA, Ali R (2022) Enhanced computerised diagnosis of Alzheimer’s disease from brain MRI images using a classifier merger strategy. Inter J Inform Technol 144:1791–1803

    Article  Google Scholar 

  43. Kiran BR, Thomas DM, Parakkal R (2018) An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. J Imag 42:36

    Article  Google Scholar 

  44. Wang, Lin, Fuqiang Zhou, Zuoxin Li, Wangxia Zuo, and Haishu Tan. 2018 Abnormal event detection in videos using hybrid spatio-temporal autoencoder. In 2018 25th IEEE International Conference on Image Processing. 2276–2280.

  45. Nawaratne R, Alahakoon D, De Silva D, Xinghuo Y (2019) Spatiotemporal anomaly detection using deep learning for real-time video surveillance. IEEE Trans Indus Inform 161:393–402

    Google Scholar 

  46. Aslam N, Kolekar MH (2022) Unsupervised anomalous event detection in videos using spatio-temporal inter-fused autoencoder. Multi Tool Appl. 81:1–26

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. ITCA Department, University of Technology, Madan Mohan Malaviya, Gorakhpur, U.P., India

    Rajeev Kumar Chaurasia & Umesh Chandra Jaiswal

Authors
  1. Rajeev Kumar Chaurasia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Umesh Chandra Jaiswal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rajeev Kumar Chaurasia.

Ethics declarations

Conflicts of interest

No.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chaurasia, R.K., Jaiswal, U.C. Spatio-temporal based video anomaly detection using deep neural networks. Int. j. inf. tecnol. 15, 1569–1581 (2023). https://doi.org/10.1007/s41870-023-01193-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41870-023-01193-y

Keywords

Search

Navigation