Skip to main content
SpringerLink
Log in

Unsupervised anomalous event detection in videos using spatio-temporal inter-fused autoencoder

  • 1220: Visual and Sensory Data Processing for Real Time Intelligent Surveillance System
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Automatic detection, localization and interpretation of an unusual event in a sequence of video is a challenging task due to its equivocal and complex nature. The development of deep neural networks have paved the way for more efficient recognition and analysis of anomalous events in video data. With the introduction of convolutional neural network (CNN) and Long short-term memory (LSTM), the spatial and temporal features extraction became easier. In this paper, we propose an end-to-end trainable Inter-fused Autoencoder (IFA) which is designed using the assemblage of CNN and LSTM layers to detect the unwonted events in a video sequence. The proposed architecture is capable of exploiting both the spatial and temporal variation of video data. The reconstruction error is computed in terms of both MSE and PSNR for each testing video. A comparison is also carried out between MSE and PSNR to show that which assessment technique is better for a reconstructive model for recreating the video sequence. A well-optimized threshold is calculated which decides the fate of testing event i.e. either usual or unusual event. Using benchmark datasets, multiple experiments were carried out to demonstrate the efficacy of proposed architecture.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Adam A, Rivlin E, Shimshoni I, Reinitz D (2008) Robust real-time unusual event detection using multiple fixed-location monitors. IEEE Trans Pattern Anal Mach Intell 30(3):555–560

    Article  Google Scholar 

  2. Akcay S, Atapour-Abarghouei A, Breckon TP (2018) Ganomaly: semi-supervised anomaly detection via adversarial training. In: Asian conference on computer vision, Springer, pp 622–637

  3. Amraee S, Vafaei A, Jamshidi K, Adibi P (2018) Anomaly detection and localization in crowded scenes using connected component analysis. Multimed Tools Appl 77(12):14767–14782

    Article  Google Scholar 

  4. Beddiar DR, Nini B, Sabokrou M, Hadid A (2020) Vision-based human activity recognition: a survey. Multimed Tools Appl 79(41):30509–30555

    Article  Google Scholar 

  5. Bhatnagar S, Ghosal D, Kolekar MH (2017) Classification of fashion article images using convolutional neural networks. In: 2017 Fourth international conference on image information processing (ICIIP), IEEE, pp 1–6

  6. Chakraborty P, Sharma A, Hegde C (2018) Freeway traffic incident detection from cameras: a semi-supervised learning approach. In: 2018 21st international conference on intelligent transportation systems (ITSC), IEEE, pp 1840–1845

  7. Cho C-J, Han DK, Ko H et al (2018) Hierarchical spatial object detection for atm vandalism surveillance. In: 2018 15th IEEE international conference on advanced video and signal based surveillance (AVSS), IEEE, pp 1–5

  8. Chong YS, Tay YH (2017) Abnormal event detection in videos using spatiotemporal autoencoder. In: International symposium on neural networks, Springer, pp 189–196

  9. Cong Y, Yuan J, Liu J (2011) Sparse reconstruction cost for abnormal event detection. In: CVPR 2011, IEEE, pp 3449–3456

  10. Del Giorno A, Bagnell JA, Hebert M (2016) A discriminative framework for anomaly detection in large videos. In: European conference on computer vision, Springer, pp 334–349

  11. Hasan M, Choi J, Neumann J, Roy-Chowdhury AK, Davis LS (2016) Learning temporal regularity in video sequences. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 733–742

  12. Jain NK, Saini R, Mittal P (2019) A review on traffic monitoring system techniques. In: Soft computing: theories and applications. Springer, pp 569–577

  13. Jiang F, Wu Y, Katsaggelos AK (2009) A dynamic hierarchical clustering method for trajectory-based unusual video event detection. IEEE Trans Image Process 18(4):907–913

    Article  MathSciNet  MATH  Google Scholar 

  14. Jiang F, Yuan J, Tsaftaris SA, Katsaggelos AK (2011) Anomalous video event detection using spatiotemporal context. Comput Vis Image Underst 115(3):323–333

    Article  Google Scholar 

  15. Kim H, Lee S, Kim Y, Lee S, Lee D, Ju J, Myung H (2016) Weighted joint-based human behavior recognition algorithm using only depth information for low-cost intelligent video-surveillance system. Exp Syst Appl 45:131–141

    Article  Google Scholar 

  16. Kim J, Grauman K (2009) Observe locally, infer globally: a space-time mrf for detecting abnormal activities with incremental updates. In: 2009 IEEE conference on computer vision and pattern recognition, IEEE, pp 2921–2928

  17. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412:6980

  18. Kratz L, Nishino K (2009) Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In: 2009 IEEE Conference on computer vision and pattern recognition, IEEE, pp 1446–1453

  19. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105

    Google Scholar 

  20. Li W, Mahadevan V, Vasconcelos N (2013) Anomaly detection and localization in crowded scenes. IEEE Trans Pattern Anal Mach Intell 36(1):18–32

    Google Scholar 

  21. Liu H, Chen S, Kubota N (2013) Intelligent video systems and analytics: a survey. IEEE Trans Ind Inf 9(3):1222–1233

    Article  Google Scholar 

  22. Liu W, Luo W, Lian D, Gao S (2018) Future frame prediction for anomaly detection–a new baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6536–6545

  23. Lu C, Shi J, Jia J (2013) Abnormal event detection at 150 fps in matlab. In: Proceedings of the IEEE international conference on computer vision, pp 2720–2727

  24. Luo W, Liu W, Gao S (2017) Remembering history with convolutional lstm for anomaly detection. In: 2017 IEEE international conference on multimedia and expo (ICME), IEEE, pp 439–444

  25. Luo W, Liu W, Gao S (2017) A revisit of sparse coding based anomaly detection in stacked rnn framework. In: Proceedings of the IEEE international conference on computer vision, pp 341–349

  26. Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. In: 2010 IEEE computer society conference on computer vision and pattern recognition, IEEE, pp 1975–1981

  27. Mathieu M, Couprie C, LeCun Y (2015)

  28. Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. In: 2009 IEEE conference on computer vision and pattern recognition, IEEE, pp 935–942

  29. Nanni L, Ghidoni S, Brahnam S (2017) Handcrafted vs. non-handcrafted features for computer vision classification. Pattern Recogn 71:158–172

    Article  Google Scholar 

  30. Nawaratne R, Alahakoon D, De Silva D, Yu X (2019) Spatiotemporal anomaly detection using deep learning for real-time video surveillance. IEEE Trans Ind Inf 16(1):393–402

    Article  Google Scholar 

  31. Patraucean V, Handa A, Cipolla R (2015) Spatio-temporal video autoencoder with differentiable memory. arXiv:1511.06309

  32. Piciarelli C, Foresti GL (2006) On-line trajectory clustering for anomalous events detection. Pattern Recogn Lett 27(15):1835–1842

    Article  Google Scholar 

  33. Ravanbakhsh M, Nabi M, Sangineto E, Marcenaro L, Regazzoni C, Sebe N (2017) Abnormal event detection in videos using generative adversarial nets. In: 2017 IEEE international conference on image processing (ICIP), IEEE, pp 1577–1581

  34. Sabokrou M, Fathy M, Hoseini M (2016) Video anomaly detection and localisation based on the sparsity and reconstruction error of auto-encoder. Electron Lett 52(13):1122–1124

    Article  Google Scholar 

  35. Shi X, Chen Z, Wang H, Yeung D-Y, Wong W-K, Woo WC (2015) Convolutional lstm network: a machine learning approach for precipitation nowcasting. Adv Neural Inf Process Syst 28:802–810

    Google Scholar 

  36. Singh VK, Kolekar MH (2021) Deep learning empowered covid-19 diagnosis using chest ct scan images for collaborative edge-cloud computing platform. Multimed Tools Appl 81(1):3–30

    Article  Google Scholar 

  37. Smeureanu S, Ionescu RT, Popescu M, Alexe B (2017) Deep appearance features for abnormal behavior detection in video. In: International conference on image analysis and processing, Springer, pp 779–789

  38. Sobhani F, Straccia U (2019)

  39. Sreenu G, Durai MS (2019) Intelligent video surveillance: a review through deep learning techniques for crowd analysis. J Big Data 6(1):1–27

    Article  Google Scholar 

  40. Srivastava N, Mansimov E, Salakhudinov R (2015). In: International conference on machine learning, pp 843–852

  41. Tung F, Zelek JS, Clausi DA (2011) Goal-based trajectory analysis for unusual behaviour detection in intelligent surveillance. Image Vis Comput 29(4):230–240

    Article  Google Scholar 

  42. Ullah H, Altamimi AB, Uzair M, Ullah M (2018) Anomalous entities detection and localization in pedestrian flows. Neurocomputing 290:74–86

    Article  Google Scholar 

  43. Wang G, Yuan X, Zheng A, Hsu H-M, Hwang J-N (2019) Anomaly candidate identification and starting time estimation of vehicles from traffic videos. In: CVPR workshops, pp 382–390

  44. Wang L, Zhou F, Li Z, Zuo W, Tan H (2018) Abnormal event detection in videos using hybrid spatio-temporal autoencoder. In: 2018 25th IEEE international conference on image processing (ICIP), IEEE, pp 2276–2280

  45. Wang X, Xie W, Song J (2018) Learning spatiotemporal features with 3dcnn and convgru for video anomaly detection. In: 2018 14th IEEE International Conference on Signal Processing (ICSP), IEEE, pp 474–479

  46. Xu L, Ren JS, Liu C, Jia J (2014) Deep convolutional neural network for image deconvolution. Adv Neural Inf Process Syst 27:1790–1798

    Google Scholar 

  47. Yan S, Smith JS, Lu W, Zhang B (2018) Abnormal event detection from videos using a two-stream recurrent variational autoencoder. IEEE Trans Cogn Dev Syst 12(1):30–42

    Article  Google Scholar 

  48. Yang Y, Fu Z, Naqvi SM (2019) Enhanced adversarial learning based video anomaly detection with object confidence and position. In: 2019 13th international conference on signal processing and communication systems (ICSPCS), IEEE, pp 1–5

  49. Zhao B, Fei-Fei L, Xing EP (2011) Online detection of unusual events in videos via dynamic sparse coding. In: CVPR 2011, IEEE, pp 3313–3320

  50. Zhou JT, Du J, Zhu H, Peng X, Liu Y, Goh RSM (2019) Anomalynet: an anomaly detection network for video surveillance. IEEE Trans Inf Forensic Secur 14(10):2537–2550

    Article  Google Scholar 

  51. Zhou S, Shen W, Zeng D, Fang M, Wei Y, Zhang Z (2016) Spatial–temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Process Image Commun 47:358–368

    Article  Google Scholar 

  52. Zhou Y, Yan S, Huang TS (2007) Detecting anomaly in videos from trajectory similarity analysis. In: 2007 IEEE international conference on multimedia and expo, IEEE, pp 1087–1090

  53. Zhu X, Liu J, Wang J, Li C, Lu H (2014) Sparse representation for robust abnormality detection in crowded scenes. Pattern Recogn 47(5):1791–1799

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Indian Institute of Technology Patna, Bihta, India

    Nazia Aslam & Maheshkumar H Kolekar

  2. Video Surveillance Lab, Department of Electrical Engineering, IIT Patna, Bihta, PIN- 801106, Bihar, India

    Nazia Aslam

Authors
  1. Nazia Aslam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maheshkumar H Kolekar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nazia Aslam.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aslam, N., Kolekar, M.H. Unsupervised anomalous event detection in videos using spatio-temporal inter-fused autoencoder. Multimed Tools Appl 81, 42457–42482 (2022). https://doi.org/10.1007/s11042-022-13496-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13496-6

Keywords

Search

Navigation