Skip to main content
SpringerLink
Log in

Video anomaly detection with memory-guided multilevel embedding

  • Regular Paper
  • Published:
International Journal of Multimedia Information Retrieval Aims and scope Submit manuscript

Abstract

Playing a vitally important role in the operation of intelligent video surveillance system and smart city, video anomaly detection (VAD) has been widely practiced and studied in both industrial circles and academia. In the present study, a new anomaly detection method is proposed for multi-level memory embedding. According to the novel method, the feature prototype of the sample is stored in the memory pool, which enhances the diversity of the sample feature prototype paradigm. Besides, the memory is embedded in the decoder in a hierarchical integrating manner, which makes the feature information of the object more complete and improves the quality of features. At the end of the model, modeling is performed for the channel relationship between the features of the object in the channel dimension, thus making the model capable of more efficient anomaly detection. This method is verified by conducting evaluation on three publicly available datasets: UCSD Ped2, CUHK Avenue, ShanghaiTech.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

Not applicable.

References

  1. Tao X, Gong X, Zhang X et al (2022) Deep learning for unsupervised anomaly localization in industrial images: a survey. IEEE Trans Instrum Meas 71:1–21

    Google Scholar 

  2. Suarez JJP, Naval Jr P C (2020) A survey on deep learning techniques for video anomaly detection. arXiv preprint arXiv:2009.14146

  3. Saligrama V, Konrad J, Jodoin PM (2010) Video anomaly identification. IEEE Signal Process Mag 27(5):18–33

    Article  Google Scholar 

  4. Gong MG, Zeng HM, Xie Y et al (2020) Local distinguishability aggrandizing network for human anomaly detection. Neural Netw 122:364–373

    Article  Google Scholar 

  5. Redmon J, Divvala S, Girshick R, et al (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 779–788

  6. Srivastava N, Mansimov E, Salakhudinov R (2015) Unsupervised learning of video representations using lstms, In: International conference on machine learning, pp 843–852

  7. Zou Y F (2109) Recognition and research about abnormal behavior of human based on video, Yunnan University, Kunming

  8. Ristea NC, Madan N, Ionescu RT, et al (2022) Self-supervised predictive convolutional attentive block for anomaly detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 13576–13586

  9. Peng J, Zhao Y, Wang L (2021) Research on video abnormal behavior detection based on deep learning. Prog Laser Optoelectron 58(6):0600004

    Article  Google Scholar 

  10. Liu W, Luo W, Lian D, et al (2018) Future frame prediction for anomaly detection–a new baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6536–6545

  11. Wang G, Wang Y, Qin J, et al (2022) Video anomaly detection by solving decoupled spatio-temporal jigsaw puzzles. In: Proceedings of the ieee conference on european conference on computer vision, pp 494–511

  12. Zaheer MZ, Mahmood A, Khan MH, et al (2022) Generative cooperative learning for unsupervised video anomaly detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 14744–14754

  13. Yang J, Shi Y, Qi Z (2022) Learning deep feature correspondence for unsupervised anomaly detection and segmentation. Patt Recogn 132:108874

    Article  Google Scholar 

  14. Zong B, Song Q, Min MR, et al (2018). Deep autoencoding gaussian mixture model for unsupervised anomaly detection, In: International conference on learning representations

  15. Lu Y, Kumar KM, Shahabeddin Nabavi S, et al (2019). Future frame prediction using convolutional vrnn for anomaly detection. In: 2019 16th IEEE international conference on advanced video and signal based surveillance, pp 1–8

  16. Yu G, Wang S, Cai Z, et al (2020) Cloze test helps: effective video anomaly detection via learning to complete video events. In: Proceedings of the 28th ACM international conference on multimedia, pp 583–591

  17. Paffenroth R, Du Toit P, Nong R et al (2013) Space-time signal processing for distributed pattern detection in sensor networks. IEEE J Select Topics Sign Process 7(1):38–49

    Article  Google Scholar 

  18. Hoffmann H (2007) Kernel PCA for novelty detection. Pattern Recogn 40(3):863–874

    Article  MATH  Google Scholar 

  19. Laxhammar R, Falkman G, Sviestins E (2009) Anomaly detection in sea traffic-a comparison of the gaussian mixture model and the kernel density estimator In: 2009 12th international conference on information fusion. pp 756–763

  20. Latecki LJ, Lazarevic A, Pokrajac D (2007) Outlier detection with kernel density functions. MLDM 7:61–75

    Google Scholar 

  21. Ma J, Perkins S (2003) Time-series novelty detection using one-class support vector machines. In: Proceedings of the international joint conference on neural networks, pp 1741–1745

  22. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  23. Hasan M, Choi J, Neumann J, et al (2016) Learning temporal regularity in video sequences In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 733–742

  24. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. pp 234–241

  25. Goodfellow IJ, Pouget-Abadie J, Mirza M et al (2014) Generative adversarial networks. Adv Neural Inf Process Syst 3:2672–2680

    Google Scholar 

  26. Hasan M, Choi J, Neumann J, et al (2016) Learning temporal regularity in video sequences. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 733–742

  27. Zhao Y, Deng B, Shen C, et al (2017) Spatio temporal autoencoder for video anomaly detection In: Proceedings of the 25th ACM international conference on multimedia. pp 1933–1941

  28. Zaheer MZ, Lee JH, Astrid M, et al (2020) Old is gold: redefining the adversarially learned one-class classifier training paradigm. In: 2020 IEEE/CVF Conference on computer vision and pattern recognition, pp 14183–14193

  29. Zhang C, Song D, Chen Y et al (2019) A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. Proceed AAAI Conf Artif Intell 33(1):1409–1416

    Google Scholar 

  30. Park H, Noh J, Ham B (2020) Learning memory-guided normality for anomaly detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 14372–14381

  31. Medel J R, Savakis A (2016) Anomaly detection in video using predictive convolutional long short-term memory networks. arXiv preprint arXiv:1612.00390

  32. Zhao Y, Deng B, Shen C, et al (2017) Spatio-temporal autoencoder for video anomaly detection. In: Proceedings of the 25th ACM international conference on Multimedia. pp 1933–1941

  33. Le VT, Kim YG (2022). Attention-based residual autoencoder for video anomaly detection. Applied Intelligence, pp 1–15

  34. Liu Z, Nie Y, Long C, et al (2021) A hybrid video anomaly detection framework via memory-augmented flow reconstruction and flow-guided frame prediction. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 13588–13597

  35. Chen D, Wang P, Yue L et al (2020) Anomaly detection in surveillance video based on bidirectional prediction. Image Vis Comput 98:103915

    Article  Google Scholar 

  36. Zhang XX, Zhu Z, Zhao Y, Chang DX (2017) Learning a general assignment model for video analytics. IEEE Trans Circuits Syst Video Technol 28(10):3066–3076

    Article  Google Scholar 

  37. Xia GY, Chen BJ, Sun HJ, Liu QS (2020) Nonconvex low-rank kernel sparse subspace learning for keyframe extraction and motion segmentation. IEEE Trans Neural Netw Learn Sys 32(4):1612–1626

    Article  MathSciNet  Google Scholar 

  38. Meng J, Wang H, Yuan J, et al (2016) From keyframes to key objects: video summarization by representative object proposal selection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 1039–1048

  39. Wang H, Kawahara Y, Weng C et al (2017) Representative selection with structured sparsity. Pattern Recogn 63:268–278

    Article  Google Scholar 

  40. Gong D, Liu L, Le V, et al (2019) Memorizing normality to detect anomaly: memory- augmented deep autoencoder for unsupervised anomaly detection. In: Proceedings of the IEEE/CVF international conference on computer vision. pp 1705–1714

  41. Graves A, Wayne G, Reynolds M et al (2016) Hybrid computing using a neural network with dynamic external memory. Nature 538(7626):471–476

    Article  Google Scholar 

  42. Lv H, Chen C, Cui Z, et al (2021) Learning normal dynamics in videos with meta prototype network. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 15425–15434

  43. Zhang Y, Wang J, Chen Y et al (2022) Adaptive memory networks with self-supervised learning for unsupervised anomaly detection. IEEE Trans Knowl Data Eng. https://doi.org/10.1109/TKDE.2021.3139916

    Article  Google Scholar 

  44. Hendrycks D, Gimpel K (2016) Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415

  45. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 7132–7141

  46. Mahadevan V, Li W, Bhalodia V, et al (2010) Anomaly detection in crowded scenes. In: 2010 IEEE computer society conference on computer vision and pattern recognition. pp 1975–1981

  47. Lu C, Shi J, Jia J (2013) Abnormal event detection at 150 fps in matlab. In: Proceedings of the IEEE international conference on computer vision. pp 2720–2727

  48. Luo W, Liu W, Gao S (2017) A revisit of sparse coding based anomaly detection in stacked rnn framework. In: Proceedings of the IEEE international conference on computer vision. pp 341–349

  49. Paszke A, Gross S, Chintala S, et al (2017) Automatic differentiation in pytorch. In: Advances in neural information processing systems 30

  50. Kim J, Grauman K (2009) Observe locally, infer globally: a space-time MRF for detecting abnormal activities with incremental updates. In: 2009 IEEE conference on computer vision and pattern recognition, pp 2921–2928

  51. Xu D, Ricci E, Yan Y, et al (2015) Learning deep representations of appearance and motion for anomalous event detection. arXiv preprint arXiv:1510.01553

  52. Tudor Ionescu R, Smeureanu S, Alexe B, et al (2017) Unmasking the abnormal events in video. In: Proceedings of the IEEE international conference on computer vision. pp 2895–2903

  53. Hinami R, Mei T, Satoh S (2017) Joint detection and recounting of abnormal events by learning deep generic knowledge. In: Proceedings of the IEEE international conference on computer vision. pp 3619–3627

  54. Nguyen T N, Meunier J (2019) Anomaly detection in video sequence with appearance-motion correspondence. In: Proceedings of the IEEE/CVF international conference on computer vision. pp 1273–1283.

  55. Chang Y, Tu Z, Xie W, et al (2020) Clustering driven deep autoencoder for video anomaly detection. In: European conference on computer vision. pp 329–345

  56. Tang Y, Zhao L, Zhang S et al (2020) Integrating prediction and reconstruction for anomaly detection. Pattern Recogn Lett 129:123–130

    Article  Google Scholar 

  57. Kanu-Asiegbu A M, Vasudevan R, Du X (2021) Leveraging trajectory prediction for pedestrian video anomaly detection. In: 2021 IEEE symposium series on computational intelligence, pp 01–08

  58. Li B, Leroux S, Simoens P (2021) Decoupled appearance and motion learning for efficient anomaly detection in surveillance video. Comput Vis Image Underst 210:103249

    Article  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

The Young Innovative Talents Project of Guangdong Province (No.2020KQNCX198); Basic and Applied Basic Research Project of Guangzhou Basic Research Program.

Author information

Authors and Affiliations

  1. School of Information Engineering, Guang Zhou Railway Ploytechnic, Guangzhou, 510430, China

    Liuping Zhou & Jing Yang

  2. St. Paul University Phillippines, Tuguegarao City, Cagayan, 3500, Philippines

    Jing Yang

Authors
  1. Liuping Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jing Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, datacollection and analysis were performed by Liuping Zhou. The first draft of the manuscript was written by Jing Yang and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Liuping Zhou.

Ethics declarations

Conflict of interest

The author declare that they have no conflict of interest.

Consent for publication

Yes.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, L., Yang, J. Video anomaly detection with memory-guided multilevel embedding. Int J Multimed Info Retr 12, 6 (2023). https://doi.org/10.1007/s13735-023-00272-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13735-023-00272-x

Keywords

Search

Navigation