Skip to main content
SpringerLink
Log in

Video anomaly detection with autoregressive modeling of covariance features

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

In this paper, we propose three different methods for anomaly detection in surveillance videos based on modeling of observation likelihoods. By means of the methods we propose, normal (typical) events in a scene are learned in a probabilistic framework by estimating the features of consecutive frames taken from the surveillance camera. The proposed methods are based on long short-term memory (LSTM) and linear regression. To decide whether an observation sequence (i.e., a small video patch) contains an anomaly or not, its likelihood under the modeled typical observation distribution is thresholded. An anomaly is decided to be present if the threshold is exceeded. Due to its effectiveness in object detection and action recognition applications, covariance features are used in this study to compactly reduce the dimensionality of the shape and motion cues of spatiotemporal patches obtained from the video segments. The two most successful methods are based on the final state vector of LSTM and support vector regression applied to mean covariance features and achieve an average performance of up to 0.95 area under curve on benchmark datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Abati, D., Porrello, A., Calderara, S., Cucchiara, R.: Latent space autoregression for novelty detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 481–490 (2019)

  2. Basharat, A., Gritai, A., Shah, M.: Learning object motion patterns for anomaly detection and improved object detection. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)

  3. Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. 41(3), 1–58 (2009)

    Article  Google Scholar 

  4. Chong, Y.S., Tay, Y.H.: Abnormal event detection in videos using spatiotemporal autoencoder. In: International Symposium on Neural Networks, pp. 189–196. Springer (2017)

  5. Cong, Y., Yuan, J., Liu, J.: Sparse reconstruction cost for abnormal event detection. In: CVPR 2011, pp. 3449–3456. IEEE (2011)

  6. Ergezer, H., Leblebicioğlu, K.: Anomaly detection and activity perception using covariance descriptor for trajectories. In: European Conference on Computer Vision, pp. 728–742 (2016)

  7. Guo, K., Ishwar, P., Konrad, J.: Action recognition from video using feature covariance matrices. IEEE Trans. Image Process. 22(6), 2479–2494 (2013)

    Article  MathSciNet  Google Scholar 

  8. Habiboğlu, Y.H., Günay, O., Çetin, A.E.: Covariance matrix-based fire and flame detection method in video. Mach. Vis. Appl. 23(6), 1103–1113 (2012)

    Article  Google Scholar 

  9. Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A.K., Davis, L.S.: Learning temporal regularity in video sequences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 733–742 (2016)

  10. Hu, W., Xiao, X., Fu, Z., Xie, D., Tan, T., Maybank, S.: A system for learning statistical motion patterns. IEEE Trans. Pattern Anal. Mach. Intell. 28(9), 1450–1464 (2006)

    Article  Google Scholar 

  11. Ionescu, R.T., Khan, F.S., Georgescu, M.I., Shao, L.: Object-centric auto-encoders and dummy anomalies for abnormal event detection in video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7842–7851 (2019)

  12. Javan Roshtkhari, M., Levine, M.D.: Online dominant and anomalous behavior detection in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2611–2618 (2013)

  13. Junejo, I.N., Javed, O., Shah, M.: Multi feature path modeling for video surveillance. In: Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004, vol. 2, pp. 716–719. IEEE (2004)

  14. Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 18–32 (2013)

    Google Scholar 

  15. Liu, W., Luo, W., Lian, D., Gao, S.: Future frame prediction for anomaly detection—a new baseline. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6536–6545 (2018)

  16. Oord, A.v.d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., Kavukcuoglu, K.: Wavenet: a generative model for raw audio. arXiv:1609.03499 (2016)

  17. Paisitkriangkrai, S., Shen, C., Zhang, J.: Fast pedestrian detection using a cascade of boosted covariance features. IEEE Trans. Circuits Syst. Video Technol. 18(8), 1140–1151 (2008)

    Article  Google Scholar 

  18. Porikli, F., Kocak, T.: Robust license plate detection using covariance descriptor in a neural network framework. In: 2006 IEEE International Conference on Video and Signal Based Surveillance, pp. 107–107. IEEE (2006)

  19. Porikli, F., Tuzel, O., Meer, P.: Covariance tracking using model update based on lie algebra. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 1, 728–735 (2006)

    Google Scholar 

  20. Sabokrou, M., Fayyaz, M., Fathy, M., Moayed, Z., Klette, R.: Deep-anomaly: fully convolutional neural network for fast anomaly detection in crowded scenes. Comput. Vis. Image Underst. 172, 88–97 (2018)

    Article  Google Scholar 

  21. Saligrama, V., Konrad, J., Jodoin, P.M.: Video anomaly identification. IEEE Signal Process. Mag. 27(5), 18–33 (2010)

    Article  Google Scholar 

  22. Sanin, A., Sanderson, C., Harandi, M.T., Lovell, B.C.: Spatio-temporal covariance descriptors for action and gesture recognition. In: 2013 IEEE Workshop on Applications of Computer Vision (WACV), pp. 103–110. IEEE (2013)

  23. Schölkopf, B., Williamson, R.C., Smola, A.J., Shawe-Taylor, J., Platt, J.C., et al.: Support vector method for novelty detection. In: NIPS, vol. 12, pp. 582–588. Citeseer (1999)

  24. Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.c.: Convolutional lstm network: a machine learning approach for precipitation nowcasting. arXiv:1506.04214 (2015)

  25. Shih, K.J., Dundar, A., Garg, A., Pottorf, R., Tao, A., Catanzaro, B.: Video interpolation and prediction with unsupervised landmarks. arXiv:1909.02749 (2019)

  26. Smola, A.J., Schölkopf, B.: A tutorial on support vector regression. Stat. Comput. 14(3), 199–222 (2004)

    Article  MathSciNet  Google Scholar 

  27. Tuzel, O., Porikli, F., Meer, P.: Region covariance: a fast descriptor for detection and classification. In: European Conference on Computer Vision, pp. 589–600. Springer (2006)

  28. Tuzel, O., Porikli, F., Meer, P.: Pedestrian detection via classification on Riemannian manifolds. IEEE Trans. Pattern Anal. Mach. Intell. 30(10), 1713–1727 (2008)

    Article  Google Scholar 

  29. Wang, S., Zhu, E., Yin, J., Porikli, F.: Video anomaly detection and localization by local motion based joint video representation and OCELM. Neurocomputing 277, 161–175 (2018)

    Article  Google Scholar 

  30. Wang, T., Qiao, M., Zhu, A., Niu, Y., Li, C., Snoussi, H.: Abnormal event detection via covariance matrix for optical flow based feature. Multim. Tools Appl. 77(13), 17375–17395 (2018)

    Article  Google Scholar 

  31. Wang, X., Tieu, K., Grimson, E.: Learning semantic scene models by trajectory analysis. In: European Conference on Computer Vision, pp. 110–123. Springer (2006)

  32. Xu, D., Ricci, E., Yan, Y., Song, J., Sebe, N.: Learning deep representations of appearance and motion for anomalous event detection. arXiv:1510.01553 (2015)

  33. Zhao, M., Saligrama, V.: Anomaly detection with score functions based on nearest neighbor graphs. Adv. Neural Inf. Process. Syst. 22, 2250–2258 (2009)

    Google Scholar 

Download references

Acknowledgements

This work was supported by The Scientific and Technological Research Council of Turkey (TUBITAK), Contract:118E268. Senior students A. Ozalp and M. S. Yavuz contributed to this study as a part of their graduation project under the supervision of H. Ozkan.

Author information

Authors and Affiliations

  1. Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Turkey

    Ali Enver Bilecen, Alp Ozalp, M. Sami Yavuz & Huseyin Ozkan

Authors
  1. Ali Enver Bilecen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alp Ozalp
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. Sami Yavuz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Huseyin Ozkan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ali Enver Bilecen.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bilecen, A.E., Ozalp, A., Yavuz, M.S. et al. Video anomaly detection with autoregressive modeling of covariance features. SIViP 16, 1027–1034 (2022). https://doi.org/10.1007/s11760-021-02049-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-021-02049-3

Keywords

Search

Navigation