Skip to main content
SpringerLink
Log in

Surveillance video anomaly detection via non-local U-Net frame prediction

  • 1183: Multimedia Processing to Tackle the Dark Side of Social Life
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Anomaly detection of surveillance video has become a critical concern in computer vision. It can be used for real-time monitoring and the timely generation of alarms and is widely applied in transportation systems and security systems. An unsupervised anomaly detection method for surveillance video based on frame prediction is implemented in this paper. Generative Adversarial Network (GAN) is used to generate the high-quality frame. Two generators are designed to predict the next future frame. Non-local U-Net is proposed as Generator 1 for frame prediction to predict the global information. Generator 2 obtains more related past frame features and large contour information. The predicted frame and the ground truth are compared to determine anomalies. We take spatial constraints during generative adversarial training, including gradient loss and intensity loss, and time constraints, such as optical flow loss, into account. We experimentally verify that the proposed method has better accuracy in surveillance videos than some other state-of-the-art anomaly detection algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Anala M, Makker M, Ashok A (2019) Anomaly detection in surveillance videos. 2019 26th International Conference on High Performance Computing. Data and Analytics Workshop (HiPCW), IEEE, pp 93–98

  2. Cong Y, Yuan J, Liu J (2011) Sparse reconstruction cost for abnormal event detection. In: CVPR 2011. IEEE, pp 3449–3456

  3. Dhole H, Sutaone M, Vyas V (2019) Anomaly detection using convolutional spatiotemporal autoencoder. In: 2019 10th International conference on computing, communication and networking technologies (ICCCNT). IEEE, pp 1–5

  4. Dong F, Zhang Y, Nie X (2020) Dual discriminator generative adversarial network for video anomaly detection. IEEE Access 8:88170–88176

    Article  Google Scholar 

  5. Dosovitskiy A, Fischer P, Ilg E, Hausser P, Hazirbas C, Golkov V, Van Der Smagt P, Cremers D, Brox T (2015) Flownet: Learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 2758–2766

  6. Fan S, Meng F (2020) Video prediction and anomaly detection algorithm based on dual discriminator. In: 2020 5th International conference on computational intelligence and applications (ICCIA). IEEE, pp 123–127

  7. Fu J, Fan W, Bouguila N (2018) A novel approach for anomaly event detection in videos based on autoencoders and SE networks. In: 2018 9th International symposium on signal, image, video and communications (ISIVC). IEEE, pp 179–184

  8. Ganokratanaa T, Aramvith S, Sebe N (2020) Unsupervised anomaly detection and localization based on deep spatiotemporal translation network. IEEE Access 8:50312–50329

    Article  Google Scholar 

  9. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  10. Gu X, Cui J, Zhu Q (2014) Abnormal crowd behavior detection by using the particle entropy. Optik 125(14):3428–3433

    Article  Google Scholar 

  11. Hasan M, Choi J, Neumann J, Roy-Chowdhury AK, Davis LS (2016) Learning temporal regularity in video sequences. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 733–742

  12. Hinami R, Mei T, Satoh S (2017) Joint detection and recounting of abnormal events by learning deep generic knowledge. In: Proceedings of the IEEE International conference on computer vision, pp 3619–3627

  13. Hu J, Zhu E, Wang S, Wang S, Liu X, Yin J (2019) Two-stage unsupervised video anomaly detection using low-rank based unsupervised one-class learning with ridge regression. In: 2019 International joint conference on neural networks (IJCNN). IEEE, pp 1–8

  14. Ismail Y, Hammad M, El-Medany W (2018) Homeland security video surveillance system for smart cities. In: 2018 International conference on innovation and intelligence for informatics, computing, and technologies (3ICT). IEEE, pp 1–4

  15. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  16. Kim H, Lee S, Kim Y, Lee S, Lee D, Ju J, Myung H (2016) Weighted joint-based human behavior recognition algorithm using only depth information for low-cost intelligent video-surveillance system. Expert Sys Appl 45:131–141

    Article  Google Scholar 

  17. Kim J, Grauman K (2009) Observe locally, infer globally: a space-time mrf for detecting abnormal activities with incremental updates. In: 2009 IEEE conference on computer vision and pattern recognition. IEEE, pp 2921–2928

  18. Koshti D, Kamoji S, Kalnad N, Sreekumar S, Bhujbal S (2020) Video anomaly detection using inflated 3d convolution network. In: 2020 International conference on inventive computation technologies (ICICT). IEEE, pp 729–733

  19. Kwon YH, Park MG (2019) Predicting future frames using retrospective cycle gan. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1811–1820

  20. Li N, Chang F, Liu C (2020) Spatial-temporal cascade autoencoder for video anomaly detection in crowded scenes. IEEE Trans Multimed 23:203–215

    Article  Google Scholar 

  21. Li S, Fang J, Xu H, Xue J (2020) Video frame prediction by deep multi-branch mask network. IEEE Trans Circuits Sys Vid Technol 31(4):1283–1295

    Article  Google Scholar 

  22. Li W, Mahadevan V, Vasconcelos N (2013) Anomaly detection and localization in crowded scenes. IEEE Trans Pattern Anal Mach Intell 36(1):18–32

    Google Scholar 

  23. Li Y, Cai Y, Liu J, Lang S, Zhang X (2019) Spatio-temporal unity networking for video anomaly detection. IEEE Access 7:172425–172432

    Article  Google Scholar 

  24. Liang X, Lee L, Dai W, Xing EP (2017) Dual motion gan for future-flow embedded video prediction. In: Proceedings of the IEEE International conference on computer vision, pp 1744–1752

  25. Liu W, Luo W, Lian D, Gao S (2018) Future frame prediction for anomaly detection–a new baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6536–6545

  26. Lu C, Shi J, Jia J (2013) Abnormal event detection at 150 fps in matlab. In: Proceedings of the IEEE international conference on computer vision, pp 2720–2727

  27. Lu Y, Kumar KM, shahabeddin Nabavi S, Wang Y (2019) Future frame prediction using convolutional VRNN for anomaly detection. In: 2019 16th IEEE International conference on advanced video and signal based surveillance (AVSS). IEEE, pp 1–8

  28. Luo W, Liu W, Gao S (2017) Remembering history with convolutional lstm for anomaly detection. In: 2017 IEEE International conference on multimedia and expo (ICME). IEEE, pp 439–444

  29. Luo W, Liu W, Gao S (2017) A revisit of sparse coding based anomaly detection in stacked rnn framework. In: Proceedings of the IEEE international conference on computer vision, pp 341–349

  30. Luo W, Liu W, Lian D, Tang J, Duan L, Peng X, Gao S (2021) Video anomaly detection with sparse coding inspired deep neural networks. IEEE Pattern Anal Mach Intell 43(3):1070–1084

    Article  Google Scholar 

  31. Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. In: 2010 IEEE computer society conference on computer vision and pattern recognition, IEEE, pp 1975–1981

  32. Mathieu M, Couprie C, LeCun Y (2015) Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:151105440

  33. Medel JR, Savakis A (2016) Anomaly detection in video using predictive convolutional long short-term memory networks. arXiv preprint arXiv: 161200390

  34. Morais R, Le V, Tran T, Saha B, Mansour M, Venkatesh S (2019) Learning regularity in skeleton trajectories for anomaly detection in videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11996–12004

  35. Nawaratne R, Alahakoon D, De Silva D, Yu X (2019) Spatiotemporal anomaly detection using deep learning for real-time video surveillance. IEEE Trans Industrial Informatics 16(1):393–402

    Article  Google Scholar 

  36. Nayak R, Pati UC, Das SK (2020) Video anomaly detection using convolutional spatiotemporal autoencoder. In: 2020 International conference on contemporary computing and applications (IC3A). IEEE, pp 175–180

  37. Persia F, D’Auria D, Pilato G (2020) An overview of video surveillance approaches. In: 2020 IEEE 14th International conference on semantic computing (ICSC). IEEE, pp 287–294

  38. Ravanbakhsh M, Nabi M, Sangineto E, Marcenaro L, Regazzoni C, Sebe N (2017) Abnormal event detection in videos using generative adversarial nets. In: 2017 IEEE International conference on image processing (ICIP). IEEE, pp 1577–1581

  39. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241

  40. Sabzalian B, Marvi H, Ahmadyfard A (2019) Deep and sparse features for anomaly detection and localization in video. In: 2019 4th International conference on pattern recognition and image analysis (IPRIA). IEEE, pp 173–178

  41. Singh P, Pankajakshan V (2018) A deep learning based technique for anomaly detection in surveillance videos. In: 2018 Twenty fourth national conference on communications (NCC). IEEE, pp 1–6

  42. Sultani W, Chen C, Shah M (2018) Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6479–6488

  43. Wang J, Wang W, Gao W (2018) Predicting diverse future frames with local transformation-guided masking. IEEE Trans Circuits Sys Vid Technol 29(12):3531–3543

    Article  Google Scholar 

  44. Wang X, Girshick R, Gupta A, He K (2018) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794–7803

  45. Yang Y, Fu Z, Naqvi SM (2019) Enhanced adversarial learning based video anomaly detection with object confidence and position. In: 2019 13th International conference on signal processing and communication systems (ICSPCS). IEEE, pp 1–5

  46. Zhang C, Chen T, Liu H, Shen Q, Ma Z (2019) Looking-ahead: Neural future video frame prediction. In: 2019 IEEE International conference on image processing (ICIP). IEEE, pp 1975–1979

  47. Zhou JT, Du J, Zhu H, Peng X, Liu Y, Goh RSM (2019) Anomalynet: An anomaly detection network for video surveillance. IEEE Trans Inf Forensics Secur 14(10):2537–2550

    Article  Google Scholar 

  48. Zhou S, Shen W, Zeng D, Fang M, Wei Y, Zhang Z (2016) Spatial-temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Process Image Commun 47:358–368

    Article  Google Scholar 

  49. Zong X, Zhang L, Du J, Wei L, Huang Q (2019) Abnormal event detection in video based on SVDD. In: 2019 10th IEEE International conference on intelligent data acquisition and advanced computing systems: technology and applications (IDAACS), vol 1. IEEE, pp 368–371

Download references

Acknowledgements

This work was supported by Research Fund of Guangxi Key Lab of Multi-source Information Mining & Security (MIMS19-03), the National Natural Science Foundation of China under Grants (62072295) and the Natural Science Foundation of Shanghai under Grant 19ZR1419000.

Author information

Authors and Affiliations

  1. School of Communication and Information Engineering, Shanghai University, 200444, Shanghai, China

    Qianqian Zhang, Guorui Feng & Hanzhou Wu

  2. Guangxi Key Lab of Multi-source Information Mining & Security, Guangxi Normal University, 541004, Guilin, China

    Guorui Feng

Authors
  1. Qianqian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guorui Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hanzhou Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guorui Feng.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Q., Feng, G. & Wu, H. Surveillance video anomaly detection via non-local U-Net frame prediction. Multimed Tools Appl 81, 27073–27088 (2022). https://doi.org/10.1007/s11042-021-11550-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11550-3

Keywords

Search

Navigation