Skip to main content
SpringerLink
Log in

Mutuality-oriented reconstruction and prediction hybrid network for video anomaly detection

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Anomaly detection in surveillance videos aiming to locate anomalous events is a very challenging task, since when training a detector, only normal video samples are available. And thus, most existing approaches address this problem in a semi-supervised way by either predicting or reconstructing the video frames and then compute anomaly scores by comparing the generated frames with reference frames. However, reconstruction-based methods usually lead to mis-detection due to the excessively powerful reconstruction abilities yet the incapable capturing of temporal information, while prediction-based methods are able to perceive temporal information but insufficient to produce realistic future frames. To overcome these problems, we propose a novel Mutuality-Oriented Reconstruction and Prediction Hybrid Network (MORPH-Net) for detecting anomalous events. In the MORPH-Net, a new Mutuality-oriented Training (MO-Training) mechanism is introduced to better combine the advantages of prediction-based models and reconstruction-based models. Compared to traditional single training mechanism or simple fusion mechanism, the MO-Training mechanism can prompt the generator module to produce temporally discriminative and realistic frames which benefit the anomaly detection. The experiments evaluated on three large-scale video surveillance datasets show the efficacy of our method compared with the state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Biswas, S., Babu, R.V.: Sparse representation based anomaly detection with enhanced local dictionaries. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 5532–5536. IEEE (2014)

  2. Chen, T., Hou, C., Wang, Z., Chen, H.: Anomaly detection in crowded scenes using motion energy model. Multimed. Tools Appl. 77(11), 14137–14152 (2018)

    Article  Google Scholar 

  3. Chong, Y.S., Tay, Y.H.: Abnormal event detection in videos using spatiotemporal autoencoder. In: International Symposium on Neural Networks, pp. 189–196. Springer (2017)

  4. Cong, Y., Yuan, J., Liu, J.: Sparse reconstruction cost for abnormal event detection. In: CVPR 2011, pp 3449–3456. IEEE (2011)

  5. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 886–893. IEEE (2005)

  6. Dalal, N., Triggs, B., Schmid, C.: Human detection using oriented histograms of flow and appearance. In: European Conference on Computer Vision, pp. 428–441. Springer (2006)

  7. Del Giorno, A., Bagnell, J.A., Hebert, M.: A discriminative framework for anomaly detection in large videos. In: European Conference on Computer Vision, pp. 334–349. Springer (2016)

  8. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020)

    Article  MathSciNet  Google Scholar 

  9. Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A.K., Davis, L.S.: Learning temporal regularity in video sequences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 733–742 (2016)

  10. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

  11. Kingma, D.P., Welling, M.: Auto-encoding variational bayes (2013). arXiv preprint arXiv:1312.6114

  12. Lee, S., Kim, H.G., Ro, Y.M.: Stan: spatio-temporal adversarial networks for abnormal event detection. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1323–1327. IEEE (2018)

  13. Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 18–32 (2013)

    Google Scholar 

  14. Liu, W., Luo, W., Lian, D., Gao, S.: Future frame prediction for anomaly detection—a new baseline. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6536–6545 (2018)

  15. Lu, C., Shi, J., Jia, J.: Abnormal event detection at 150 fps in matlab. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2720–2727 (2013)

  16. Luo, W., Liu, W., Gao, S.: Remembering history with convolutional LSTM for anomaly detection. In: 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 439–444. IEEE (2017)

  17. Mahadevan, V., Li, W., Bhalodia, V., Vasconcelos, N.: Anomaly detection in crowded scenes. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1975–1981. IEEE (2010)

  18. Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error (2015). arXiv preprint arXiv:1511.05440

  19. Nguyen, T.N., Meunier, J.: Anomaly detection in video sequence with appearance-motion correspondence. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1273–1283 (2019)

  20. Sabokrou, M., Khalooei, M., Fathy, M., Adeli, E.: Adversarially learned one-class classifier for novelty detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3379–3388 (2018)

  21. Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6479–6488 (2018)

  22. Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6479–6488 (2018)

  23. Tang, Y., Zhao, L., Zhang, S., Gong, C., Li, G., Yang, J.: Integrating prediction and reconstruction for anomaly detection. Pattern Recognit. Lett. 129, 123–130 (2020)

    Article  Google Scholar 

  24. Tudor, I.R., Smeureanu, S., Alexe, B., Popescu, M.: Unmasking the abnormal events in video. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2895–2903 (2017)

  25. Wang, W., Chang, F., Mi, H.: Intermediate fused network with multiple timescales for anomaly detection. Neurocomputing 433, 37–49 (2021)

    Article  Google Scholar 

  26. Zhao, J., Yi, Z., Pan, S., Zhao, Y., Zhao, Z., Su, F., Zhuang, B.: Unsupervised traffic anomaly detection using trajectories. In: CVPR Workshops, pp. 133–140 (2019)

  27. Zhou, J.T., Du, J., Zhu, H., Peng, X., Liu, Y., Goh, R.S.M.: Anomalynet: an anomaly detection network for video surveillance. IEEE Trans. Inf. Forensics Secur. 14(10), 2537–2550 (2019)

    Article  Google Scholar 

Download references

Acknowledgements

The project is supported by National Key R&D Program of China (NO.2018YFB1305300), National Natural Science Foundation of China (62176138, 62176136), Shandong Provincial Key Research and Development Program (Major Scientific and Technological Innovation Project ) (NO. 2019JZZY010130, 2020CXGC010207).

Author information

Authors and Affiliations

  1. School of Control Science and Engineering, Shandong University, 17923 Jingshi Road, Jinan, 250061, People’s Republic of China

    Wenqian Wang, Faliang Chang & Chunsheng Liu

Authors
  1. Wenqian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Faliang Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chunsheng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Faliang Chang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, W., Chang, F. & Liu, C. Mutuality-oriented reconstruction and prediction hybrid network for video anomaly detection. SIViP 16, 1747–1754 (2022). https://doi.org/10.1007/s11760-021-02131-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-021-02131-w

Keywords

Search

Navigation