Skip to main content
SpringerLink
Log in

Real-time adversarial GAN-based abnormal crowd behavior detection

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Detecting abnormal events in the crowd is a challenging problem. Insufficient samples make those traditional model-based methods cannot cope with sophisticated anomaly monitoring. Therefore, we design a real-time generative adversarial network plus an add-on encoder to deal with the continually changing environment. After the generator reconstructs the compressed pattern to generate the design to the latent vector, a discriminator is used to construct better videos by minimizing the adversarial loss function. We calculated the abnormal score by the distance between the two underlying patterns encoded by the first and the second encoders. The unusual event is detected when the anomaly score is above the threshold. To accelerate the processing efficiency, we introduced the grouped pointwise convolution method to decrease the computing complexity. The frame-level and video-level experiments on the benchmark dataset show the accuracy and reliance of our approach. The acceleration approach can increase the efficiency of the network with only limited accuracy loss.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Danelljan, M., Bhat, G., Gladh, S., Khan, F.S., Felsberg, M.: Deep motion and appearance cues for visual tracking. Pattern Recogn. Lett. 124, 74–81 (2018)

    Article  Google Scholar 

  2. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434v2

  3. Akcay, S., Atapour-Abarghouei, A., Breckon, T.P.: GANomaly: semi-supervised anomaly detection via adversarial training. In: Asian Conference on Computer Vision (ACCV), pp. 622–637 (2019)

  4. Runsheng, Y., Zhenyu, S., Qiongxiong, M., Laiyun, Q.: Predictive learning: using future representation learning variantial autoencoder for human action prediction. arXiv:1711.09265 (2016)

  5. Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017, pp. 5967–5976 (2017)

  6. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. arXiv:1606.03498

  7. Ihaddadene, N., Djeraba, C.: Real-time crowd motion analysis. In: International Conference on Pattern Recognition, pp. 1–4 (2008)

  8. Zhang, X., Zhang, Q., Hu, S., Guo, C., Yu, H.: Energy level-based abnormal crowd behavior detection. Sensors, 18, 423 (2018)

    Article  Google Scholar 

  9. Saligrama, V., Chen, Z.: Video anomaly detection based on local statistical aggregates. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2112–2119 (2012)

  10. Ali, S., Waqas, M., Chen, N., Chen, D., Han, Y., Boateng, B., Xiong, J., Han, J., He, W.: Three-dimensional twisted fiber composite as high-loading cathode support for lithium sulfur batteries. Compos. B Eng. 174, 107025 (2019)

    Article  Google Scholar 

  11. Tian, Y., Cheng, G., Gelernter, J., Yu, S., Song, C., Yang, B.: Joint temporal context exploitation and active learning for video segmentation. Pattern Recogn. 100, 107158 (2020)

    Article  Google Scholar 

  12. Tian, Y., Gelernter, J., Wang, X., Li, J., Yu, Y.: Traffic sign detection using a multi-scale recurrent attention network. IEEE Trans. Intell. Transp. Syst. 20, 4466–4475 (2019)

    Article  Google Scholar 

  13. Tian, Y., Wang, X., Wu, J., Wang, R., Yang, B.: Multi-scale hierarchical residual network for dense captioning. J. Artif. Intell. Res. 64, 181–196 (2019)

    Article  MathSciNet  Google Scholar 

  14. Tian, Y., Hu, W., Jiang, H., Wu, J.: Densely connected attentional pyramid residual network for human pose estimation. Neurocomputing 347, 13–23 (2019)

    Article  Google Scholar 

  15. Tian, Y., Chen, T., Cheng, G., Yu, S., Li, X., Li, J., Yang, B.: Global context assisted structure-aware vehicle retrieval. IEEE Trans. Intell. Transp. Syst., 1–10 (2020)

  16. Wang, X., Tian, Y., Zhao, X., Yang, T., Gelernter, J., Wang, J., Cheng, G., Hu, W.: Multi-person pose estimation by mask-aware deep reinforcement learning. ACM Trans. Multimedia Comput. Commun. Appl., 84–100 (2020)

  17. Tian, Y., Zhang, K., Li, J., Lin, X., Yang, B.: LSTM based traffic flow prediction with missing data. Neurocomputing 318, 297–305 (2018)

    Article  Google Scholar 

  18. Tian, Y., Zhang, Y., Zhou, D., Cheng, G., Chen, W.-G., Wang, R.: Triple attention network for video segmentation. Neurocomputing 417, 202–211 (2020)

    Article  Google Scholar 

  19. Tian, Y., Jia, Y., Shi, Y., Liu, Y., Ji, H., Sigal, L.: Inferring 3D body pose using variational semi-parametric regression. In: 18th IEEE International Conference on Image Processing, pp. 29–32 (2011)

  20. Yang, B., Sun, S., Li, J., Lin, X., Tian, Y.: Traffic flow prediction using LSTM with feature enhancement. Neurocomputing 332, 320–327 (2019)

    Article  Google Scholar 

  21. Zeng, Q., Martin, R.R., Wang, L., Quinn, J.A., Sun, Y., Tu, C.: Region-based bas-relief generation from a single image. Graph. Models 76, 140–151 (2014)

    Article  Google Scholar 

  22. Chen, W., Sun, T., Li, M., Jiang, H., Zhou, C.: A new image co-segmentation method using saliency detection for surveillance image of coal miners. Comput. Electric. Eng. 40, 227–235 (2014)

    Article  Google Scholar 

  23. Yuan, S., Zhou, W., Chen, L.: Epileptic seizure prediction using diffusion distance and bayesian linear discriminate analysis on intracranial EEG. Int. J. Neural Syst. 28, 1750043 (2018)

    Article  Google Scholar 

  24. Zhou, C., Liu, C.: An efficient segmentation method using saliency object detection. Multimedia Tools Appl. 74, 5623–5634 (2015)

    Article  Google Scholar 

  25. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 1, 4489–4497 (2015)

    Google Scholar 

  26. Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A.K., Davis, L.S.: Learning temporal regularity in video sequences. In: 2015 IEEE Workshop on Applications of Computer Vision, pp. 148–155 (2015)

  27. Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. 36, 18–32 (2013)

    Google Scholar 

  28. Xie, H., Yang, D., Sun, N., Chen, Z., Zhang, Y.: Automated pulmonary nodule detection in CT images using deep convolutional neural networks. Pattern Recogn. 85, 109–119 (2019)

    Article  Google Scholar 

  29. Girdhar, R., Ramanan, D., Gupta, A., Sivic, J., Russell, B.C.: ActionVLAD: learning spatio-temporal aggregation for action classification. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017, pp. 3165–3174 (2017)

  30. Qi, L., Dai, P., Yu, J., Zhou, Z., Xu, Y.: Time–location–frequency–aware internet of things service selection based on historical records. Int. J. Distrib. Sens. Netw. 13, 155014771668869 (2017)

    Article  Google Scholar 

  31. Hou, M., Gao, Y., Liu, J., Dai, L., Kong, X., Shang, J.: Network analysis based on low-rank method for mining information on integrated data of multi-cancers. Comput. Biol. Chem. 78, 468–473 (2019)

    Article  Google Scholar 

  32. Zhou, C., Liu, C.: Co-segmentation of multiple similar images using saliency detection and region merging. IET Comput. Vision 8, 254–261 (2014)

    Article  Google Scholar 

  33. Wang, J., Liu, J., Zheng, C., Wang, Y., Kong, X., Wen, C.: A mixed-norm Laplacian regularized low-rank representation method for tumor samples clustering. IEEE/ACM Trans. Comput. Biol. Bioinf. 16, 172–182 (2019)

    Article  Google Scholar 

  34. Wei, C., Wang, P., Zhang, Y.: Entropy, similarity measure of interval-valued intuitionistic fuzzy sets and their applications. Inf. Sci. 181, 4273–4286 (2011)

    Article  MathSciNet  Google Scholar 

  35. Schlegl, T., Seebck, P., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G.: Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In: The International Conference on Information Processing in Medical Imaging (IPMI), pp. 146–157 (2017)

  36. Dimokranitou, A.: Adversarial autoencoders for anomalous event detection in images, Doctoral Dissertation (2017)

  37. Larsen, A.B.L., Sonderby, S.K., Larochelle, H., Winther, O.: Autoencoding beyond pixels using a learned similarity metric. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning  (ICML’16), vol. 48, pp. 1558–1566 (2016)

  38. Li, R., Sturtivant, C., Yu, J., Cheng, X.: A novel secure and efficient data aggregation scheme for IoT. IEEE Internet Things J. 6, 1551–1560 (2019)

    Article  Google Scholar 

  39. Chen, W., Wilson, J.T., Tyree, S., Weinberger, K.Q., Chen, Y.: Compressing neural networks with the hashing trick. arXiv:1504.04788

  40. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.: MobileNetV2: inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)

  41. Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: FitNets: hints for thin deep nets. arXiv:1412.6550

  42. Unusual Crowd Activity Dataset of the University of Minnesota. http://mha.cs.umn.edu/proj_events.shtml

  43. Hassner, T., Itcher, Y., Klipergross, O.: Violent flows: real-time detection of violent crowd behavior. In: 3rd IEEE International Workshop on Computer Vision and Pattern Recognition (CVPR), pp. 1–6 (2012)

  44. Tian, W., Snoussi, H.: Histograms of optical flow orientation for visual abnormal events detection. In: IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, pp. 13–18 (2012)

  45. Laptev, I., Marszalek, M., Schmid, C., Rozenfeld, B.: Learning realistic human actions from movies. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)

  46. Sabokrou, M., Fathy, M., Hoseini, M., Klette, R.: Real-time anomaly detection and localization in crowded scenes. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Boston, MA, USA, 7–12 June 2015, pp. 56–62 (2015)

  47. Wu, S., Moore, B.E., Shah, M.: Chaotic invariants of Lagrangian particle trajectories for anomaly detection in crowded scenes. In: Twenty-third IEEE Conference on Computer Vision Pattern Recognition, pp. 2054–2060 (2010)

  48. Cong, Y., Yuan, J., Liu, J.: Sparse reconstruction cost for abnormal event detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3449–3456 (2011)

  49. Mousavi, H., Nabi, M., Kiani, H., Perina, A., Murino, V.: Crowd motion monitoring using tracklet-based commotion measure. In: IEEE International Conference on Image Processing (ICIP), pp. 2354–2358 (2015)

  50. Mousavi, H., Mohammadi, S., Perina, A., Chellali, R., Mur, V.: Analyzing tracklets for the detection of abnormal crowd behavior. In: IEEE Winter Conference on Applications of Computer Vision Workshop (WACV), pp. 148–155 (2015)

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 61972351, in part by the Natural Science Foundation of Zhejiang Province under Grant LY19F030005 and Grant LY18F020008, in part by the Opening Foundation of State Key Laboratory of Virtual Reality Technology and System of Beihang University under Grant VRLAB2020B15.

Author information

Authors and Affiliations

  1. School of Computer Science, Changchun University, Changchun, 130022, People’s Republic of China

    Qiulei Han, Qinsheng Du & Nianfeng Li

  2. Beijing Institute of Computer Technology and Applications, Beijing, 110000, People’s Republic of China

    Haofeng Wang, Lin Yang, Min Wu & Jinqiao Kou

Authors
  1. Qiulei Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haofeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Min Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jinqiao Kou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qinsheng Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Nianfeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qiulei Han.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, Q., Wang, H., Yang, L. et al. Real-time adversarial GAN-based abnormal crowd behavior detection. J Real-Time Image Proc 17, 2153–2162 (2020). https://doi.org/10.1007/s11554-020-01029-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-020-01029-z

Keywords

Search

Navigation