Skip to main content
SpringerLink
Log in

Motion-shape-based deep learning approach for divergence behavior detection in high-density crowd

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

We propose a novel method of abnormal crowd behavior detection in surveillance videos. Mainly, our work focuses on detecting crowd divergence behavior that can lead to serious disasters like a stampede. We introduce a notion of physically capturing motion in the form of images and classify crowd behavior using a convolution neural network (CNN) trained on motion-shape images (MSIs). First, the optical flow (OPF) is computed, and finite-time Lyapunov exponent (FTLE) field is obtained by integrating OPF. Lagrangian coherent structure (LCS) in the FTLE field represents crowd-dominant motion. A ridge extraction scheme is proposed for the conversion of LCS-to-grayscale MSIs. Lastly, a supervised training approach is utilized with CNN to predict normal or divergence behavior for any unknown image. We test our method on six real-world low- as well as high-density crowd datasets and compare performance with state-of-the-art methods. Experimental results show that our method is not only robust for any type of scene but also outperform existing state-of-the-art methods in terms of accuracy. We also propose a divergence localization method that not only identifies divergence starting (source) points but also comes with a new feature of generating a ‘localization mask’ around the diverging crowd showing the size of divergence. Finally, we also introduce two new datasets containing videos of crowd normal and divergence behaviors at the high density.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29

Similar content being viewed by others

References

  1. Illiyas, F.T., Mani, S.K., Pradeepkumar, A.P., Mohan, K.: Human stampedes during religious festivals: a comparative review of mass gathering emergencies in India. Int. J. Disaster Risk Reduct. 5, 10–18 (2013). https://doi.org/10.1016/j.ijdrr.2013.09.003

    Article  Google Scholar 

  2. Batty, M., Desyllas, J., Duxbury, E.: The discrete dynamics of small-scale spatial events: agent-based models of mobility in carnivals and street parades. Int. J. Geogr. Inf. Sci. 17(7), 673–697 (2003). https://doi.org/10.1080/1365881031000135474

    Article  Google Scholar 

  3. Dong, Y.-H., Liu, F., Liu, Y.-M., Jiang, X.-R., Zhao, Z.-X.: Emergency preparedness for mass gatherings: lessons of ‘12.31’ stampede in Shanghai Bund. Chin. J. Traumatol. 20(4), 240–242 (2017). https://doi.org/10.1016/j.cjtee.2016.08.005

    Article  Google Scholar 

  4. Helbing, D., Mukerji, P.: Crowd disasters as systemic failures: analysis of the Love Parade disaster. EPJ Data Sci. 1(1), 1–40 (2012). https://doi.org/10.1140/epjds7

    Article  Google Scholar 

  5. Johansson, A., Helbing, D., Al-Abideen, H.Z., Al-Bosta, S.: From crowd dynamics to crowd safety: a video-based analysis (2008) [Online]. http://arxiv.org/abs/0810.4590

  6. Cong, Y., Yuan, J., Liu, J.: Sparse reconstruction cost for abnormal event detection. In: Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 3449–3456 (2011). https://doi.org/10.1109/CVPR.2011.5995434

  7. Wu, S., Wong, H.S., Yu, Z.: A bayesian model for crowd escape behavior detection. IEEE Trans. Circuits Syst. Video Technol. 24(1), 85–98 (2014). https://doi.org/10.1109/TCSVT.2013.2276151

    Article  Google Scholar 

  8. Chen, C.Y., Shao, Y.: Crowd escape behavior detection and localization based on divergent centers. IEEE Sens. J. 15(4), 2431–2439 (2015). https://doi.org/10.1109/JSEN.2014.2381260

    Article  Google Scholar 

  9. https://www.worldbulletin.net/asia-pacific/stampede-at-hindu-festival-kills-18-in-south-india-h162104.html

  10. https://www.oasys-software.com/products/pedestrian-simulation/massmotion/

  11. Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1981), 185–203 (1981)

    Article  Google Scholar 

  12. Brox, T., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. Comput. Vis.—ECCV 2004(4), 25–36 (2004). https://doi.org/10.1007/978-3-540-24673-2_3

    Article  MATH  Google Scholar 

  13. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. Proc. Imaging Underst. Work. 130, 121–130 (1981)

    Google Scholar 

  14. Fortun, D., Bouthemy, P., Kervrann, C., Fortun, D., Bouthemy, P., Kervrann, C.: Optical flow modeling and computation : a survey. Comput. Vis. Image Underst. 134, 1–21 (2015)

    Article  Google Scholar 

  15. Lawal, I.A., Poiesi, F., Anguita, D., Cavallaro, A.: Support vector motion clustering. IEEE Trans. Circuits Syst. Video Technol. 27(11), 1–1 (2016). https://doi.org/10.1109/TCSVT.2016.2580401

    Article  Google Scholar 

  16. Cheriyadat, A.M., Radke, R.J.: Detecting dominant motions in dense crowds. IEEE J. Sel. Top. Signal Process. 2(4), 568–581 (2008). https://doi.org/10.1109/JSTSP.2008.2001306

    Article  Google Scholar 

  17. Benabbas, Y., Ihaddadene, N., Djeraba, C.: Motion pattern extraction and event detection for automatic visual surveillance. Eurasip J. Image Video Process. 2011, 1–15 (2011)

    Article  Google Scholar 

  18. Ali, S., Shah, M.: A Lagrangian particle dynamics approach for crowd flow simulation and stability analysis. In: IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, pp. 1–6 (2007). https://doi.org/10.1109/CVPR.2007.382977

  19. Hu, M.H.M., Ali, S., Shah, M.: Learning motion patterns in crowded scenes using motion flow field. In: 2008 19th Int. Conf. Pattern Recognit., pp. 2–6 (2008). https://doi.org/10.1109/ICPR.2008.4761183

  20. Solmaz, B., Moore, B.E., Shah, M.: Identifying behaviors in crowd scenes using stability analysis for dynamical systems. IEEE Trans. Pattern Anal. Mach. Intell. 34, 2064–2070 (2012). https://doi.org/10.1109/TPAMI.2012.123

    Article  Google Scholar 

  21. Chen, D.Y., Huang, P.C.: Motion-based unusual event detection in human crowds. J. Vis. Commun. Image Represent. 22(2), 178–186 (2011). https://doi.org/10.1016/j.jvcir.2010.12.004

    Article  Google Scholar 

  22. Hu, W., Xiao, X., Fu, Z., Xie, D., Tan, T., Maybank, S.: A system for learning statistical motion patterns. IEEE Trans. Pattern Anal. Mach. Intell. 28(9), 1450–1464 (2006). https://doi.org/10.1109/TPAMI.2006.176

    Article  Google Scholar 

  23. Kratz, L., Nishino, K.: Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In: 2009 IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Work. CVPR Work, pp. 1446–1453 (2009). https://doi.org/10.1109/CVPRW.2009.5206771

  24. Kratz, L., Member, S., Nishino, K.: Spatio-temporal motion patterns in extremely crowded scenes. Analysis 34(5), 987–1002 (2012)

    Google Scholar 

  25. Cong, Y., Yuan, J., Tang, Y.: Video anomaly search in crowded scenes via spatio-temporal motion context. IEEE Trans. Inf. Forens. Secur. 8(10), 1590–1599 (2013). https://doi.org/10.1109/TIFS.2013.2272243

    Article  Google Scholar 

  26. Wu, Y., Ye, Y., Zhao, C.: Coherent motion detection with collective density clustering. In: Proceedings of the 23rd ACM international conference on Multimedia—MM’15, 2015, vol. 1, no. 1, pp. 361–370. https://doi.org/10.1145/2733373.2806227

  27. Ali, S., Shah, M.: A Lagrangian particle dynamics approach for crowd flow segmentation and stability analysis. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–6 (2007)

  28. Zitouni, M.S., Bhaskar, H., Dias, J., Al-Mualla, M.E.: Advances and trends in visual crowd analysis: a systematic survey and evaluation of crowd modelling techniques. Neurocomputing 186, 139–159 (2015). https://doi.org/10.1016/j.neucom.2015.12.070

    Article  Google Scholar 

  29. Wu, S., Moore, B.E., Shah, M.: Chaotic invariants of Lagrangian particle trajectories for anomaly detection in crowded scenes. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2054–2060 (2010)

  30. Wang, X., Gao, M., He, X., Wu, X., Li, Y.: An abnormal crowd behavior detection algorithm based on fluid mechanics. J. Comput. 9(5), 1144–1149 (2014). https://doi.org/10.4304/jcp.9.5.1144-1149

    Article  Google Scholar 

  31. Wang, X., Yang, X., He, X., Teng, Q., Gao, M.: A high accuracy flow segmentation method in crowded scenes based on streakline. Opt.—Int. J. Light Electron Opt. 125(3), 924–929 (2014). https://doi.org/10.1016/j.ijleo.2013.07.166

    Article  Google Scholar 

  32. Mehran, R., Moore, B.E., Shah, M.: A streakline representation of flow in crowded scenes. Lect. Notes Comput. Sci. (incl. Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinf.) 6313 LNCS(PART 3), 439–452 (2010). https://doi.org/10.1007/978-3-642-15558-1_32

    Article  Google Scholar 

  33. Pereira, E.M., Cardoso, J.S., Morla, R.: Long-range trajectories from global and local motion representations. J. Vis. Commun. Image Represent. 40, 265–287 (2016). https://doi.org/10.1016/j.jvcir.2016.06.020

    Article  Google Scholar 

  34. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.A.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)

    MathSciNet  MATH  Google Scholar 

  35. Huang, S., Huang, D., Zhou, X.: Learning multimodal deep representations for crowd anomaly event detection. Math. Probl. Eng. (2018). https://doi.org/10.1155/2018/6323942.

  36. Xu, J., Ren, D., Zhang, L., Zhang, D.: Patch group based bayesian learning for blind image denoising. Lect. Notes Comput. Sci. (incl. Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinf.) 10116 LNCS, 79–95 (2017). https://doi.org/10.1007/978-3-319-54407-6_6

    Article  Google Scholar 

  37. Li, X., Shen, H., Li, H., Zhang, L.: Patch matching-based multitemporal group sparse representation for the missing information reconstruction of remote-sensing images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 9(8), 3629–3641 (2016). https://doi.org/10.1109/JSTARS.2016.2533547

    Article  Google Scholar 

  38. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: EEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), pp. 886–893 (2005)

  39. Barnich, O., Van Droogenbroeck, M.: ViBe: A universal background subtraction algorithm for video sequences. IEEE Trans. Image Process. 20(6), 1709–1724 (2011). https://doi.org/10.1109/TIP.2010.2101613

    Article  MathSciNet  MATH  Google Scholar 

  40. Mehran, R., Oyama, A., Shah, M.: Abnormal crowd behavior detection using social force model. In: 2009 IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Work. CVPR Work, no. 2, pp. 935–942 (2009). https://doi.org/10.1109/CVPRW.2009.5206641

  41. Shadden, S.C., Lekien, F., Marsden, J.E.: Definition and properties of Lagrangian coherent structures from finite-time Lyapunov exponents in two-dimensional aperiodic flows. Phys. D Nonlinear Phenom. 212(3–4), 271–304 (2005). https://doi.org/10.1016/j.physd.2005.10.007

    Article  MathSciNet  MATH  Google Scholar 

  42. Lipinski, D., Mohseni, K.: A ridge tracking algorithm and error estimate for efficient computation of Lagrangian coherent structures. Chaos Interdiscip. J. Nonlinear Sci. 20(1), 017504 (2010). https://doi.org/10.1063/1.3270049

    Article  MathSciNet  MATH  Google Scholar 

  43. Peikert, R., Schindler, B., Carnecky, R.: Ridge surface methods for the visualization of Lagrangian coherent structures. Semseg.Org [Online]. http://www.semseg.org/results/_files/2012-ICFD-PeikertEtAl-RidgeSurfaceMethods.pdf%5Cnpapers3://publication/uuid/7F60AE35-CBAF-4E7B-B59F-23A81EADD827

  44. Zeiler, M.D., et al.: On rectified linear units for speech processing New York University, USA Google Inc., USA University of Toronto, Canada. In: IEEE International Conference on Acoustic Speech and Signal Processing (ICASSP 2013), pp. 3–7 (2013)

  45. Krausz, B., Bauckhage, C.: Loveparade 2010: automatic video analysis of a crowd disaster. Comput. Vis. Image Underst. 116(3), 307–319 (2012). https://doi.org/10.1016/j.cviu.2011.08.006

    Article  Google Scholar 

  46. Wu, S., Moore, B.E., Shah, M.: Chaotic invariants of Lagrangian particle trajectories for anomaly detection in crowded scenes. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2054–2060 (2010)

  47. U. C. F. Crowd Dataset. https://www.crcv.ucf.edu/data/crowd.php

  48. U. M. N. Crowd Dataset. http://mha.cs.umn.edu/proj_events.shtml#crowd

  49. 2009 Dataset, PETS. http://www.cvg.reading.ac.uk/PETS2009/a.html

  50. NGSIM. https://ops.fhwa.dot.gov/trafficanalysistools/ngsim.htm

  51. Wu, S., Yang, H., Zheng, S., Su, H., Fan, Y., Yang, M.H.: Crowd behavior analysis via curl and divergence of motion trajectories. Int. J. Comput. Vis. 123(3), 1–21 (2017). https://doi.org/10.1007/s11263-017-1005-y

    Article  MathSciNet  MATH  Google Scholar 

  52. Direkoglu, C.: Abnormal crowd behavior detection using motion information images and convolutional neural networks. IEEE Access 8, 80408–80416 (2020). https://doi.org/10.1109/ACCESS.2020.2990355

    Article  Google Scholar 

  53. Ravanbakhsh, M., Nabi, M., Sangineto, E., Marcenaro, L., Regazzoni, C., Sebe, N.: Abnormal event detection in videos using generative adversarial nets. In: Proceedings—International Conference on Image Processing, ICIP, vol. 2017, pp. 1577–1581 (2018). https://doi.org/10.1109/ICIP.2017.8296547

  54. Ravanbakhsh, M., Nabi, M., Mousavi, H., Sangineto, E., Sebe, N.: Plug-and-play CNN for crowd motion analysis: an application in abnormal event detection. In: Proceedings—2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, vol. 2018, pp. 1689–1698 (2018). https://doi.org/10.1109/WACV.2018.00188

  55. Ahmed, F., Tarlow, D., Batra, D.: Optimizing expected intersection-over-union with candidate-constrained CRFs. In: Proc. IEEE Int. Conf. Comput. Vis., vol. 2015 Inter, pp. 1850–1858 (2015). https://doi.org/10.1109/ICCV.2015.215

Download references

Acknowledgments

We thank Dr. Yasir Salih for sharing high-density crowd dataset and valuable suggestions on motion estimation at high-density crowd. The MassMotion crowd simulation software is supported by the Center for Intelligent Signal for Imaging Research (CISIR) under PO Number 3920089787/30.10.2017.

Author information

Authors and Affiliations

  1. Center for Intelligent Signal and Imaging Research (CISIR), Universiti Teknologi PETRONAS, 32610, Bandar Seri Iskandar, Perak Darul Ridzuan, Malaysia

    Muhammad Umer Farooq & Mohamad Naufal M. Saad

  2. Department of Computer Science, National University of Technology, Islamabad, Pakistan

    Sultan Daud Khan

Authors
  1. Muhammad Umer Farooq
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamad Naufal M. Saad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sultan Daud Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Umer Farooq.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Farooq, M.U., Saad, M.N.M. & Khan, S.D. Motion-shape-based deep learning approach for divergence behavior detection in high-density crowd. Vis Comput 38, 1553–1577 (2022). https://doi.org/10.1007/s00371-021-02088-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02088-4

Keywords

Search

Navigation