Skip to main content
SpringerLink
Log in

Towards understanding socio-cognitive behaviors of crowds from visual surveillance data

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The problem of understanding socio-cognitive aspects of crowd behavior is a challenging yet critical task particularly for human-computer interaction applications. This issue is considered an important component of both current surveillance systems and futuristic interactions between intelligent agents and crowds of humans. In this paper, a probabilistic formulation of different categories of socio-cognitive crowd behavior is proposed. This framework models in a hierarchical manner relationships between the movements of entities within the crowd to differentiate between various crowd behaviors. Functionally, the framework can be considered a mid-level layer between purely visual analysis (i.e. detection and tracking of crowd components) and detailed semantics of crowd behaviors. In the presented study, the proposed framework extends a Gaussian Mixture Model of Dynamic Textures detection (GMM-of-DT) technique using Kalman filtering for the hierarchical motion representation of the crowds, simultaneously at micro (individual crowd members) and macro (groups of individuals) levels. Thereon, socio-cognitive crowd behaviors are categorized into individual, group, leader-follower and social interaction types. The proposed approach is validated on exemplary sequences from a benchmark PETS dataset (and, preliminarily, on other publicly available datasets), where the actual detection/tracking results are employed to evaluate probabilities of various socio-cognitive behaviors, and to compare the outputs of the proposed models to manually annotated ground-truth data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Baig MW, Barakova EI, Marcenaro L, Regazzoni CS, Rauterberg M (2014) Bio-inspired probabilistic model for crowd emotion detection. In: International Joint Conference on Neural Networks (IJCNN), pp 3966–3973

  2. Benfold B, Reid I (2011) Stable multi-target tracking in real-time surveillance video. In: CVPR, pp 3457–3464

  3. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition, vol 1, pp 886–893

  4. Dehghan A, Idrees H, Zamir AR, Shah M (2014) Automatic detection and tracking of pedestrians in videos with various crowd densities. In: Pedestrian and Evacuation Dynamics 2012, Springer, pp 3–19

  5. Dehghan A, Assari SM, Shah M (2015) Gmmcp tracker: Globally optimal generalized maximum multi clique problem for multiple object tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4091–4099

  6. Dollár P, Appel R, Belongie S, Perona P (2014) Fast feature pyramids for object detection. IEEE Trans Pattern Anal Mach Intell 36(8):1532–1545

    Article  Google Scholar 

  7. Doretto G, Chiuso A, Wu YN, Soatto S (2003) Dynamic textures. Int J Comput Vis 51:91–109

    Article  Google Scholar 

  8. Ferryman J, Shahrokni A (2009) Pets2009: Dataset and challenge. In: Twelfth IEEE International Workshop on Performance Evaluation of Tracking and Surveillance

  9. Fridman N, Kaminka GA (2010) Modeling pedestrian crowd behavior based on a cognitive model of social comparison theory. Comput Math Organ Theory 16 (4):348–372

    Article  Google Scholar 

  10. Gonen M, Alpaydin E (2011) Multiple kernel learning algorithms. J Mach Res 12:2211–2268

    MathSciNet  MATH  Google Scholar 

  11. Jinhuan W, Nan L, Lei Z (2015) Small group behaviors and their impacts on pedestrian evacuation. In: The 27th Chinese Control and Decision Conference, pp 232–237

  12. Leach M, Baxter R, Robertson N, Sparks E (2014) Detecting social groups in crowded surveillance videos using visual attention. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 467–473

  13. Li Y (2018) A deep spatiotemporal perspective for understanding crowd behavior. IEEE Trans Multimedia 20(12):3289–3297

    Article  Google Scholar 

  14. Luber M, Stork JA, Tipaldi GD, Arras KO (2010) People tracking with human motion predictions from social forces. In: IEEE International Conference on Robotics and Automation (ICRA), pp 464–469

  15. Madrigal F, Hayet JB, Lerasle F (2014) Intention-aware multiple pedestrian tracking. In: 2014 22nd International Conference on Pattern Recognition (ICPR), pp 4122–4127

  16. Mazzon R, Poiesi F, Cavallaro A (2013) Detection and tracking of groups in crowd. In: IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp 202–207

  17. Mehran R, Oyama A, Shah M (June 2009) Abnormal crowd behavior detection using social force model. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 935–942

  18. Milan A, Roth S, Schindler K (2014) Continuous energy minimization for multitarget tracking. IEEE Trans Pattern Anal Mach Intell 36(1):58–72

    Article  Google Scholar 

  19. Park SI, Quek F, Cao Y (Dec 2012) Modeling social groups in crowds using common ground theory. In: Proceedings of the 2012 Winter Simulation Conference (WSC), pp 1–12

  20. Ravanbakhsh M, Nabi M, Mousavi H, Sangineto E, Sebe N (2018) Plug-and-play cnn for crowd motion analysis: an application in abnormal event detection. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp 1689–1698

  21. Solera F, Calderara S, Cucchiara R (2013) Structured learning for detection of social groups in crowd. In: IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp 7–12

  22. Solera F, Calderara S, Cucchiara R (2015) Learning to identify leaders in crowd. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 43–48

  23. Tian S, Yuan F, Xia G-S (2016) Multi-object tracking with inter-feedback between detection and tracking. Neurocomputing 171:768–780

    Article  Google Scholar 

  24. Lu W, Yung NHC, Xu L (2014) Multiple-human tracking by iterative data association and detection update. IEEE Trans Intell Transp Syst 15(5):1886–1899

    Article  Google Scholar 

  25. Wei H, Xiao Y, Li R, Liu X (2018) Crowd abnormal detection using two-stream fully convolutional neural networks. In: 10Th International Conference on Measuring Technology and Mechatronics Automation (ICMTMA), pp 332–336

  26. Wen L, Lei Z, Lyu S, Li SZ, Yang MH (2016) Exploiting hierarchical dense structures on hypergraphs for multi-object tracking. IEEE Trans Pattern Anal Mach Intell 38(10):1983–1996

    Article  Google Scholar 

  27. Wen L, Li W, Yan J, Lei Z, Yi D, Li SZ (2014) Multiple target tracking based on undirected hierarchical relation hypergraph. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 1282–1289

  28. Zaki MH, Sayed T (2018) Automated analysis of pedestrian group behavior in urban settings. IEEE Trans Intell Transp Syst 19(6):1880–1889

    Article  Google Scholar 

  29. Zhang Y, Qin L, Ji R, Yao H, Huang Q (2015) Social attribute-aware force model: Exploiting richness of interaction for abnormal crowd detection. IEEE Trans Circuits Syst Video Technol 25(7):1231–1245

    Article  Google Scholar 

  30. Zhang Y, Qin L, Ji R, Zhao S, Huang Q, Luo J (2017) Exploring coherent motion patterns via structured trajectory learning for crowd mood modeling. IEEE Trans Circuits Syst Video Technol 27(3):635–648

    Article  Google Scholar 

  31. Zhang Y, Qin L, Yao H, Huang Q (2012) Abnormal crowd behavior detection based on social attribute-aware force model. In: IEEE International Conference on Image Processing, pp 2689–2692

  32. Zhao J, Xu Y, Yang X, Yan Q (2011) Crowd instability analysis using velocity-field based social force model. In: IEEE Visual Communications and Image Processing (VCIP), pp 1–4

  33. Zhao S, Yao H, Gao Y, Ding G, Chua T (Oct 2018) Predicting personalized image emotion perceptions in social networks. IEEE Trans Affect Comput 9(4):526–540

    Article  Google Scholar 

  34. Zhao S, Yao H, Gao Y, Ji R, Ding G (2017) Continuous probability distribution prediction of image emotions via multitask shared sparse regression. IEEE Trans Multimedia 19(3):632–645

    Article  Google Scholar 

  35. Zhou T, Yang J, Loza A, Bhaskar H, Al-Mualla M (2015) A crowd modelling framework using fast head detection and shape-aware matching. Journal of Electronic Imaging, p 24

  36. Sami Zitouni M, Bhaskar H, Al-Mualla ME (2016) Robust background modeling and foreground detection using dynamic textures. In: VISIGRAPP (4: VISAPP), pp 403–410

  37. Sami Zitouni M, Bhaskar H, Dias J, Al-Mualla ME (2016) Advances and trends in visual crowd analysis: a systematic survey and evaluation of crowd modelling techniques. Neurocomputing 186:139–159

    Article  Google Scholar 

  38. Sami Zitouni M, Dias J, Al-Mualla M, Bhaskar H (2015) Hierarchical crowd detection and representation for big data analytics in visual surveillance. In: IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, pp 1827–1832

  39. Sami Zitouni M, Sluzek A, Bhaskar H (2019) Visual analysis of socio-cognitive crowd behaviors for surveillance: a survey and categorization of trends and methods. Eng Appl Artif Intell 82:294–312

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates

    M. Sami Zitouni & Andrzej Sluzek

  2. Warsaw University of Life Sciences-SGGW, Warszawa, Poland

    Andrzej Sluzek

  3. Zero One Infinity Consulting (ZOIC) Services Ltd., Ontario, Canada

    Harish Bhaskar

Authors
  1. M. Sami Zitouni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrzej Sluzek
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Harish Bhaskar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. Sami Zitouni.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zitouni, M.S., Sluzek, A. & Bhaskar, H. Towards understanding socio-cognitive behaviors of crowds from visual surveillance data. Multimed Tools Appl 79, 1781–1799 (2020). https://doi.org/10.1007/s11042-019-08201-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08201-z

Keywords

Search

Navigation