Advertisement

Towards understanding socio-cognitive behaviors of crowds from visual surveillance data

  • M. Sami ZitouniEmail author
  • Andrzej Sluzek
  • Harish Bhaskar
Article
First Online:
  • 34 Downloads

Abstract

The problem of understanding socio-cognitive aspects of crowd behavior is a challenging yet critical task particularly for human-computer interaction applications. This issue is considered an important component of both current surveillance systems and futuristic interactions between intelligent agents and crowds of humans. In this paper, a probabilistic formulation of different categories of socio-cognitive crowd behavior is proposed. This framework models in a hierarchical manner relationships between the movements of entities within the crowd to differentiate between various crowd behaviors. Functionally, the framework can be considered a mid-level layer between purely visual analysis (i.e. detection and tracking of crowd components) and detailed semantics of crowd behaviors. In the presented study, the proposed framework extends a Gaussian Mixture Model of Dynamic Textures detection (GMM-of-DT) technique using Kalman filtering for the hierarchical motion representation of the crowds, simultaneously at micro (individual crowd members) and macro (groups of individuals) levels. Thereon, socio-cognitive crowd behaviors are categorized into individual, group, leader-follower and social interaction types. The proposed approach is validated on exemplary sequences from a benchmark PETS dataset (and, preliminarily, on other publicly available datasets), where the actual detection/tracking results are employed to evaluate probabilities of various socio-cognitive behaviors, and to compare the outputs of the proposed models to manually annotated ground-truth data.

Keywords

Crowd analysis Socio-cognitive behaviors Behavior analysis Detection Tracking 
This is a preview of subscription content, log in to check access.

Notes

References

  1. 1.
    Baig MW, Barakova EI, Marcenaro L, Regazzoni CS, Rauterberg M (2014) Bio-inspired probabilistic model for crowd emotion detection. In: International Joint Conference on Neural Networks (IJCNN), pp 3966–3973Google Scholar
  2. 2.
    Benfold B, Reid I (2011) Stable multi-target tracking in real-time surveillance video. In: CVPR, pp 3457–3464Google Scholar
  3. 3.
    Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition, vol 1, pp 886–893Google Scholar
  4. 4.
    Dehghan A, Idrees H, Zamir AR, Shah M (2014) Automatic detection and tracking of pedestrians in videos with various crowd densities. In: Pedestrian and Evacuation Dynamics 2012, Springer, pp 3–19Google Scholar
  5. 5.
    Dehghan A, Assari SM, Shah M (2015) Gmmcp tracker: Globally optimal generalized maximum multi clique problem for multiple object tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4091–4099Google Scholar
  6. 6.
    Dollár P, Appel R, Belongie S, Perona P (2014) Fast feature pyramids for object detection. IEEE Trans Pattern Anal Mach Intell 36(8):1532–1545CrossRefGoogle Scholar
  7. 7.
    Doretto G, Chiuso A, Wu YN, Soatto S (2003) Dynamic textures. Int J Comput Vis 51:91–109CrossRefGoogle Scholar
  8. 8.
    Ferryman J, Shahrokni A (2009) Pets2009: Dataset and challenge. In: Twelfth IEEE International Workshop on Performance Evaluation of Tracking and SurveillanceGoogle Scholar
  9. 9.
    Fridman N, Kaminka GA (2010) Modeling pedestrian crowd behavior based on a cognitive model of social comparison theory. Comput Math Organ Theory 16 (4):348–372CrossRefGoogle Scholar
  10. 10.
    Gonen M, Alpaydin E (2011) Multiple kernel learning algorithms. J Mach Res 12:2211–2268MathSciNetzbMATHGoogle Scholar
  11. 11.
    Jinhuan W, Nan L, Lei Z (2015) Small group behaviors and their impacts on pedestrian evacuation. In: The 27th Chinese Control and Decision Conference, pp 232–237Google Scholar
  12. 12.
    Leach M, Baxter R, Robertson N, Sparks E (2014) Detecting social groups in crowded surveillance videos using visual attention. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 467–473Google Scholar
  13. 13.
    Li Y (2018) A deep spatiotemporal perspective for understanding crowd behavior. IEEE Trans Multimedia 20(12):3289–3297CrossRefGoogle Scholar
  14. 14.
    Luber M, Stork JA, Tipaldi GD, Arras KO (2010) People tracking with human motion predictions from social forces. In: IEEE International Conference on Robotics and Automation (ICRA), pp 464–469Google Scholar
  15. 15.
    Madrigal F, Hayet JB, Lerasle F (2014) Intention-aware multiple pedestrian tracking. In: 2014 22nd International Conference on Pattern Recognition (ICPR), pp 4122–4127Google Scholar
  16. 16.
    Mazzon R, Poiesi F, Cavallaro A (2013) Detection and tracking of groups in crowd. In: IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp 202–207Google Scholar
  17. 17.
    Mehran R, Oyama A, Shah M (June 2009) Abnormal crowd behavior detection using social force model. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 935–942Google Scholar
  18. 18.
    Milan A, Roth S, Schindler K (2014) Continuous energy minimization for multitarget tracking. IEEE Trans Pattern Anal Mach Intell 36(1):58–72CrossRefGoogle Scholar
  19. 19.
    Park SI, Quek F, Cao Y (Dec 2012) Modeling social groups in crowds using common ground theory. In: Proceedings of the 2012 Winter Simulation Conference (WSC), pp 1–12Google Scholar
  20. 20.
    Ravanbakhsh M, Nabi M, Mousavi H, Sangineto E, Sebe N (2018) Plug-and-play cnn for crowd motion analysis: an application in abnormal event detection. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp 1689–1698Google Scholar
  21. 21.
    Solera F, Calderara S, Cucchiara R (2013) Structured learning for detection of social groups in crowd. In: IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp 7–12Google Scholar
  22. 22.
    Solera F, Calderara S, Cucchiara R (2015) Learning to identify leaders in crowd. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 43–48Google Scholar
  23. 23.
    Tian S, Yuan F, Xia G-S (2016) Multi-object tracking with inter-feedback between detection and tracking. Neurocomputing 171:768–780CrossRefGoogle Scholar
  24. 24.
    Lu W, Yung NHC, Xu L (2014) Multiple-human tracking by iterative data association and detection update. IEEE Trans Intell Transp Syst 15(5):1886–1899CrossRefGoogle Scholar
  25. 25.
    Wei H, Xiao Y, Li R, Liu X (2018) Crowd abnormal detection using two-stream fully convolutional neural networks. In: 10Th International Conference on Measuring Technology and Mechatronics Automation (ICMTMA), pp 332–336Google Scholar
  26. 26.
    Wen L, Lei Z, Lyu S, Li SZ, Yang MH (2016) Exploiting hierarchical dense structures on hypergraphs for multi-object tracking. IEEE Trans Pattern Anal Mach Intell 38(10):1983–1996CrossRefGoogle Scholar
  27. 27.
    Wen L, Li W, Yan J, Lei Z, Yi D, Li SZ (2014) Multiple target tracking based on undirected hierarchical relation hypergraph. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 1282–1289Google Scholar
  28. 28.
    Zaki MH, Sayed T (2018) Automated analysis of pedestrian group behavior in urban settings. IEEE Trans Intell Transp Syst 19(6):1880–1889CrossRefGoogle Scholar
  29. 29.
    Zhang Y, Qin L, Ji R, Yao H, Huang Q (2015) Social attribute-aware force model: Exploiting richness of interaction for abnormal crowd detection. IEEE Trans Circuits Syst Video Technol 25(7):1231–1245CrossRefGoogle Scholar
  30. 30.
    Zhang Y, Qin L, Ji R, Zhao S, Huang Q, Luo J (2017) Exploring coherent motion patterns via structured trajectory learning for crowd mood modeling. IEEE Trans Circuits Syst Video Technol 27(3):635–648CrossRefGoogle Scholar
  31. 31.
    Zhang Y, Qin L, Yao H, Huang Q (2012) Abnormal crowd behavior detection based on social attribute-aware force model. In: IEEE International Conference on Image Processing, pp 2689–2692Google Scholar
  32. 32.
    Zhao J, Xu Y, Yang X, Yan Q (2011) Crowd instability analysis using velocity-field based social force model. In: IEEE Visual Communications and Image Processing (VCIP), pp 1–4Google Scholar
  33. 33.
    Zhao S, Yao H, Gao Y, Ding G, Chua T (Oct 2018) Predicting personalized image emotion perceptions in social networks. IEEE Trans Affect Comput 9(4):526–540CrossRefGoogle Scholar
  34. 34.
    Zhao S, Yao H, Gao Y, Ji R, Ding G (2017) Continuous probability distribution prediction of image emotions via multitask shared sparse regression. IEEE Trans Multimedia 19(3):632–645CrossRefGoogle Scholar
  35. 35.
    Zhou T, Yang J, Loza A, Bhaskar H, Al-Mualla M (2015) A crowd modelling framework using fast head detection and shape-aware matching. Journal of Electronic Imaging, p 24Google Scholar
  36. 36.
    Sami Zitouni M, Bhaskar H, Al-Mualla ME (2016) Robust background modeling and foreground detection using dynamic textures. In: VISIGRAPP (4: VISAPP), pp 403–410Google Scholar
  37. 37.
    Sami Zitouni M, Bhaskar H, Dias J, Al-Mualla ME (2016) Advances and trends in visual crowd analysis: a systematic survey and evaluation of crowd modelling techniques. Neurocomputing 186:139–159CrossRefGoogle Scholar
  38. 38.
    Sami Zitouni M, Dias J, Al-Mualla M, Bhaskar H (2015) Hierarchical crowd detection and representation for big data analytics in visual surveillance. In: IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, pp 1827–1832Google Scholar
  39. 39.
    Sami Zitouni M, Sluzek A, Bhaskar H (2019) Visual analysis of socio-cognitive crowd behaviors for surveillance: a survey and categorization of trends and methods. Eng Appl Artif Intell 82:294–312CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • M. Sami Zitouni
    • 1
    Email author
  • Andrzej Sluzek
    • 1
    • 2
  • Harish Bhaskar
    • 3
  1. 1.Khalifa University of Science and TechnologyAbu DhabiUnited Arab Emirates
  2. 2.Warsaw University of Life Sciences-SGGWWarszawaPoland
  3. 3.Zero One Infinity Consulting (ZOIC) Services Ltd.OntarioCanada

Personalised recommendations

Cite article

Buy options