Skip to main content
SpringerLink
Log in

Human crowd behaviour analysis based on video segmentation and classification using expectation–maximization with deep learning architectures

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In recent years, the demand for automatic crowd behavior analysis has surged, driven by the need to ensure public safety and minimize casualties during events of public and religious significance. However, effectively analyzing the nonlinearities present in real-world crowd images and videos remains a challenge. To address this, research proposes a novel approach leveraging deep learning (DL) architectures for the segmentation and classification of human crowd behavior. Our method begins by collecting input from surveillance videos capturing crowd activity, which is then processed to remove noise and extract the crowd scene. Subsequently, we employ an expectation–maximization-based ZFNet architecture for accurate video segmentation. The segmented video is then classified using transfer exponential Conjugate Gradient Neural Networks, enhancing the precision of crowd behavior characterization. Our method has been proven effective in experimental analysis on many human crowd datasets, with significant results of average mean precision (MAP) of 59%, the mean square error (MSE) of 61%, accuracy in the training of 95%, validation precision of 95%, and selectivity of 88%. The potential of DL-based methods to advance crowd behavior analysis for improved privacy and security is highlighted by this study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

Abbreviations

[d0,…, d9] :

Local Descriptor

[D0…, Dt-1] :

SSIM value

I :

Incomplete

g :

Dimensional

z :

Component memberships

\(1,{\pi }_{1},\dots ,{\pi }_{k}\) :

Mulr multinomial distribution

\(Q\left(\Psi :{\widehat{\Psi }}^{n}\right)\) :

Conditional expectation

v :

Observed data

\({z}_{i}^{n}\) :

Posterior probability

\({\Gamma }_{i}\) :

Kth component of the mixture

\(\widehat{\psi }\) :

Parameter set

\(\widehat{\Psi }(t+1)\) :

Fresh parameter

\({\dot{\pi }}_{k}^{n+1)}\) :

New estimates

\({z}_{n}^{n}\) :

Digamma function

h × w :

Height, width

d :

Depth

\({x}_{ij}\) :

Input vector

\({f}_{ks}\) :

Function

\({y}_{ij}\) :

Vector output

\({I}_{n}(g)\) :

Average of a function g

\(g\left({{\varvec{x}}}_{i}\right)\) :

Independent random variables

\({W}_{n}\) :

Weight matrix

\({B}_{m(j)}\) :

Bias and m is the number of inputs

\(\Delta (f*)\) :

Constant

\(\underset{j}{{\text{inf}}} {\int }_{{\mathbb{R}}^{d}}\) :

Fourier transform extension

W :

Output weights

\({{\varvec{w}}}_{j}^{0}\) :

Random variables

\((a1,\dots ,am)\) :

Coefficients

U(k) :

Projected convergence rate

V(k) :

Observed consensus error

\({z}_{i}(k)\) :

Iterates

z :

First term

z* :

Expected optimization error

\(R(k)\) :

SGD's effectiveness

\({\eta }_{t}\) :

Rate of learning

\({\mathbf{d}}_{\mathbf{t}}\) :

Sample drawn

d :

Random sample

\(\Omega\) :

Sample space

\({h}_{t+1}-{h}_{t}\) :

Random variable

\({\mathbf{w}}^{t}\) :

The separation between the existing solution

\({\mathbf{w}}^{*}\) :

Ideal solution

\({h}_{t}\) :

Random variable

References

  1. Tyagi B, Nigam S, Singh R (2022) A review of deep learning techniques for crowd behaviour analysis. Arch Computat Methods Eng 29(7):5427–5455

    Article  Google Scholar 

  2. Chaudhary D, Kumar S, Dhaka VS (2022) Video based human crowd analysis using machine learning: a survey. Comput Methods Biomech Biomed Eng: Imaging Vis 10(2):113–131

    Google Scholar 

  3. Bruno A, Ferjani M, Sabeur Z, Arbab-Zavar B, Cetinkaya D, Johnstone L, ... Benaouda D (2022) High-level feature extraction for crowd behaviour analysis: a computer vision approach. In Image Analysis and Processing. ICIAP 2022 Workshops: ICIAP International Workshops, Lecce, Italy, May 23–27, 2022, Revised Selected Papers, Part II (pp. 59–70). Springer International Publishing, Cham

  4. Kong YX, Wu RJ, Zhang YC, Shi GY (2023) Utilizing statistical physics and machine learning to discover collective behaviour on temporal social networks. Inf Process Manage 60(2):103190

    Article  Google Scholar 

  5. Farooq MU, Mohamad Saad MN, Saleh Y, Daud Khan S (2022) Deep learning approach for divergence behaviour detection at high density crowd. In International Conference on Artificial Intelligence for Smart Community: AISC 2020, 17–18 December, UniversitiTeknologi Petronas, Malaysia (pp. 875–888). Springer Nature Singapore, Singapore

  6. Sharma V, Mir RN, Singh C (2023) Scale-aware CNN for crowd density estimation and crowd behaviour analysis. Comput Electr Eng 106:108569

    Article  Google Scholar 

  7. Bahamid A, Mohd Ibrahim A (2022) A review on crowd analysis of evacuation and abnormality detection based on machine learning systems. Neural Comput Appl 34(24):21641–21655

    Article  Google Scholar 

  8. Bhuiyan MR, Abdullah J, Hashim N, Al Farid F (2022) Video analytics using deep learning for crowd analysis: a review. Multimed Tools Appl 81(19):27895–27922

  9. Matkovic F, Ivasic-Kos M, Ribaric S (2022) A new approach to dominant motion pattern recognition at the macroscopic crowd level. Eng Appl Artif Intell 116:105387

    Article  Google Scholar 

  10. Hou H, Wang L (2022) Measuring dynamics in evacuation behaviour with deep learning. Entropy 24(2):198

    Article  PubMed  PubMed Central  ADS  Google Scholar 

  11. Pattan P, Arjunagi S (2022) A human behaviour analysis model to track object behaviour in surveillance videos. Measurement: Sensors 24:100454

    Google Scholar 

  12. Abpeikar S, Kasmarik K, Garratt M, Hunjet R, Khan MM, Qiu H (2022) Automatic collective motion tuning using actor-critic deep reinforcement learning. Swarm Evol Comput 72:101085

    Article  Google Scholar 

  13. Zhang D, Li W, Gong J, Huang L, Zhang G, Shen S, ... Ma H (2022) HDRLM3D: a deep reinforcement learning-based model with human-like perceptron and policy for crowd evacuation in 3D environments. ISPRS Int J Geo-Inform 11(4):255

  14. Lu Y, Ruan X, Huang J (2022) Deep reinforcement learning based on social spatial-temporal graph convolution network for crowd navigation. Machines 10(8):703

    Article  Google Scholar 

  15. Liu T, Zheng Q, Tian L (2022) Application of distributed probability model in sports based on deep learning: deep belief network (DL-DBN) algorithm for human behaviour analysis. Comput Intell Neurosci 2022

  16. Ha D, Tang Y (2022) Collective intelligence for deep learning: a survey of recent developments. Collective Intell 1(1):26339137221114870

    Article  Google Scholar 

  17. Liang Z, Li L, Wang L (2022) Crowd-oriented behaviour simulation: reinforcement learning framework embedded with emotion model. In Artificial Intelligence: Second CAAI International Conference, CICAI 2022, Beijing, China, August 27–28, 2022, Revised Selected Papers, Part III (pp. 195–207). Springer Nature Switzerland, Cham

  18. Choi T, Pyenson B, Liebig J, Pavlic TP (2022) Beyond tracking: using deep learning to discover novel interactions in biological swarms. Artif Life Robot 27(2):393–400

    Article  Google Scholar 

  19. Poon KH, Wong PKY, Cheng JC (2022) Long-time gap crowd prediction using time series deep learning models with two-dimensional single attribute inputs. Adv Eng Inform 51:101482

    Article  Google Scholar 

  20. Tiwari RG, Yadav SK, Misra A, Sharma A (2022) Classification of swarm collective motion using machine learning. In Human-Centric Smart Computing: Proceedings of ICHCSC 2022. Springer Nature Singapore, Singapore, pp 173–181

  21. Chakole PD, Satpute VR, Cheggoju N (2022) Crowd behaviour anomaly detection using correlation of optical flow magnitude. J Phys: Conf Ser 2273(1):012023 (IOP Publishing)

    Google Scholar 

  22. Guo B, Liu Y, Liu S, Yu Z, Zhou X (2022) CrowdHMT: crowd intelligence with the deep fusion of human, machine, and IoT. IEEE Internet Things J 9(24):24822–24842

    Article  Google Scholar 

  23. Tripathi SK (2022) Design and development of some methods and models for crowd analysis using computer vision and deep learning approaches.

  24. Lalit R, Purwar RK (2022) Crowd abnormality detection using optical flow and glcm-based texture features. J Inform Technol Res (JITR) 15(1):1–15

    Article  Google Scholar 

  25. Pai AK, Chandrahasan P, Raghavendra U, Karunakar AK (2023) Motion pattern-based crowd scene classification using histogram of angular deviations of trajectories. Vis Comput 39(2):557–567

  26. Bala B, Kadurka RS, Negasa G (2022) Recognizing unusual activity with the deep learning perspective in crowd segment. In: A Fusion of Artificial Intelligence and Internet of Things for Emerging Cyber Systems. Springer, Cham, pp 171–181

  27. Vidhyalakshmi M, Ramesh S, Bharathi ML, Kshirsagar PR, Rajaram A, Tirth V (2023) A comparative recognition research on excretory organism in medical applications using neural networks. Multimed Tools Appl 1–18

  28. Shafiq M, Tian Z, Bashir AK, Du X, Guizani M (2020) CorrAUC: A malicious bot-IoT traffic detection method in IoT network using machine-learning techniques. IEEE Internet Things J 8(5):3242–3254

    Article  Google Scholar 

  29. Shafiq M, Tian Z, Bashir AK, Du X, Guizani M (2020) IoT malicious traffic identification using wrapper-based feature selection mechanisms. Comput Secur 94:101863

    Article  Google Scholar 

  30. Shafiq M, Tian Z, Bashir AK, Jolfaei A, Yu X (2020) Data mining and machine learning methods for sustainable smart cities traffic classification: a survey. Sustain Cities Soc 60:102177

    Article  Google Scholar 

  31. Singh D, Kaur M, Alanazi JM, AlZubi AA, Lee HN (2022) Efficient evolving deep ensemble medical image captioning network. IEEE J Biomed Health Inform 27(2):1016–1025

    Article  Google Scholar 

  32. Raina R, Gondhi NK, Chaahat, Singh D, Kaur M, Lee HN (2023) A systematic review on acute leukemia detection using deep learning techniques. Arch Computat Methods Eng 30(1):251–270

    Article  Google Scholar 

Download references

Funding

No funding is involved in this work.

Author information

Authors and Affiliations

  1. Department of CSE, Birla Institute of Technology, Mesra, Jharkhand, 835215, India

    Shruti Garg

  2. Department of DSE, SIT, Manipal University Jaipur, Jaipur, India

    Sudhir Sharma

  3. Department of IT, SIT, Manipal University Jaipur, Jaipur, India

    Sumit Dhariwal

  4. Department of Computer Science and Engineering, Saveetha School of Engineering, Saveetha Institute of Medical and Technical Sciences (SIMATS), Saveetha University, Chennai, India

    W. Deva Priya

  5. Department of E&TC, Symbiosis Institute of Technology, Symbiosis International (Deemed University), Pune, Maharashtra, 412115, India

    Mangal Singh

  6. Department of Networking and Communications, School of Computing, Faculty of Engineering and Technology, SRM Institute of Science and Technology Kattankulathur, Chennai, India

    S. Ramesh

Authors
  1. Shruti Garg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sudhir Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sumit Dhariwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. W. Deva Priya
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mangal Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. S. Ramesh
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors are contributed equally to this work.

Corresponding author

Correspondence to S. Ramesh.

Ethics declarations

Ethics approval and consent to participate

No participation of humans takes place in this implementation process.

Human and animal rights

No violation of Human and Animal Rights is involved.

Conflict of interest

Conflict of interest is not applicable in this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Garg, S., Sharma, S., Dhariwal, S. et al. Human crowd behaviour analysis based on video segmentation and classification using expectation–maximization with deep learning architectures. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-18630-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-024-18630-0

Keywords

Search

Navigation