Skip to main content
SpringerLink
Log in

IE-GAN: a data-driven crowd simulation method via generative adversarial networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Crowd simulation has been widely used in evacuation exercises, games or movie manufacturing, and many other fields. How to plan reasonable trajectories for pedestrians in a scene is always one of the critical problems in crowd simulation. Traditional simulation methods have the problem of large differences between simulated and actual trajectories, and it is difficult to generate near-real and reasonable multimodal pedestrian trajectories. In this paper, we propose a novel method utilizing generative models for crowd simulation: GAN with Incubator and Extender (IE-GAN). This data-driven model learns the movement laws of pedestrians from real datasets, and simulates a full movement trajectory for the “dummy” without corresponding situations in the dataset through a unique model architecture. In our method, the generated initial trajectory and further trajectories constitute the full trajectory of the “dummy”. Incubator networks based on long-term memory network (LSTM) are used to generate the initial trajectory, and the further trajectory is generated by the Extender, which is based on a generative adversarial network (GAN). The experimental results show that the trajectories generated by our model can approach real human’s trajectories.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Availability of data and materials

Our model is mainly performed on two common pedestrian datasets, ETH and UCY [12, 40], to verify the prediction effect of IE-GAN.

References

  1. Mirza M, Osindero S (2014) Conditional generative adversarial nets. https://doi.org/10.48550/arXiv.1411.1784, arXiv:1411.1784

  2. Helbing D, Molnár P (1995) Social force model for pedestrian dynamics. Phys Rev E 51(5):4282–4286. https://doi.org/10.1103/PhysRevE.51.4282

    Article  Google Scholar 

  3. Gupta A, Johnson J, Fei-Fei L, et al (2018) Social GAN: socially acceptable trajectories with generative adversarial networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition. IEEE, Salt Lake City, UT, pp 2255–2264, https://doi.org/10.1109/CVPR.2018.00240

  4. Sadeghian A, Kosaraju V, Sadeghian A, et al (2019) SoPhie: an attentive gan for predicting paths compliant to social and physical constraints. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, Long Beach, CA, USA, pp 1349–1358, https://doi.org/10.1109/CVPR.2019.00144

  5. Amirian J, Hayet JB, Pettre J (2019) Social ways: learning multi-modal distributions of pedestrian trajectories with GANs. In: 2019 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW). IEEE, Long Beach, CA, USA, pp 2964–2972, https://doi.org/10.1109/CVPRW.2019.00359

  6. Xu P, Hayet JB, Karamouzas I (2022) SocialVAE: human trajectory prediction using timewise latents. In: Avidan S, Brostow G, Cissé M et al (eds) Computer vision - ECCV 2022. Lecture Notes in Computer Science. Springer Nature Switzerland, Cham, pp 511–528. https://doi.org/10.1007/978-3-031-19772-7_30

    Chapter  Google Scholar 

  7. Shah A, Chavan P, Jadhav D (2022) Convolutional neural network-based image segmentation techniques. In: Reddy VS, Prasad VK, ang J et al (eds) Soft computing and signal processing. Advances in Intelligent Systems and Computing. Springer, Singapore, pp 553–561. https://doi.org/10.1007/978-981-16-1249-7_52

    Chapter  Google Scholar 

  8. Balani N, Chavan P, Ghonghe M (2022) Design of high-speed blockchain-based sidechaining peer to peer communication protocol over 5G networks. Multimed Tools Appl 81(25):36,699-36,713. https://doi.org/10.1007/s11042-021-11604-6

    Article  Google Scholar 

  9. Mohamed A, Qian K, Elhoseiny M, et al (2020) Social-STGCNN: a social spatio-temporal graph convolutional neural network for human trajectory prediction. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, Seattle, WA, USA, pp 14,412–14,420, https://doi.org/10.1109/CVPR42600.2020.01443

  10. Shi L, Wang L, Long C, et al (2021) SGCN:sparse graph convolution network for pedestrian trajectory prediction. In: 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, Nashville, TN, USA, pp 8990–8999, https://doi.org/10.1109/CVPR46437.2021.00888

  11. Lee KH, Choi MG, Hong Q, et al (2007) Group behavior from video: a data-driven approach to crowd simulation. In: Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on computer animation. Eurographics Association, Goslar, DEU, SCA ’07, pp 109–118, https://doi.org/10.1145/1272690.1272706

  12. Lerner A, Chrysanthou Y, Lischinski D (2007) Crowds by Example. Comput Graph. Forum 26(3):655–664. https://doi.org/10.1111/j.1467-8659.2007.01089.x

    Article  Google Scholar 

  13. Alahi A, Goel K, Ramanathan V, et al (2016) Social LSTM: human trajectory prediction in crowded spaces. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Las Vegas, NV, USA, pp 961–971, https://doi.org/10.1109/CVPR.2016.110

  14. Fernando T, Denman S, Sridharan S et al (2018) Soft + Hardwired attention: An LSTM framework for human trajectory prediction and abnormal event detection. Neural Netw 108:466–478. https://doi.org/10.1016/j.neunet.2018.09.002

    Article  Google Scholar 

  15. Hug R, Becker S, Hübner W, et al (2018) Particle-based pedestrian path prediction using LSTM-MDL models. In: 2018 21st international conference on intelligent transportation systems (ITSC), pp 2684–2691, https://doi.org/10.1109/ITSC.2018.8569478, iSSN: 2153-0017

  16. Wei X, Lu W, Zhu L et al (2018) Learning motion rules from real data: neural network for crowd simulation. Neurocomputing 310:125–134. https://doi.org/10.1016/j.neucom.2018.05.022

    Article  Google Scholar 

  17. Liu B, Liu H, Zhang H et al (2018) A social force evacuation model driven by video data. Simul Model Pract Theory 84:190–203. https://doi.org/10.1016/j.simpat.2018.02.007

    Article  Google Scholar 

  18. Zhao M, Turner SJ, Cai W (2013) A data-driven crowd simulation model based on clustering and classification. In: 2013 IEEE/ACM 17th international symposium on distributed simulation and real time applications, pp 125–134, https://doi.org/10.1109/DS-RT.2013.21, iSSN: 1550-6525

  19. Zhang G, Yu Z, Jin D, et al (2022) Physics-infused machine learning for crowd simulation. In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. Association for Computing Machinery, New York, NY, USA, KDD ’22, pp 2439–2449, https://doi.org/10.1145/3534678.3539440

  20. Amirian J, van Toll W, Hayet JB, et al (2019) Data-driven crowd simulation with generative adversarial networks. In: Proceedings of the 32nd international conference on computer animation and social agents. Association for Computing Machinery, New York, NY, USA, CASA ’19, pp 7–10, https://doi.org/10.1145/3328756.3328769

  21. Pang SM, Cao JX, Jian MY et al (2022) BR-GAN: a pedestrian trajectory prediction model combined with behavior recognition. IEEE Trans Intell Trans Syst 23(12):24,609-24,620. https://doi.org/10.1109/TITS.2022.3193442

    Article  Google Scholar 

  22. Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. In: 2009 IEEE conference on computer vision and pattern recognition, pp 935–942, https://doi.org/10.1109/CVPR.2009.5206641, iSSN: 1063-6919

  23. Luber M, Stork JA, Tipaldi GD, et al (2010) People tracking with human motion predictions from social forces. In: 2010 IEEE international conference on robotics and automation, pp 464–469, https://doi.org/10.1109/ROBOT.2010.5509779, iSSN: 1050-4729

  24. Pellegrini S, Ess A, Van Gool L (2010) Improving data association by joint modeling of pedestrian trajectories and groupings. In: Daniilidis K, Maragos P, Paragios N (eds) Computer vision - ECCV 2010. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, pp 452–465. https://doi.org/10.1007/978-3-642-15549-9_33

    Chapter  Google Scholar 

  25. Yamaguchi K, Berg AC, Ortiz LE et al (2011) Who are you with and where are you going? In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 1345–1352. https://doi.org/10.1109/CVPR.2011.5995468

  26. Treuille A, Cooper S, Popović Z (2006) Continuum crowds. ACM Trans Graph 25(3):1160–1168. https://doi.org/10.1145/1141911.1142008

    Article  Google Scholar 

  27. Antonini G, Bierlaire M, Weber M (2006) Discrete choice models of pedestrian walking behavior. Transport Res B-METH 40(8):667–687. https://doi.org/10.1016/j.trb.2005.09.006

    Article  Google Scholar 

  28. Rostami M, Berahmand K, Nasiri E et al (2021) Review of swarm intelligence-based feature selection methods. Eng Appl Artif Intell 100(104):210. https://doi.org/10.1016/j.engappai.2021.104210

    Article  Google Scholar 

  29. Sheikhpour R, Berahmand K, Forouzandeh S (2023) Hessian-based semi-supervised feature selection using generalized uncorrelated constraint. Knowl Based Syst 269(110):521. https://doi.org/10.1016/j.knosys.2023.110521

    Article  Google Scholar 

  30. Xu K, Ba J, Kiros R, et al (2015) Show, attend and tell: neural image caption generation with visual attention. In: Proceedings of the 32nd international conference on machine learning. PMLR, pp 2048–2057

  31. Duan J, Wang L, Long C et al (2022) Complementary attention gated network for pedestrian trajectory prediction. Proc AAAI Conf Artif Intell 36(1):542–550. https://doi.org/10.1609/aaai.v36i1.19933

    Article  Google Scholar 

  32. Chen K, Song X, Yuan H et al (2022) Fully convolutional encoder-decoder with an attention mechanism for practical pedestrian trajectory prediction. IEEE trans Intell Transp Syst 23(11):20,046-20,060. https://doi.org/10.1109/TITS.2022.3170874

    Article  Google Scholar 

  33. Zhou L, Zhao Y, Yang D et al (2022) GCHGAT: pedestrian trajectory prediction using group constrained hierarchical graph attention networks. Appl Intell 52(10):11,434-11,447. https://doi.org/10.1007/s10489-021-02997-w

    Article  Google Scholar 

  34. Lv P, Wang W, Wang Y et al (2023) Ssagcn: social soft attention graph convolution network for pedestrian trajectory prediction. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2023.3250485

  35. Goodfellow I, Pouget-Abadie J, Mirza M et al (2020) Generative adversarial networks. Commun ACM 63(11):139–144. https://doi.org/10.1145/3422622

    Article  MathSciNet  Google Scholar 

  36. Graves A (2014) Generating Sequences With Recurrent Neural Networks. https://doi.org/10.48550/arXiv.1308.0850, arXiv:1308.0850

  37. Graves A, Jaitly N (2014) Towards end-to-end speech recognition with recurrent neural networks. In: Proceedings of the 31st international conference on machine learning. PMLR, pp 1764–1772, https://doi.org/10.5555/3044805.3045089

  38. Kooij JFP, Schneider N, Flohr F et al (2014) Context-based pedestrian path prediction. In: Fleet D, Pajdla T, Schiele B et al (eds) Computer vision- ECCV 2014. Lecture Notes in Computer Science. Springer International Publishing, Cham, pp 618–633. https://doi.org/10.1007/978-3-319-10599-4_40

    Chapter  Google Scholar 

  39. Lee N, Choi W, Vernaza P, et al (2017) Desire: distant future prediction in dynamic scenes with interacting agents. pp 336–345, https://openaccess.thecvf.com/content_cvpr_2017/papers/Lee_DESIRE_Distant_Future_CVPR_2017_paper.pdf

  40. Pellegrini S, Ess A, Schindler K, et al (2009) You’ll never walk alone: modeling social behavior for multi-target tracking. In: 2009 IEEE 12th international conference on computer vision, pp 261–268, https://doi.org/10.1109/ICCV.2009.5459260, iSSN: 2380-7504

  41. Kingma DP, Ba J (2017) Adam: a method for stochastic optimization. https://doi.org/10.48550/arXiv.1412.6980, arXiv:1412.6980

  42. Konev S, Brodt K, Sanakoyeu A (2022) MotionCNN: a strong baseline for motion prediction in autonomous driving. https://doi.org/10.48550/arXiv.2206.02163,https://ui.adsabs.harvard.edu/abs/2022arXiv220602163K

  43. Ye J, Zhao Z, Wu M (2007) Discriminative K-means for clustering. In: advances in neural information processing systems, vol 20. Curran Associates, Inc., https://proceedings.neurips.cc/paper_files/paper/2007/hash/a5cdd4aa0048b187f7182f1b9ce7a6a7-Abstract.html

Download references

Funding

This paper is supported by National Key R &D Program of China (No.2021ZD0111902), NSFC (No.62072015, U21B2038, U19B2039), Beijing Natural Science Foundation (No. 4222021), R &D Program of Beijing Municipal Education Commission (No. KZ202210005008).

Author information

Authors and Affiliations

  1. Beijing Key Laboratory of Multimedia and Intelligent Software Technology, Beijing Artificial Intelligence Institute, Faculty of Information Technology, Beijing University of Technology, No. 100, Ping Leyuan Community, Beijing, 100124, China

    Xuanqi Lin, Yuchen Liang, Yong Zhang, Yongli Hu & Baocai Yin

Authors
  1. Xuanqi Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuchen Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yongli Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Baocai Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

In summary, the main contributions of this paper are the following: \(\bullet \) IE-GAN, a general network including Incubator and Extender, is presented to generate full trajectories for any pedestrian from any feasible place to any destination in the simulation scene. \(\bullet \) Those trajectories generated by IE-GAN can flexibly avoid collision with obstacles through an attention mechanism. \(\bullet \) IE-GAN can simulate movement for several different types of pedestrians through the results of clustering, which has an increment on the authenticity of the simulation.

Corresponding author

Correspondence to Yong Zhang.

Ethics declarations

Conflict of interest/Competing interests

The authors declare that there is no conflict of interest/Competing interests regarding the publication of this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, X., Liang, Y., Zhang, Y. et al. IE-GAN: a data-driven crowd simulation method via generative adversarial networks. Multimed Tools Appl 83, 45207–45240 (2024). https://doi.org/10.1007/s11042-023-17346-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-17346-x

Keywords

Search

Navigation