Skip to main content

Advertisement

SpringerLink
Log in

Environment-sensitive crowd behavior modeling method based on reinforcement learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Most existing crowd evacuation methods focus on internal factors and do not consider the influence of the external environment factors, producing unrealistic global behavior occurs when individuals are moving through the crowded space. As an essential part of a building, safety indication signs (SISs) are a form of environmental information and perceptual access that play an important role in promoting wayfinding by virtue of their guiding role in movement direction and route selection by providing guidance, warning and mandatory message to people. In this paper, we propose an innovative crowd simulation method guided by SISs with reinforcement learning strategy for use in emergencies. To illustratethe guiding function of the SISs, we establish a guidance field for each SIS and add the attractive force to the guidance field. Besides, we formulate the multi-agent (crowd) navigation problem as an action-selection problem and design a novel reinforcement learning strategy for driving individuals to accomplish collision-free movement more efficiently. Particularly, we define a state transition strategy between the SIS area and the non-SIS area to achieve continuity of guidance. We use extensive simulations to highlight the potential of our method in different scenarios and evaluate the results in terms of evacuation efficiency and the reasonableness of SIS placement. In practice, our system runs at interactive rates and can solve complex planning problems involving one or more groups.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Sutton RS, Barto AG (2018) Reinforcement learning: An introduction. MIT press

  2. Zhou M, Dong H, Ioannou PA, Zhao Y, Wang F-Y (2019) Guided crowd evacuation: approaches and challenges. IEEE/CAA J Autom Sinica 6(5):1081–1094

    Article  Google Scholar 

  3. Liu J, Chen Y, Chen Y (2021) Emergency and disaster manage- ment-crowd evacuation research. J Ind Inf Integr 21:100191

    Google Scholar 

  4. Sharbini H, Sallehuddin R, Haron H (2021) Crowd evacuation simulation model with soft computing optimization techniques: a systematic literature review. J Manage Analy 8(3):443–485

    Google Scholar 

  5. Burstedde C, Klauck K, Schadschneider A, Zittartz J (2001) Simulation of pedestrian dynamics using a two-dimensional cellular automaton. Physica A 295(3-4):507–525

    Article  MATH  Google Scholar 

  6. Li Y, Chen M, Dou Z, Zheng X, Cheng Y, Mebarki A (2019) A review of cellular automata models for crowd evacuation. Physica A 526:120752

    Article  Google Scholar 

  7. Thalmann D, Musse SR (2012) Crowd simulation. Springer Science & Business Media

  8. Pelechano N, Allbeck JM, Badler NI (2008) Virtual crowds: methods, simulation, and control. Synth Lect Comput Graph Animat 3(1):1–176

    Google Scholar 

  9. Henderson L (1971) The statistics of crowd fluids. Nature 229(5284):381–383

    Article  Google Scholar 

  10. Henderson LF (1974) On the fluid mechanics of human crowd motion. Transp Res 8(6):509–515

    Article  Google Scholar 

  11. Kerr W, Spears D (2005) Robotic simulation of gases for a surveillance task. In: 2005 IEEE/RSJ international conference on intelligent robots and systems. IEEE, pp 2905–2910

  12. Reynolds CW (1987) Flocks, herds and schools: A distributed behavioral model. In: Proceedings of the 14th annual conference on Computer graphics and interactive techniques, pp 25–34

  13. Reynolds CW et al (1999) Steering behaviors for autonomous characters. In: Game developers conference, vol 1999. Citeseer, pp 763–782

  14. Kari J (2005) Theory of cellular automata: a survey. Theoretical Comput Sci 334(1-3):3–33

    Article  MathSciNet  MATH  Google Scholar 

  15. Padovani D, Neto JJ, Cereda PRM (2018) Modeling pedestrian dynamics with adaptive cellular automata. Procedia Comput Sci 130:1120–1127

    Article  Google Scholar 

  16. Zhou X, Hu J, Ji X, Xiao X (2019) Cellular automaton simulation of pedestrian flow considering vision and multi-velocity. Physica A 514:982–992

    Article  Google Scholar 

  17. Felcman J, Kubera P (2021) A cellular automaton model for a pedestrian flow problem. Math Model Nat Phenom 16:11

    Article  MathSciNet  MATH  Google Scholar 

  18. Van den Berg J, Lin M, Manocha D (2008) Reciprocal velocity obstacles for real-time multi-agent navigation. In: 2008 IEEE International Conference on Robotics and Automation. IEEE, pp 1928–1935

  19. Muhammad F, Juniastuti S, Nugroho SMS, Hariadi M (2018) Crowds evacuation simulation on heterogeneous agent using agent-based reciprocal velocity obstacle. In: 2018 international seminar on intelligent technology and its applications (ISITIA). IEEE, pp 275–280

  20. Douthwaite JA, Zhao S, Mihaylova LS (2019) Velocity obstacle approaches for multi-agent collision avoidance. Unmanned Syst 7(01):55–64

    Article  Google Scholar 

  21. Li J, Zhang H (2021) Crowd evacuation simulation research based on improved reciprocal velocity obstacles (rvo) model with path planning and emotion contagion. Transportation Research Record, 03611981211056910

  22. Helbing D, Molnar P (1995) Social force model for pedestrian dynamics. Phys Rev E 51 (5):4282

    Article  Google Scholar 

  23. Liu B, Liu H, Zhang H, Qin X (2018) A social force evacuation model driven by video data. Simul Model Pract Theory 84:190–203

    Article  Google Scholar 

  24. Li X, Liang Y, Zhao M, Wang C, Bai H, Jiang Y (2019) Simulation of evacuating crowd based on deep learning and social force model. IEEE Access 7:155361–155371

    Article  Google Scholar 

  25. Li Q, Liu Y, Kang Z, Li K, Chen L (2020) Improved social force model considering conflict avoidance. Chaos: An Interdisciplinary Journal of Nonlinear Science 30(1):013129

    Article  MathSciNet  MATH  Google Scholar 

  26. Zhao Y, Liu H, Gao K (2021) An evacuation simulation method based on an improved artificial bee colony algorithm and a social force model. Appl Intell 51(1):100–123

    Article  Google Scholar 

  27. Liu Y, Lyu L (2019) Diversified crowd evacuation method in large public places. IEEE Access 7:144874–144884

    Article  Google Scholar 

  28. Nilsson D, Thompson P, McGrath D, Boyce K, Frantzich H (2020) Crowd safety: prototyping for the future: summary report showing how the science for “pedestrian flow” can keep up with demographic change

  29. Mitchell TM, Mitchell TM (1997) Machine learning, vol 1. McGraw-hill New York

  30. Adamatzky A (2018) Unconventional computing: A Volume in the Encyclopedia of Complexity and Systems Science. Springer

  31. Longley P, Batty M (2003) Advanced spatial analysis: the CASA book of GIS, ESRI Inc.

  32. Yao Z, Zhang G, Lu D, Liu H (2019) Data-driven crowd evacuation: a reinforcement learning method. Neurocomputing 366:314–327

    Article  Google Scholar 

  33. Zhang Z, Lu D, Li J, Liu P, Zhang G (2021) Crowd evacuation simulation using hierarchical deep reinforcement learning. In: 2021 IEEE 24th international conference on computer supported cooperative work in design (CSCWD). IEEE, pp 563–568

  34. Malebary SJ, Basori AH et al (2021) Reinforcement learning for pedestrian evacuation simulation and optimization during pandemic and panic situation. In: Journal of Physics: Conference Series, vol 1817. IOP Publishing, p 012008

  35. Xue Y, Wu R, Liu J, Tang X (2021) Crowd evacuation guidance based on combined action reinforcement learning. Algorithms 14(1):26

    Article  MathSciNet  Google Scholar 

  36. Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3 (1):9–44

    Article  Google Scholar 

  37. Audibert J-Y, Munos R, Szepesvári C (2009) Exploration–exploitation tradeoff using variance estimates in multi-armed bandits. Theor Comput Sci 410(19):1876–1902

    Article  MathSciNet  MATH  Google Scholar 

  38. Macready WG, Wolpert DH (1998) Bandit problems and the exploration/exploitation tradeoff. IEEE Trans Evol Comput 2(1):2–22

    Article  Google Scholar 

  39. Koppell J (2011) International organization for standardization. Handb Transnatl Gov Inst Innov 41:289

    Google Scholar 

  40. Chen N, Zhao M, Gao K, Zhao J (2020) The physiological experimental study on the effect of different color of safety signs on a virtual subway fire escape—an exploratory case study of zijing mountain subway station. Int J Environ Res Public Health 17(16):5903

    Article  Google Scholar 

  41. Zhu Y, Chen T, Ding N, Chraibi M, Fan W-C (2020) Follow the evacuation signs or surrounding people during building evacuation, an experimental study. Physica A 560:125156

    Article  Google Scholar 

  42. Kavraki LE, Svestka P, Latombe J-C, Overmars MH (1996) Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Trans Robot Autom 12(4):566–580

    Article  Google Scholar 

  43. LaValle SM et al (1998) Rapidly-exploring random trees: A new tool for path planning

  44. Van Den Berg J, Guy SJ, Lin M, Manocha D (2011) Reciprocal n-body collision avoidance. In: Robotics research. Springer, pp 3–19

  45. Lo S, Fang Z, Lin P, Zhi G (2004) An evacuation model: the sgem package. Fire Safety J 39(3):169–190

    Article  Google Scholar 

  46. Wang Q, Liu H, Gao K, Zhang L (2019) Improved multi-agent reinforcement learning for path planning-based crowd simulation. IEEE Access 7:73841–73855

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 61976127.

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Shandong Normal University, Jinan, 250358, China

    Chen Pang, Lei Lyu, Qinglin Zhou & Limei Zhou

  2. Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, 250358, China

    Lei Lyu

Authors
  1. Chen Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Lyu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qinglin Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Limei Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei Lyu.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pang, C., Lyu, L., Zhou, Q. et al. Environment-sensitive crowd behavior modeling method based on reinforcement learning. Appl Intell 53, 19356–19371 (2023). https://doi.org/10.1007/s10489-023-04509-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04509-4

Keywords

Search

Navigation