Autonomous Agents and Multi-Agent Systems

, Volume 30, Issue 5, pp 990–1019 | Cite as

Learning behavior patterns from video for agent-based crowd modeling and simulation

  • Jinghui ZhongEmail author
  • Wentong Cai
  • Linbo Luo
  • Mingbi Zhao


This paper proposes a novel data-driven modeling framework to construct agent-based crowd model based on real-world video data. The constructed crowd model can generate crowd behaviors that match those observed in the video and can be used to predict trajectories of pedestrians in the same scenario. In the proposed framework, a dual-layer architecture is proposed to model crowd behaviors. The bottom layer models the microscopic collision avoidance behaviors, while the top layer models the macroscopic crowd behaviors such as the goal selection patterns and the path navigation patterns. An automatic learning algorithm is proposed to learn behavior patterns from video data. The learned behavior patterns are then integrated into the dual-layer architecture to generate realistic crowd behaviors. To validate its effectiveness, the proposed framework is applied to two different real world scenarios. The simulation results demonstrate that the proposed framework can generate crowd behaviors similar to those observed in the videos in terms of crowd density distribution. In addition, the proposed framework can also offer promising performance on predicting the trajectories of pedestrians.


Agent-based modeling Crowd modeling and simulation Data-driven modeling Behavior pattern 



The research reported in this paper is financially supported by the Tier 1 Academic Research Fund (AcRF) under Project Number RG23/14. Linbo Luo is supported by National Natural Science Foundation of China (Grant No. 61502370), China 111 Project (No. B16037) and Fundamental Research Funds for the Central Universities (Grant No. JB150305).


  1. 1.
    Ali, S., Nishino, K., Manocha, D., & Shah, M. (2013). Modeling, Simulation and Visual Analysis of Crowds. New York: Springer.CrossRefGoogle Scholar
  2. 2.
    Ali, S., & Shah, M. (2007). A lagrangian particle dynamics approach for crowd flow segmentation and stability analysis. In IEEE Conference on Computer Vision and Pattern Recognition, 2007 (CVPR’07) (pp. 1–6). Piscataway: IEEE.Google Scholar
  3. 3.
    Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In Computer Vision-ECCV 2004 (pp. 25–36). Berlin: Springer.Google Scholar
  4. 4.
    Finnsson, H., & Björnsson, Y. (2008). Simulation-based approach to general game playing. In Proceedings of the 23rd national conference on Artificial intelligence (Vol. 1, pp. 259–264).Google Scholar
  5. 5.
    Ge, W., Collins, R., & Ruback, R. (2012). Vision-based analysis of small groups in pedestrian crowds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(5), 1003–1016.CrossRefGoogle Scholar
  6. 6.
    Gratch, J., & Marsella, S. (2004). A domain-independent framework for modeling emotion. Cognitive Systems Research, 5(4), 269–306.CrossRefGoogle Scholar
  7. 7.
    Guy, S.J., Lin, M.C., & Manocha, D. (2010). Modeling collision avoidance behavior for virtual humans. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems(AAMAS’10) (pp. 575–582).Google Scholar
  8. 8.
    Guy, S. J., Van Den Berg, J., Liu, W., Lau, R., Lin, M. C., & Manocha, D. (2012). A statistical similarity measure for aggregate crowd dynamics. ACM Transactions on Graphics (TOG), 31(6), 190.CrossRefGoogle Scholar
  9. 9.
    Helbing, D., Johansson, A., & Al-Abideen, H. Z. (2007). Dynamics of crowd disasters: An empirical study. Physical Review E, 75(4), 046109.CrossRefGoogle Scholar
  10. 10.
    Helbing, D., & Molnar, P. (1995). Social force model for pedestrian dynamics. Physical Review E, 51(5), 4282.CrossRefGoogle Scholar
  11. 11.
    Jin, X., Xu, J., Wang, C. C., Huang, S., & Zhang, J. (2008). Interactive control of large-crowd navigation in virtual environments using vector fields. IEEE Computer Graphics and Applications, 6, 37–46.Google Scholar
  12. 12.
    Johansson, A., Helbing, D., & Shukla, P. K. (2007). Specification of the social force pedestrian model by evolutionary adjustment to video tracking data. Advances in Complex Systems, 10(supp02), 271–288.MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Ju, E., Choi, M. G., Park, M., Lee, J., Lee, K. H., & Takahashi, S. (2010). Morphable crowds. ACM Transactions on Graphics, 29, 140.CrossRefGoogle Scholar
  14. 14.
    Kaminka, G. A., & Fridman, N. (2006). A cognitive model of crowd behavior based on social comparison theory. In Proceedings of the AAAI-2006 workshop on cognitive modeling.Google Scholar
  15. 15.
    Lee, K.H., Choi, M.G., Hong, Q., & Lee, J. (2007). Group behavior from video: A data-driven approach to crowd simulation. In Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on Computer animation (pp. 109–118). Eurographics Association.Google Scholar
  16. 16.
    Lerner, A., Chrysanthou, Y., & Lischinski, D. (2007). Crowds by example. Computer Graphics Forum, 26, 655–664.CrossRefGoogle Scholar
  17. 17.
    Lerner, A., Fitusi, E., Chrysanthou, Y., & Cohen-Or, D. (2009). Fitting behaviors to pedestrian simulations. In Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (pp. 199–208). New York: ACM.Google Scholar
  18. 18.
    Lin, D., Grimson, E., & Fisher, J. (2009). Learning visual flows: A lie algebraic approach. In IEEE Conference on Computer Vision and Pattern Recognition, 2009 (CVPR’09) (pp. 747–754). Piscataway: IEEE.Google Scholar
  19. 19.
    Lin, D., Grimson, E., & Fisher, J. (2010). Modeling and estimating persistent motion with geometric flows. In IEEE Conference on Computer Vision and Pattern Recognition, 2010 (CVPR’10 ) (pp. 1–8). Piscataway: IEEE.Google Scholar
  20. 20.
    Luo, L., Zhou, S., Cai, W., Low, M. Y. H., Tian, F., Wang, Y., et al. (2008). Agent-based human behavior modeling for crowd simulation. Computer Animation and Virtual Worlds, 19(3–4), 271–281.CrossRefGoogle Scholar
  21. 21.
    Mehran, R., Moore, B.E., & Shah, M. (2010). A streakline representation of flow in crowded scenes. In European conference on computer vision, 2010 (ECCV’10) (pp. 439–452). Berlin: Springer.Google Scholar
  22. 22.
    Musse, S. R., Jung, C. R., Jacques, J., & Braun, A. (2007). Using computer vision to simulate the motion of virtual agents. Computer Animation and Virtual Worlds, 18(2), 83–93.CrossRefGoogle Scholar
  23. 23.
    Musse, S. R., & Thalmann, D. (2001). Hierarchical model for real time simulation of virtual human crowds. IEEE Transactions on Visualization and Computer Graphics, 7(2), 152–164.CrossRefGoogle Scholar
  24. 24.
    Pan, X., Han, C. S., Dauber, K., & Law, K. H. (2007). A multi-agent based framework for the simulation of human and social behaviors during emergency evacuations. AI & Society, 22(2), 113–132.CrossRefGoogle Scholar
  25. 25.
    Pellegrini, S., Ess, A., Schindler, K., & Van Gool, L. (2009). You’ll never walk alone: Modeling social behavior for multi-target tracking. In IEEE International Conference on Computer Vision, 2009 (ICCV’09) (pp. 261–268). Piscataway: IEEE.Google Scholar
  26. 26.
    Schmidt, M., & Lipson, H. (2009). Distilling free-form natural laws from experimental data. Science, 324(5923), 81–85.CrossRefGoogle Scholar
  27. 27.
    Scovanner, P., & Tappen, M.F. (2009). Learning pedestrian dynamics from the real world. In IEEE International Conference on Computer Vision, 2009 (ICCV’09) (Vol. 9, pp. 381–388).Google Scholar
  28. 28.
    Shao, W., & Terzopoulos, D. (2005). Autonomous pedestrians. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation (pp. 19–28).Google Scholar
  29. 29.
    Thalmann, D. (2007). Crowd Simulation. New York: Wiley Online Library.CrossRefGoogle Scholar
  30. 30.
    Tomasi, C., & Kanade, T. (1991). Detection and Tracking of Point Features. Pittsburgh: School of Computer Science, Carnegie Mellon University.Google Scholar
  31. 31.
    Tsai, J., Fridman, N., Bowring, E., Brown, M., Epstein, S., Kaminka, G., Marsella, S., Ogden, A., Rika, I., Sheel, A., et al. (2011). Escapes: Evacuation simulation with children, authorities, parents, emotions, and social comparison. In Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’11) (pp. 457–464).Google Scholar
  32. 32.
    Van den Berg, J., Lin, M., & Manocha, D. (2008). Reciprocal velocity obstacles for real-time multi-agent navigation. In IEEE International Conference on Robotics and Automation, 2008 (ICRA’08) (pp. 1928–1935). Piscataway: IEEE.Google Scholar
  33. 33.
    Welford, B. (1962). Note on a method for calculating corrected sums of squares and products. Technometrics, 4(3), 419–420.MathSciNetCrossRefGoogle Scholar
  34. 34.
    Wolinski, D., Guy, S. J., Olivier, A.-H., Lin, M., Manocha, D., & Pettré, J. (2014). Parameter estimation and comparative evaluation of crowd simulations. Computer Graphics Forum, 33, 303–312.CrossRefGoogle Scholar
  35. 35.
    Yamaguchi, K., Berg, A. C., Ortiz, L. E., & Berg, T. L. (2011). Who are you with and where are you going? In IEEE Conference on Computer Vision and Pattern Recognition, 2011 (CVPR’11) (pp. 1345–1352). Piscataway: IEEE.Google Scholar
  36. 36.
    Zhao, M., Turner, S. J., & Cai, W. (2013). A data-driven crowd simulation model based on clustering and classification. In 2013 IEEE/ACM 17th International Symposium on Distributed Simulation and Real Time Applications (DS-RT) (pp. 125–134). Piscataway: IEEE.Google Scholar
  37. 37.
    Zhong, J., & Cai, W. (2015). Differential evolution with sensitivity analysis and the powell’s method for crowd model calibration. Journal of Computational Science, 9, 26–32.CrossRefGoogle Scholar
  38. 38.
    Zhong, J., Cai, W., Luo, L., & Lees, M. (2014). Ea-based evacuation planning using agent-based crowd simulation. In Proceedings of the 2014 Winter Simulation Conference (pp. 395–406). Piscataway: IEEE Press.Google Scholar
  39. 39.
    Zhong, J., Cai, W., Luo, L., & Yin, H. (2015). Learning behavior patterns from video: A data-driven framework for agent-based crowd modeling. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS’15) (pp. 801–809). International Foundation for Autonomous Agents and Multiagent Systems.Google Scholar
  40. 40.
    Zhong, J., Hu, N., Cai, W., Lees, M., & Luo, L. (2015). Density-based evolutionary framework for crowd model calibration. Journal of Computational Science, 6, 11–22.CrossRefGoogle Scholar
  41. 41.
    Zhong, J., Luo, L., Cai, W., & Lees, M. (2014). Automatic rule identification for agent-based crowd models through gene expression programming. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems (AAMAS’14) (pp. 1125–1132). International Foundation for Autonomous Agents and Multiagent Systems.Google Scholar
  42. 42.
    Zhong, J., Ong, Y.-S., & Cai, W. (2016). Self-learning gene expression programming. IEEE Transactions on Evolutionary Computation, 20(1), 65–80.CrossRefGoogle Scholar
  43. 43.
    Zhou, B., Wang, X., & Tang, X. (2012). Understanding collective crowd behaviors: Learning a mixture model of dynamic pedestrian-agents. In IEEE Conference on Computer Vision and Pattern Recognition, 2012 CVPR’12 (pp. 2871–2878). Piscataway: IEEE.Google Scholar
  44. 44.
    Zhou, S., Chen, D., Cai, W., Luo, L., Low, M. Y. H., Tian, F., et al. (2010). Crowd modeling and simulation technologies. ACM Transactions on Modeling and Computer Simulation, 20(4), 20.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2016

Authors and Affiliations

  1. 1.School of Computer EngineeringNanyang Technological UniversitySingaporeSingapore
  2. 2.School of Cyber EngineeringXidian UniversityXi’anChina

Personalised recommendations