This paper proposes a novel data-driven modeling framework to construct agent-based crowd model based on real-world video data. The constructed crowd model can generate crowd behaviors that match those observed in the video and can be used to predict trajectories of pedestrians in the same scenario. In the proposed framework, a dual-layer architecture is proposed to model crowd behaviors. The bottom layer models the microscopic collision avoidance behaviors, while the top layer models the macroscopic crowd behaviors such as the goal selection patterns and the path navigation patterns. An automatic learning algorithm is proposed to learn behavior patterns from video data. The learned behavior patterns are then integrated into the dual-layer architecture to generate realistic crowd behaviors. To validate its effectiveness, the proposed framework is applied to two different real world scenarios. The simulation results demonstrate that the proposed framework can generate crowd behaviors similar to those observed in the videos in terms of crowd density distribution. In addition, the proposed framework can also offer promising performance on predicting the trajectories of pedestrians.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
The source code, training data and simulation videos can be downloaded from: http://crowds.sce.ntu.edu.sg/resource.html.
The original video and extracted trajectories are downloaded from http://www.ee.cuhk.edu.hk/~xgwang/grandcentral.html.
The video and extracted trajectories are downloaded from http://www.vision.ee.ethz.ch/datasets/index.en.html.
Ali, S., Nishino, K., Manocha, D., & Shah, M. (2013). Modeling, Simulation and Visual Analysis of Crowds. New York: Springer.
Ali, S., & Shah, M. (2007). A lagrangian particle dynamics approach for crowd flow segmentation and stability analysis. In IEEE Conference on Computer Vision and Pattern Recognition, 2007 (CVPR’07) (pp. 1–6). Piscataway: IEEE.
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In Computer Vision-ECCV 2004 (pp. 25–36). Berlin: Springer.
Finnsson, H., & Björnsson, Y. (2008). Simulation-based approach to general game playing. In Proceedings of the 23rd national conference on Artificial intelligence (Vol. 1, pp. 259–264).
Ge, W., Collins, R., & Ruback, R. (2012). Vision-based analysis of small groups in pedestrian crowds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(5), 1003–1016.
Gratch, J., & Marsella, S. (2004). A domain-independent framework for modeling emotion. Cognitive Systems Research, 5(4), 269–306.
Guy, S.J., Lin, M.C., & Manocha, D. (2010). Modeling collision avoidance behavior for virtual humans. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems(AAMAS’10) (pp. 575–582).
Guy, S. J., Van Den Berg, J., Liu, W., Lau, R., Lin, M. C., & Manocha, D. (2012). A statistical similarity measure for aggregate crowd dynamics. ACM Transactions on Graphics (TOG), 31(6), 190.
Helbing, D., Johansson, A., & Al-Abideen, H. Z. (2007). Dynamics of crowd disasters: An empirical study. Physical Review E, 75(4), 046109.
Helbing, D., & Molnar, P. (1995). Social force model for pedestrian dynamics. Physical Review E, 51(5), 4282.
Jin, X., Xu, J., Wang, C. C., Huang, S., & Zhang, J. (2008). Interactive control of large-crowd navigation in virtual environments using vector fields. IEEE Computer Graphics and Applications, 6, 37–46.
Johansson, A., Helbing, D., & Shukla, P. K. (2007). Specification of the social force pedestrian model by evolutionary adjustment to video tracking data. Advances in Complex Systems, 10(supp02), 271–288.
Ju, E., Choi, M. G., Park, M., Lee, J., Lee, K. H., & Takahashi, S. (2010). Morphable crowds. ACM Transactions on Graphics, 29, 140.
Kaminka, G. A., & Fridman, N. (2006). A cognitive model of crowd behavior based on social comparison theory. In Proceedings of the AAAI-2006 workshop on cognitive modeling.
Lee, K.H., Choi, M.G., Hong, Q., & Lee, J. (2007). Group behavior from video: A data-driven approach to crowd simulation. In Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on Computer animation (pp. 109–118). Eurographics Association.
Lerner, A., Chrysanthou, Y., & Lischinski, D. (2007). Crowds by example. Computer Graphics Forum, 26, 655–664.
Lerner, A., Fitusi, E., Chrysanthou, Y., & Cohen-Or, D. (2009). Fitting behaviors to pedestrian simulations. In Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (pp. 199–208). New York: ACM.
Lin, D., Grimson, E., & Fisher, J. (2009). Learning visual flows: A lie algebraic approach. In IEEE Conference on Computer Vision and Pattern Recognition, 2009 (CVPR’09) (pp. 747–754). Piscataway: IEEE.
Lin, D., Grimson, E., & Fisher, J. (2010). Modeling and estimating persistent motion with geometric flows. In IEEE Conference on Computer Vision and Pattern Recognition, 2010 (CVPR’10 ) (pp. 1–8). Piscataway: IEEE.
Luo, L., Zhou, S., Cai, W., Low, M. Y. H., Tian, F., Wang, Y., et al. (2008). Agent-based human behavior modeling for crowd simulation. Computer Animation and Virtual Worlds, 19(3–4), 271–281.
Mehran, R., Moore, B.E., & Shah, M. (2010). A streakline representation of flow in crowded scenes. In European conference on computer vision, 2010 (ECCV’10) (pp. 439–452). Berlin: Springer.
Musse, S. R., Jung, C. R., Jacques, J., & Braun, A. (2007). Using computer vision to simulate the motion of virtual agents. Computer Animation and Virtual Worlds, 18(2), 83–93.
Musse, S. R., & Thalmann, D. (2001). Hierarchical model for real time simulation of virtual human crowds. IEEE Transactions on Visualization and Computer Graphics, 7(2), 152–164.
Pan, X., Han, C. S., Dauber, K., & Law, K. H. (2007). A multi-agent based framework for the simulation of human and social behaviors during emergency evacuations. AI & Society, 22(2), 113–132.
Pellegrini, S., Ess, A., Schindler, K., & Van Gool, L. (2009). You’ll never walk alone: Modeling social behavior for multi-target tracking. In IEEE International Conference on Computer Vision, 2009 (ICCV’09) (pp. 261–268). Piscataway: IEEE.
Schmidt, M., & Lipson, H. (2009). Distilling free-form natural laws from experimental data. Science, 324(5923), 81–85.
Scovanner, P., & Tappen, M.F. (2009). Learning pedestrian dynamics from the real world. In IEEE International Conference on Computer Vision, 2009 (ICCV’09) (Vol. 9, pp. 381–388).
Shao, W., & Terzopoulos, D. (2005). Autonomous pedestrians. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation (pp. 19–28).
Thalmann, D. (2007). Crowd Simulation. New York: Wiley Online Library.
Tomasi, C., & Kanade, T. (1991). Detection and Tracking of Point Features. Pittsburgh: School of Computer Science, Carnegie Mellon University.
Tsai, J., Fridman, N., Bowring, E., Brown, M., Epstein, S., Kaminka, G., Marsella, S., Ogden, A., Rika, I., Sheel, A., et al. (2011). Escapes: Evacuation simulation with children, authorities, parents, emotions, and social comparison. In Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’11) (pp. 457–464).
Van den Berg, J., Lin, M., & Manocha, D. (2008). Reciprocal velocity obstacles for real-time multi-agent navigation. In IEEE International Conference on Robotics and Automation, 2008 (ICRA’08) (pp. 1928–1935). Piscataway: IEEE.
Welford, B. (1962). Note on a method for calculating corrected sums of squares and products. Technometrics, 4(3), 419–420.
Wolinski, D., Guy, S. J., Olivier, A.-H., Lin, M., Manocha, D., & Pettré, J. (2014). Parameter estimation and comparative evaluation of crowd simulations. Computer Graphics Forum, 33, 303–312.
Yamaguchi, K., Berg, A. C., Ortiz, L. E., & Berg, T. L. (2011). Who are you with and where are you going? In IEEE Conference on Computer Vision and Pattern Recognition, 2011 (CVPR’11) (pp. 1345–1352). Piscataway: IEEE.
Zhao, M., Turner, S. J., & Cai, W. (2013). A data-driven crowd simulation model based on clustering and classification. In 2013 IEEE/ACM 17th International Symposium on Distributed Simulation and Real Time Applications (DS-RT) (pp. 125–134). Piscataway: IEEE.
Zhong, J., & Cai, W. (2015). Differential evolution with sensitivity analysis and the powell’s method for crowd model calibration. Journal of Computational Science, 9, 26–32.
Zhong, J., Cai, W., Luo, L., & Lees, M. (2014). Ea-based evacuation planning using agent-based crowd simulation. In Proceedings of the 2014 Winter Simulation Conference (pp. 395–406). Piscataway: IEEE Press.
Zhong, J., Cai, W., Luo, L., & Yin, H. (2015). Learning behavior patterns from video: A data-driven framework for agent-based crowd modeling. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS’15) (pp. 801–809). International Foundation for Autonomous Agents and Multiagent Systems.
Zhong, J., Hu, N., Cai, W., Lees, M., & Luo, L. (2015). Density-based evolutionary framework for crowd model calibration. Journal of Computational Science, 6, 11–22.
Zhong, J., Luo, L., Cai, W., & Lees, M. (2014). Automatic rule identification for agent-based crowd models through gene expression programming. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems (AAMAS’14) (pp. 1125–1132). International Foundation for Autonomous Agents and Multiagent Systems.
Zhong, J., Ong, Y.-S., & Cai, W. (2016). Self-learning gene expression programming. IEEE Transactions on Evolutionary Computation, 20(1), 65–80.
Zhou, B., Wang, X., & Tang, X. (2012). Understanding collective crowd behaviors: Learning a mixture model of dynamic pedestrian-agents. In IEEE Conference on Computer Vision and Pattern Recognition, 2012 CVPR’12 (pp. 2871–2878). Piscataway: IEEE.
Zhou, S., Chen, D., Cai, W., Luo, L., Low, M. Y. H., Tian, F., et al. (2010). Crowd modeling and simulation technologies. ACM Transactions on Modeling and Computer Simulation, 20(4), 20.
The research reported in this paper is financially supported by the Tier 1 Academic Research Fund (AcRF) under Project Number RG23/14. Linbo Luo is supported by National Natural Science Foundation of China (Grant No. 61502370), China 111 Project (No. B16037) and Fundamental Research Funds for the Central Universities (Grant No. JB150305).
About this article
Cite this article
Zhong, J., Cai, W., Luo, L. et al. Learning behavior patterns from video for agent-based crowd modeling and simulation. Auton Agent Multi-Agent Syst 30, 990–1019 (2016). https://doi.org/10.1007/s10458-016-9334-8
- Agent-based modeling
- Crowd modeling and simulation
- Data-driven modeling
- Behavior pattern