Capturing Human Movements for Simulation Environment

  • Chengxin WangEmail author
  • Muhammad Shalihin Bin OthmanEmail author
  • Gary TanEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1094)


In this paper, we proposed a novel data-driven framework to translate human movements from real-life video feeds into a virtual simulator in Unity 3D. In the proposed framework, YOLOv3 is used for pedestrian detection. Thereafter, a modified offline tracking algorithm with the min-cost flow was built to associate detected pedestrians from frame to frame. Finally, 2D trajectories are produced where a script would translate them into the Unity 3D platform. The proposed framework has the ability to display realistic behavior patterns where we would be able to introduce threats and analyze different strategies for improving evacuation and rescue in disaster situations.


Computer vision Multiple object tracking Simulation 


  1. 1.
    Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)Google Scholar
  2. 2.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)Google Scholar
  3. 3.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)Google Scholar
  4. 4.
    Liu, W., et al.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). Scholar
  5. 5.
    Redmon, J., Farhadi, A.: Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018)
  6. 6.
    He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. CoRR, vol. abs/1703.06870 (2017).
  7. 7.
    Fang, K., Xiang, Y., Li, X., Savarese, S.: Recurrent autoregressive networks for online multi-object tracking. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 466–475. IEEE (2018)Google Scholar
  8. 8.
    Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online and realtime tracking. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 3464–3468. IEEE (2016)Google Scholar
  9. 9.
    Kalman, R.E.: A new approach to linear filtering and prediction problems. J. Basic Eng. 82(1), 35–45 (1960)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Kuhn, H.W.: The hungarian method for the assignment problem. Naval Res. Logistics Q. 2(1–2), 83–97 (1955)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Yu, F., Li, W., Li, Q., Liu, Y., Shi, X., Yan, J.: POI: multiple object tracking with high performance detection and appearance feature. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9914, pp. 36–42. Springer, Cham (2016). Scholar
  12. 12.
    Wen, L., Li, W., Yan, J., Lei, Z., Yi, D., Li, S.Z.: Multiple target tracking based on undirected hierarchical relation hypergraph. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1282–1289 (2014)Google Scholar
  13. 13.
    Zhang, L., Li, Y., Nevatia, R.: Global data association for multi-object tracking using network flows. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)Google Scholar
  14. 14.
    Luke, S., Cioffi-Revilla, C., Panait, L., Sullivan, K., Balan, G.: Mason: a multiagent simulation environment. Simulation 81(7), 517–527 (2005)CrossRefGoogle Scholar
  15. 15.
    Cordasco, G., De Chiara, R., Mancuso, A., Mazzeo, D., Scarano, V., Spagnuolo, C.: A framework for distributing agent-based simulations. In: Alexander, M., et al. (eds.) Euro-Par 2011. LNCS, vol. 7155, pp. 460–470. Springer, Heidelberg (2012). Scholar
  16. 16.
    Sklar, E.: Netlogo, a multi-agent simulation environment (2007)CrossRefGoogle Scholar
  17. 17.
    Unity. The world’s leading real-time creation platform (2019).
  18. 18.
    Zhong, J., Cai, W., Luo, L., Zhao, M.: Learning behavior patterns from video for agent-based crowd modeling and simulation. Auton. Agent. Multi-Agent Syst. 30(5), 990–1019 (2016)CrossRefGoogle Scholar
  19. 19.
    van den Berg, J., Guy, S.J., Snape, J., Lin, M.C., Manocha, D.: Rvo2 library: Reciprocal collision avoidance for real-time multi-agent simulation (2011)Google Scholar
  20. 20.
    Luo, L., Chai, C., Ma, J., Zhou, S., Cai, W.: Proactivecrowd: modelling proactive steering behaviours for agent-based crowd simulation. In: Computer Graphics Forum, vol. 37, no. 1, pp. 375–388. Wiley Online Library (2018)Google Scholar
  21. 21.
    Luo, L., et al.: Agent-based human behavior modeling for crowd simulation. Comput. Animation Virtual Worlds 19(3–4), 271–281 (2008)CrossRefGoogle Scholar
  22. 22.
    Snape, J., Guy, S.J., Vembar, D., Lake, A., Lin, M.C., Manocha, D.: Reciprocal collision avoidance and navigation for video games. In: Game Developers Conference, San Francisco (2012)Google Scholar
  23. 23.
  24. 24.
    Benfold, B., Reid, I.: Stable multi-target tracking in real-time surveillance video. In: CVPR, pp. 3457–3464, June 2011Google Scholar
  25. 25.
    Moussaïd, M., Perozo, N., Garnier, S., Helbing, D., Theraulaz, G.: The walking behaviour of pedestrian social groups and its impact on crowd dynamics. PLoS ONE 5(4), e10047 (2010)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.School of ComputingNational University of SingaporeSingaporeSingapore

Personalised recommendations