Signal, Image and Video Processing

, Volume 13, Issue 4, pp 643–650 | Cite as

Augmentation of virtual agents in real crowd videos

  • Yalım Doğan
  • Serkan Demirci
  • Uğur GüdükbayEmail author
  • Hamdi Dibeklioğlu
Original Paper


Augmenting virtual agents in real crowd videos is an important task for different applications from simulations of social environments to modeling abnormalities in crowd behavior. We propose a framework for this task, namely for augmenting virtual agents in real crowd videos. We utilize pedestrian detection and tracking algorithms to automatically locate the pedestrians in video frames and project them into our simulated environment, where the navigable area of the simulated environment is available as a navigation mesh. We represent the real pedestrians in the video as simple three-dimensional (3D) models in our simulation environment. 3D models representing real agents and the augmented virtual agents are simulated using local path planning coupled with a collision avoidance algorithm. The virtual agents augmented into the real video move plausibly without colliding with static and dynamic obstacles, including other virtual agents and the real pedestrians.


Data-driven simulation Pedestrian detection Pedestrian tracking Crowd simulation Collision avoidance Augmented reality 


Supplementary material

11760_2018_1392_MOESM1_ESM.pdf (5.4 mb)
Supplementary material 1 (pdf 5533 KB)

Supplementary material 2 (mp4 107983 KB)


  1. 1.
    Lerner, A., Chrysanthou, Y., Lischinski, D.: Crowds by example. Comput. Gr. Forum 26(3), 655–664 (2007)CrossRefGoogle Scholar
  2. 2.
    Charalambous, P., Karamouzas, I., Guy, S.J., Chrysanthou, Y.: A data-driven framework for visual crowd analysis. Comput. Gr. Forum 33(7), 41–50 (2014)CrossRefGoogle Scholar
  3. 3.
    Musse, S.R., Jung, C.R., Jacques, J., Braun, A.: Using computer vision to simulate the motion of virtual agents. Comput. Anim. Virtual Worlds 18(2), 83–93 (2007)CrossRefGoogle Scholar
  4. 4.
    Kim, S., Bera, A., Best, A., Chabra, R., Manocha, D.: Interactive and adaptive data-driven crowd simulation. In: Proceedings of IEEE Virtual Reality, ser. VR’16, pp. 29–38 (2016)Google Scholar
  5. 5.
    Narahara, T., Kobayashi, Y.: Crowd mapper: projection-based interactive pedestrian agents for collective design in architecture. In: Proceedings of the eCAADe 33rd Annual Conference. Education and Research in Computer Aided Architectural Design in Europe (eCAADe), pp. 191–200 (2015)Google Scholar
  6. 6.
    Bulbul, A., Dahyot, R.: Populating virtual cities using social media. Comput. Anim. Virtual Worlds 28(5), e1742 (2017)CrossRefGoogle Scholar
  7. 7.
    Turkay, C., Koc, E., Balcisoy, S.: Integrating information theory in agent-based crowd simulation behavior models. Comput. J. 54(11), 1800–1820 (2011)CrossRefGoogle Scholar
  8. 8.
    Thalmann, N.M., Thalmann, D.: Animating virtual actors in real environments. Multimed. Syst. 5(2), 113–125 (1997)CrossRefGoogle Scholar
  9. 9.
    Zheng, F., Li, H.: ARCrowd-a tangible interface for interactive crowd simulation. In: Proceedings of the 16th International Conference on Intelligent User Interfaces. ACM, pp. 427–430 (2011)Google Scholar
  10. 10.
    Obaid, M., Damian, I., Kistler, F., Endrass, B., Wagner, J., André, E.: Cultural behaviors of virtual agents in an augmented reality environment. In: Proceedings of the International Conference on Intelligent Virtual Agents. Springer, pp. 412–418 (2012)Google Scholar
  11. 11.
    Fernández, C., Baiget, P., Roca, F.X., Gonzàlez, J.: Augmenting video surveillance footage with virtual agents for incremental event evaluation. Pattern Recognit. Lett. 32(6), 878–889 (2011)CrossRefGoogle Scholar
  12. 12.
    Baiget, P., Fernández, C., Roca, X., Gonzàlez, J.: Generation of augmented video sequences combining behavioral animation and multi-object tracking. Comput. Anim. Virtual Worlds 20(4), 473–489 (2009)CrossRefGoogle Scholar
  13. 13.
    Bera, A., Randhavane, T., Prinja, R., Manocha, D.: Sociosense: robot navigation amongst pedestrians with social and psychological constraints. arXiv:1706.01102 (2017)
  14. 14.
    Van den Berg, J., Lin, M., Manocha, D.: Reciprocal velocity obstacles for real-time multi-agent navigation. In: Proceedings of the IEEE International Conference on Robotics and Automation, ser. ICRA’08. IEEE, pp. 1928–1935 (2008)Google Scholar
  15. 15.
    Zhang, G., Qin, X., An, X., Chen, W., Bao, H.: As-consistent-as-possible compositing of virtual objects and video sequences. Comput. Anim. Virtual Worlds 17(3–4), 305–314 (2006)CrossRefGoogle Scholar
  16. 16.
    Unity Technologies, Inc., Unity manual: navigation and pathfinding. [Online].
  17. 17.
    Pelechano, N., Fuentes, C.: Hierarchical path-finding for navigation meshes (HNA*). Comput. Gr. 59, 68–78 (2016)CrossRefGoogle Scholar
  18. 18.
    Olivier, A.-H., Bruneau, J., Kulpa, R., Pettré, J.: Walking with virtual people: evaluation of locomotion interfaces in dynamic environments. IEEE Trans. Vis. Comput. Gr. 24(7), 2251–2263 (2018)CrossRefGoogle Scholar
  19. 19.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, ser. CVPR’05, IEEE, vol. 1, pp. 886–893 (2005)Google Scholar
  20. 20.
    Zivkovic, Z., van der Heijden, F.: Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recognit. Lett. 27(7), 773–780 (2006)CrossRefGoogle Scholar
  21. 21.
    Leal-Taixé, L., Milan, A., Reid, I., Roth, S., Schindler, K.: MOTChallenge 2015: towards a benchmark for multi-target tracking. arXiv:1504.01942 [cs] (2015)
  22. 22.
    Takahashi, K.: Shadowdrawer (2015). [Online]. Available:
  23. 23.
    García-Martín, Á., Sánchez-Matilla, R., Martínez, J.M.: Hierarchical detection of persons in groups. Signal Image Video Process. 11(7), 1181–1188 (2017)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer EngineeringBilkentAnkaraTurkey

Personalised recommendations