Integration of Multi-modal Cues in Synthetic Attention Processes to Drive Virtual Agent Behavior

  • Sven SeeleEmail author
  • Tobias Haubrich
  • Tim Metzler
  • Jonas Schild
  • Rainer Herpers
  • Marcin Grzegorzek
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10498)


Simulations and serious games require realistic behavior of multiple intelligent agents in real-time. One particular issue is how attention and multi-modal sensory memory can be modeled in a natural but effective way, such that agents controllably react to salient objects or are distracted by other multi-modal cues from their current intention. We propose a conceptual framework that provides a solution with adherence to three main design goals: natural behavior, real-time performance, and controllability. As a proof of concept, we implement three major components and showcase effectiveness in a real-time game engine scenario. Within the exemplified scenario, a visual sensor is combined with static saliency probes and auditory cues. The attention model weighs bottom-up attention against intention-related top-down processing, controllable by a designer using memory and attention inhibitor parameters. We demonstrate our case and discuss future extensions.


Intelligent virtual agents Synthetic perception Virtual attention 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Andrist, S., Pejsa, T., Mutlu, B., Gleicher, M.: Designing effective gaze mechanisms for virtual agents. In: Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, CHI 2012, pp. 705–714. ACM, New York (2012)Google Scholar
  2. 2.
    Balint, T., Allbeck, J.M.: What’s going on? multi-sense attention for virtual agents. In: Aylett, R., Krenn, B., Pelachaud, C., Shimodaira, H. (eds.) IVA 2013. LNCS, vol. 8108, pp. 349–357. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40415-3_31CrossRefGoogle Scholar
  3. 3.
    Blumberg, B.M.: Old Tricks, New Dogs: Ethology and Interactive Creatures. Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, MA, USA (1997)Google Scholar
  4. 4.
    Bordeux, C., Boulic, R., Thalmann, D.: An efficient and flexible perception pipeline for autonomous agents. Computer Graphics Forum 18, 23–30 (1999). Scholar
  5. 5.
    Bruce, N.D.B., Tsotsos, J.K.: Saliency, attention, and visual search: An information theoretic approach. Journal of Vision 9(3), 1–24 (2009)CrossRefGoogle Scholar
  6. 6.
    Brunnhuber, M., Schrom-Feiertag, H., Luksch, C., Matyus, T., Hesina, G.: Bridging the gap between visual exploration and agent-based pedestrian simulation in a virtual environment. In: Proc. of the 18th ACM Symposium on Virtual Reality Software and Technology, VRST 2012, NY, USA, pp. 9–16 (2012).
  7. 7.
    Conde, T., Thalmann, D.: An integrated perception for autonomous virtual agents: Active and predictive perception. Computer Animation and Virtual Worlds 17, 457–468 (2006). Scholar
  8. 8.
    Frintrop, S.: Computational Visual Attention. Sprinter (2011)Google Scholar
  9. 9.
    Haubrich, T., Seele, S., Herpers, R., Bauckhage, C., Becker, P.: Synthetic perception for intelligent virtual agents. In: Proc. of the 1st ACM SIGCHI Annual Symposium on Computer-Human Interaction in Play (CHI PLAY) (2014)Google Scholar
  10. 10.
    Haubrich, T., Seele, S., Herpers, R., Bauckhage, C., Becker, P.: Modeling sensation for an intelligent virtual agent’s perception process. In: Brinkman, W.-P., Broekens, J., Heylen, D. (eds.) IVA 2015. LNCS, vol. 9238, pp. 87–97. Springer, Cham (2015). doi: 10.1007/978-3-319-21996-7_9CrossRefGoogle Scholar
  11. 11.
    Herpers, R., Kattner, H., Rodax, H., Sommer, G.: Gaze: An attentive processing strategy to detect and analyze the prominent facial regions. In: Workshop on Automatic Face and Gesture Recognition, Zurich, Switzerland (1995)Google Scholar
  12. 12.
    Itti, L., Dhavale, N., Pighin, F.: Realistic avatar eye and head animation using a neurobiological model of visual attention. In: Bosacchi, B., Fogel, D.B., Bezdek, J.C. (eds.) Proc. SPIE 48th Annual Int. Symposium on Optical Science and Technology, vol. 5200, pp. 64–78. SPIE Press, Bellingham (2003)Google Scholar
  13. 13.
    Kim, Y., van Velsen, M., Hill Jr., R.W.: Modeling dynamic perceptual attention in complex virtual environments. In: Panayiotopoulos, T., Gratch, J., Aylett, R., Ballin, D., Olivier, P., Rist, T. (eds.) IVA 2005. LNCS, vol. 3661, pp. 266–277. Springer, Heidelberg (2005). doi: 10.1007/11550617_23CrossRefGoogle Scholar
  14. 14.
    Krueger, F., Seele, S., Herpers, R., Becker, P., Bauckhage, C.: Adaptive decision making in microsimulations of urban traffic in virtual environments. In: Pisan, Y., Sgouros, N.M., Marsh, T. (eds.) Entertainment Computing - ICEC 2014. LNCS, vol. 8770, pp. 220–222. Springer, Heidelberg (2014). doi: 10.1007/978-3-662-45212-7_28CrossRefGoogle Scholar
  15. 15.
    Kuiper, D.M., Wenkstern, R.Z.: Virtual agent perception combination in multi agent based systems. In: Proc. of the 2013 Int. Conf. on Autonomous Agents and Multi-agent Systems (2013).
  16. 16.
    Kuiper, D.M., Wenkstern, R.Z.: Agent vision in multi-agent based simulation systems. Autonomous Agents and Multi-Agent Systems 29(2), 161–191 (2015)CrossRefGoogle Scholar
  17. 17.
    Leonard, T.: Building an AI sensory system (2003). Accessed 2017–07-16.
  18. 18.
    Luck, M., Aylett, R.: Applying artificial intelligence to virtual reality: Intelligent virtual environments. Applied Artificial Intelligence 14, 3–32 (2000)CrossRefGoogle Scholar
  19. 19.
    Noser, H., Renault, O., Thalmann, D., Magnenat-Thalmann, N.: Navigation for digital actors based on synthetic vision, memory, and learning. Computers & Graphics 19, 7–19 (1995)CrossRefGoogle Scholar
  20. 20.
    van Oijen, J., Dignum, F.: Scalable perception for bdi-agents embodied in virtual environments. In: Proc. of the 2011 IEEE/WIC/ACM Int. Conf. on Web Intelligence and Intelligent Agent Technology, vol. 02 (2011)Google Scholar
  21. 21.
    van Oijen, J., Vanhée, L., Dignum, F.: CIGA: a middleware for intelligent agents in virtual environments. In: Beer, M., Brom, C., Dignum, F., Soo, V.-W. (eds.) AEGS 2011. LNCS, vol. 7471, pp. 22–37. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-32326-3_2CrossRefGoogle Scholar
  22. 22.
    Ondřej, J., Pettré, J., Olivier, A.H., Donikian, S.: A synthetic-vision based steering approach for crowd simulation. ACM TOG 29(4), 123:1–123:9 (2010).
  23. 23.
    Pejsa, T., Andrist, S., Gleicher, M., Mutlu, B.: Gaze and attention management for embodied conversational agents. ACM Trans. Interact. Intell. Syst. 5(1), 3:1–3:34 (2015)CrossRefGoogle Scholar
  24. 24.
    Peters, C., O’Sullivan, C.: Synthetic vision and memory for autonomous virtual humans. Computer Graphics Forum 21, 743–752 (2002)CrossRefGoogle Scholar
  25. 25.
    Peters, C., Castellano, G., Rehm, M., André, E., Raouzaiou, A., Rapantzikos, K., Karpouzis, K., Volpe, G., Camurri, A., Vasalou, A.: Fundamentals of agent perception and attention modelling, pp. 293–319. Springer, Heidelberg (2011)Google Scholar
  26. 26.
    Reynolds, C.W.: Flocks, herds and schools: A distributed behavioral model. SIGGRAPH Computer Graphics 21(4), 25–34 (1987)CrossRefGoogle Scholar
  27. 27.
    Seele, S., Herpers, R., Bauckhage, C.: Cognitive agents for microscopic traffic simulations in virtual environments. In: Entertainment Computing, ICEC 2012Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Sven Seele
    • 1
    Email author
  • Tobias Haubrich
    • 1
  • Tim Metzler
    • 1
  • Jonas Schild
    • 1
    • 2
  • Rainer Herpers
    • 1
    • 3
    • 4
  • Marcin Grzegorzek
    • 5
  1. 1.Institute of Visual ComputingBonn-Rhein-Sieg University of Applied SciencesSankt AugustinGermany
  2. 2.Hochschule Hannover – University of Applied Sciences and ArtsHannoverGermany
  3. 3.University of New BrunswickNew BrunswickCanada
  4. 4.York UniversityTorontoCanada
  5. 5.Research Group for Pattern RecognitionUniversity of SiegenSiegenGermany

Personalised recommendations