International Journal of Speech Technology

, Volume 5, Issue 4, pp 343–354 | Cite as

Multimodal Communication in Inhabited Virtual Environments

  • Anton Nijholt
  • Dirk Heylen


This paper reports on ongoing research of various aspects of multi-modal interactions in a Virtual Reality environment in which visitors can interact in diverse ways with agents and objects in the environment. The virtual environment is a copy of a theatre and is inhabited by both embodied and non-embodied agents that can assist the visitor: providing information about the performances, selling tickets, or helping the visitor navigate through the building.

Interactions in this environment are typically multimodal. Several agents are capable of natural language interactions with the user. However, because the interactions take place in a virtual reality environment, a lot of information can also be presented visually.

natural language 3D visualization multi-modal interaction agents interest communities 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Argyle, M. and Cook, M. (1976). Gaze and Mutual Gaze. Cambridge: Cambridge University Press.Google Scholar
  2. Baus, J., Butz, A., and Kr¨uger, A. (2000). Incorporating a virtual presenter in resource adaptive navigational help system. ProceedingsWorkshop on Guiding Users through Interactive Experiences: Usability Centred Design and Evaluation of Virtual 3D Environments, Paderborn, Germany, April 2000.Google Scholar
  3. Bertolo, M., Maninetti, P., and Marini, D. (1999). Baroque dance animation with virtual dancers. Eurographics '99, Short Papers and Demos, Milan, pp. 117–120.Google Scholar
  4. Cockburn, A. (1997). Using goal-based use cases-Transitioning from theory to practice. J. Object-Oriented Programming, 10(7): 56–62.Google Scholar
  5. Dahlbäck, N., Reithinger, N., and Walker, M.A. (1997). Standards for dialogue coding in natural language processing. Report on a Dagstuhl-Seminar.Google Scholar
  6. Darken, R.P. and Silbert, J.L. (1996). Way finding strategies and behaviors in virtual worlds. Proc. CHI'96, pp. 142–149.Google Scholar
  7. Evers, M. and Nijholt, A. (2000). Jacob-An animated instruction agent for virtual reality. In T. Tan, Y. Shi, and W. Gao (Eds.), Advances in Multimodal Interfaces-ICMI 2000, Proc. Third International Conference on Multimodal Interfaces, Beijing, China, Lecture Notes in Computer Science, Vol. 1948. Berlin: Springer-Verlag, pp. 526–533.Google Scholar
  8. Gibbon, D., Moore, R., and Winski, R. (1997). Handbook of Standards and Resources for Spoken Language Systems. Berlin: Mouton de Gruyter.Google Scholar
  9. Höök, K., et al. (1988). Towards a framework for design and evaluation of navigation in electronic spaces. Persona Deliverable for the EC.Google Scholar
  10. Kragtwijk, M., Nijholt, A., and Zwiers, J. (2001). Implementation of a 3D virtual drummer. In M. Magnenat-Thalmann and D. Thalmann (Eds.), Proceedings CAS2001, EurographicsWorkshop on Animation and Simulation 2001, Manchester, UK. New York: Springer-Verlag, pp. 15–26.Google Scholar
  11. Lie, D., Hulstijn, J., Op den Akker, R., and Nijholt, A. (1998). A transformational approach to NL understanding in dialogue systems. Proc. NLP and Industrial Applications, Moncton, pp. 163–168.Google Scholar
  12. Nijholt, A. and Hulstijn, J. (2000). Multimodal interactions with agents in virtual worlds. In N. Kasabov (Ed.), Future Directions for Intelligent Information Systems and Information Science, Studies in Fuzziness and Soft Computing. Physica-Verlag, pp. 148–173.Google Scholar
  13. Rao, A. and Georgeff, M. (1992). An abstract architecture for rational agents. Proc. of the 3rd International Conference on Principles of Knowledge Representation and Reasoning, pp. 439–449.Google Scholar
  14. Reitmayr, G., Carroll, S., Reitemeyer, A., and Wagner, M.G. (1999). Deep matrix: An open technology based virtual environment system. The Visual Computer Journal, 15:395–412.Google Scholar
  15. Torres, O., Cassell, J., and Prevost, S. (1997). Modeling gaze behavior as a function of discourse structure. First International Workshop on Human Computer Conversations, Bellagio, Italy.Google Scholar
  16. Van Es, I. (2001). Kijkgedrag voor virtual agents in gesprek met de mens. Nuttig? University of Twente, Department of Computer Science, Internal Report.Google Scholar
  17. Van Es, I., Heylen, D., Van Dijk, B., and Nijholt, A. (2002). Gaze behavior of talking agents makes a difference. Proceedings CHI 2002, extended abstracts.Google Scholar
  18. Van Luin, J., Op den Akker, R., and Nijholt, A. (2001). A dialogue agent for navigation support in virtual reality. In J. Jacko and A. Sears (Eds.), Proceedings ACM SIGCHI Conference CHI 2001: Anyone. Anywhere, extended abstracts. Seattle: Association for Computing Machinery, pp. 117–118.Google Scholar
  19. Van Noord, G., Bouma, G., Koeling, R., and Nederhof, M.-J. (1999). Robust grammatical analysis for spoken dialogue systems. Natural Language Engineering, 5(1):45–93.Google Scholar
  20. Vertegaal, R., Slagter, R., Van der Veer, G., and Nijholt, A. (2001). Eye gaze patterns in conversations: There is more to conversational agents than meets the eyes. In J. Jacko, A. Sears, M. Beaudouin-Lafon, and R.J.K. Jacob (Eds.), Proceedings ACM SIGCHI Conference CHI 2001: Anyone. Anywhere. Association for Computing Machinery, pp. 301–308.Google Scholar

Copyright information

© Kluwer Academic Publishers 2002

Authors and Affiliations

  • Anton Nijholt
    • 1
  • Dirk Heylen
    • 1
  1. 1.Centre of Telematics and Information Technology (CTIT)University of TwenteEnschedeThe Netherlands

Personalised recommendations