Abstract
This paper reports on ongoing research of various aspects of multi-modal interactions in a Virtual Reality environment in which visitors can interact in diverse ways with agents and objects in the environment. The virtual environment is a copy of a theatre and is inhabited by both embodied and non-embodied agents that can assist the visitor: providing information about the performances, selling tickets, or helping the visitor navigate through the building.
Interactions in this environment are typically multimodal. Several agents are capable of natural language interactions with the user. However, because the interactions take place in a virtual reality environment, a lot of information can also be presented visually.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Argyle, M. and Cook, M. (1976). Gaze and Mutual Gaze. Cambridge: Cambridge University Press.
Baus, J., Butz, A., and Kr¨uger, A. (2000). Incorporating a virtual presenter in resource adaptive navigational help system. ProceedingsWorkshop on Guiding Users through Interactive Experiences: Usability Centred Design and Evaluation of Virtual 3D Environments, Paderborn, Germany, April 2000.
Bertolo, M., Maninetti, P., and Marini, D. (1999). Baroque dance animation with virtual dancers. Eurographics '99, Short Papers and Demos, Milan, pp. 117–120.
Cockburn, A. (1997). Using goal-based use cases-Transitioning from theory to practice. J. Object-Oriented Programming, 10(7): 56–62.
Dahlbäck, N., Reithinger, N., and Walker, M.A. (1997). Standards for dialogue coding in natural language processing. Report on a Dagstuhl-Seminar.
Darken, R.P. and Silbert, J.L. (1996). Way finding strategies and behaviors in virtual worlds. Proc. CHI'96, pp. 142–149.
Evers, M. and Nijholt, A. (2000). Jacob-An animated instruction agent for virtual reality. In T. Tan, Y. Shi, and W. Gao (Eds.), Advances in Multimodal Interfaces-ICMI 2000, Proc. Third International Conference on Multimodal Interfaces, Beijing, China, Lecture Notes in Computer Science, Vol. 1948. Berlin: Springer-Verlag, pp. 526–533.
Gibbon, D., Moore, R., and Winski, R. (1997). Handbook of Standards and Resources for Spoken Language Systems. Berlin: Mouton de Gruyter.
Höök, K., et al. (1988). Towards a framework for design and evaluation of navigation in electronic spaces. Persona Deliverable for the EC.
Kragtwijk, M., Nijholt, A., and Zwiers, J. (2001). Implementation of a 3D virtual drummer. In M. Magnenat-Thalmann and D. Thalmann (Eds.), Proceedings CAS2001, EurographicsWorkshop on Animation and Simulation 2001, Manchester, UK. New York: Springer-Verlag, pp. 15–26.
Lie, D., Hulstijn, J., Op den Akker, R., and Nijholt, A. (1998). A transformational approach to NL understanding in dialogue systems. Proc. NLP and Industrial Applications, Moncton, pp. 163–168.
Nijholt, A. and Hulstijn, J. (2000). Multimodal interactions with agents in virtual worlds. In N. Kasabov (Ed.), Future Directions for Intelligent Information Systems and Information Science, Studies in Fuzziness and Soft Computing. Physica-Verlag, pp. 148–173.
Rao, A. and Georgeff, M. (1992). An abstract architecture for rational agents. Proc. of the 3rd International Conference on Principles of Knowledge Representation and Reasoning, pp. 439–449.
Reitmayr, G., Carroll, S., Reitemeyer, A., and Wagner, M.G. (1999). Deep matrix: An open technology based virtual environment system. The Visual Computer Journal, 15:395–412.
Torres, O., Cassell, J., and Prevost, S. (1997). Modeling gaze behavior as a function of discourse structure. First International Workshop on Human Computer Conversations, Bellagio, Italy.
Van Es, I. (2001). Kijkgedrag voor virtual agents in gesprek met de mens. Nuttig? University of Twente, Department of Computer Science, Internal Report.
Van Es, I., Heylen, D., Van Dijk, B., and Nijholt, A. (2002). Gaze behavior of talking agents makes a difference. Proceedings CHI 2002, extended abstracts.
Van Luin, J., Op den Akker, R., and Nijholt, A. (2001). A dialogue agent for navigation support in virtual reality. In J. Jacko and A. Sears (Eds.), Proceedings ACM SIGCHI Conference CHI 2001: Anyone. Anywhere, extended abstracts. Seattle: Association for Computing Machinery, pp. 117–118.
Van Noord, G., Bouma, G., Koeling, R., and Nederhof, M.-J. (1999). Robust grammatical analysis for spoken dialogue systems. Natural Language Engineering, 5(1):45–93.
Vertegaal, R., Slagter, R., Van der Veer, G., and Nijholt, A. (2001). Eye gaze patterns in conversations: There is more to conversational agents than meets the eyes. In J. Jacko, A. Sears, M. Beaudouin-Lafon, and R.J.K. Jacob (Eds.), Proceedings ACM SIGCHI Conference CHI 2001: Anyone. Anywhere. Association for Computing Machinery, pp. 301–308.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Nijholt, A., Heylen, D. Multimodal Communication in Inhabited Virtual Environments. International Journal of Speech Technology 5, 343–354 (2002). https://doi.org/10.1023/A:1020913109256
Issue Date:
DOI: https://doi.org/10.1023/A:1020913109256