Multimodal Communication in Inhabited Virtual Environments
This paper reports on ongoing research of various aspects of multi-modal interactions in a Virtual Reality environment in which visitors can interact in diverse ways with agents and objects in the environment. The virtual environment is a copy of a theatre and is inhabited by both embodied and non-embodied agents that can assist the visitor: providing information about the performances, selling tickets, or helping the visitor navigate through the building.
Interactions in this environment are typically multimodal. Several agents are capable of natural language interactions with the user. However, because the interactions take place in a virtual reality environment, a lot of information can also be presented visually.
Unable to display preview. Download preview PDF.
- Argyle, M. and Cook, M. (1976). Gaze and Mutual Gaze. Cambridge: Cambridge University Press.Google Scholar
- Baus, J., Butz, A., and Kr¨uger, A. (2000). Incorporating a virtual presenter in resource adaptive navigational help system. ProceedingsWorkshop on Guiding Users through Interactive Experiences: Usability Centred Design and Evaluation of Virtual 3D Environments, Paderborn, Germany, April 2000.Google Scholar
- Bertolo, M., Maninetti, P., and Marini, D. (1999). Baroque dance animation with virtual dancers. Eurographics '99, Short Papers and Demos, Milan, pp. 117–120.Google Scholar
- Cockburn, A. (1997). Using goal-based use cases-Transitioning from theory to practice. J. Object-Oriented Programming, 10(7): 56–62.Google Scholar
- Dahlbäck, N., Reithinger, N., and Walker, M.A. (1997). Standards for dialogue coding in natural language processing. Report on a Dagstuhl-Seminar.Google Scholar
- Darken, R.P. and Silbert, J.L. (1996). Way finding strategies and behaviors in virtual worlds. Proc. CHI'96, pp. 142–149.Google Scholar
- Evers, M. and Nijholt, A. (2000). Jacob-An animated instruction agent for virtual reality. In T. Tan, Y. Shi, and W. Gao (Eds.), Advances in Multimodal Interfaces-ICMI 2000, Proc. Third International Conference on Multimodal Interfaces, Beijing, China, Lecture Notes in Computer Science, Vol. 1948. Berlin: Springer-Verlag, pp. 526–533.Google Scholar
- Gibbon, D., Moore, R., and Winski, R. (1997). Handbook of Standards and Resources for Spoken Language Systems. Berlin: Mouton de Gruyter.Google Scholar
- Höök, K., et al. (1988). Towards a framework for design and evaluation of navigation in electronic spaces. Persona Deliverable for the EC.Google Scholar
- Kragtwijk, M., Nijholt, A., and Zwiers, J. (2001). Implementation of a 3D virtual drummer. In M. Magnenat-Thalmann and D. Thalmann (Eds.), Proceedings CAS2001, EurographicsWorkshop on Animation and Simulation 2001, Manchester, UK. New York: Springer-Verlag, pp. 15–26.Google Scholar
- Lie, D., Hulstijn, J., Op den Akker, R., and Nijholt, A. (1998). A transformational approach to NL understanding in dialogue systems. Proc. NLP and Industrial Applications, Moncton, pp. 163–168.Google Scholar
- Nijholt, A. and Hulstijn, J. (2000). Multimodal interactions with agents in virtual worlds. In N. Kasabov (Ed.), Future Directions for Intelligent Information Systems and Information Science, Studies in Fuzziness and Soft Computing. Physica-Verlag, pp. 148–173.Google Scholar
- Rao, A. and Georgeff, M. (1992). An abstract architecture for rational agents. Proc. of the 3rd International Conference on Principles of Knowledge Representation and Reasoning, pp. 439–449.Google Scholar
- Reitmayr, G., Carroll, S., Reitemeyer, A., and Wagner, M.G. (1999). Deep matrix: An open technology based virtual environment system. The Visual Computer Journal, 15:395–412.Google Scholar
- Torres, O., Cassell, J., and Prevost, S. (1997). Modeling gaze behavior as a function of discourse structure. First International Workshop on Human Computer Conversations, Bellagio, Italy.Google Scholar
- Van Es, I. (2001). Kijkgedrag voor virtual agents in gesprek met de mens. Nuttig? University of Twente, Department of Computer Science, Internal Report.Google Scholar
- Van Es, I., Heylen, D., Van Dijk, B., and Nijholt, A. (2002). Gaze behavior of talking agents makes a difference. Proceedings CHI 2002, extended abstracts.Google Scholar
- Van Luin, J., Op den Akker, R., and Nijholt, A. (2001). A dialogue agent for navigation support in virtual reality. In J. Jacko and A. Sears (Eds.), Proceedings ACM SIGCHI Conference CHI 2001: Anyone. Anywhere, extended abstracts. Seattle: Association for Computing Machinery, pp. 117–118.Google Scholar
- Van Noord, G., Bouma, G., Koeling, R., and Nederhof, M.-J. (1999). Robust grammatical analysis for spoken dialogue systems. Natural Language Engineering, 5(1):45–93.Google Scholar
- Vertegaal, R., Slagter, R., Van der Veer, G., and Nijholt, A. (2001). Eye gaze patterns in conversations: There is more to conversational agents than meets the eyes. In J. Jacko, A. Sears, M. Beaudouin-Lafon, and R.J.K. Jacob (Eds.), Proceedings ACM SIGCHI Conference CHI 2001: Anyone. Anywhere. Association for Computing Machinery, pp. 301–308.Google Scholar