A Unified Framework for Constructing Multimodal Experiments and Applications
Inspired by a Wizard of Oz (WOZ) simulation experiment, we developed a working prototype of a system that enables users to interact with a map display through synergistic combinations of pen and voice. To address many of the issues raised by multimodal fusion, our implementation employed a distributed multi-agent framework to coordinate parallel competition and cooperation among processing components. Since then, the agent-based infrastructure has been enhanced with a collaboration technology, creating a framework in which multiple humans and automated agents can naturally interact within the same graphical workspace.
Our current endeavour is the leveraging of this architecture to create a unified implementation framework for simultaneously developing both WOZ simulated systems and their fully-automated counterparts. Bootstrapping effects made possible by such an approach are illustrated by an experiment currently under way in our laboratory: as a naive subject draws, writes, and speaks requests to a (simulated) interactive map, a hidden Wizard responds as efficiently as possible using our best fully-automated system, through either standard graphical interface devices or multimodal combinations of pen and voice. The input choices made by both subject and Wizard are invaluable, and the data collected from each can be applied directly to evaluating and improving the automated part of the system.
KeywordsMultimodal Interface Collaboration Technology Distribute Software System Multimodal Fusion Distribute Data Service
Unable to display preview. Download preview PDF.
- Cheyer, A. (1998) MVIEWS: Multimodal tools for the video analyst. In: Proceedings ofIUI’98, San Francisco (USA), 55–62.Google Scholar
- Cheyer A. and Julia L. (1995) Multimodal maps: An agent-based approach. In: Proceedings of the International COnference on Cooperative Multimodal Communication CMC’95, Eindhoven (The Netherlands), May 1995, 103–113.Google Scholar
- Cohen, P., Cheyer, A., Wang, M., and Baeg, S. (1998) An Open Agent Architecture. In: M.N. Huhns and M.P. Singh (eds.), Readings in Agents, San Francisco: Morgan Kaufmann Publishers, 197–204.Google Scholar
- Guzzoni, D., Cheyer, A., Julia, L., and Konolige, K. (1997) Many robots make short work. AI Magazine, Vol. 18, No. 1, Spring 1997, 55–64.Google Scholar
- Julia, L. and Cheyer, A. (1997) Speech: a privileged modality. In: Proceedings of EuroSpeech’97, Rhodes (Greece), vol. 4, 1843–1846.Google Scholar
- Julia, L., Cheyer, A., Neumeyer, L., Dowding, J., and Charafeddine, M. (1997) http://www.speech.sri.com/demos/atis.html. In: Proceedings of AAAI’97, Stanford (USA), 72–76.
- Kehler A., Martin J.C., Cheyer A., Julia L., Hobbs J., and Bear J. (1998) On representing Salience and Reference in Multimodal Human-Computer Interaction. In: Proceedings of AAAI’98 (Representations for Multi-Modal Human-Computer Interaction), Madison (USA), 33–39.Google Scholar
- Martin, D., Cheyer, A., and Moran, D. (1999) The Open Agent Architecture: A framework for building distributed software systems. Applied Artificial Intelligence: An International Journal. (13,1–2).Google Scholar
- Martin, D., Cheyer, A., and Moran, D. (1998) Building Distributed Software Systems with the Open Agent Architecture. See http://www.ai.sri.com/~oaa and “Publications”, 1998.
- Martin, D., Cheyer, A., and Lee, GL. (1996) Agent development tools for the Open Agent Architecture. In: Proceedings of the International Conference on the Practical Application of Intelligent Agents and Multi-Agent Technology, London, April 1996.Google Scholar
- Martin, D., Oohama, H., Moran, D., and Cheyer, A. (1997) Information brokering in an agent architecture. In: Proceedings of the Second International Conference on the Practical Application of Intelligent Agents and Multi-Agent Technology, London, April 1997.Google Scholar
- Martin, J.C., Julia, L., and Cheyer, A. (1998) A theoretical framework for multimodal user studies. In: Proceedings of the Second International COnference on Cooperative Multimodal Communication CMC/98, Tilburg (The Netherlands), January 1998, 104–110.Google Scholar
- Moore, R., Dowding, J., Bratt, H., Gawron, J.M., and Cheyer, A. (1997) Com-mandTalk: A spoken-language interface for battlefield simulations. In: Proceedings of Fifth Conference on Applied Natural Language Processing, Washington, D.C., April 1997.Google Scholar
- Moran, D., Cheyer, A., Julia, L., and Park, S. (1997) Multimodal user interfaces in the Open Agent Architecture. In: Proceedings of IUI’97, Orlando (USA), 61–68.Google Scholar
- Oviatt, S. (1996) Multimodal interfaces for dynamic interactive maps. In: Proceedings of CHI’96, April 13–18, 1996. 95–102.Google Scholar
- Oviatt, S., De Angeli, A., and Kuhn, K. (1997) Integration and synchronization of input modes during multimodal human-computer interaction. In: Proceedings of the workshop “Referring Phenomena in a Multimedia Context and their Computational Treatment”, ACL/EACL’97 (Madrid), 1–13. Also: http://www.dfki.uni-sb.de/imedia/workshops/mm-references.html