Advertisement

A Unified Framework for Constructing Multimodal Experiments and Applications

  • Adam Cheyer
  • Luc Julia
  • Jean-Claude Martin
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2155)

Abstract

Inspired by a Wizard of Oz (WOZ) simulation experiment, we developed a working prototype of a system that enables users to interact with a map display through synergistic combinations of pen and voice. To address many of the issues raised by multimodal fusion, our implementation employed a distributed multi-agent framework to coordinate parallel competition and cooperation among processing components. Since then, the agent-based infrastructure has been enhanced with a collaboration technology, creating a framework in which multiple humans and automated agents can naturally interact within the same graphical workspace.

Our current endeavour is the leveraging of this architecture to create a unified implementation framework for simultaneously developing both WOZ simulated systems and their fully-automated counterparts. Bootstrapping effects made possible by such an approach are illustrated by an experiment currently under way in our laboratory: as a naive subject draws, writes, and speaks requests to a (simulated) interactive map, a hidden Wizard responds as efficiently as possible using our best fully-automated system, through either standard graphical interface devices or multimodal combinations of pen and voice. The input choices made by both subject and Wizard are invaluable, and the data collected from each can be applied directly to evaluating and improving the automated part of the system.

Keywords

Multimodal Interface Collaboration Technology Distribute Software System Multimodal Fusion Distribute Data Service 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Cheyer, A. (1998) MVIEWS: Multimodal tools for the video analyst. In: Proceedings ofIUI’98, San Francisco (USA), 55–62.Google Scholar
  2. Cheyer A. and Julia L. (1995) Multimodal maps: An agent-based approach. In: Proceedings of the International COnference on Cooperative Multimodal Communication CMC’95, Eindhoven (The Netherlands), May 1995, 103–113.Google Scholar
  3. Cheyer A. and Julia L. (1998) Multimodal maps: An agent-based approach. In: H. Bunt, R.J. Beun, and T. Borghuis (eds.), Multimodal Human-Computer Communication; Systems, Techniques and Experiments. Lecture Notes in Artificial Intelligence 1374, Berlin: Springer, 111–121.CrossRefGoogle Scholar
  4. Cohen, P., Cheyer, A., Wang, M., and Baeg, S. (1998) An Open Agent Architecture. In: M.N. Huhns and M.P. Singh (eds.), Readings in Agents, San Francisco: Morgan Kaufmann Publishers, 197–204.Google Scholar
  5. Guzzoni, D., Cheyer, A., Julia, L., and Konolige, K. (1997) Many robots make short work. AI Magazine, Vol. 18, No. 1, Spring 1997, 55–64.Google Scholar
  6. Julia, L. and Cheyer, A. (1997) Speech: a privileged modality. In: Proceedings of EuroSpeech’97, Rhodes (Greece), vol. 4, 1843–1846.Google Scholar
  7. Julia, L., Cheyer, A., Neumeyer, L., Dowding, J., and Charafeddine, M. (1997) http://www.speech.sri.com/demos/atis.html. In: Proceedings of AAAI’97, Stanford (USA), 72–76.
  8. Kehler A., Martin J.C., Cheyer A., Julia L., Hobbs J., and Bear J. (1998) On representing Salience and Reference in Multimodal Human-Computer Interaction. In: Proceedings of AAAI’98 (Representations for Multi-Modal Human-Computer Interaction), Madison (USA), 33–39.Google Scholar
  9. Martin, D., Cheyer, A., and Moran, D. (1999) The Open Agent Architecture: A framework for building distributed software systems. Applied Artificial Intelligence: An International Journal. (13,1–2).Google Scholar
  10. Martin, D., Cheyer, A., and Moran, D. (1998) Building Distributed Software Systems with the Open Agent Architecture. See http://www.ai.sri.com/~oaa and “Publications”, 1998.
  11. Martin, D., Cheyer, A., and Lee, GL. (1996) Agent development tools for the Open Agent Architecture. In: Proceedings of the International Conference on the Practical Application of Intelligent Agents and Multi-Agent Technology, London, April 1996.Google Scholar
  12. Martin, D., Oohama, H., Moran, D., and Cheyer, A. (1997) Information brokering in an agent architecture. In: Proceedings of the Second International Conference on the Practical Application of Intelligent Agents and Multi-Agent Technology, London, April 1997.Google Scholar
  13. Martin, J.C., Julia, L., and Cheyer, A. (1998) A theoretical framework for multimodal user studies. In: Proceedings of the Second International COnference on Cooperative Multimodal Communication CMC/98, Tilburg (The Netherlands), January 1998, 104–110.Google Scholar
  14. Moore, R., Dowding, J., Bratt, H., Gawron, J.M., and Cheyer, A. (1997) Com-mandTalk: A spoken-language interface for battlefield simulations. In: Proceedings of Fifth Conference on Applied Natural Language Processing, Washington, D.C., April 1997.Google Scholar
  15. Moran, D., Cheyer, A., Julia, L., and Park, S. (1997) Multimodal user interfaces in the Open Agent Architecture. In: Proceedings of IUI’97, Orlando (USA), 61–68.Google Scholar
  16. Oviatt, S. (1996) Multimodal interfaces for dynamic interactive maps. In: Proceedings of CHI’96, April 13–18, 1996. 95–102.Google Scholar
  17. Oviatt, S., De Angeli, A., and Kuhn, K. (1997) Integration and synchronization of input modes during multimodal human-computer interaction. In: Proceedings of the workshop “Referring Phenomena in a Multimedia Context and their Computational Treatment”, ACL/EACL’97 (Madrid), 1–13. Also: http://www.dfki.uni-sb.de/imedia/workshops/mm-references.html

Copyright information

© Springer-Verlag Berlin Heidelberg 2001

Authors and Affiliations

  • Adam Cheyer
    • 1
  • Luc Julia
    • 1
  • Jean-Claude Martin
    • 2
  1. 1.SRI InternationalMenlo ParkUSA
  2. 2.LIMSI-CNRSOrsayFrance

Personalised recommendations