A Unified Framework for Constructing Multimodal Experiments and Applications

  • Adam Cheyer
  • Luc Julia
  • Jean-Claude Martin
Conference paper

DOI: 10.1007/3-540-45520-5_14

Volume 2155 of the book series Lecture Notes in Computer Science (LNCS)
Cite this paper as:
Cheyer A., Julia L., Martin JC. (2001) A Unified Framework for Constructing Multimodal Experiments and Applications. In: Bunt H., Beun R.J. (eds) Cooperative Multimodal Communication. CMC 1998. Lecture Notes in Computer Science, vol 2155. Springer, Berlin, Heidelberg

Abstract

Inspired by a Wizard of Oz (WOZ) simulation experiment, we developed a working prototype of a system that enables users to interact with a map display through synergistic combinations of pen and voice. To address many of the issues raised by multimodal fusion, our implementation employed a distributed multi-agent framework to coordinate parallel competition and cooperation among processing components. Since then, the agent-based infrastructure has been enhanced with a collaboration technology, creating a framework in which multiple humans and automated agents can naturally interact within the same graphical workspace.

Our current endeavour is the leveraging of this architecture to create a unified implementation framework for simultaneously developing both WOZ simulated systems and their fully-automated counterparts. Bootstrapping effects made possible by such an approach are illustrated by an experiment currently under way in our laboratory: as a naive subject draws, writes, and speaks requests to a (simulated) interactive map, a hidden Wizard responds as efficiently as possible using our best fully-automated system, through either standard graphical interface devices or multimodal combinations of pen and voice. The input choices made by both subject and Wizard are invaluable, and the data collected from each can be applied directly to evaluating and improving the automated part of the system.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2001

Authors and Affiliations

  • Adam Cheyer
    • 1
  • Luc Julia
    • 1
  • Jean-Claude Martin
    • 2
  1. 1.SRI InternationalMenlo ParkUSA
  2. 2.LIMSI-CNRSOrsayFrance