Multimodal maps: An agent-based approach

  • Adam Cheyer
  • Luc Julia
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1374)


In this paper, we discuss how multiple input modalities may be combined to produce more natural user interfaces. To illustrate this technique, we present a prototype map-based application for a travel planning domain. The application is distinguished by a synergistic combination of handwriting, gesture and speech modalities; access to existing data sources including the World Wide Web; and a mobile handheld interface. To implement the described application, a hierarchical distributed network of heterogeneous software agents was augmented by appropriate functionality for developing synergistic multimodal applications.


Speech Recognition Direct Manipulation Speech Recognition System Interagent Communication Multimodal Interface 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Allegayer, J., Jansen-Winkeln, R., Reddig, C. and Reithinger, N. (1989) Bidirectional use of knowledge in the multi-modal NL access system XTRA. In Proceedings of IJCAI-89, Detroit, pp. 1492–1497.Google Scholar
  2. Bolt, R. (1980) Put that there: Voice and Gesture at the Graphic Interface, Computer Graphics, 14(3), pp. 262–270.MathSciNetCrossRefGoogle Scholar
  3. Cohen, M., Murveit, H., Bernstein, J., Price, P., and Weintraub, M. (1990) The DE-CIPHER Speech Recognition System. In 1990 IEEE ICASSP, pp. 77–80.Google Scholar
  4. Cohen, P. (1992) The role of natural language in a multimodal interface. In Proceedings of UIST'92, pp. 143–149.Google Scholar
  5. Cohen, P.R., Cheyer, A., Wang, M. and Baeg, S.C. (1994) An Open Agent Architecture. In Proceedings AAAI'94-SA, Stanford, pp. 1–8.Google Scholar
  6. Dauphin DTR-1 User's Manual, Dauphin Technology, Inc., Lombard, Ill 60148.Google Scholar
  7. Faure, C. and Julia, L. (1994) An Agent-Based Architecture for a Multimodal Interface. In Proceedings AAAI'94 — IM4S, Stanford, pp. 82–86.Google Scholar
  8. Genesereth, M. and Singh, N.P. (1994) A knowledge sharing approach to software interoperation, unpublished manuscript, Computer Science Department, Stanford University.Google Scholar
  9. Telescript Product Documentation (1995), General Magic Inc.Google Scholar
  10. Koons, D.B., Sparrell, C.J., and Thorisson, K.R. (1993) Integrating Simultaneous Input from Speech, Gaze and Hand Gestures. In Intelligent Multimedia Interfaces, Maybury, M.T. (ed.), Menlo Park: AAAI Press/MIT Press.Google Scholar
  11. Maybury, M.T. (ed.) (1993) Intelligent Multimedia Interfaces, Menlo Park: AAAI Press/MIT Press.Google Scholar
  12. Neal, J.G., and Shapiro, S.C. (1991) Intelligent Multi-media Interface Technology. In Intelligent User Interfaces, Sullivan, J.W. and Tyler, S.W. (eds.), Reading: Addison-Wesley Pub. Co., pp. 11–43.Google Scholar
  13. Nigay, L. and Coutaz, J. (1993) A Design Space for Multimodal Systems: Concurrent Processing and Data Fusion. In Proceedings InterCHI'93, Amsterdam, ACM Press, pp. 172–178.Google Scholar
  14. Object Management Group (1991) The Common Object Request Broker: Architecture and Specification, OMG Document Number 91.12.1.Google Scholar
  15. Oviatt, S. (1994) Toward Empirically-Based Design of Multimodal Dialogue Systems. In Proceedings of AAAI'94 — IM4S, Stanford, pp. 30–36.Google Scholar
  16. Oviatt, S. and Olsen, E. (1994) Integration Themes in Multimodal Human-Computer Interaction. In Proceedings of ICSLP'94, Yokohama, pp. 551–554.Google Scholar
  17. Park, S.K., Choi J.M., Myeong-Wuk J., Lee G.L., and Lim Y.H. (submitted for publication), MASCOS: A Multi-Agent System as the Computer Secretary.Google Scholar
  18. Rhyne J. (1987) Dialogue Management for Gestural Interfaces, Computer Graphics, 21(2), pp. 137–142.MathSciNetCrossRefGoogle Scholar
  19. Schwartz, D.G. (1993) Cooperating heterogeneous systems: A blackboard-based meta approach, Technical Report 93-112, Center for Automation and Intelligent Systems Research, Case Western Reserve University, Cleveland Ohio, (unpublished PhD. thesis).Google Scholar
  20. Sullivan, J. and Tyler, S. (eds.) (1991) Intelligent User Interfaces, Reading: Addison-Wesley Pub. Co.zbMATHGoogle Scholar
  21. Warren, D. and Pereira, F. (1982) An Efficient Easily Adaptable System for Interpreting Natural Language Queries, American Journal of Computational Linguistics, 8(3), pp. 110–123.Google Scholar

Copyright information

© Springer-Verlag 1998

Authors and Affiliations

  • Adam Cheyer
    • 1
  • Luc Julia
    • 1
  1. 1.SRI InternationalMenlo ParkUSA

Personalised recommendations