Skip to main content

A Unified Approach for Semantic-Based Multimodal Interaction

Part of the Cognitive Technologies book series (COGTECH)


This article gives an overview of the basic dialog shell for multimodal interaction that was created as the foundation for the software components developed in the Core Technology Cluster of the THESEUS research program. It explains the basic setup of systems based on the dialog shell for specific use cases. The general processing paradigm for interaction in our framework using semantic interface elements as well as several dialog phenomena handled by the dialog shell are described. Finally, a selection of different demonstrator systems that were built for a range of applications is presented briefly. They offer access to quite varied use case scenarios featuring multimodal input and output, including gestures, multi-touch, spoken input and output, and interactions across mobile and stationary devices.


  • Semantic Representation
  • Semantic Interpretation
  • Multimodal Interaction
  • Speech Recognizer
  • Service Invocation

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-06755-1_11
  • Chapter length: 14 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   99.00
Price excludes VAT (USA)
  • ISBN: 978-3-319-06755-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Hardcover Book
USD   119.99
Price excludes VAT (USA)
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9


  1. 1.

    Generally, an essential action of any rule is to remove the WME that triggered it, to avoid endless re-firing.


  • J. Alexandersson, T. Becker, Overlay as the basic operation for discourse processing in a multimodal dialogue system, in Proceedings of the IJCAI Workshop Knowledge and Reasoning in Practical Dialogue Systems, Seattle, 2001

    Google Scholar 

  • S. Bergweiler, M. Deru, D. Porta, Integrating a multitouch Kiosk system with mobile devices and multimodal interaction, in Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS ’10), Saarbrücken (ACM, New York, 2010), pp. 245–246,

  • B. Carpenter, The Logic of Typed Feature Structures (Cambridge University Press, Cambridge, UK, 1992)

    Google Scholar 

  • M.T. Maybury, W. Wahlster, Readings in Intelligent User Interfaces (Morgan Kaufmann Publishers, San Francisco, 1998)

    Google Scholar 

  • N. Reithinger, S. Bergweiler, R. Engel, G. Herzog, N. Pfleger, M. Romanelli, D. Sonntag, A look under the hood – design and development of the first smartweb system demonstrator, in Proceedings of the 7th International Conference on Multimodal Interfaces (ICMI ’05), Trento (ACM, New York, 2005), pp. 159–166

    Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Markus Löckelt .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Löckelt, M., Deru, M., Schulz, C.H., Bergweiler, S., Becker, T., Reithinger, N. (2014). A Unified Approach for Semantic-Based Multimodal Interaction. In: Wahlster, W., Grallert, HJ., Wess, S., Friedrich, H., Widenka, T. (eds) Towards the Internet of Services: The THESEUS Research Program. Cognitive Technologies. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-06754-4

  • Online ISBN: 978-3-319-06755-1

  • eBook Packages: Computer ScienceComputer Science (R0)