Skip to main content
Book cover

Towards the Internet of Services: The THESEUS Research Program pp 131–144Cite as

A Unified Approach for Semantic-Based Multimodal Interaction

Part of the Cognitive Technologies book series (COGTECH)

Abstract

This article gives an overview of the basic dialog shell for multimodal interaction that was created as the foundation for the software components developed in the Core Technology Cluster of the THESEUS research program. It explains the basic setup of systems based on the dialog shell for specific use cases. The general processing paradigm for interaction in our framework using semantic interface elements as well as several dialog phenomena handled by the dialog shell are described. Finally, a selection of different demonstrator systems that were built for a range of applications is presented briefly. They offer access to quite varied use case scenarios featuring multimodal input and output, including gestures, multi-touch, spoken input and output, and interactions across mobile and stationary devices.

Keywords

  • Semantic Representation
  • Semantic Interpretation
  • Multimodal Interaction
  • Speech Recognizer
  • Service Invocation

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-06755-1_11
  • Chapter length: 14 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   99.00
Price excludes VAT (USA)
  • ISBN: 978-3-319-06755-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Hardcover Book
USD   119.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Notes

  1. 1.

    Generally, an essential action of any rule is to remove the WME that triggered it, to avoid endless re-firing.

References

  • J. Alexandersson, T. Becker, Overlay as the basic operation for discourse processing in a multimodal dialogue system, in Proceedings of the IJCAI Workshop Knowledge and Reasoning in Practical Dialogue Systems, Seattle, 2001

    Google Scholar 

  • S. Bergweiler, M. Deru, D. Porta, Integrating a multitouch Kiosk system with mobile devices and multimodal interaction, in Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS ’10), Saarbrücken (ACM, New York, 2010), pp. 245–246, http://doi.acm.org/10.1145/1936652.1936698

  • B. Carpenter, The Logic of Typed Feature Structures (Cambridge University Press, Cambridge, UK, 1992)

    Google Scholar 

  • M.T. Maybury, W. Wahlster, Readings in Intelligent User Interfaces (Morgan Kaufmann Publishers, San Francisco, 1998)

    Google Scholar 

  • N. Reithinger, S. Bergweiler, R. Engel, G. Herzog, N. Pfleger, M. Romanelli, D. Sonntag, A look under the hood – design and development of the first smartweb system demonstrator, in Proceedings of the 7th International Conference on Multimodal Interfaces (ICMI ’05), Trento (ACM, New York, 2005), pp. 159–166

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. DFKI GmbH, German Research Center for Artificial Intelligence, Saarbrücken, Germany

    Markus Löckelt, Matthieu Deru, Simon Bergweiler & Tilman Becker

  2. DFKI GmbH, German Research Center for Artificial Intelligence, Berlin, Germany

    Christian H. Schulz & Norbert Reithinger

Authors
  1. Markus Löckelt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matthieu Deru
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christian H. Schulz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Simon Bergweiler
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tilman Becker
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Norbert Reithinger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Markus Löckelt .

Editor information

Editors and Affiliations

  1. Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI) GmbH, Saarbrücken, Germany

    Wolfgang Wahlster

  2. Fraunhofer Heinrich-Hertz-Institut, Berlin, Germany

    Hans-Joachim Grallert

  3. Empolis Information Management GmbH, Kaiserslautern, Germany

    Stefan Wess

  4. Corporate Technology, Siemens AG, München, Germany

    Hermann Friedrich

  5. Strategy Advisory, SAP Deutschland AG & Co. KG, Walldorf, Germany

    Thomas Widenka

Rights and permissions

Reprints and Permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Löckelt, M., Deru, M., Schulz, C.H., Bergweiler, S., Becker, T., Reithinger, N. (2014). A Unified Approach for Semantic-Based Multimodal Interaction. In: Wahlster, W., Grallert, HJ., Wess, S., Friedrich, H., Widenka, T. (eds) Towards the Internet of Services: The THESEUS Research Program. Cognitive Technologies. Springer, Cham. https://doi.org/10.1007/978-3-319-06755-1_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-06755-1_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-06754-4

  • Online ISBN: 978-3-319-06755-1

  • eBook Packages: Computer ScienceComputer Science (R0)