This article gives an overview of the basic dialog shell for multimodal interaction that was created as the foundation for the software components developed in the Core Technology Cluster of the THESEUS research program. It explains the basic setup of systems based on the dialog shell for specific use cases. The general processing paradigm for interaction in our framework using semantic interface elements as well as several dialog phenomena handled by the dialog shell are described. Finally, a selection of different demonstrator systems that were built for a range of applications is presented briefly. They offer access to quite varied use case scenarios featuring multimodal input and output, including gestures, multi-touch, spoken input and output, and interactions across mobile and stationary devices.
- Semantic Representation
- Semantic Interpretation
- Multimodal Interaction
- Speech Recognizer
- Service Invocation
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, access via your institution.
Generally, an essential action of any rule is to remove the WME that triggered it, to avoid endless re-firing.
J. Alexandersson, T. Becker, Overlay as the basic operation for discourse processing in a multimodal dialogue system, in Proceedings of the IJCAI Workshop Knowledge and Reasoning in Practical Dialogue Systems, Seattle, 2001
S. Bergweiler, M. Deru, D. Porta, Integrating a multitouch Kiosk system with mobile devices and multimodal interaction, in Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS ’10), Saarbrücken (ACM, New York, 2010), pp. 245–246, http://doi.acm.org/10.1145/1936652.1936698
B. Carpenter, The Logic of Typed Feature Structures (Cambridge University Press, Cambridge, UK, 1992)
M.T. Maybury, W. Wahlster, Readings in Intelligent User Interfaces (Morgan Kaufmann Publishers, San Francisco, 1998)
N. Reithinger, S. Bergweiler, R. Engel, G. Herzog, N. Pfleger, M. Romanelli, D. Sonntag, A look under the hood – design and development of the first smartweb system demonstrator, in Proceedings of the 7th International Conference on Multimodal Interfaces (ICMI ’05), Trento (ACM, New York, 2005), pp. 159–166
Editors and Affiliations
Rights and permissions
© 2014 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Löckelt, M., Deru, M., Schulz, C.H., Bergweiler, S., Becker, T., Reithinger, N. (2014). A Unified Approach for Semantic-Based Multimodal Interaction. In: Wahlster, W., Grallert, HJ., Wess, S., Friedrich, H., Widenka, T. (eds) Towards the Internet of Services: The THESEUS Research Program. Cognitive Technologies. Springer, Cham. https://doi.org/10.1007/978-3-319-06755-1_11
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-06754-4
Online ISBN: 978-3-319-06755-1
eBook Packages: Computer ScienceComputer Science (R0)