A Unified Approach for Semantic-Based Multimodal Interaction
Abstract
This article gives an overview of the basic dialog shell for multimodal interaction that was created as the foundation for the software components developed in the Core Technology Cluster of the THESEUS research program. It explains the basic setup of systems based on the dialog shell for specific use cases. The general processing paradigm for interaction in our framework using semantic interface elements as well as several dialog phenomena handled by the dialog shell are described. Finally, a selection of different demonstrator systems that were built for a range of applications is presented briefly. They offer access to quite varied use case scenarios featuring multimodal input and output, including gestures, multi-touch, spoken input and output, and interactions across mobile and stationary devices.
Keywords
Semantic Representation Semantic Interpretation Multimodal Interaction Speech Recognizer Service InvocationReferences
- J. Alexandersson, T. Becker, Overlay as the basic operation for discourse processing in a multimodal dialogue system, in Proceedings of the IJCAI Workshop Knowledge and Reasoning in Practical Dialogue Systems, Seattle, 2001Google Scholar
- S. Bergweiler, M. Deru, D. Porta, Integrating a multitouch Kiosk system with mobile devices and multimodal interaction, in Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS ’10), Saarbrücken (ACM, New York, 2010), pp. 245–246, http://doi.acm.org/10.1145/1936652.1936698
- B. Carpenter, The Logic of Typed Feature Structures (Cambridge University Press, Cambridge, UK, 1992)Google Scholar
- M.T. Maybury, W. Wahlster, Readings in Intelligent User Interfaces (Morgan Kaufmann Publishers, San Francisco, 1998)Google Scholar
- N. Reithinger, S. Bergweiler, R. Engel, G. Herzog, N. Pfleger, M. Romanelli, D. Sonntag, A look under the hood – design and development of the first smartweb system demonstrator, in Proceedings of the 7th International Conference on Multimodal Interfaces (ICMI ’05), Trento (ACM, New York, 2005), pp. 159–166Google Scholar