Integrating Multimodal Cues Using Grammar Based Models
- Cite this paper as:
- Giuliani M., Knoll A. (2007) Integrating Multimodal Cues Using Grammar Based Models. In: Stephanidis C. (eds) Universal Access in Human-Computer Interaction. Ambient Interaction. UAHCI 2007. Lecture Notes in Computer Science, vol 4555. Springer, Berlin, Heidelberg
Multimodal systems must process several input streams efficiently and represent the input in a way that allows the establishment of connections between modalities. This paper describes a multimodal system that uses Combinatory Categorial Grammars to parse several input streams and translate them into logical formulas. These logical formulas are expressed in Hybrid Logic, which is very suitable for multimodal integration because it can represent temporal relationships between modes in an abstract way. This level of abstraction makes it possible to define rules for multimodal processing in a straightforward way.
Unable to display preview. Download preview PDF.