Integrating Multimodal Cues Using Grammar Based Models

  • Manuel Giuliani
  • Alois Knoll
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4555)


Multimodal systems must process several input streams efficiently and represent the input in a way that allows the establishment of connections between modalities. This paper describes a multimodal system that uses Combinatory Categorial Grammars to parse several input streams and translate them into logical formulas. These logical formulas are expressed in Hybrid Logic, which is very suitable for multimodal integration because it can represent temporal relationships between modes in an abstract way. This level of abstraction makes it possible to define rules for multimodal processing in a straightforward way.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Shimazu, H., Arita, S., Takashima, Y.: Multi-modal definite clause grammar. In: Proceedings of the 15th Conference on Computational linguistics, pp. 832–836. Association for Computational Linguistics, Morristown, NJ, USA (1994)Google Scholar
  2. 2.
    Johnston, M., Bangalore, S.: Finite-state multimodal parsing and understanding. In: Proceedings of COLING-2000, Saarbruecken, Germany (2000)Google Scholar
  3. 3.
    Nevatia, R., Zhao, T., Hongeng, S.: Hierarchical language-based representation of events in video streams. In: IEEE Workshop on Event Mining (2003)Google Scholar
  4. 4.
    Ryoo, M.S., Aggarwal, J.K.: Recognition of composite human activities through context-free grammar based representation. In: CVPR 2006: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, pp. 1709–1718. IEEE Computer Society Press, Los Alamitos (2006)Google Scholar
  5. 5.
    Ades, A.E., Steedman, M.J.: On the order of words. Linguistics and philosophy 4, 517–558 (1982)CrossRefGoogle Scholar
  6. 6.
    Steedman, M.: The syntactic process. MIT Press, Cambridge, MA (2000)Google Scholar
  7. 7.
    Ajdukiewicz, K.: Die syntaktische konnexität. Studia Philosophica 1, 1–27 (1935)Google Scholar
  8. 8.
    Bar-Hillel, Y.: A quasi-arithmetic notation for syntactic description. Language 29, 47–58 (1953)CrossRefGoogle Scholar
  9. 9.
    White, M.: Efficient realization of coordinate structures in combinatory categorial grammar. Research on Language & Computation 4(1), 39–75 (2006)CrossRefGoogle Scholar
  10. 10.
    Baldridge, J., Kruijff, G.J.: Coupling ccg and hybrid logic dependency semantics. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002), University of Pennsylania, Philadelphia, PA (2002)Google Scholar
  11. 11.
    Prior, A.: Past, Present and Future. Oxford University Press, Oxford (1967)zbMATHGoogle Scholar
  12. 12.
    Brugman, H., Russel, A.: Annotating multi-media/multi-modal resources with elan. In: 4th International Conference on Language Resources and Evaluation (LREC 2004), Lisbon (26.05.2004-28.05.2004 2004), pp. 2065–2068 (2004)Google Scholar
  13. 13.
    Schmidt, T., Wörner, K.: Erstellen und analysieren von gesprächskorpora mit exmaralda. Gesprächsforschung - Online-Zeitschrift zur verbalen Interaktion Ausgabe 6, 171–195 (2005)Google Scholar
  14. 14.
    Allen, J.F.: Maintaining knowledge about temporal intervals. Commun. ACM 26(11), 832–843 (1983)zbMATHCrossRefGoogle Scholar
  15. 15.
    Foster, M.E., By, T., Rickert, M., Knoll, A.: Human-robot dialogue for joint construction tasks. In: Proceedings, Eighth International Conference on Multimodal Interfaces (ICMI 2006), Banff (November 2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Manuel Giuliani
    • 1
  • Alois Knoll
    • 1
  1. 1.Robotics and Embedded Systems Group, Department of Informatics, Technische Universität München, Boltzmannstraße 3, D-85748 Garching bei MünchenGermany

Personalised recommendations