Two-Dimensional Visual Language Grammar

  • Siska Fitrianie
  • Leon J. M. Rothkrantz
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4188)


Visual language refers to the idea that communication occurs through visual symbols, as opposed to verbal symbols or words. Contrast to a sentence construction in spoken language with a linear ordering of words, a visual language has a simultaneous structure with a parallel temporal and spatial configuration. Inspired by Deikto [5], we propose a two-dimensional string or sentence construction of visual expressions, i.e. spatial arrangements of symbols, which represent concepts. A proof of concept communication interface has been developed, which enables users to create visual messages to represent concepts or ideas in their mind. By the employment of ontology, the interface constructs both the syntax and semantics of a 2D visual string using a Lexicalized Tree Adjoining Grammar (LTAG) into (natural language) text. This approach captures elegantly the interaction between pragmatic and syntactic descriptions in a 2D sentence, and the inferential interactions between multiple possible meanings generated by the sentence. From our user test results, we conclude that our developed visual language interface could serve as a communication mediator.


Crisis Situation Thematic Role Visual Language Inferential Interaction Natural Language Text 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bevelier, D., Corina, D.P., Neville, H.J.: Brain and Language: a Perspective from Sign Language. Neuron 21, 275–278 (1998)CrossRefGoogle Scholar
  2. 2.
    Chandler, D.: Semiotics, the Basic. Routledge (2001)Google Scholar
  3. 3.
    Chang, S.K., Polese, G., Orefice, S., Tucci, M.: A Methodology and Interactive Environment for Iconic Language Design. International Journal of Human Computer Studies 41, 683–716 (1994)CrossRefGoogle Scholar
  4. 4.
    Cohn, N.: Visual Syntactic Structures, Towards a Generative Grammar of Visual Language (2003),
  5. 5.
    Crawford, C.: Erasmatron (2005),
  6. 6.
    Frutiger, A.: Sign and Symbols, Their Design and Meaning. van Nostrand Reinholt, New York (1989)Google Scholar
  7. 7.
    Fillmore, C.J.: The Case for Case. In: Bach, E., Harms, R. (eds.) Universals in Linguistic Theory. Holt, Rinehart and Winston, New York (1968)Google Scholar
  8. 8.
    Fitrianie, S., Rothkrantz, L.J.M.: Language-Independent Communication using Icons on a PDA. In: Matoušek, V., Mautner, P., Pavelka, T. (eds.) TSD 2005. LNCS (LNAI), vol. 3658, pp. 404–411. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  9. 9.
    Joshi, A.K., Levy, L., Takahashi, M.: Tree Adjunct Grammars. Journal of the Computer and System Sciences 10, 136–163 (1975)MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Kipper-Schuler, K.: VerbNet: A Broad-Coverage, Comprehensive Verb Lexicon. Ph.D. thesis proposal, University of Pennsylvania (2003)Google Scholar
  11. 11.
    Kjeldskov, J., Kolbe, N.: Interaction Design for Handheld Computers. In: Proc. of APCHI 2002. Science Press, China (2002)Google Scholar
  12. 12.
    Lester, P.M.: Visual Communication: Images with Messages, 4th edn. Wadsworth/Thompson, Belmont (2006)Google Scholar
  13. 13.
    Leemans, N.E.M.P.: VIL, A Visual Inter Lingua. Doctoral Dissertation, Worcester Polytechnic Institute, USA (2001)Google Scholar
  14. 14.
    Littlejohn, S.W.: Theories of Human Communication, 5th edn. Wadsworth (1996)Google Scholar
  15. 15.
    Perlovsky, L.I.: Emotions, Learning and Control. In: Proc. of Intl. Symposium: Intelligent Control, Intelligent System and Semiotics (1999)Google Scholar
  16. 16.
    Ryant, N., Kipper, K.: Assigning Xtag Trees to VerbNet. In: 7th International Workshop on Tree Adjoining Grammar and Related Formalisms (TAG+7), Vancouver, Canada (2004)Google Scholar
  17. 17.
    Schank, R.: Computer Models of Thought and Language. W.H. Freeman, San Francisco (1973)MATHGoogle Scholar
  18. 18.
    XTAG Research Group: A Lexicalized Tree Adjoining Grammar for English. Technical Report IRCS-01-03. IRCS, University of Pennsylvania (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Siska Fitrianie
    • 1
  • Leon J. M. Rothkrantz
    • 1
  1. 1.Man-Machine-Interaction GroupDelft University of TechnologyDelftThe Netherlands

Personalised recommendations