Abstract
Visual language refers to the idea that communication occurs through visual symbols, as opposed to verbal symbols or words. Contrast to a sentence construction in spoken language with a linear ordering of words, a visual language has a simultaneous structure with a parallel temporal and spatial configuration. Inspired by Deikto [5], we propose a two-dimensional string or sentence construction of visual expressions, i.e. spatial arrangements of symbols, which represent concepts. A proof of concept communication interface has been developed, which enables users to create visual messages to represent concepts or ideas in their mind. By the employment of ontology, the interface constructs both the syntax and semantics of a 2D visual string using a Lexicalized Tree Adjoining Grammar (LTAG) into (natural language) text. This approach captures elegantly the interaction between pragmatic and syntactic descriptions in a 2D sentence, and the inferential interactions between multiple possible meanings generated by the sentence. From our user test results, we conclude that our developed visual language interface could serve as a communication mediator.
The research reported here is part of the Interactive Collaborative Information Systems (ICIS) project, supported by the Dutch Ministry of Economic Affairs, grant nr: BSIK03024.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bevelier, D., Corina, D.P., Neville, H.J.: Brain and Language: a Perspective from Sign Language. Neuron 21, 275–278 (1998)
Chandler, D.: Semiotics, the Basic. Routledge (2001)
Chang, S.K., Polese, G., Orefice, S., Tucci, M.: A Methodology and Interactive Environment for Iconic Language Design. International Journal of Human Computer Studies 41, 683–716 (1994)
Cohn, N.: Visual Syntactic Structures, Towards a Generative Grammar of Visual Language (2003), http://www.emaki.net/essays/
Crawford, C.: Erasmatron (2005), http://www.erasmatazz.com
Frutiger, A.: Sign and Symbols, Their Design and Meaning. van Nostrand Reinholt, New York (1989)
Fillmore, C.J.: The Case for Case. In: Bach, E., Harms, R. (eds.) Universals in Linguistic Theory. Holt, Rinehart and Winston, New York (1968)
Fitrianie, S., Rothkrantz, L.J.M.: Language-Independent Communication using Icons on a PDA. In: Matoušek, V., Mautner, P., Pavelka, T. (eds.) TSD 2005. LNCS (LNAI), vol. 3658, pp. 404–411. Springer, Heidelberg (2005)
Joshi, A.K., Levy, L., Takahashi, M.: Tree Adjunct Grammars. Journal of the Computer and System Sciences 10, 136–163 (1975)
Kipper-Schuler, K.: VerbNet: A Broad-Coverage, Comprehensive Verb Lexicon. Ph.D. thesis proposal, University of Pennsylvania (2003)
Kjeldskov, J., Kolbe, N.: Interaction Design for Handheld Computers. In: Proc. of APCHI 2002. Science Press, China (2002)
Lester, P.M.: Visual Communication: Images with Messages, 4th edn. Wadsworth/Thompson, Belmont (2006)
Leemans, N.E.M.P.: VIL, A Visual Inter Lingua. Doctoral Dissertation, Worcester Polytechnic Institute, USA (2001)
Littlejohn, S.W.: Theories of Human Communication, 5th edn. Wadsworth (1996)
Perlovsky, L.I.: Emotions, Learning and Control. In: Proc. of Intl. Symposium: Intelligent Control, Intelligent System and Semiotics (1999)
Ryant, N., Kipper, K.: Assigning Xtag Trees to VerbNet. In: 7th International Workshop on Tree Adjoining Grammar and Related Formalisms (TAG+7), Vancouver, Canada (2004)
Schank, R.: Computer Models of Thought and Language. W.H. Freeman, San Francisco (1973)
XTAG Research Group: A Lexicalized Tree Adjoining Grammar for English. Technical Report IRCS-01-03. IRCS, University of Pennsylvania (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Fitrianie, S., Rothkrantz, L.J.M. (2006). Two-Dimensional Visual Language Grammar. In: Sojka, P., Kopeček, I., Pala, K. (eds) Text, Speech and Dialogue. TSD 2006. Lecture Notes in Computer Science(), vol 4188. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11846406_72
Download citation
DOI: https://doi.org/10.1007/11846406_72
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-39090-9
Online ISBN: 978-3-540-39091-6
eBook Packages: Computer ScienceComputer Science (R0)