Multimodal Comprehension of Graphics with Textual Annotations: The Role of Graphical Means Relating Annotations and Graph Lines
Graphs are often accompanied by text, i.e. linguistically coded information, augmenting the information presented diagrammatically. Thus, graph comprehension by humans often constitutes comprehension and integration of information provided by different representational modalities, namely graphical elements and verbal constituents. In this study we focus on textual annotations to line graphs providing information about events, processes and their temporal properties as well as temporal relations about the events and processes in question. We present results of an experimental investigation on parameters which influence subject’s interpretations concerning the temporal properties of the annotated events and on eye movement behavior. In particular, we discuss the role of graph shape and the role of graphical means for relating textual annotations and determined parts of the graph line.
Keywordstext-graphics comprehension annotations line graphs temporal relations
Unable to display preview. Download preview PDF.
- 2.Hegarty, M.: The Mechanics of Comprehension and Comprehension of Mechanics. In: Rayner, K. (ed.) Eye Movements and Visual Cognition: Scene Perception and Reading, pp. 428–443. Springer, New York (1992)Google Scholar
- 7.Pinker, S.: A Theory of Graph Comprehension. In: Freedle, R. (ed.) Artificial intelligence and the Future of Testing, pp. 73–126. Erlbaum, Hillsdale (1990)Google Scholar
- 9.Kosslyn, S.M.: Image and Mind. Harvard University Press, Cambridge (1980)Google Scholar
- 11.Tversky, B.: Semantics, Syntax, and Pragmatics of Graphics. In: Holmqvist, K., Ericsson, Y. (eds.) Language and Visualisation, pp. 141–158. Lund University Press, Lund (2004)Google Scholar
- 12.Habel, C., Acarturk, C.: On Reciprocal Improvement in Multimodal Generation: Co-reference by Text and Information Graphics. In: van der Sluis, I., Theune, M., Reiter, E., Krahmer, E. (eds.) Workshop on Multimodal Output Generation (MOG 2007), Aberdeen, United Kingdom, pp. 69–80 (2007)Google Scholar
- 13.Acarturk, C., Habel, C., Cagiltay, K., Alacam, O.: Multimodal Comprehension of Language and Graphics: Graphs with and without Annotations (in press, accepted for publication in Journal of Eye Movement Research)Google Scholar
- 14.Underwood, G., Jebbett, L., Roberts, K.: Inspecting Pictures for Information to Verify a Sentence: Eye Movements in General Encoding and in Focused search. The Quarterly Journal of Experimental Psychology 57A, 165–182 (2004)Google Scholar