Multimodal Comprehension of Graphics with Textual Annotations: The Role of Graphical Means Relating Annotations and Graph Lines

  • Cengiz Acarturk
  • Christopher Habel
  • Kursat Cagiltay
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5223)


Graphs are often accompanied by text, i.e. linguistically coded information, augmenting the information presented diagrammatically. Thus, graph comprehension by humans often constitutes comprehension and integration of information provided by different representational modalities, namely graphical elements and verbal constituents. In this study we focus on textual annotations to line graphs providing information about events, processes and their temporal properties as well as temporal relations about the events and processes in question. We present results of an experimental investigation on parameters which influence subject’s interpretations concerning the temporal properties of the annotated events and on eye movement behavior. In particular, we discuss the role of graph shape and the role of graphical means for relating textual annotations and determined parts of the graph line.


text-graphics comprehension annotations line graphs temporal relations 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bernsen, N.O.: Foundations of Multimodal Representations: A Taxonomy of Representational Modalities. Interacting with Computers 6, 347–371 (1994)CrossRefGoogle Scholar
  2. 2.
    Hegarty, M.: The Mechanics of Comprehension and Comprehension of Mechanics. In: Rayner, K. (ed.) Eye Movements and Visual Cognition: Scene Perception and Reading, pp. 428–443. Springer, New York (1992)Google Scholar
  3. 3.
    Butcher, K.R.: Learning from Text with Diagrams: Promoting Mental Model Development and Inference Generation. Journal of Educational Psychology 98, 182–197 (2006)CrossRefGoogle Scholar
  4. 4.
    Narayanan, N.H., Hegarty, M.: On Designing Comprehensible Interactive Hypermedia Manuals. International Journal of Human Computer Studies 48, 267–301 (1998)CrossRefGoogle Scholar
  5. 5.
    Scaife, M., Rogers, Y.: External Cognition: How Do Graphical Representations Work? International Journal of Human Computer Studies 45, 185–213 (1996)CrossRefGoogle Scholar
  6. 6.
    Peebles, D.J., Cheng, P.C.-H.: Extending Task Analytic Models of Graph-based Reasoning: A Cognitive Model of Problem Solving with Cartesian Graphs in ACT-R/PM. Cognitive Systems research 3, 77–86 (2002)CrossRefGoogle Scholar
  7. 7.
    Pinker, S.: A Theory of Graph Comprehension. In: Freedle, R. (ed.) Artificial intelligence and the Future of Testing, pp. 73–126. Erlbaum, Hillsdale (1990)Google Scholar
  8. 8.
    Mautone, P.D., Mayer, R.E.: Cognitive Aids for Guiding Graph Comprehension. Journal of Educational Psychology 99, 640–652 (2007)CrossRefGoogle Scholar
  9. 9.
    Kosslyn, S.M.: Image and Mind. Harvard University Press, Cambridge (1980)Google Scholar
  10. 10.
    Kosslyn, S.M.: Understanding Charts and Graphs. Applied Cognitive Psychology 3, 185–226 (1989)CrossRefGoogle Scholar
  11. 11.
    Tversky, B.: Semantics, Syntax, and Pragmatics of Graphics. In: Holmqvist, K., Ericsson, Y. (eds.) Language and Visualisation, pp. 141–158. Lund University Press, Lund (2004)Google Scholar
  12. 12.
    Habel, C., Acarturk, C.: On Reciprocal Improvement in Multimodal Generation: Co-reference by Text and Information Graphics. In: van der Sluis, I., Theune, M., Reiter, E., Krahmer, E. (eds.) Workshop on Multimodal Output Generation (MOG 2007), Aberdeen, United Kingdom, pp. 69–80 (2007)Google Scholar
  13. 13.
    Acarturk, C., Habel, C., Cagiltay, K., Alacam, O.: Multimodal Comprehension of Language and Graphics: Graphs with and without Annotations (in press, accepted for publication in Journal of Eye Movement Research)Google Scholar
  14. 14.
    Underwood, G., Jebbett, L., Roberts, K.: Inspecting Pictures for Information to Verify a Sentence: Eye Movements in General Encoding and in Focused search. The Quarterly Journal of Experimental Psychology 57A, 165–182 (2004)Google Scholar
  15. 15.
    Just, M.A., Carpenter, P.A.: A Theory of Reading: From Eye Fixations to Comprehension. Psychological Review 87, 329–354 (1980)CrossRefGoogle Scholar
  16. 16.
    Anderson, J.R., Bothell, D., Douglass, S.: Eye Movements Do Not Reflect Retrieval Processes: Limits of the Eye-mind Hypothesis. Psychological Science 15, 225–231 (2004)CrossRefGoogle Scholar
  17. 17.
    Cohen, E., Singh, M.: Geometric Determinants of Shape Segmentation: Tests Using Segment Identification. Vision Research 47, 2825–2840 (2007)CrossRefGoogle Scholar
  18. 18.
    De Winter, J., Wagemans, J.: Segmentation of Object Outlines into Parts: A Large-scale Integrative Study. Cognition 99, 275–325 (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Cengiz Acarturk
    • 1
  • Christopher Habel
    • 1
  • Kursat Cagiltay
    • 2
  1. 1.Computer ScienceUniversity of HamburgHamburgGermany
  2. 2.Computer Education and Instructional TechnologyMiddle East Technical UniversityAnkaraTurkey

Personalised recommendations