Abstract
When students read for learning, they frequently are required to integrate text and graphics information into coherent knowledge structures. The following study aimed at analyzing how students deal with texts and how they deal with graphics when they try to integrate the two sources of information. Furthermore, the study investigated differences between students from different school types and grades. Forty students from grades 5 and 8 from higher track and lower track of the German school system were asked to process and integrate texts and graphics in order to answer items from different levels of a text–picture integration taxonomy. Students’ eye movements were recorded and analyzed. Results suggest fundamentally different functions of text and graphics, which are associated with different processing strategies. Texts are more likely to be used according to a coherence-formation strategy, whereas graphics are more likely to be used on demand as visual cognitive tools according to an information-selection strategy. Students from different tracks of schooling revealed different adaptivity with regard to the requirements of combining text and graphic information.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
In the original picture, the different parts of the legs are not labeled by color names, but by the corresponding colors. Within the original text, colors are used instead of color names as reference to the picture.
It should be noted that although the hierarchy between the levels A, B and C in the taxonomy is based on the principle of lower level structures being embedded into higher level structures, it is possible to construct A-items, B-items and C-items from different hierarchies within the same text–picture unit. In this case, the items are located at different taxonomy levels, but are nevertheless stochastically independent.
References
Ainsworth, S. (1999). The functions of multiple representations. Computers in Education, 33, 131–152.
Baddeley, A. D. (1992). Working memory. Science, 255, 556–559.
Eggen, T. J. H. M. (2004). Contributions to the theory of practice of computerized adaptive testing. Enschede: University of Twente.
Falkenheiner, B., Forbus, K. D., & Gentner, D. (1989/1990). The structure-mapping engine: algorithm and examples. Artificial Intelligence, 41, 1–63.
Flavell, J. H., & Wellman, H. M. (1977). Metamemory. In R. V. Kail & J. W. Hagen (Eds.), Perspectives on the development of memory and cognition. Hillsdale: Erlbaum.
Fletcher, J. D., & Tobias, S. (2005). The multimedia principle. In R. E. Mayer (Ed.), Cambridge handbook of multimedia learning (pp. 117–133). Cambridge: Cambridge University Press.
Gentner, D. (1989). The mechanisms of analogical learning. In S. Vosniadou & A. Ortony (Eds.), Similarity and analogical reasoning (pp. 199–241). London: Cambridge University Press.
Goldman, S. R., & Rakestraw, J. A. (2000). Structural aspects of constructing meaning from text. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. III, pp. 311–335). Hillsdale: Erlbaum.
Graesser, A. C., Millis, K. K., & Zwaan, R. A. (1997). Discourse comprehension. Annual Review of Psychology, 48, 163–189.
Hannus, M., & Hyönä, J. (1999). Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary Educational Psychology, 24, 95–123.
Hartman, H. J. (2001). Metacognition in learning and instruction: Theory, research and practice. Dordrecht: Kluwer.
Hasselhorn, M. (1996). Kategoriales Organisieren bei Kindern: Zur Entwicklung einer Gedächtnisstrategie. Göttingen: Hogrefe.
Hegarty, M. (2004). Mechanical reasoning as mental simulation. Trends in Cognitive Sciences, 8, 280–285.
Hegarty, M., & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal of Memory and Language, 32, 717–742.
Hyönä, J., Radach, R., & Deubel, H. (Eds.). (2003). The mind’s eye: cognitive and applied aspects of eye movement research. Amsterdam: North Holland.
Just, M. A., & Carpenter, P. A. (1980). A theory of reading: from eye fixations to comprehension. Psychological Review, 87, 329–354.
Kaakinen, J. K., & Hyönä, J. (2005). Perspective effects on expository text comprehension: evidence from think-aloud protocols, eyetracking and recall. Discourse Processes, 40, 239–257.
Keehner, M., Hegarty, M., Cohen, C. A., Khooshabeh, P., & Montello, D. R. (2008). Spatial reasoning with external visualizations: what matters is what you see, not whether you interact. Cognitive Science, 32, 1099–1132.
Kintsch, W. (1998). Comprehension: a paradigm for cognition. New York: Cambridge University Press.
Kruley, P., Sciama, S. C., & Glenberg, A. M. (1994). On-line processing of textual illustrations in the visuospatial sketchpad: evidence from dual-task studies. Memory & Cognition, 22, 261–272.
Kulhavy, R. W., Stock, W. A., & Kealy, W. A. (1993). How geographic maps increase recall of instructional text. Educational Technology Research and Development, 41, 47–62.
Kulhavy, R. W., Stock, W. A., & Caterino, L. C. (1994). Reference maps as a framework for remembering text. In W. Schnotz & R. Kulhavy (Eds.), Comprehension of graphics (pp. 153–162). Amsterdam: North Holland.
Levie, H. W., & Lentz, R. (1982). Effects of text illustration: a review of research. Educational Communication and Technology Journal, 30, 195–232.
Levin, J. R., Anglin, G. J., & Carney, R. N. (1987). On empirically validating functions of pictures in prose. In D. M. Willows & H. A. Houghton (Eds.), The psychology of illustration (pp. 51–85). New York: Springer.
Mayer, R. E. (1997). Multimedia learning: are we asking the right questions? Educational Psychologist, 32, 1–19.
Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.
Mayer, R. E. (2005a). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 31–48). New York: Cambridge University Press.
Mayer, R. E. (2005b). The Cambridge handbook of multimedia learning. Cambridge: Cambridge University Press.
Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press.
McNamara, D. S. (Ed.). (2007). Reading comprehension strategies: theories, interventions, and technologies. Mahwah: Erlbaum.
Mokros, J. R., & Tinker, R. F. (1987). The impact of microcomputer based labs on children’s ability to interpret graphs. Journal of Research in Science Teaching, 24(4), 369–383.
Paivio, A. (1986). Mental representations: a dual coding approach. New York: Oxford University Press.
Pinker, S. (1990). A theory of graph comprehension. In R. Freedle (Ed.), Artificial intelligence and the future of testing (pp. 73–126). Hillsdale: Erlbaum.
Rickards, J. P., & Denner, P. R. (1978). Inserted questions as aids to reading text. Instructional Science, 7, 313–346.
Rigney, D. (2010). The Matthew effect: how advantage begets further advantage. New York: Columbia University Press.
Rouet, J. F. (2006). The skills of document use: from text comprehension to Web-based learning. Mahwah: Erlbaum.
Rouet, J. F., Britt, M. A., Mason, R. A., & Perfetti, C. A. (1996). Using multiple sources of evidence to reason about history. Journal of Educational Psychology, 88, 478–493.
Rozenblit, L., Spivey, M., & Wojslawowicz, J. (1998). Mechanical reasoning about gear-and-belt diagrams: Do eye movements predict performance? In Proceedings of Mind III: The Annual Conference of the Cognitive Science Society of Ireland, 158–165.
Schnotz, W. (1993). On the relation between dual coding and mental models in graphics comprehension. Learning and Instruction, 3, 247–249.
Schnotz, W. (2002). Towards an integrated view of learning from text and visual displays. Educational Psychology Review, 14(2), 101–120.
Schnotz, W. (2005). An integrated model of text and picture comprehension. In R. E. Mayer (Ed.), Cambridge handbook of multimedia learning (pp. 49–69). Cambridge: Cambridge University Press.
Schnotz, W., & Bannert, M. (1999). Einfluesse der Visualisierungsform auf die Konstruktion mentaler Modelle beim Text- und Bildverstehen [Influence of the type of visualization on the construction of mental models during picture and text comprehension]. Zeitschrift fuer Experimentelle Psychologie, 46, 217–236.
Schnotz, W., & Bannert, M. (2003). Construction and interference in learning from multiple representations. Learning and Instruction, 13, 141–156.
Schnotz, W., Horz, H., McElvany, N., Schroeder, S., Ullrich, M., Baumert, J., Hachfeld, A., & Richter, T. (2010). Das BITE-Projekt: integrative Verarbeitung von Bildern und Texten in der Sekundarstufe I. Zeitschrift für Pädagogik, 56, 143–153.
Schnotz, W., Ullrich, M., Hochpöchler, U., Horz, H., McElvany, N., Schroeder, S., & Baumert, J. (2011). What makes text–picture integration difficult? A structural and procedural analysis of textbook requirements. Ricerche di Psicologia, 1, 103–135.
Sims, V. K., & Hegarty, M. (1997). Mental animation in the visuospatial sketchpad: evidence from dual-tasks studies. Memory & Cognition, 25, 321–332.
Stanovich, K. E. (1986). Matthew effects in reading: some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21(4), 360–407.
Stanovich, K. E. (2000). Progress in understanding reading: Scientific Foundations and New frontiers. New York: Guilford Press.
Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and Instruction, 12, 185–233.
Ullman, S. (1984). Visual routines. Cognition, 18, 97–159.
van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic.
Wainer, H. (1992). Understanding graphs and tables. Educational Researcher, 21(1), 14–23.
Weaver, C. A., III, Mannes, S., & Fletcher, C. R. (Eds.). (1995). Discourse comprehension. Hillsdale: Erlbaum.
Weidenmann, B. (1989). When good pictures fail: An information-processing approach to the effects of illustrations. In H. Mandl & J. R. Levin (Eds.), Knowledge acquisition from text and pictures (pp. 157–170). Amsterdam: North-Holland.
Wertheimer, M. (1938). Laws of organization in perceptual forms in a source book for Gestalt Psychology. London: Routledge & Kegan Paul.
Winn, W. D. (1990). A theoretical framework for research on learning from graphics. International Journal of Educational Research, 14, 553–564.
Zelazny, G. (2006). Say it with charts: the executive’s guide to visual communication. New York: McGraw-Hill.
Zenisky, A. L., Hambleton, R. K., & Sireci, S. G. (2002). Identification of local item dependencies in the Medical College Admissions Test. Journal of Educational Measurement, 39(4), 291–301.
Acknowledgments
This research has been performed as part of the BITE-Project supported by the German Science Foundation (Grant Number GZ: SCHN 665/3-1, SCHN 665/6-1) within the Special Research Program ‘Competence models for assessing individual learning results and for balancing of educational processes’. We are grateful to Dipl.-Psych. Wienke Wannagat for her help in running the empirical studies described in this article.
Author information
Authors and Affiliations
Corresponding author
Additional information
Ulrike Hochpöchler. Faculty of Psychology, University of Koblenz-Landau, Fortstraße 7, 76829 Landau, Germany. E-mail: hochpoechler@uni-landau.de
Current themes of research:
Text–picture integration.
Wolfgang Schnotz. Faculty of Psychology, University of Koblenz-Landau, Fortstraße 7, 76829 Landau, Germany. E-mail: schnotz@uni-landau.de
Current themes of research:
Text–Picture Integration, Knowledge Acquisition from Multiple Representations, Knowledge Acquisition with Conflicting Information, Multimedia Learning, Learning with Hypermedia, Learning with Animation.
Most relevant publications in the field of Psychology of Education:
Schnotz, W. (2011). Colorful bouquets in multimedia research: A closer look at the modality effect. Zeitschrift für Pädagogische Psychologie, 25, 269–276.
Schnotz, W., Baadte, C., Müller, A., & Rasch, R. (2010). Creative thinking and problem solving with depictive and descriptive representations. In L. Verschaffel, E. De Corte, T. de Jong & J. Elen (Eds.) Use of representations in reasoning and problem solving (pp. 11–35). London: Routledge.
Schnotz, W., & Heiß, A. (2009). Semantic scaffolds in hypermedia learning environments. Computers in Human Behavior, 25(2), 371–380.
Schnotz, W., & Kürschner, C. (2007). A reconsideration of cognitive load theory. Educational Psychology Review, 19(4), 469–508.
Schnotz, W., & Lowe, R.K. (2008). A unified view of learning from animated and static graphics. In R.K. Lowe & W. Schnotz (Eds.), Learning with animation. Research implications for design (pp. 304–356). New York: Cambridge University Press.
Thorsten Rasch. Faculty of Psychology, University of Koblenz-Landau, Fortstraße 7, 76829 Landau, Germany. E-mail: rasch@uni-landau.de
Current themes of research:
Text–picture integration, learning from animation, executive functions of working memory.
Most relevant publications in the field of Psychology of Education:
Lowe, R.K., Rasch, T., & Schnotz W. (2010). Aligning affordances of graphics with learning task requirements. Applied Cognitive Psychology. doi:10.1002/acp.1712.
Meyer, K., Rasch, T., & Schnotz, W. (2010). Effects of animation’s speed of presentation on perceptual processing and learning. Learning and Instruction, 20(2), 136–145.
Rasch, T. (2006). Verstehen abstrakter Sachverhalte. Semantische Gestalten in der Konstruktion mentaler Modelle. Berlin: wvb.
Rasch, T., & Schnotz, W. (2009). Interactive and non-interactive pictures in multimedia learning environments: Effects on learning outcomes and learning efficiency. Learning and Instruction, 19, 411–422.
Schnotz, W., & Rasch, T. (2005). Enabling, facilitating, and inhibiting effects of animations in multimedia learning: why reduction of cognitive load can have negative results on learning. Educational Technology: Research and Development, 53(3), 47–58.
Mark Ullrich. Institute of Psychology, University of Frankfurt, Mertonstrasse 17 (Jügel-Haus), 60325 Frankfurt, Germany. E-mail: m.ullrich@psych.uni-frankfurt.de
Current themes of research:
Learning from text and pictures, adult education.
Most relevant publications in the field of Psychology of Education:
Lintorf, K., McElvany, N., Rjosk, C., Schroeder, S., Baumert, J., Schnotz, W., Horz, H., & Ullrich, M. (2011). Zuverlässigkeit von diagnostischen Lehrerurteilen—Realiabilität verschiedener Urteilsmaße bei der Einschätzung von Aufgabenschwierigkeiten. Unterrichtswissenschaft, 39, 102–120.
McElvany, N., Schroeder, S., Hachfeld, A., Baumert, J., Richter, T., Schnotz, W., Horz, H., & Ullrich, M., & (2010). Diagnostische Fähigkeiten von Lehrkräften bei der Einschätzung von Schülerleistungen und Aufgabenschwierigkeiten bei Lernmedien mit instruktionalen Bildern. Zeitschrift für Pädagogische Psychologie, 23(3–4), 223–235.
McElvany, N., Schroeder, S., Richter, T., Hachfeld, A., Baumert, J., Schnotz, W., Horz, H., & Ullrich, M., (2010). Texte mit instruktionalen Bildern als Unterrichtsmaterial - Kompetenzen der Lehrkräfte. Unterrichtswissenschaft, 38, 98–116.
Schnotz, W., Horz, H., McElvany, N., Schroeder, S., Ullrich, M., Baumert, J., Hachfeld, A., & Richter, T. (2010). Das BITE-Projekt: Integrative Verarbeitung von Texten und Bildern in der Sekundarstufe I. Zeitschrift für Pädagogik, 56, 143–153. In Klieme, Eckhard/Leutner, Detlev/Kenk, Martina: Kompetenzmodellierung. Zwischenbilanz des DFG-Schwerpunktprogramms und Perspektiven des Forschungsansatzes. 56. Beiheft der Zeitschrift für Pädagogik, Weinheim u.a.: Beltz.
Holger Horz. Institute of Psychology, University of Frankfurt, Mertonstrasse 17 (Jügel-Haus), 60325 Frankfurt, Germany. E-mail: horz@psych.uni-frankfurt.de
Current themes of research:
Learning from text and pictures, instructional media, adult education, cognitive load, instructional prompts.
Most relevant publications in the field of Psychology of Education:
Horz, H. (2010). Lernen mit Medien. In Reinders, H., Ditton, H., Gräsel, C., & Gniewosz, B. (Hrsg.), Lehrbuch Empirische Bildungsforschung (S. 21–32). Wiesbaden: VS-Verlag für Sozialwissenschaft.
Horz, H. (2012). Situated prompts in authentic learning environments. In N. M. Seel (Ed.), Encyclopedia of the sciences of learning (pp. 3086–3087). Heidelberg: Springer.
Horz, H., & Schnotz, W. (2010). Cognitive load in learning with multiple representations. In J. L. Plass, R. Moreno, & R. Bruenken (Eds.), Cognitive load: Theory & application (pp. 229–252). New York: Cambridge University Press.
Horz, H., Winter C., & Fries, S. (2009). Differential benefits of instructional prompts. Computers in Human Behavior, 25, 818–828.
Ullrich, M., Schnotz, W., Horz, H., McElvany, N., Schroeder, S., & Baumert, J. (2012). Kognitionspsychologische Aspekte der Bild-Text-Integration. Psychologische Rundschau, 63, 11–17.
Nele McElvany. Institut für Schulentwicklungsforschung (IFS), Technical University of Dortmund, Vogelpothsweg 78, 44227 Dortmund, Germany. E-mail: McElvany@ifs.tu-dortmund.de
Current themes of research:
Lehr-Lernforschung im schulischen Kontext und im Elternhaus, Lesekompetenz, Kompetenzen von Lehrkräften, Bildung und Migration, Übergang von der Grundschule auf die weiterführende Schule.
Most relevant publications in the field of Psychology of Education:
Lintorf, K., McElvany, N., Rjosk, C., Schroeder, S., Baumert, J., Schnotz, W., Horz, H., & Ullrich, M. (2011). Zuverlässigkeit von diagnostischen Lehrerurteilen - Reliabilität verschiedener Urteilsmaße bei der Einschätzung von Schülerleistungen und Aufgabenschwierigkeiten. Unterrichtswissenschaft, 39(2), 102–120.
McElvany, N. (2011). Familiäre Bedingungsfaktoren von Lesekompetenz und Effektivität systematischer Förderung. In: Bundesverband Alphabetisierung und Grundbildung e.V./Bothe, Joachim (Hrsg.). Funktionaler Analphabetismus im Kontext von Familie und Partnerschaft. Alphabetisierung und Grundbildung Band 8, (S. 62–71). Münster/NewYork/München/Berlin: Waxmann.
Rjosk, C., McElvany, N., Anders, Y., & Becker, M. (2011). Diagnostische Fähigkeiten von Lehrkräften bei der Einschätzung der basalen Lesefähigkeit ihrer Schülerinnen und Schüler. Psychologie in Erziehung und Unterricht, 58(2), 92–105.
Schroeder, S., Richter, T., McElvany, N., Hachfeld, A., Baumert, J., Schnotz, W., Horz, H., & Ullrich, M. 547 (2011). Teachers’ beliefs, instructional behaviours, and students’ engagement in learning from texts with 548 instructional pictures. Learning and Instruction, 21(3), 403–415.
van Steensel, R.C., McElvany, N., Kurvers, J.J., & Herppich, S. (2011). How effective are family literacy programs?: results of a meta-analysis. Review of Educational Research, 81, 69–96. March 2011
Jürgen Baumert. Max-Planck-Institute of Human Development, Lentzeallee 94, 14195 Berlin, Germany. E-mail: jmpobaumert@mpib-berlin.mpg.de
Current themes of research:
Learning and instruction, Teacher expertise, Large-scale assessments, Development of educational systems, Cognitive and motivational development in childhood and adolescence.
Most relevant publications in the field of Psychology of Education:
Baumert, J., Nagy, G., & Lehmann, R. (2012). Cumulative Advantages and the Emergence of Social and Ethnic Inequality: Matthew Effects in Reading and Mathematics Development Within Elementary Schools? Child Development, 83(4), 1347-1367. doi:10.1111/j.1467-8624.2012.01779.x.
Stanat, P., Becker, M., Baumert, J., Lüdtke, O., & Eckhardt, A. G. (2012). Improving second language skills of immigrant students: a field trial study evaluating the effects of a summer learning program. Learning and Instruction, 22(3), 159-170.
Becker, M., Lüdtke, O., Trautwein, U., Köller, O., & Baumert, J. (2012). The differential effects of school tracking on psychometric intelligence: Do academic-track schools make students smarter? Journal of Educational Psychology, 104(3), 682-699.
Baumert, J., Kunter, M., Blum, W., Brunner, M., Voss, T., Jordan, A., Klusmann, U., Krauss, S., Neubrand, M., & Tsai, Y. (2010). Teachers’ mathematical knowledge, cognitive activation in the classroom, and student progress. American Educational Research Journal, 47(1), 133-180. doi:10.3102/0002831209345157.
Baumert, J., Becker, M., Neumann, M., & Nikolova, R. (2010). Besondere Förderung von Kern- kompetenzen an Spezialgymnasien? Der Frühübergang in grundständige Gymnasien in Berlin. Zeitschrift für Pädagogische Psychologie, 24(1), 5-22. doi:10.1024/1010-0652.a000001.
Baumert, J., Lüdtke, O., Trautwein, U., & Brunner, M. (2009). Large-scale student assessment studies measure the results of processes of knowledge acquisition: evidence in support of the distinction between intelligence and student achievement. Educational Research Review, 4(3), 165-176.
Rights and permissions
About this article
Cite this article
Hochpöchler, U., Schnotz, W., Rasch, T. et al. Dynamics of mental model construction from text and graphics. Eur J Psychol Educ 28, 1105–1126 (2013). https://doi.org/10.1007/s10212-012-0156-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10212-012-0156-z