Skip to main content

Advertisement

SpringerLink
Log in

Integrative processing of verbal and graphical information during re-reading predicts learning from illustrated text: an eye-movement study

  • Published:
Reading and Writing Aims and scope Submit manuscript

Abstract

Printed or digital textbooks contain texts accompanied by various kinds of visualisation. Successful comprehension of these materials requires integrating verbal and graphical information. This study investigates the time course of processing an illustrated text through eye-tracking methodology in the school context. The aims were to identify patterns of first- and second-pass reading and to examine whether the integrative processing of text and picture during the less automatic and more purposeful second-pass reading predicts learning, after controlling for reading comprehension, prior knowledge, and self-concept. Forty-three 7th graders read an illustrated science text while their eye movements were recorded. A cluster analysis revealed two processing patterns during the first-pass reading, which differed for the time spent on the main concepts in the text and picture. During re-reading, two patterns of stronger and weaker integrative processing emerged. Integration of verbal and graphical information was revealed by the frequency of second-pass transitions from text to picture and from picture to text, and the duration of picture re-inspecting while re-reading text information (look-from text to picture) and re-reading text information while re-inspecting the visualised information (look-from picture to text). A series of hierarchical regression analyses indicated that only the patterns of integrative processing during the second-pass reading uniquely predict verbal and graphical recalls, and the transfer of knowledge. The study provides evidence that the delayed processing which integrates text and graphics contributes to text retention and the application of newly learned knowledge, over and above individual characteristics. The educational significance is outlined.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Ainsworth, S. (2006). DeFT: A conceptual framework for considering learning with multiple representations. Learning and Instruction, 16(3), 183–198. doi:10.1016/j.learninstruc.2006.03.001.

    Article  Google Scholar 

  • Alexander, P. A. (2012). Reading into the future: Competence for the 21st century. Educational Psychologist, 47(4), 259–280. doi:10.1080/00461520.2012.722511.

    Article  Google Scholar 

  • Bartholomé, T., & Bromme, R. (2009). Coherence formation when learning from text and pictures: What kind of support for whom? Journal of Educational Psychology, 101(2), 282–293. doi:10.1037/a0014312.

  • Bråten, I., Ferguson, L. E., Anmarkrud, O., & Strømsø, H. I. (2013). Prediction of learning and comprehension when adolescents read multiple texts: The roles of word-level processing, strategic approach, and reading motivation. Reading and Writing, 26(3), 321–348.

    Article  Google Scholar 

  • Butcher, K. R. (2006). Learning from text with diagrams: Promoting mental model development and inference generation. Journal of Educational Psychology, 98, 182–197. doi:10.1037/0022-0663.98.1.182

  • Butcher, K. R. (2014). The multimedia principle. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 174–205). New York: Cambridge University Press.

    Chapter  Google Scholar 

  • Cornoldi, C., & Colpo, G. (1995). Nuove prove MT per la scuola media [New MT tests of reading comprehension for the middle school] Florence. Italy: Organizzazioni Speciali.

    Google Scholar 

  • Cromley, J. G., Snyder-Hogan, L. E., & Luciw-Dubas, U. A. (2010a). Cognitive activities in complex science text and diagrams. Contemporary Educational Psychology, 35(1), 59–74.

    Article  Google Scholar 

  • Cromley, J. G., Snyder-Hogan, L. E., & Luciw-Dubas, U. A. (2010b). Reading comprehension of scientific text: A domain-specific test of the direct and inferential mediation model of reading comprehension. Journal of Educational Psychology, 102, 687–700. doi:10.1037/a0019452.

    Article  Google Scholar 

  • Eitel, A., Scheitel, K. (2014). Picture or text first? Explaining sequence effects when learning with pictures and text. Educational Psychology Review. doi:0.1007/s10648-014-9264-4

  • Eitel, A., Scheitel, K., & Schüler, A. (2013). How inspecting a picture affects processing of text in multimedia learning. Applied Cognitive Psychology, 27(4), 451–461. doi:10.1002/acp.2922.

    Article  Google Scholar 

  • Eitel, A., Scheitel, K., Schüler, A., Nyström, M., & Holmqvist, K. (2014). How a picture facilitates the process of learning from text: Evidence for scaffolding. Learning and Instruction, 28, 48–63.

    Article  Google Scholar 

  • Florax, M., & Ploetzner, R. (2010). What contributes to the split-attention effect? The role of text segmentation, picture labelling, and spatial proximity. Learning and Instruction, 20(3), 216–224. doi:10.1016/j.learninstruc.2009.02.021.

    Article  Google Scholar 

  • Hannus, M., & Hyönä, J. (1999). Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary Educational Psychology, 24, 95–123. doi:10.1006/ceps.1998.0987.

    Article  Google Scholar 

  • Hegarty, M., & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal of Memory and Language, 32, 717–742. doi:10.1006/jmla.1993.1036

  • Hyönä, J. (2010). The use of eye movements in the study of multimedia learning. Learning and Instruction, 20, 172–176. doi:10.1016/j.learninstruc.2009.02.013.

    Article  Google Scholar 

  • Hyönä, J., Lorch, R. F., & Kaakinen, J. (2002). Individual differences in reading to summarize expository text: Evidence from eye fixation patterns. Journal of Educational Psychology, 94, 44–55. doi:10.1037/0022-0663.94.1.44.

    Article  Google Scholar 

  • Hyönä, J., & Nurminen, A.-M. (2006). Do adult readers know how they read? Evidence from eye movement patterns and verbal reports. British Journal of Educational Psychology, 97(1), 31–50. doi:10.1348/000712605x53678.

    Google Scholar 

  • Johnson, C. I., & Mayer, R. E. (2012). An eye movement analysis of the spatial contiguity effect in multimedia learning. Journal of Experimental psychology: Applied, 18(2), 178–191. doi:10.1037/a0026923.

    Google Scholar 

  • Katzir, T., Lesaux, N. K., & Kim, Y.-S. (2009). The role of reading self-concept and home literacy practices in fourth grade reading comprehension. Reading and Writing, 22(3), 261–279. doi:10.1007/s11145-007-9112-8.

    Article  Google Scholar 

  • Kendeou, P., & van den Broek, P. (2007). The effects of prior knowledge and text structure on comprehension processes during reading of scientific texts. Memory and Cognition, 35, 1567–1577. doi:10.3758/BF03193491.

    Article  Google Scholar 

  • Kim, Y.-S., Petscher, Y., & Foorman, B. (2013). The unique relation of silent reading fluency to end-of-year reading comprehension: Understanding individual differences at the student, classroom, school, and district levels. Reading and Writing,. doi:10.1007/s11145-013-9455-2.

    Google Scholar 

  • Marsh, H. W. (1990). The structure of academic self-concept: The Marsh-Shavelson model. Journal of Educational Psychology, 82(4), 623–633. doi:10.1037/0022-0663.82.4.623.

    Article  Google Scholar 

  • Marsh, H. W., Trautwein, U., Lütke, O., Köller, O., & Baumert, J. (2005). Academic self-concept, interest, grades, and standardized test scores: Reciprocal effects models of causal ordering. Child Development, 76(2), 397–416. doi:10.1111/j.1467-8624.2005.00853.x.

    Article  Google Scholar 

  • Mason, L., Boscolo, P., Tornatora, M. C., & Ronconi, L. (2013a). Besides knowledge: A cross-sectional study on the relations between epistemic beliefs, achievement goals, self-beliefs, and achievement in science. Instructional Science, 41(1), 49–79. doi:10.1007/s11251-012-9210-0.

    Article  Google Scholar 

  • Mason, L., Pluchino, P., & Tornatora, M. C. (2013b). Effects of picture labeling on illustrated science text processing and learning: Evidence from eye movements. Reading Research Quarterly, 48(2), 199–214. doi:10.1002/rrq.41.

    Article  Google Scholar 

  • Mason, L., Pluchino, P., & Tornatora, M. C. (2015). Eye-movement modeling of text and picture integration during reading: Effects on processing and learning. Contemporary Educational Psychology,. doi:10.1016/j.cedpsych.2015.01.004.

    Google Scholar 

  • Mason, L., Pluchino, P., Tornatora, M. C., & Ariasi, N. (2013c). An eye-tracking study of learning from science text with concrete and abstract illustrations. Journal of Experimental Education, 81(3), 356–384. doi:10.1080/00220973.2012.727885.

    Article  Google Scholar 

  • Mason, L., Tornatora, M. C., & Pluchino, P. (2013d). Do fourth graders integrate text and picture in processing and learning from an illustrated science text? Evidence from eye-movement patterns. Computers & Education, 60(1), 95–109. doi:10.1016/j.compedu.2012.07.011.

    Article  Google Scholar 

  • Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press.

    Book  Google Scholar 

  • Mayer, R. E. (2010). Unique contributions of eye-tracking research to the study of learning with graphics. Learning and Instruction, 20, 167–171. doi:10.1016/j.learninstruc.2009.02.012.

    Article  Google Scholar 

  • Mayer, R. E. (2014). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 43–71). New York: Cambridge University Press.

    Chapter  Google Scholar 

  • McNamara, D. S., & Kintsch, W. (1996). Learning from texts: Effects of prior knowledge and text coherence. Discourse Processes, 22, 247–288. doi:10.1080/01638539609544975.

    Article  Google Scholar 

  • Mikkilä-Erdmann, M., Penttinen, M., Anto, E., & Olkinuora, E. (2008). Constructing mental models during learning from science text. Eye tracking methodology meets conceptual change. In D. Ifenthaler, P. Pirnay-Dummer, & J. Michael Spector (Eds.), Understanding models for learning and instruction: Essays in honor of Norbert M. Seel. (pp. 63–79). New York: Springer.

  • Norman, R. R. (2012). Reading the graphics: What is the relationship between graphical reading processes and student comprehension? Reading and Writing, 25, 739–774. doi:10.1007/s11145-011-9298-7.

    Article  Google Scholar 

  • Ozuru, Y., Dempsey, K., & McNamara, D. (2009). Prior knowledge, reading skill, and text cohesion in the comprehension of science texts. Learning and Instruction, 19(3), 228–242. doi:10.1016/j.learninstruc.2008.04.003.

    Article  Google Scholar 

  • Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62(8), 1457–1506. doi:10.1080/17470210902816461.

    Article  Google Scholar 

  • Rayner, K., Chace, K. H., Slattery, T. J., & Ashby, J. (2006). Eye movements as reflections of comprehension processes in reading. Scientific Studies of Reading, 10(3), 241–255. doi:10.1207/s1532799xssr1003_3.

    Article  Google Scholar 

  • Schellings, G., Aarnoutse, C., & van Leeuwe, J. (2006). Third-graders’ think-aloud protocols: Types of reading activities in reading an expository text. Learning and Instruction, 16(6), 549–568. doi:10.1016/j.learninstruc.2006.10.004.

    Article  Google Scholar 

  • Schlag, S., & Ploetzner, R. (2011). Supporting learning from illustrated texts: Conceptualizing and evaluating a learning strategy. Instructional Science, 39(6), 921–937. doi:10.1007/s11251-010-9160-3.

    Article  Google Scholar 

  • Schnotz, W. (2014). Integrated model of text and picture comprehension. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 72–103). New York: Cambridge University Press.

    Chapter  Google Scholar 

  • Schnotz, W., & Bannert, M. (2003). Construction and interference in learning from multiple representations. Learning and Instruction, 13, 141–156. doi:10.1016/S0959-4752(02)00017-8.

    Article  Google Scholar 

  • Schnotz, W., Ludewig, U., Ulrich, M., Horz, H., McElvany, N., & Baumert, J. (2014). Strategy shifts during learning from texts and picture. Journal of Educational Psychology. Advance online publication. doi/10.1037/a0037054

  • Schroeder, S., Richter, T., McElvany, N., Hachfeld, A., Baumert, J., Schnotz, W., et al. (2011). Teachers’ beliefs, instructional behaviors, and students’ engagement in learning from texts with instructional pictures. Learning and Instruction, 21(3), 403–415. doi:10.1016/j.learninstruc.2010.06.001.

    Article  Google Scholar 

  • Schunk, D. A., & Pajares, F. (2005). Competence perceptions. In A. J. Elliot & C. S. Dweck (Eds.), Handbook of competence and motivation (pp. 85–104). New York: Guilford Press.

    Google Scholar 

  • Serra, M. J., & Dunlosky, J. (2010). Metacognitive judgments reflect the belief that diagrams improve learning from text. Memory, 18(7), 698–711. doi:10.1080/09658211.2010.506441.

    Article  Google Scholar 

  • Stalbovs, K., Eitel, A., & Scheiter, K. (2013). Which cognitive processes predict successful learning with multimedia? A comparison of eye tracking parameters. Paper presented at the 15th conference of the European Association for Research on Learning and Instruction, Munich, Germany.

  • Taboada, A., Tonks, S. M., Wigfiled, A., & Guthrie, J. T. (2009). Effects of motivational and cognitive variables on reading comprehension. Reading and Writing, 22(1), 85–106. doi:10.1007/s11145-008-9133-y.

    Article  Google Scholar 

  • Trautwein, U., Lüdtke, O., Köller, O., & Baumert, J. J. (2006). Self-esteem, academic self-concept, and achievement: How the learning environment moderates the dynamics of self-concept. Journal of Personality and Social Psychology, 90(2), 334–349. doi:10.1037/0022-3514.90.2.334.

    Article  Google Scholar 

  • van Gog, T., & Scheiter, K. (2010). Eye tracking as a tool to study and enhance multimedia learning. Learning and Instruction, 20, 95–99. doi:10.1016/j.learninstruc.2009.02.009.

    Article  Google Scholar 

Download references

Author information

Author notes

  1. Patrik Pluchino

    Present address: Department of General Psychology, University of Padova, via Venezia 8, 35131, Padova, Italy

Authors and Affiliations

  1. Department of Developmental Psychology and Socialization, University of Padova, via Venezia 8, 35131, Padova, Italy

    Lucia Mason, Maria Caterina Tornatora & Patrik Pluchino

Authors
  1. Lucia Mason
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Caterina Tornatora
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Patrik Pluchino
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lucia Mason.

Additional information

The study is part of a research project on learning difficulties in the science domain funded by a grant to the first author (STPD08HANE_001) from the University of Padova, Italy, under the funding program for “Strategic Projects”. We are very grateful to all the students, their parents and teachers, and the school principal, who made this study possible.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mason, L., Tornatora, M.C. & Pluchino, P. Integrative processing of verbal and graphical information during re-reading predicts learning from illustrated text: an eye-movement study. Read Writ 28, 851–872 (2015). https://doi.org/10.1007/s11145-015-9552-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11145-015-9552-5

Keywords

Search

Navigation