Reading and Writing

, Volume 28, Issue 6, pp 851–872 | Cite as

Integrative processing of verbal and graphical information during re-reading predicts learning from illustrated text: an eye-movement study

  • Lucia MasonEmail author
  • Maria Caterina Tornatora
  • Patrik Pluchino


Printed or digital textbooks contain texts accompanied by various kinds of visualisation. Successful comprehension of these materials requires integrating verbal and graphical information. This study investigates the time course of processing an illustrated text through eye-tracking methodology in the school context. The aims were to identify patterns of first- and second-pass reading and to examine whether the integrative processing of text and picture during the less automatic and more purposeful second-pass reading predicts learning, after controlling for reading comprehension, prior knowledge, and self-concept. Forty-three 7th graders read an illustrated science text while their eye movements were recorded. A cluster analysis revealed two processing patterns during the first-pass reading, which differed for the time spent on the main concepts in the text and picture. During re-reading, two patterns of stronger and weaker integrative processing emerged. Integration of verbal and graphical information was revealed by the frequency of second-pass transitions from text to picture and from picture to text, and the duration of picture re-inspecting while re-reading text information (look-from text to picture) and re-reading text information while re-inspecting the visualised information (look-from picture to text). A series of hierarchical regression analyses indicated that only the patterns of integrative processing during the second-pass reading uniquely predict verbal and graphical recalls, and the transfer of knowledge. The study provides evidence that the delayed processing which integrates text and graphics contributes to text retention and the application of newly learned knowledge, over and above individual characteristics. The educational significance is outlined.


Text processing Reading comprehension Integrative processing Multimedia learning Eye movements 


  1. Ainsworth, S. (2006). DeFT: A conceptual framework for considering learning with multiple representations. Learning and Instruction, 16(3), 183–198. doi: 10.1016/j.learninstruc.2006.03.001.CrossRefGoogle Scholar
  2. Alexander, P. A. (2012). Reading into the future: Competence for the 21st century. Educational Psychologist, 47(4), 259–280. doi: 10.1080/00461520.2012.722511.CrossRefGoogle Scholar
  3. Bartholomé, T., & Bromme, R. (2009). Coherence formation when learning from text and pictures: What kind of support for whom? Journal of Educational Psychology, 101(2), 282–293. doi: 10.1037/a0014312.
  4. Bråten, I., Ferguson, L. E., Anmarkrud, O., & Strømsø, H. I. (2013). Prediction of learning and comprehension when adolescents read multiple texts: The roles of word-level processing, strategic approach, and reading motivation. Reading and Writing, 26(3), 321–348.CrossRefGoogle Scholar
  5. Butcher, K. R. (2006). Learning from text with diagrams: Promoting mental model development and inference generation. Journal of Educational Psychology, 98, 182–197. doi: 10.1037/0022-0663.98.1.182
  6. Butcher, K. R. (2014). The multimedia principle. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 174–205). New York: Cambridge University Press.CrossRefGoogle Scholar
  7. Cornoldi, C., & Colpo, G. (1995). Nuove prove MT per la scuola media [New MT tests of reading comprehension for the middle school] Florence. Italy: Organizzazioni Speciali.Google Scholar
  8. Cromley, J. G., Snyder-Hogan, L. E., & Luciw-Dubas, U. A. (2010a). Cognitive activities in complex science text and diagrams. Contemporary Educational Psychology, 35(1), 59–74.CrossRefGoogle Scholar
  9. Cromley, J. G., Snyder-Hogan, L. E., & Luciw-Dubas, U. A. (2010b). Reading comprehension of scientific text: A domain-specific test of the direct and inferential mediation model of reading comprehension. Journal of Educational Psychology, 102, 687–700. doi: 10.1037/a0019452.CrossRefGoogle Scholar
  10. Eitel, A., Scheitel, K. (2014). Picture or text first? Explaining sequence effects when learning with pictures and text. Educational Psychology Review. doi:0.1007/s10648-014-9264-4Google Scholar
  11. Eitel, A., Scheitel, K., & Schüler, A. (2013). How inspecting a picture affects processing of text in multimedia learning. Applied Cognitive Psychology, 27(4), 451–461. doi: 10.1002/acp.2922.CrossRefGoogle Scholar
  12. Eitel, A., Scheitel, K., Schüler, A., Nyström, M., & Holmqvist, K. (2014). How a picture facilitates the process of learning from text: Evidence for scaffolding. Learning and Instruction, 28, 48–63.CrossRefGoogle Scholar
  13. Florax, M., & Ploetzner, R. (2010). What contributes to the split-attention effect? The role of text segmentation, picture labelling, and spatial proximity. Learning and Instruction, 20(3), 216–224. doi: 10.1016/j.learninstruc.2009.02.021.CrossRefGoogle Scholar
  14. Hannus, M., & Hyönä, J. (1999). Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary Educational Psychology, 24, 95–123. doi: 10.1006/ceps.1998.0987.CrossRefGoogle Scholar
  15. Hegarty, M., & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal of Memory and Language, 32, 717–742. doi: 10.1006/jmla.1993.1036
  16. Hyönä, J. (2010). The use of eye movements in the study of multimedia learning. Learning and Instruction, 20, 172–176. doi: 10.1016/j.learninstruc.2009.02.013.CrossRefGoogle Scholar
  17. Hyönä, J., Lorch, R. F., & Kaakinen, J. (2002). Individual differences in reading to summarize expository text: Evidence from eye fixation patterns. Journal of Educational Psychology, 94, 44–55. doi: 10.1037/0022-0663.94.1.44.CrossRefGoogle Scholar
  18. Hyönä, J., & Nurminen, A.-M. (2006). Do adult readers know how they read? Evidence from eye movement patterns and verbal reports. British Journal of Educational Psychology, 97(1), 31–50. doi: 10.1348/000712605x53678.Google Scholar
  19. Johnson, C. I., & Mayer, R. E. (2012). An eye movement analysis of the spatial contiguity effect in multimedia learning. Journal of Experimental psychology: Applied, 18(2), 178–191. doi: 10.1037/a0026923.Google Scholar
  20. Katzir, T., Lesaux, N. K., & Kim, Y.-S. (2009). The role of reading self-concept and home literacy practices in fourth grade reading comprehension. Reading and Writing, 22(3), 261–279. doi: 10.1007/s11145-007-9112-8.CrossRefGoogle Scholar
  21. Kendeou, P., & van den Broek, P. (2007). The effects of prior knowledge and text structure on comprehension processes during reading of scientific texts. Memory and Cognition, 35, 1567–1577. doi: 10.3758/BF03193491.CrossRefGoogle Scholar
  22. Kim, Y.-S., Petscher, Y., & Foorman, B. (2013). The unique relation of silent reading fluency to end-of-year reading comprehension: Understanding individual differences at the student, classroom, school, and district levels. Reading and Writing,. doi: 10.1007/s11145-013-9455-2.Google Scholar
  23. Marsh, H. W. (1990). The structure of academic self-concept: The Marsh-Shavelson model. Journal of Educational Psychology, 82(4), 623–633. doi: 10.1037/0022-0663.82.4.623.CrossRefGoogle Scholar
  24. Marsh, H. W., Trautwein, U., Lütke, O., Köller, O., & Baumert, J. (2005). Academic self-concept, interest, grades, and standardized test scores: Reciprocal effects models of causal ordering. Child Development, 76(2), 397–416. doi: 10.1111/j.1467-8624.2005.00853.x.CrossRefGoogle Scholar
  25. Mason, L., Boscolo, P., Tornatora, M. C., & Ronconi, L. (2013a). Besides knowledge: A cross-sectional study on the relations between epistemic beliefs, achievement goals, self-beliefs, and achievement in science. Instructional Science, 41(1), 49–79. doi: 10.1007/s11251-012-9210-0.CrossRefGoogle Scholar
  26. Mason, L., Pluchino, P., & Tornatora, M. C. (2013b). Effects of picture labeling on illustrated science text processing and learning: Evidence from eye movements. Reading Research Quarterly, 48(2), 199–214. doi: 10.1002/rrq.41.CrossRefGoogle Scholar
  27. Mason, L., Pluchino, P., & Tornatora, M. C. (2015). Eye-movement modeling of text and picture integration during reading: Effects on processing and learning. Contemporary Educational Psychology,. doi: 10.1016/j.cedpsych.2015.01.004.Google Scholar
  28. Mason, L., Pluchino, P., Tornatora, M. C., & Ariasi, N. (2013c). An eye-tracking study of learning from science text with concrete and abstract illustrations. Journal of Experimental Education, 81(3), 356–384. doi: 10.1080/00220973.2012.727885.CrossRefGoogle Scholar
  29. Mason, L., Tornatora, M. C., & Pluchino, P. (2013d). Do fourth graders integrate text and picture in processing and learning from an illustrated science text? Evidence from eye-movement patterns. Computers & Education, 60(1), 95–109. doi: 10.1016/j.compedu.2012.07.011.CrossRefGoogle Scholar
  30. Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press.CrossRefGoogle Scholar
  31. Mayer, R. E. (2010). Unique contributions of eye-tracking research to the study of learning with graphics. Learning and Instruction, 20, 167–171. doi: 10.1016/j.learninstruc.2009.02.012.CrossRefGoogle Scholar
  32. Mayer, R. E. (2014). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 43–71). New York: Cambridge University Press.CrossRefGoogle Scholar
  33. McNamara, D. S., & Kintsch, W. (1996). Learning from texts: Effects of prior knowledge and text coherence. Discourse Processes, 22, 247–288. doi: 10.1080/01638539609544975.CrossRefGoogle Scholar
  34. Mikkilä-Erdmann, M., Penttinen, M., Anto, E., & Olkinuora, E. (2008). Constructing mental models during learning from science text. Eye tracking methodology meets conceptual change. In D. Ifenthaler, P. Pirnay-Dummer, & J. Michael Spector (Eds.), Understanding models for learning and instruction: Essays in honor of Norbert M. Seel. (pp. 63–79). New York: Springer.Google Scholar
  35. Norman, R. R. (2012). Reading the graphics: What is the relationship between graphical reading processes and student comprehension? Reading and Writing, 25, 739–774. doi: 10.1007/s11145-011-9298-7.CrossRefGoogle Scholar
  36. Ozuru, Y., Dempsey, K., & McNamara, D. (2009). Prior knowledge, reading skill, and text cohesion in the comprehension of science texts. Learning and Instruction, 19(3), 228–242. doi: 10.1016/j.learninstruc.2008.04.003.CrossRefGoogle Scholar
  37. Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62(8), 1457–1506. doi: 10.1080/17470210902816461.CrossRefGoogle Scholar
  38. Rayner, K., Chace, K. H., Slattery, T. J., & Ashby, J. (2006). Eye movements as reflections of comprehension processes in reading. Scientific Studies of Reading, 10(3), 241–255. doi: 10.1207/s1532799xssr1003_3.CrossRefGoogle Scholar
  39. Schellings, G., Aarnoutse, C., & van Leeuwe, J. (2006). Third-graders’ think-aloud protocols: Types of reading activities in reading an expository text. Learning and Instruction, 16(6), 549–568. doi: 10.1016/j.learninstruc.2006.10.004.CrossRefGoogle Scholar
  40. Schlag, S., & Ploetzner, R. (2011). Supporting learning from illustrated texts: Conceptualizing and evaluating a learning strategy. Instructional Science, 39(6), 921–937. doi: 10.1007/s11251-010-9160-3.CrossRefGoogle Scholar
  41. Schnotz, W. (2014). Integrated model of text and picture comprehension. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 72–103). New York: Cambridge University Press.CrossRefGoogle Scholar
  42. Schnotz, W., & Bannert, M. (2003). Construction and interference in learning from multiple representations. Learning and Instruction, 13, 141–156. doi: 10.1016/S0959-4752(02)00017-8.CrossRefGoogle Scholar
  43. Schnotz, W., Ludewig, U., Ulrich, M., Horz, H., McElvany, N., & Baumert, J. (2014). Strategy shifts during learning from texts and picture. Journal of Educational Psychology. Advance online publication. doi/ 10.1037/a0037054
  44. Schroeder, S., Richter, T., McElvany, N., Hachfeld, A., Baumert, J., Schnotz, W., et al. (2011). Teachers’ beliefs, instructional behaviors, and students’ engagement in learning from texts with instructional pictures. Learning and Instruction, 21(3), 403–415. doi: 10.1016/j.learninstruc.2010.06.001.CrossRefGoogle Scholar
  45. Schunk, D. A., & Pajares, F. (2005). Competence perceptions. In A. J. Elliot & C. S. Dweck (Eds.), Handbook of competence and motivation (pp. 85–104). New York: Guilford Press.Google Scholar
  46. Serra, M. J., & Dunlosky, J. (2010). Metacognitive judgments reflect the belief that diagrams improve learning from text. Memory, 18(7), 698–711. doi: 10.1080/09658211.2010.506441.CrossRefGoogle Scholar
  47. Stalbovs, K., Eitel, A., & Scheiter, K. (2013). Which cognitive processes predict successful learning with multimedia? A comparison of eye tracking parameters. Paper presented at the 15th conference of the European Association for Research on Learning and Instruction, Munich, Germany.Google Scholar
  48. Taboada, A., Tonks, S. M., Wigfiled, A., & Guthrie, J. T. (2009). Effects of motivational and cognitive variables on reading comprehension. Reading and Writing, 22(1), 85–106. doi: 10.1007/s11145-008-9133-y.CrossRefGoogle Scholar
  49. Trautwein, U., Lüdtke, O., Köller, O., & Baumert, J. J. (2006). Self-esteem, academic self-concept, and achievement: How the learning environment moderates the dynamics of self-concept. Journal of Personality and Social Psychology, 90(2), 334–349. doi: 10.1037/0022-3514.90.2.334.CrossRefGoogle Scholar
  50. van Gog, T., & Scheiter, K. (2010). Eye tracking as a tool to study and enhance multimedia learning. Learning and Instruction, 20, 95–99. doi: 10.1016/j.learninstruc.2009.02.009.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  • Lucia Mason
    • 1
    Email author
  • Maria Caterina Tornatora
    • 1
  • Patrik Pluchino
    • 1
    • 2
  1. 1.Department of Developmental Psychology and SocializationUniversity of PadovaPadovaItaly
  2. 2.Department of General PsychologyUniversity of PadovaPadovaItaly

Personalised recommendations