Skip to main content
SpringerLink
Log in

Focus of attention and choice of text modality in multimedia learning

  • Published:
European Journal of Psychology of Education Aims and scope Submit manuscript

Abstract

The term ‘modality effect’ in multimedia learning means that students learn better from pictures combined with spoken rather than written text. The most prominent explanations refer to the split attention between visual text reading and picture observation which could affect transfer of information into working memory, maintenance of information in working memory or the effective size of working memory. The assumption of a continuous need for split attention is questionable, however. Learners can keep pictorial information in working memory, when they have seen the picture before, especially if they have higher prior knowledge. Instead of suffering from a permanent split attention, learners frequently show tendencies to simply ignore pictures. This suggests guiding learners towards picture analysis by picture-related text paragraphs. We assume that these paragraphs are associated with stronger modality effects than content-related paragraphs, especially if the pictures are new to learners. These assumptions were tested in an experiment with 120 students learning about volcanism from illustrated text consisting of segments each including a content-related paragraph followed by a picture-related paragraph describing the accompanying visualization. Content-related and picture-related paragraphs were presented as visual or auditory texts leading to 2x2 conditions of text presentation. Picture novelty was manipulated by presenting a picture throughout the whole segment or only when the picture-related paragraph was read. As expected, picture-related paragraphs were associated with stronger modality effects than content-related paragraphs if picture novelty is high. The distinction between different kinds of paragraphs seems to be important for the prediction of modality effects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Amstad, T. (1978). Wie verständlich sind unsere Zeitungen. Dissertation. Universität Zürich.

  • Ayres, P., & Sweller, J. (2005). The split-attention principle in multimedial learning. In R. E. Mayer (Ed.), The cambridge handbook of multimedia learning (pp. 135–146). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • Baddeley, A. D. (1986). Working memory. Oxford: Oxford University Press.

    Google Scholar 

  • Baddeley, A. D., & Logie, R. H. (1999). Working memory: The multiple-component model. In A. Miyake & P. Shah (Eds.), Models of working memory: Mechanisms of active maintenance and executive control (pp. 28–61). Cambridge: Cambridge Univ. Press.

    Chapter  Google Scholar 

  • Bernard, R. M. (1990). Using extended captions to improve learning from instructional text. British Journal of Educational Technology, 21(3), 215–225.

    Article  Google Scholar 

  • Chafe, W. L. (1994). Discourse, consciousness, and time. Chicago: University of Chicago Press.

    Google Scholar 

  • Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8(4), 293–332.

    Google Scholar 

  • Clements, P. (1979). The effects of staging on recall from prose. In R. O. Freedle (Ed.), New directions in discourse processing. Vol. II in the series Advances in discourse processes (pp. 287–330). Norwood: Ablex.

    Google Scholar 

  • Cowan, N. (2001). The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87–185.

    Article  Google Scholar 

  • Danks, J. H. (1980). Comprehension in listening and reading: Same or different? In J. Danks & K. Pezdek (Eds.), Reading and understanding (pp. 25–40). Newark, Delware: International Reading Association.

    Google Scholar 

  • Danks, J. H., & End, L. J. (1987). Processing strategies for reading and listening. In R. Horowitz & S. J. Samuels (Eds.), Comprehending oral and written language (pp. 271–294). San Diego: Academic Press.

    Google Scholar 

  • Diakidoy, I. A., Stylianou, P., Karefillidou, C., & Papageorgiou, P. (2012). The relationship between listening and reading comprehension of different types of text at increasing grade levels. Reading Psychology. doi:10.1080/02702710590910584.

    Google Scholar 

  • Ellis, A. W., & Young, A. W. (1996). Human cognitive neuropsychology. A textbook with readings. Hove: Psychology Press.

    Google Scholar 

  • Engelkamp, J. (1982). Given and new information: Theoretical positions and empirical evidence. Saarbrücken: Universität des Saarlandes, Fachrichtung Psychologie im Fachbereich 6, Report No.79.

  • Ferreira, F., & Anes, M. (1994). Why study spoken language? In M. A. Gernsbacher (Ed.), Handbook of psycholinguistics (pp. 34–36). San Diego: Academic Press.

    Google Scholar 

  • Ginns, P. (2005). Meta-analysis of the modality effect. Learning and Instruction, 15, 313–331. doi:10.1016/j.learninstruc.2005.07.001.

    Article  Google Scholar 

  • Givón, T. (Ed.). (1983). Topic continuity in discourse: A quantitative cross-language study. Amsterdam: Benjamins.

    Google Scholar 

  • Green, R. (1981). Remembering ideas form text: the effect of modality of presentation. British Journal of Educational Psychology, 51, 83–89.

    Article  Google Scholar 

  • Grimes, J. E. (1975). The thread of discourse. The Hague: Mouton.

    Google Scholar 

  • Gyselinck, V., Jamet, E., & Dubois, V. (2008). The role of working memory components in multimedia comprehension. Applied Cognitive Psychology, 22, 353–374.

    Article  Google Scholar 

  • Halliday, M. A. K. (1970). Language structure and language function. In J. Lyons (Ed.), New horizons in linguistics (pp. 140–164). Harmondsworth: Penguin.

    Google Scholar 

  • Hannus, M., & Hyönä, J. (1999). Utilization of illustrations during learning of science: textbook passages among low- and high-ability children. Contemporary Educational Psychology, 24, 95–123.

    Article  Google Scholar 

  • Haviland, S. E., & Clark, H. H. (1974). What’s new? Acquiring new information as a process in comprehension. Journal of Verbal Learning and Verbal Behavior, 13, 512–521.

    Article  Google Scholar 

  • Hegarty, M., & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal of Memory and Language, 32, 717–742.

    Article  Google Scholar 

  • Hron, A., Kurbjuhn, I., Mandl, H., & Schnotz, W. (1985). Structural inferences in reading and listening. In G. Rickheit & H. Strohner (Eds.), Inferences in text processing (pp. 221–245). Amsterdam: North-Holland.

    Chapter  Google Scholar 

  • Jeung, H.-J., Chandler, P., & Sweller, J. (1997). The role of visual indicators in dual sensory mode instruction. Educational Psychology, 17, 329–343. doi:10.1080/0144341970170307.

    Article  Google Scholar 

  • Kalyuga, S. (2000). When using sound with a text or picture is not beneficial for learning. Australian Journal of Educational Technology, 16(2), 161–172.

    Google Scholar 

  • Kalyuga, S., Chandler, P., & Sweller, J. (1998). Levels of expertise and instructional design. Human Factors, 40(1), 1–17.

    Article  Google Scholar 

  • Kalyuga, S., Chandler, P., & Sweller, J. (2000). Incorporating learner experience into the design of multimedia instruction. Journal of Educational Psychology, 92, 126–136.

    Article  Google Scholar 

  • Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38, 23–31.

    Article  Google Scholar 

  • Kozma, R. (2003). The material features of multiple representations and their cognitive and social affordances for science understanding. Learning and Instruction, 13, 205–226.

    Article  Google Scholar 

  • Kulhavy, R. W., Lee, B. J., & Caterino, L. C. (1985). Conjoint retention of maps and related discourse. Contemporary Educational Psychology, 10, 28–37.

    Article  Google Scholar 

  • Kürschner, C., & Schnotz, W. (2008). Das Verhältnis gesprochener und geschriebener Sprache bei der Konstruktion mentaler Repräsentationen. Psychologische Rundschau, 59(3), 139–149.

    Article  Google Scholar 

  • Landerl, K. (2001). Lesegeschwindigkeitstest (national und international). In G. Haider & B. Lang (Eds.), PISA PLUS 2000 (pp. 119–130). Studien Verlag: Innsbruck.

    Google Scholar 

  • Leahy, W., & Sweller, J. (2011). Cognitive load theory, modality of presentation and the transient information effect. Applied Cognitive Psychology, 25(6), 943–951.

    Article  Google Scholar 

  • Leahy, W., Chandler, P., & Sweller, J. (2003). When auditory presentations should and should not be a component of multimedia instruction. Applied Cognitive Psychology, 17(4), 401–418. doi:10.1002/Acp.877.

    Article  Google Scholar 

  • Levie, H. W., & Lentz, R. (1982). Effects of text illustrations: a review of research. Educational Communication and Technology Journal, 30, 195–232.

    Google Scholar 

  • Levin, J. R., Anglin, G. J., & Carney, R. N. (1987). On empirically validating functions of pictures in prose. In H. A. Houghton & D. M. Willows (Eds.), The psychology of illustration (pp. 51–86). New York: Springer.

    Chapter  Google Scholar 

  • Lindow, S., Fuchs, H. M., Fürstenberg, A., Kleber, J., Schweppe, J., & Rummer, R. (2011). On the robustness of the modality effect: Attempting to replicate a basic finding. Zeitschrift für Pädagogische Psychologie, 25(4), 231–243.

    Google Scholar 

  • Low, R., & Sweller, J. (2005). The modality principle in multimedia learning. In R. E. Mayer (Ed.), The cambridge handbook of multimedia learning (pp. 147–158). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • Lowe, R. K. (1999). Extracting information from an animation during complex visual learning. European Journal of Psychology of Education, 14, 225–244.

    Article  Google Scholar 

  • Lowe, R. K. (2004). Interrogation of a dynamic visualisation during learning. Learning and Instruction, 14, 257–274.

    Article  Google Scholar 

  • Massaro, D. W. (1979). Reading and listening. In P. A. Kolers, M. E. Wrolstad, & H. Bouma (Eds.), Processing of visible language (pp. 331–370). New York: Plenum Press.

    Chapter  Google Scholar 

  • Mayer, R. E. (1997). Multimedia learning: are we asking the right questions? Educational Psychologist, 32, 1–19.

    Article  Google Scholar 

  • Mayer, R. E. (2005). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 31–48). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press.

    Book  Google Scholar 

  • Mayer, R. E., & Moreno, R. (1998). Split-attention effect in multimedia learning: evidence for dual processing systems in working memory. Journal of Educational Psychology, 90, 312–320.

    Article  Google Scholar 

  • Mayrath, M. C., Nihalani, P. K., & Robinson, D. H. (2011). Varying tutorial modality and interface restrictions to maximize transfer in a complex simulation environment. Journal of Educational Psychology, 103, 257–268.

    Article  Google Scholar 

  • McDonald, T., & Thornley, C. (2002). Unlocking meaning in text: some thoughts on the literacy challenges our students face. English in Aotearoa, 48, 54–60.

    Google Scholar 

  • Miller, G. A. (1956). The magical number seven, plus or minus two: some limits on our capacity to process information. Psychological Review, 63, 81–97.

    Article  Google Scholar 

  • Mokros, J. R., & Tinker, R. F. (1987). The Impact of microcomputer-based labs on children's ability to interpret graphs. Journal of Research in Science Teaching, 24, 369–383.

    Article  Google Scholar 

  • Moreno, R., & Durán, R. (2004). Do multiple representations need explanations? The role of verbal guidance and individual differences in multimedia mathematics learning. Journal of Educational Psychology, 96(3), 492–503.

    Article  Google Scholar 

  • Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: the role of modality and contiguity. Journal of Educational Psychology, 91, 358–368.

    Article  Google Scholar 

  • Mousavi, S. Y., Low, R., & Sweller, J. (1995). Reducing cognitive load by mixing auditory and visual presentation modes. Journal of Educational Psychology, 87, 319–334. doi:10.1037/0022-0663.87.2.319.

    Article  Google Scholar 

  • Peterson, L., & Peterson, M. (1959). Short-term retention of individual verbal items. Journal of Experimental Psychology, 58, 193–198.

    Article  Google Scholar 

  • Ploetzner, R., Lowe, R., & Schlag, S. (2013). A systematic characterization of cognitive techniques for learning from textual and pictorial representations. Journal of Education and Learning, 2(2), 78–95.

    Article  Google Scholar 

  • Rieben, L., & Perfetti, C. (1991). Learning to read: Basic research and its implications. Hillsdale: Erlbaum.

    Google Scholar 

  • Rubin, D. L., Hafer, T., & Arata, K. (2000). Reading and listening to oral-based versus literate-based discourse. Communication Education, 49(2), 121–133.

    Article  Google Scholar 

  • Rummer, R., Schweppe, J., Fürstenberg, A., Seufert, T., & Brünken, R. (2010). Working memory interference during processing texts and pictures: implications for the explanation of the modality effect. Applied Cognitive Psychology, 24, 164–176.

    Article  Google Scholar 

  • Schmidt-Weigand, F. (2011). Does animation amplify the modality effect – or is there any modality effect at all? Zeitschrift für Pädagogische Psychologie.

  • Schmidt-Weigand, F., Kohnert, A., & Glowalla, U. (2010a). Explaining the modality and contiguity effects: New insights from investigating Students' viewing behaviour. Applied Cognitive Psychology, 24, 226–237. doi:10.1002/Acp.1554.

    Article  Google Scholar 

  • Schmidt-Weigand, F., Kohnert, A., & Glowalla, U. (2010b). A closer look at split visual attention in system- and self-paced instruction in multimedia learning. Learning and Instruction, 20, 100–110. doi:10.1016/j.learninstruc.2009.02.011.

    Article  Google Scholar 

  • Schnotz, W. (2005). An integrated model of text and picture comprehension. In R. E. Mayer (Ed.), Cambridge handbook of multimedia learning (pp. 49–69). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • Schnotz, W. (2011). Colorful bouquets in multimedia research: a closer look at the modality effect. Zeitschrift für Pädagogische Psychologie, 25, 269–276.

    Article  Google Scholar 

  • Schüler, A., Scheiter, K., & Schmidt-Weigand, F. (2011). Boundary conditions and constraints of the modality effect. Zeitschrift für Pädagogische Psychologie, 25, 211–220.

    Article  Google Scholar 

  • Sticht, T. G., & James, J. H. (1984). Listening and reading. In P. D. Pearson, R. Barr, M. L. Kamil, & P. Mosenthal (Eds.), Handbook of reading research (pp. 293–317). New York: Longmann.

    Google Scholar 

  • Stiller, K. D., Freitag, A., Zinnbauer, P., & Freitag, C. (2009). How pacing of multimedia instructions can influence modality effects: a case of superiority of visual texts. Australasian Journal of Educational Technology, 25, 184–203.

    Google Scholar 

  • Tabbers, H. K., & van der Spoel, W. (2011). Where did the modality principle in multimedia learning go? A double replication failure that questions both theory and practical use. Zeitschrift für Pädagogische Psychologie, 25(4), 221–230.

    Article  Google Scholar 

  • Tabbers, H. K., Martens, R. L., & van Merriënboer, J. J. G. (2004). Multimedia instructions and cognitive load theory: effects of modality and cueing. British Journal of Educational Psychology, 74, 71–81.

    Article  Google Scholar 

  • Tindall-Ford, S., Chandler, P., & Sweller, J. (1997). When two sensory modes are better than one. Journal of Experimental Psychology: Applied, 3, 257–287.

    Google Scholar 

  • Weidenmann, B. (1989). When good pictures fail: An information-processing approach to the effects of illustrations. In H. Mandl & J. R. Levin (Eds.), Knowledge acquisition from text and pictures (pp. 157–170). Amsterdam: Elsevier.

    Chapter  Google Scholar 

  • Yeung, A. S., Jin, P., & Sweller, J. (1997). Cognitive load and learner expertise: split-attention and redundancy effects in reading with explanatory notes. Contemporary Educational Psychology, 23, 1–21.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Psychology, General and Educational Psychology, University of Koblenz-Landau, Fortstraße 7, 76829, Landau, Germany

    Wolfgang Schnotz & Christiane Baadte

  2. Philosophical Faculty II, University of Würzburg, Oswald-Külpe-Weg 82, 97074, Würzburg, Germany

    Christoph Mengelkamp

  3. Communication and Environment, Rhine-Waal University of Applied Sciences, Südstraße 8, 47475, Kamp-Lintfort, Germany

    Georg Hauck

Authors
  1. Wolfgang Schnotz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christoph Mengelkamp
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christiane Baadte
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Georg Hauck
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wolfgang Schnotz.

Additional information

Wolfgang Schnotz. Faculty of Psychology, General and Educational Psychology, University of Koblenz-Landau, Fortstraße 7, 76829 Landau, Germany. schnotz@uni-landau.de. Phone: +49-634128034234; Fax: +49-634128034240; Email: schnotz@uni-landau.de

Current themes of research:

Text-Picture Integration, Knowledge Acquisition from Multiple Representations, Knowledge Acquisition with Conflicting Information, Multimedia Learning, Learning with Hypermedia, Learning with Animation

Most relevant publications in the field of Psychology of Education:

Schnotz, W. (2011). Colorful Bouquets in Multimedia Research: A Closer Look at the Modality Effect. Zeitschrift für Pädagogische Psychologie. 25, 269–276.

Schnotz, W., Baadte, C., Müller, A., & Rasch, R. (2010). Creative thinking and problem solving with depictive and descriptive representations. . In L. Verschaffel, E. De Corte, T. de Jong & J. Elen (Eds.) Use of representations in reasoning and problem solving (pp. 11–35). London: Routledge.

Schnotz, W., & Heiß, A. (2009). Semantic Scaffolds in Hypermedia Learning Environments. Computers in Human Behavior, 25(2), 371–380.

Schnotz, W. & Kürschner, C. (2007). A reconsideration of cognitive load theory. Educational Psychology Review, 19(4), 469–508.

Schnotz, W. & Lowe, R.K. (2008). A unified view of learning from animated and static graphics. In R.K. Lowe & W. Schnotz (Eds.), Learning with animation. Research implications for design (pp. 304–356). New York: Cambridge University Press.

Christoph Mengelkamp. University of Würzburg, Philosophical Faculty II, Oswald-Külpe-Weg 82, 97074 Würzburg, Germany. christoph.mengelkamp@uni-wuerzburg.de

Current themes of research:

metacognition in self-regulated learning, interactive animations, learning from pictures and texts, educational diagnostics and methodology

Most relevant publications in the field of Psychology of Education:

Mengelkamp, C., & Bannert, M. (2012). Confidence judgments in learning. In N. M. Seel (Ed.), Encyclopledia of the sciences of learning (pp. 756–759). New York: Springer. doi:10.1007/978-1-4419-1428-6 doi:10.1007/978-1-4419-1428-6#_blank

Mengelkamp, C., & Bannert, M. (2010). Accuracy of confidence judgments: Stability and generality in the learning process and predictive validity for learning outcome. Memory and Cognition, 38, 441–451. doi:10.3758/MC.38.4.441 doi:10.3758/MC.38.4.441#_blank

Bannert, M., Hildebrand, M., & Mengelkamp, C. (2009). Effects of a metacognitive support device in learning environments. Computers in Human Behaviour, 25, 829–835. doi:10.1016/j.chb.2008.07.002 doi:10.1016/j.chb.2008.07.002#_blank

Mengelkamp, C., & Bannert, M. (2009). Judgements about knowledge: searching for factors that influence their validity. Electronic Journal of Research in Educational Psychology, 7(1), 163–190.

Bannert, M., & Mengelkamp, C. (2008). Assessment of metacognitive skills by means of instruction to think aloud and reflect when prompted. Does the verbalisation method affect learning? Metacognition and Learning, 3, 39–58. doi:10.1007/s11409-007-9009-6 doi:10.1007/s11409-007-9009-6#_blank .

Christiane Baadte. University of Koblenz-Landau, Faculty of Psychology, Fortstraße 7, 76829 Landau, Germany. baadte@uni-landau.de

Current themes of research:

Diagnostic and didactic competencies of teachers, feedback with self-regulated learning with text-picture integration, working memory and multimedia learning

Most relevant publications in the field of Psychology of Education:

Baadte, C. & Dutke, S. (in press). Learning about persons: The effects of text structure and executive capacity on conceptual change. European Journal of Psychology of Education. doi:10.1007/s10212-012-0153-2

Baadte, C. & Dutke, S. (in press). Learning About Persons: The Effects of Text Structure and Executive Capacity on Conceptual Change. European Journal of Psychology of Education.

Baadte, C. & Schnotz, W. (in press). Feedback Effects on performance, motivation and mood: Are they moderated by the learner's self-concept? Scandinavian Journal of Educational Research.

Baadte, C. & Schnotz, W. (2012). Grundlagen des Verstehens von Texten mit Bildern. Weiterbildung, 6, S.35-38.

Dutke, S., Baadte, C., Hähnel, A., von Hecker, U., & Rinck, M. (2010). Using diagnostic text information to constrain situation models. Discourse Processes, 47 (6), 510–544.

Georg Hauck. Rhine-Waal University of Applied Sciences, Communication and Environment, Südstraße 8, 47475 Kamp-Lintfort, Germany. georg.hauck@hochschule-rhein-waal.de

Current themes of research:

Instructional media, communication and organizational counseling

Most relevant publications in the field of Psychology of Education:

Kürschner, C., Seufert, T., Hauck, G., Schnotz, W. & Eid, M. (2006). Konstruktion visuell-räumlicher Repräsentationen beim Hör- und Leseverstehen. Zeitschrift für Psychologie, 214(3), S. 117--132.

Kürschner, C., Schnotz, W., Eid, M. & Hauck, G. (2005). Individuelle Modalitätspräferenzen beim Textverstehen. Zeitschrift für Entwicklungspsychologie und Pädagogische Psychologie, 37(1), 2--16.

Hauck, G. (2008). Der Aufbau von Wissensstrukturen mit Bildern. Saarbrücken: VDM.

Hauck, G. (2005). Effekte von Modalitätspräferenzen beim Wissenserwerb mit multimedialen Lernsystemen. Berlin: Logos.

Appendix

Appendix

Table 3 Number of words, maximum processing time, and readability scores of the content-related and the picture-related paragraphs in the experiment
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Schnotz, W., Mengelkamp, C., Baadte, C. et al. Focus of attention and choice of text modality in multimedia learning. Eur J Psychol Educ 29, 483–501 (2014). https://doi.org/10.1007/s10212-013-0209-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10212-013-0209-y

Keywords

Search

Navigation