Abstract
Multimedia resources in the twenty-first century primarily consist of verbal and audio-visual representations that, if properly processed, make it possible to create an integrated mental model of learning. Although these tools are commonly used in language education, successful use of multimedia learning resources in second language teaching and learning contexts remains a challenge. In particular, although much of how learners learn from verbal representations is understood, there is less insight into visual knowledge processing, either alone or in conjunction with texts in language teaching. This chapter aims to provide a set of theoretical frameworks and a realistic planning structure designed to inform the planning and design of more efficient digital content integration into language teaching and learning contexts. Based on a cognitive linguistics view aligned with dual coding and cognitive load theory, the chapter critically reviews and presents the differences in the way verbal and visual texts and pictures represent knowledge to second language learners. The next segment addresses issues in building a multimedia learning culture. Also, the role of individual differences in the processing of multimedia resources is highlighted. The chapter concludes with methodological recommendations on the crucial part of interdisciplinary analysis and evaluation methods in this field.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Rahman, S.M.: Multimedia Technologies: Concepts, Methodologies, Tools, and applications, vol. 1–3 (2008). https://doi.org/10.4018/978-1-59904-953-3
De Jesus Ferreira Nobre, A.M.: Multimedia technologies and online task-based foreign language teaching-learning. Tuning J. Higher Educ. 5(2), 75–97 (2018). https://doi.org/10.18543/tjhe-5(2)-2018pp75-97
Saracho, S.N.: Handbook of Research on the Education of Young Children, Fourth edn (2019). https://doi.org/10.4324/9780429442827
Park, C.W., Kim, D., Gook, Cho, S., Han, H.J.: Adoption of multimedia technology for learning and gender difference Comput. Hum. Behav. 92, 288–296 (2019). https://doi.org/10.1016/j.chb.2018.11.029
Pincus, H., Wojcieszak, M., Boomgarden, H.: Do multimedia matter? Cognitive and affective effects of embedded multimedia journalism. J. Mass Commun. Q. 94(3), 747–771 (2017). https://doi.org/10.1177/1077699016654679
Lindner, M.A., Eitel, A., Barenthien, J., Köller, O.: An integrative study on learning and testing with multimedia: effects on students’ performance and metacognition Learn. Instr. 71 (2021) https://doi.org/10.1016/j.learninstruc.2018.01.002
Vu, N.N.: Applying conceptual metaphor in cognitive linguistics for teaching English idioms. J. Sci. HCMC Univ. Educ. 17, 53–61 (2008)
Hung, B.P., Truong, V., Nguyen, N.V.: Students’ responses to CL-based teaching of English prepositions. Egitim Arastirmalari—Eurasian J. Educ. Res (73), 41–58 (2018). https://doi.org/10.14689/ejer.2018.73.3
Langacker, R.: Cognitive Grammar: A Basic Introduction (2008). https://doi.org/10.1093/acprof:oso/9780195331967.001.0001
Langacker, R.W.: Grammar and conceptualization. (2010). https://doi.org/10.1515/9783110800524
Langacker, R.: Ten lectures on the basics of cognitive grammar. (2017). https://doi.org/10.1163/9789004347458
De Jong, T.: Cognitive load theory, educational research, and instructional design: some food for thought. Instr. Sci. 38(2), 105–134 (2010). https://doi.org/10.1007/s11251-009-9110-0
Mutlu-Bayraktar, D., Cosgun, V., Altan, T.: Cognitive load in multimedia learning environments: A systematic review. Comput. Educ. 141 (2019). https://doi.org/10.1016/j.compedu.2019.103618
Hadie, S.N.H., Hassan, A., Mohd Ismail, Z.I., Ismail, H.N., Talip, S.B., Abdul Rahim, A.F.: Empowering students’ minds through a cognitive load theory-based lecture model: a metacognitive approach. Innov. Educ. Teach. Int. 55(4), 398–407 (2018). https://doi.org/10.1080/14703297.2016.1252685
Reedy, G.B.: Using cognitive load theory to inform simulation design and practice. Clin. Simul. Nurs. 11(8), 355–360 (2015). https://doi.org/10.1016/j.ecns.2015.05.004
Kalyuga, S., Singh, A.M.: Rethinking the boundaries of cognitive load theory in complex learning. Educ. Psychol. Rev. 28(4), 831–852 (2016). https://doi.org/10.1007/s10648-015-9352-0
Vu, N.N.: Structural, orientational, ontological conceptual. J. Sci. Ho Chi Minh City Open Univ. 3(3), 67–71 (2015)
Sweller, J.: Cognitive load theory, learning difficulty, and instructional design. Learn. Instr. 4(4), 295–312 (1994). https://doi.org/10.1016/0959-4752(94)90003-5
Sweller, J.: Cognitive load theory and educational technology. Educ. Technol. Res. Develop. 68(1) (2020). https://doi.org/10.1007/s11423-019-09701-3
Atkinson, R.C., Brelsford, J.W., Shiffrin, R.M.: Multiprocess models for memory with applications to a continuous presentation task. J. Math. Psychol. 4(2), 277–300 (1967). https://doi.org/10.1016/0022-2496(67)90053-3
Kirschner, P.A., Sweller, J., Kirschner, F., Zambrano, J.R.: From cognitive load theory to collaborative cognitive load theory. Int. J. Comput.-Support. Collab. Learn. 13(2), 213–233 (2018). https://doi.org/10.1007/s11412-018-9277-y
Derry, J.: A problem for cognitive load theory—the distinctively human life-form. J. Philos. Educ. 54(1), 5–22 (2020). https://doi.org/10.1111/1467-9752.12411
Sorden, S.D.: A cognitive approach to instructional design for multimedia learning. Inf. Sci. (2005). https://doi.org/10.28945/498
Janssen, J., Kirschner, P.A.: Applying collaborative cognitive load theory to computer-supported collaborative learning: towards a research agenda. Educ. Tech. Research Dev. 68(2), 783–805 (2020). https://doi.org/10.1007/s11423-019-09729-5
Kalyuga, S.: Cognitive load theory: how many types of load does it really need? Educ. Psychol. Rev. 23(1), 1–19 (2011). https://doi.org/10.1007/s10648-010-9150-7
Sweller, J.: Cognitive load theory. Psychol. Learn. Motiv. 55, 37–76 (2011). https://doi.org/10.1016/B978-0-12-387691-1.00002-8
Paas, F., van Merriënboer, J.J.G.: Cognitive-load theory: Methods to manage working memory load in the learning of complex tasks. Curr. Dir. Psychol. Sci. 29(4), 394–398 (2020). https://doi.org/10.1177/0963721420922183
Leahy, W., Sweller, J.: Cognitive load theory and the effects of transient information on the modality effect. Instr. Sci. 44(1), 107–123 (2016). https://doi.org/10.1007/s11251-015-9362-9
Sweller, J., Paas, F.: Should self-regulated learning be integrated with cognitive load theory? A commentary. Learn. Instr. 51, 85–89 (2017). https://doi.org/10.1016/j.learninstruc.2017.05.005
Paivio, A.: Dual coding theory and the mental lexicon. Mental Lexicon 5(2), 205–230 (2010). https://doi.org/10.1075/ml.5.2.04pai
Paivio, A., Desrochers, A.: A dual-coding approach to bilingual memory. Can. J. Psychol. 34(4), 388–399 (1980). https://doi.org/10.1037/h0081101
Kusumawati, D., Rachmawati, Y.: Optimizing storytelling through dual coding theory (2017). https://doi.org/10.2991/icece-16.2017.22
Sadoski, M., Paivio, A.: A dual coding theory model of reading. In: Alvermann, D.E., Unrau, N.J., Ruddell R.B. (eds.) Theoretical Models and Processes of Reading, pp. 886–922. International Reading Association, Newark (2013)
Wong, K.M., Samudra, P.G.: L2 vocabulary learning from educational media: extending dual-coding theory to dual-language learners. Comput. Assist. Lang. Learn. (2019). https://doi.org/10.1080/09588221.2019.1666150
Jessen, F., Heun, R., Erb, M., Granath, D.O., Klose, U., Papassotiropoulos, A., Grodd, W.: The concreteness effect: evidence for dual coding and context availability. Brain Lang. 74(1), 103–112 (2000). https://doi.org/10.1006/brln.2000.2340
Vu, N.N., Van, N.T.T., Lien, N.T.H.: Cross-linguistic analysis of metonymic conceptualization of personality in English and Vietnamese idioms containing head, face, and eyes. Int. J. English Lang. Stud. 2(4), 23–32 (2020). https://doi.org/10.32996/ijels
Zhu, W., Wang, X., Gao, W.: Multimedia intelligence: When multimedia meets artificial intelligence. IEEE Trans. Multimedia 22(7), 1823–1835 (2020). https://doi.org/10.1109/TMM.2020.2969791
Mayer, R.E.: Multimedia Learning, 2nd edn Cambridge University Press, New York (2009). https://doi.org/10.1017/CBO9780511811678
Mayer, R.E.: Applying the science of learning to multimedia instruction. In: Psychology of Learning and Motivation—Advances in Research and Theory, vol. 55, pp. 77–108. Elsevier Academic Press (2011). https://doi.org/10.1016/B978-0-12-387691-1.00003-X
Rahimi, M., Allahyari, A.: Effects of multimedia learning combined with strategy-based instruction on vocabulary learning and strategy use. Sage Open, 9(2) (2019). https://doi.org/10.1177/2158244019844081
Um, E.R., Plass, J.L., Hayward, E.O., Homer, B.D.: Emotional design in multimedia learning. J. Educ. Psychol. 104(2), 485–498 (2012). https://doi.org/10.1037/a0026609
Hsu, R.W., Chang, C.Y., Ho, H.J.: A study of English writing instruction with the use of multimedia. Int. J. Inf. Educ. Technol. 10(3), 186–190 (2020). https://doi.org/10.18178/ijiet.2020.10.3.1361
Kiat, T.Y., Jumintono, Kriswanto, E.S., Sugiri, Handayani, E., Anggarini, Y., Rofik, M.: The Effectiveness of multimedia learning on academic achievement in reproduction topic science subject. Univ. J. Educ. Res. 8(8), 3625–3629. https://doi.org/10.13189/ujer.2020.080839
Aisami, R.S.: Learning styles and visual literacy for learning and performance. Procedia. Soc. Behav. Sci. 176(20), 538–545 (2015). https://doi.org/10.1016/j.sbspro.2015.01.508
Cervetti, G., Damico, J., Pearson, P.D.: Multiple literacies, new literacies, and teacher education. Theory into Practice 45(4), 378–386 (2006). https://doi.org/10.1207/s15430421tip4504_12
Hoffman, J.V., Mosley Wetzel, M., DeJulio, S.: Multiple literacy tutoring experiences across a teacher preparation program: how can practice in hybrid spaces challenge the “practice makes practice” dilemma? Action Teacher Educ. 40(1), 58–76 (2018). https://doi.org/10.1080/01626620.2018.1424660
Zhang, K., Zhong, J., Wu, S.: Multimedia application in psychology teaching in colleges. Revista Argentina de Clinica Psicologica, 29(2), 507–513 (2020). https://doi.org/10.24205/03276716.2020.270
Ahmad, N.A.: Learning reading skills independently using interactive multimedia. Univ. J. Educ. Res. 8(6), 2641–2645 (2020). https://doi.org/10.13189/ujer.2020.080647
Park, B., Plass, J.L., Brünken, R.: Cognitive and affective processes in multimedia learning. Learn. Instr. (2014). https://doi.org/10.1016/j.learninstruc.2013.05.005
Karen, B.: Knowledge building through multiple literacy learning. In: Proceedings of the 31st Annual Conference of the International Association of School Librarianship, pp. 225–237 (2002).
Bigelow, M., Vanek, J., King, K., Abdi, N.: Literacy as social(media) practice: refugeee youth and native language literacy at school. Int. J. Intercult. Relat. 60, 183–197 (2017). https://doi.org/10.1016/j.ijintrel.2017.04.002
Li, J., Antonenko, P.D., Wang, J.: Trends and issues in multimedia learning research in 1996–2016: a bibliometric analysis. Educ. Res. Rev. 28, Article 100282 (2019). https://doi.org/10.1016/j.edurev.2019.100282
Banerjee, S.: Elements of Multimedia. (2019). https://doi.org/10.1201/9780429433207
Taylor, H.A., Rapp, D.N., Brunye, T.A.D.T.: Repetition and dual coding in procedural multimedia presentations. Appl. Cogn. Psychol. 22, 877–895 (2007). https://doi.org/10.1002/acp
Lee, S.: Raising EFL learners’ awareness of L2 lexical errors and correct usage: A dual coding approach. English Teaching, 72(2), 29–50 (2017). https://doi.org/10.15858/engtea.72.2.201706.29
Mayer, R.E., Moreno, R.: Aids to computer-based multimedia learning. Learn. Instr. 12(1), 107–119 (2002). https://doi.org/10.1016/S0959-4752(01)00018-4
Liu, X., Liu, Y., Tu, J.F.: Multimedia technology and learner autonomy: an experimental study for asymmetric effects. Symmetry, 12(3) (2020). https://doi.org/10.3390/sym12030462
Knoop-van Campen, C.A.N., Segers, E., Verhoeven, L.: Effects of audio support on multimedia learning processes and outcomes in students with dyslexia. Comput. Educ. 150 (2020). https://doi.org/10.1016/j.compedu.2020.103858
Benoît, G.: Visual communication as an information activity. J. Visual Literacy 34(2), 51–68 (2015). https://doi.org/10.1080/23796529.2015.11674729
Knowlton, J.Q.: On the definition of Picture. Educ. Commun. Technol. J. 14(2), 157–183 (1966). https://doi.org/10.1007/BF02769550
Sless, D.: Learning and Visual Communication. Routledge, London (2018). https://doi.org/10.4324/9780429021909
Moriarty, S.E.: A conceptual map of visual communication. J. Visual Literacy 17(2), 9–24 (1997). https://doi.org/10.1080/23796529.1997.11674518
Oliveira, A.W., Cook, K.: Student visual communication of evolution. Res. Sci. Educ. 47(3), 519–538 (2017). https://doi.org/10.1007/s11165-016-9515-0
Chen, J.C.C.: The crossroads of English language learners, task-based instruction, and 3D multi-user virtual learning in second life. Comput. Educ. 102, 152–171 (2016). https://doi.org/10.1016/j.compedu.2016.08.004
Barry, A.M.: Perception and visual communication theory. J. Visual Literacy 22(1), 91–106 (2002). https://doi.org/10.1080/23796529.2002.11674583
Moriarty, S.E.: The symbiotics of semiotics and visual communication. J. Visual Literacy 22(1), 19–28 (2002). https://doi.org/10.1080/23796529.2002.11674579
Worth, S.: Studying Visual Communication. De Gruyter, Philadelphia (2016). https://doi.org/10.9783/9781512809282
Mayer, R.E.: Incorporating motivation into multimedia learning. Learn. Instr. 29, 171–173 (2014). https://doi.org/10.1016/j.learninstruc.2013.04.003
Kassim, H., Nicholas, H., Ng, W.: Using a multimedia learning tool to improve creative performance. Thinking Skills Creativity 13, 9–19 (2014). https://doi.org/10.1016/j.tsc.2014.02.004
Kassim, W.Z.W.: Utilizing dual coding theory and animated images to enhance ESL students vocabulary learning. English Teacher 47(3), 81–91 (2018)
Vu, N.N., Phuong, L.T.T., Lien, N.T.H., Luong, N.T.: Using interactive whiteboard for teaching vocabulary to young English learners: Students’ performance and perceptions. J. Critical Rev. 7(19), 6651–6657 (2020)
Linh, V.T., Vu, N.N.: The impact of mobile learning on EFL students’ learning behaviors and perceptions: from content delivery to blended interaction. Int. Res. Higher Educ. 5(4), 25 (2021). https://doi.org/10.5430/irhe.v5n4p25
Natadjaja, L.: A usability of graphic design with local content in the interactive multimedia design for. Nirmana (2004). https://doi.org/10.9744/nirmana.6.2
Praheto, B.E., Andayani, Rohmadi, M., Wardani, N.E.: The effectiveness of interactive multimedia in learning Indonesian language skills in higher education. Rupkatha J. Interdis. Stud. Hum. 12(1) (2020). https://doi.org/10.21659/rupkatha.v12n1.34
Mazza, R.: Introduction to Information Visualization. Springer, London (2009). https://doi.org/10.1007/978-1-84800-219-7
Hansen, C.D., Johnson, C.R.. Visualization Handbook. Elsevier Academic Press (2005)
Munzner, T.: Visualization Analysis and Design. CRC Press, New York (2014). https://doi.org/10.1201/b17511
Cao, N., Cui, W.: Introduction to text visualization. Atlantis Press (2016). https://doi.org/10.2991/978-94-6239-186-4
Paivio, A.: Intelligence, dual coding theory, and the brain. Intelligence (2014). https://doi.org/10.1016/j.intell.2014.09.002
Yanasugondha, V.: A study of English vocabulary learning using the dual coding theory. LEARN J. Language Educ. Acquis. Res. Netw. 10(1), 165–175 (2017)
Kozhevnikov, M., Kosslyn, S., Shephard, J.: Spatial versus object visualizers: a new characterization of visual cognitive style. Mem. Cognit. 33(4), 710–726 (2005). https://doi.org/10.3758/BF03195337
Logie, R.H.: Visuo-spatial Working Memory. Psychology Press, London (2014). https://doi.org/10.4324/9781315804743
Mayer, R.E., Massa, L.J.: Three facets of visual and verbal learners: cognitive ability, cognitive style, and learning preference. J. Educ. Psychol. 95(4), 833–846 (2003). https://doi.org/10.1037/0022-0663.95.4.833
Vu, N.N.: Mobile learning in language teaching context of Vietnam: An evaluation of students’ readiness. J. Sci. HCMC Univ. Educ. 7(85), 16–27 (2016). Retrieved from http://www.vjol.info/index.php/sphcm/article/viewFile/24861/21273
Paivio, A.: Dual coding theory: Retrospect and current status. Canadian J. Psychol./Revue canadienne de psychologie 45(3), 255–287 (1991). https://doi.org/10.1037/h0084295
Höffler, T.N., Schwartz, R.N.: Effects of pacing and cognitive style across dynamic and non-dynamic representations. Comput. Educ. 57(2), 1716–1726 (2011). https://doi.org/10.1016/j.compedu.2011.03.012
Domagk, S., Schwartz, R.N., Plass, J.L.: Interactivity in multimedia learning: an integrated model. Comput. Hum. Behav. 26(5), 1024–1033 (2010). https://doi.org/10.1016/j.chb.2010.03.003
Alsadhan, A.O., Alhomod, S., Shafi, M.M.: Multimedia based e-learning: design and integration of multimedia content in E-learning. Int. J. Emerg. Technol. Learn. 9(3), 26–30 (2014). https://doi.org/10.3991/ijet.v9i3.3308
Mayer, R.E.: Multimedia Learning. Cambridge University Press (2001). https://doi.org/10.1017/cbo9781139164603
Huy, C.V., Luong, N.T., Vu, N.N.: Blended learning in badminton training for professionals: students’ perceptions and performance impacts. European J. Phys. Educ. Sport Sci. 6(6), 28–36 (2020). http://dx.doi.org/10.46827/ejpe.v6i6.3226
Hyde, J.: Introduction to American literature. Peabody Journal of Education, vol. 33. HCMCUE Publishing House, Ho Chi Minh City (1956). https://doi.org/10.1080/01619565609536653
Mustofa, A., Hill, J.L.: Understanding cultural context in responding to literature: researching the teaching of literature in EFL classroom context. Engl. Lang. Teach. 11(6), 108 (2018). https://doi.org/10.5539/elt.v11n6p108
Jamet, E., Gavota, M., Quaireau, C.: Attention guiding in multimedia learning. Learn. Instr. 18(2), 135–145 (2008). https://doi.org/10.1016/j.learninstruc.2007.01.011
Heo, M., Toomey, N.: Learning with multimedia: The effects of gender, type of multimedia learning resources, and spatial ability. Comput. Educ. 146 (2020) https://doi.org/10.1016/j.compedu.2019.103747
Kaminska, P.M.: Learning Styles and Second Language Education. Cambridge Scholars Publishing (2014)
Sternberg, R.J., Zhang, L.F.: Perspectives on Thinking, Learning, and Cognitive Styles. Routledge, New York (2014). https://doi.org/10.4324/9781410605986
Reid, G.: Learning styles and inclusion. (2005). https://doi.org/10.4135/9781446212523
Manzoor, A.: Blended learning in higher education: a developing country perspective. In: Online Course Management: Concepts, Methodologies, Tools, and Applications, vol. 3, pp. 1439–1449. IGI Global (2018). https://doi.org/10.4018/978-1-5225-5472-1.ch075
Tong, C., Roberts, R., Borgo, R., Walton, S., Laramee, R.S., Wegba, K.L.A., Wang, Y., Qu, H., Luo, Q., Ma, X. (2018). Storytelling and visualization: an extended survey. Information 9(3), Article 65. https://doi.org/10.3390/info9030065
Tversky, B., Morrison, J.B., Betrancourt, M.: Animation: Can it facilitate? Int. J. Hum. Comput. Stud. 57(4), 247–262 (2002). https://doi.org/10.1006/ijhc.2002.1017
Levin, J.R., Anglin, G.J., Carney, R.N. (1987). On empirically validating functions of pictures in prose. In Willow, M.D., Houghton, H.A. (eds.) The psychology of illustration: Basic research, pp. 51–85. Springer, New York (1987). https://doi.org/10.1007/978-1-4612-4674-9_2
Lemke, J.: Toward critical multimedia literacy. In: McKenna, M.C., Labbo, L.D., Kieffer, R.D., Reinking, D. (eds.) International Handbook of Literacy and Technology. Routledge London, (2015). https://doi.org/10.4324/9780203929131.ch1
Arrow, A.W., Finch, B.T.: Multimedia literacy practices in beginning classrooms and at home: the differences in practices and beliefs. Literacy 47(3), 131–141 (2013). https://doi.org/10.1111/lit.12006
Morbey, M.L.: Beyond multimedia literacy: Supporting mastery in virtual environments. Int. J. Learning Media 2(4), 7–20 (2010). https://doi.org/10.1162/ijlm_a_00058
Semali, L.M.: Literacy in Multimedia America: Integrating Media Education Across the Curriculum. Routledge, London (2017). https://doi.org/10.4324/9781351236225
Lemke, J.: Multimedia literacy demands of the scientific curriculum. Linguist. Educ. 10(3), 247–271 (1998). https://doi.org/10.1016/S0898-5898(99)00009-1
Flores-Koulish, S.A., Deal, D., Losinger, J., McCarthy, K., Rosebrugh, E.: After media literacy course: three early childhood teachers look back. Action Teacher Educ. 33(2), 127–143 (2011). https://doi.org/10.1080/01626620.2011.569308
Baddeley, A.: Working memory. Comptes Rendus de l’Academie des Sciences—Serie III 321(2–3), 167–173 (1998). https://doi.org/10.1016/S0764-4469(97)89817-4
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Vu, N.N., Hung, B.P., Van, N.T.T., Lien, N.T.H. (2022). Theoretical and Instructional Aspects of Using Multimedia Resources in Language Education: A Cognitive View. In: Kumar, R., Sharma, R., Pattnaik, P.K. (eds) Multimedia Technologies in the Internet of Things Environment, Volume 2. Studies in Big Data, vol 93. Springer, Singapore. https://doi.org/10.1007/978-981-16-3828-2_9
Download citation
DOI: https://doi.org/10.1007/978-981-16-3828-2_9
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-3827-5
Online ISBN: 978-981-16-3828-2
eBook Packages: EngineeringEngineering (R0)