Multimodality in Mobile-Assisted Language Learning

  • Volker Eisenlauer
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 479)


Current mobile language learning applications are the latest link in a chain of learning materials that are designed to trigger self-directed and holistic learning experiences. The interactive and visually appealing learning materials provide contextualized input and offer various options for enhancing a learner’s productive and receptive language skills. Turning to practice, the multimodal potential of mobile assisted language learning applications appears far from exhausted: Various mobile Apps use multimodality only in a limited way as they either employ images only for illustrative purposes or exhibit large discrepancies between the ideational meanings of different representative modes.

To assess the potential of multimodal meaning creation for mobile learning, my paper reflects on specific semiotic characteristics of verbal and pictorial signs and investigates the semantic relations holding between these modes. The integration of insights from educational and multimodal theory with findings from mobile learning helps to identify a set of intermodal relations that are particularly suited for analyzing text-image links in mobile assisted language learning environments. Applying an empirical lens, the paper investigates patterns of multimodal meaning creation in the vocabulary tasks of one of the most popular language learning Apps, i.e. Duolingo. Finally, my descriptive and empirical findings are summarized and integrated into a set of guidelines on how to exploit text-image links for vocabulary/language learning purposes in mobile environments.


Multimodality Mobile Assisted Language Learning Digital Literacy Semiotics Text-Image Relations Multimodal Vocabulary Instruction 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barthes, R.: Image, music, text. The Noonday Press, New York (1977)Google Scholar
  2. 2.
    Beatty, K.: Beyond the classroom: Mobile learning the wider world. The International Research Foundation for English Language Education (2013), (last accessed May 05, 2014)
  3. 3.
    Bucher, H.J.: Multimodalität - ein universelles Merkmal der Medienkommunikation: Zum Verhältnis von Medienangebot und Medienrezeption. In: Bucher, H.J., Schumacher, P. (Hrsg.) Interaktionale Rezeptionsforschung. Theorie und Methode der Blickaufzeichnung in der Medienforschung, Wiesbaden, pp. 51–82 (2013)Google Scholar
  4. 4.
    Clarke, J.: Augmented reality, multimodal literacy and mobile technology: An experiment in teacher engagement. In: Proceedings of the 12th World Conference on Mobile and Contextual Learning (mLearn 2013) (2013), (last accessed May 05, 2014)
  5. 5.
    Godwin-Jones, R.: Emerging Technologies. Mobile Apps for Language Learning. Language Learning & Technology 15(2) (2011), (last accessed May 05, 2014)
  6. 6.
    Joseph, S., Uther, M.: Mobile Devices for Language Learning: Multimedia Approaches. Research and Practice in Technology Enhanced Learning 4(1), 7–32 (2009)CrossRefGoogle Scholar
  7. 7.
    Klopfer, E., Squire, K., Jenkins, H.: Environmental Detectives: PDAs as a window into a virtual simulated world. In: Kerres, M., et al. (Hrsg.) Didaktik der Notebook-Universität, Münster, pp. 259–274 (2004)Google Scholar
  8. 8.
    Kress, G.: What is a mode? In: Jewitt, C. (ed.) The Routledge Handbook of Multimodal Analysis, London, pp. 54–67 (2009)Google Scholar
  9. 9.
    Kress, G., van Leeuwen, T.: Reading Images: The Grammar of Visual Design. Routledge, London (2006)Google Scholar
  10. 10.
    Kress, G., van Leeuwen, T.: Colour as a semiotic mode: notes for a grammar of colour. Visual Communication 1(3), 343–368 (2002)CrossRefGoogle Scholar
  11. 11.
    Kukulska-Hulme, A., Shield, L.: An overview of mobile assisted language learning: From content delivery to supported collaboration and interaction. ReCALL 20(3), 271–289 (2008)CrossRefGoogle Scholar
  12. 12.
    Magal-Royo, T., et al.: Multimodal application for foreign language teaching. In: 14th International Conference on Interactive Collaborative Learning, pp. 145–148 (2011)Google Scholar
  13. 13.
    Nöth, W.: Handbuch der Semiotik. Metzler, Stuttgart (2000)Google Scholar
  14. 14.
    Ogata, H., Yano, Y.: Knowledge awareness for a computer-assisted language learning using handhelds. International Journal of Continuous Engineering Education and Lifelong Learning 14(4/5), 435–449 (2004)CrossRefGoogle Scholar
  15. 15.
    Paivio, A.: Mental representations: A dual-coding approach. Oxford University Press, New York (1986)Google Scholar
  16. 16.
    Reinfried, M.: Vom ‘Stellvertreter’ zum ‘Türöffner’ – Bilder in Fremdsprachen-Lehrwerken. In: Lieber, G. (Hrsg.) Lehren und Lernen mit Bildern. Ein Handbuch zur Bilddidaktik, pp. 198–211. Schneider, Baltmannsweiler (2008)Google Scholar
  17. 17.
    Rorty, R. (ed.): The Linguistic Turn: Essays in Philosophical Method. University of Chicago Press, Chicago (1992)Google Scholar
  18. 18.
    Royce, T.: Intersemiotic Complementarity: A Framework for Multimodal Discourse Analysis. In: Royce, T., Bowcher, W. (eds.) New Directions in the Analysis of Multimodal Discourse, pp. 63–109. Erlbaum, Mahwa (2007)Google Scholar
  19. 19.
    Sadoski, M.: A Dual Coding View of Vocabulary Learning. Reading & Writing Quarterly: Overcoming Learning Difficulties 21(3), 221–238 (2005)CrossRefGoogle Scholar
  20. 20.
    Schallert, D.: The role of illustrations in reading comprehension. In: Spiro, R., et al. (eds.) Theoretical Issues in Reading Comprehension, pp. 503–524. Lawrence Erlbaum, Hillsdale (1980)Google Scholar
  21. 21.
    Shen, H.: Imagery and verbal coding approaches in Chinese vocabulary instruction. Language Teaching Research 14(4), 485–499 (2010)CrossRefGoogle Scholar
  22. 22.
    Stöckl, H.: In Between Modes. Language and Image in Printed Media. In: Ventola, E., et al. (eds.) Perspectives on Multimodality, Amsterdam, pp. 9–30 (2004)Google Scholar
  23. 23.
    Stöckl, H.: Sprache-Bild-Texte lesen. Bausteine zur Methodik einer Grundkompetenz. In: Diekmannshenke, H., et al. (Hrsg.) Bildlinguistik, Berlin, pp. 43–70 (2010)Google Scholar
  24. 24.
    Unsworth, L.: Image/Text Relations and Intersemiosis: Towards Multimodal Text Desciption for Multiliteracies Education. In: Proceedings of the 33rd International Systemic Functional Congress, pp. 1165–1205 (2006)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Volker Eisenlauer
    • 1
  1. 1.English and Applied LinguisticsUniversity of SalzburgSalzburgAustria

Personalised recommendations