Language Resources and Evaluation

, Volume 47, Issue 2, pp 399–423 | Cite as

A corpus-based approach to the multimodal analysis of specialized knowledge

  • Juan Antonio Prieto Velasco
Original Paper


Modern communication environments have changed the cognitive patterns of individuals, who are now used to the interaction of information encoded in different semiotic modalities, especially visual and linguistic. Despite this, the main premise of Corpus Linguistics is still ruling: our perception of and experience with the world is conveyed in texts, which nowadays need to be studied from a multimodal perspective. Therefore, multimodal corpora are becoming extremely useful to extract specialized knowledge and explore the insights of specialized language and its relation to non-language-specific representations of knowledge. It is our assertion that the analysis of the image-text interface can help us understand the way visual and linguistic information converge in subject-field texts. In this article, we use Frame-based terminology to sketch a novel proposal to study images in a corpus rich in pictorial representations for their inclusion in a terminological resource on the environment. Our corpus-based approach provides the methodological underpinnings to create meaning within terminographic entries, thus facilitating specialized knowledge transfer and acquisition through images.


Frame-based terminology Knowledge visualization Multimodality Image-text interface Terminological resources 



This research is funded by the Spanish Ministry of Economy and Competitiveness under the project FFI2011-23120.


  1. Allwood, J. (2008). Multimodal corpora. In A. Lüdeling & M. Kytö (Eds.), Corpus linguistics: An international handbook (pp. 207–225). Berlin: Mouton de Gruyter.Google Scholar
  2. Baddeley, A. D. (1986). Working memory. Oxford: Oxford University Press.Google Scholar
  3. Baldry, A. P., & Thibault, P. J. (2006). Multimodal corpus linguistics. In G. Thompson & S. Hunston (Eds.), System and corpus exploring connections (pp. 164–183). London and New York: Equinox.Google Scholar
  4. Baldry, A. P., & Thibault, P. J. (2008). Applications of multimodal concordances. Journal of Language and Communication Studies, 41, 11–41.Google Scholar
  5. Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617–645.CrossRefGoogle Scholar
  6. Bowker, L. (2002). Computer-aided translation technology. A practical introduction. Ottawa: University of Ottawa Press.Google Scholar
  7. Bowker, L., & Howkins, S. (2006). Variation in the organization of medical terms. Exploring some motivations for term choice. Terminology, 12(1), 79–110.CrossRefGoogle Scholar
  8. Carney, R. N., & Levin, J. R. (2002). Pictorial illustrations still improve students’ learning from text. Educational Psychology Review, 14(1), 5–26.CrossRefGoogle Scholar
  9. Faber, P. (2009). The cognitive shift in terminology and specialized translation. MonTI. Monografías de Traducción e Interpretación, no., 1, 107–134.CrossRefGoogle Scholar
  10. Faber, P. (2011). The dynamics of specialized knowledge representation: Simulational reconstruction or the perception–action interface. Terminology, 17(1), 9–29.CrossRefGoogle Scholar
  11. Faber, P., León Araúz, P., Prieto Velasco, J. A., & Reimerink, A. (2007). Linking images and words: The description of specialized concepts. International Journal of Lexicography, 20(1), 39–65.CrossRefGoogle Scholar
  12. Faber, P., Márquez Linares, C., & Vega, M. (2005). Framing terminology: A process-oriented approach. Meta: Journal des traducteurs/Meta: Translators’ Journal, 50(4).Google Scholar
  13. Faber, P., Montero Martínez, S., Castro Prieto, M. R., Senso Ruiz, J., Prieto Velasco, J. A., León Araúz, P., et al. (2006). Process-oriented terminology management in the domain of coastal engineering. Terminology, 12(2), 189–213.CrossRefGoogle Scholar
  14. Fillmore, C. (1982). Frame semantics. In The Linguistic Society of Korea (Ed.), Linguistics in the morning calm (pp. 111–137). Seoul: Hanshin.Google Scholar
  15. Fillmore, C. (1985). Frames and the semantics of understanding. Quaderni di Semantica, 6(2), 222–254.Google Scholar
  16. Fillmore, C., & Atkins, B. (1992). Towards a frame-based organization of the lexicon: The semantics of RISK and its neighbors. In A. Lehrer & E. Kittay (Eds.), Frames, fields, and contrasts: New essays in semantics and lexical organization (pp. 75–102). Hillsdale: Lawrence Erlbaum.Google Scholar
  17. Firth, J. R. (1957). A synopsis of linguistic theory 1930–1955. Studies in Linguistic Analysis (special volume of the Philological Society) (pp. 1–32).Google Scholar
  18. Freixa, J. (2006). Causes of denominative variation in terminology: A typology proposal. Terminology, 12(1), 51–77.CrossRefGoogle Scholar
  19. Jewitt, C. (Ed.). (2009). Handbook of multimodal analysis. London: Routledge.Google Scholar
  20. Kress, G., & Van Leeuwen, T. (2002a). Reading images: The grammar of visual design. London: Routledge.Google Scholar
  21. Kress, G. R., & Van Leeuwen, T. (2002b). Multimodal discourse: The modes and media of contemporary communication. London: Edward Arnold.Google Scholar
  22. León Araúz, P., & Faber, P. (2010). Natural and contextual constraints for domain-specific relations. In V. Barbu Mititelu, V. Pekar, & E. Barbu (Eds.), Proceedings of the Workshop Semantic Relations. Theory and Applications (pp. 12–17). Valletta.Google Scholar
  23. Liu, Y., & O’Halloran, K. L. (2009). Intersemiotic texture: Analyzing cohesive devices between language and images. Social Semiotics, 19(4), 367–387.CrossRefGoogle Scholar
  24. López Rodríguez, C. I. (2001). Tipologías textuales y cohesión en la traducción biomédica inglés-español: un estudio de corpus. PhD Thesis. University of Granada. Available at Accessed Feb 2012.
  25. López Rodríguez, C. I. (2009). Extracción y representación de conocimiento a partir de corpus. In E. Valero & E. Rambla Alcina (Eds.), Terminología y sociedad del conocimiento (pp. 341–379). Bern: Peter Lang.Google Scholar
  26. López Rodríguez, C. I., Buendía Castro, M., & García Aragón, A. (2012). User needs to the test: Evaluating a terminological knowledge base on the environment by trainee translators. Jostrans. The Journal of Specialized Translation, 18, 57–76.Google Scholar
  27. Louwerse, M. M., & Jeuniaux, P. (2010). The linguistic and embodied nature of conceptual processing. Cognition, 114, 96–104.CrossRefGoogle Scholar
  28. Marsh, E. E., & White, M. D. (2003). Taxonomy of relationships between images and text. Journal of Documentation, 59(6), 647–672.CrossRefGoogle Scholar
  29. Martinec, R., & Salway, A. (2005). A system for image-text relations in new (and old) media. Visual Communication, 4(3), 337–371.CrossRefGoogle Scholar
  30. McEnery, A., & Wilson, A. (2001). Corpus linguistics. Edinburgh: Edinburgh University Press.Google Scholar
  31. Meyer, I., & Mackintosh, K. (1996). The corpus from a terminographer’s viewpoint. International Journal of Corpus Linguistics, 1(2), 257–285.CrossRefGoogle Scholar
  32. O’Halloran, K. L., & Smith B. A. (accepted for publication). Multimodal text analysis. In C. A. Chapelle (Ed.). Encyclopedia of applied linguistics. New Jersey: Wiley-Blackwell.Google Scholar
  33. O’Halloran, K. L., Tan, S., Smith, B. A., & Podlasov, A. (2010). Challenges in designing digital interfaces for the study of multimodal phenomena. Information Design Journal, 18(1), 2–12.CrossRefGoogle Scholar
  34. Paivio, A. (1971). Imagery and verbal processes. New York: Holt, Rinehart, and Winston.Google Scholar
  35. Prieto Velasco, J. A. (2008). Información gráfica y grados de especialidad en el discurso científico-técnico: un estudio de corpus. PhD Thesis, University of Granada. Available at: Accessed Feb 2012.
  36. Prieto Velasco, J. A. (2009). Traducción e imagen: la información visual en textos especializados. Granada: Tragacanto.Google Scholar
  37. Prieto Velasco, J. A., & López Rodríguez, C. I. (2009). Managing graphic information in terminological knowledge bases. Terminology, 15(2), 179–213.CrossRefGoogle Scholar
  38. Reimerink, A., García de Quesada, M., & Montero Martínez, S. (2010). Contextual information in terminological knowledge bases: A multimodal approach. Journal of Pragmatics, 42(7), 1928–1950.CrossRefGoogle Scholar
  39. Temmerman, R. (2000). Towards new ways of terminology description: The sociocognitive-approach. Amsterdam/Philadelphia: John Benjamins.Google Scholar
  40. Temmerman, R., & Kerremans, K. (2003). Termontography: Ontology building and the sociocognitive approach to terminology description. Prague CIL17-conference.Google Scholar
  41. Tercedor, M. I. (2011). The cognitive dynamics of terminological variation. Terminology, 17(2), 181–197.CrossRefGoogle Scholar
  42. Unsworth, L. (2008). Multimodal semiotics: Functional analysis in contexts of education. London and New York: Continuum.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2012

Authors and Affiliations

  1. 1.Department of Philology and TranslationUniversity Pablo de OlavideSevillaSpain

Personalised recommendations