Abstract
This chapter provides an overview of current advances in multimodal corpus linguistics. It defines what multimodal corpora are, what they can be used for, how and why they are used, and outlines some of the practical and methodological challenges and concerns faced in the construction and analysis of multimodal corpora. Examples of notable corpora are presented, alongside examples of software tools for multimodal corpus development and enquiry, and the chapter ends with reflections on possible future directions for developments within this field.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abuczki, Á., & Ghazaleh, E. B. (2013). An overview of multimodal corpora, annotation tools and schemes. Argumentum, 9, 86–98.
Adolphs, S. (2013). Corpora: Multimodal. In C. A. Chapelle (Ed.), The encyclopaedia of applied linguistics. London: Blackwell.
Adolphs, S., & Carter, R. (2013). Spoken corpus linguistics: From monomodal to multimodal. London: Routledge.
Adolphs, S., Knight, D., & Carter, R. (2011). Capturing context for heterogeneous corpus analysis: Some first steps. International Journal of Corpus Linguistics, 16(3), 305–324.
Aist, G., Allen, J., Campana, E., Galescu, L., Gomez Gallo, C.A., Stoness, S., Swift, M., & Tanenhaus, M. (2006). Software architectures for incremental understanding of human speech, 1922–1925. In: Proceedings of Interspeech 2006, Pittsburgh.
Allwood, J. (2008). Multimodal Corpora. In A. Lüdeling & M. Kytö (Eds.), Corpus linguistics: An international handbook (pp. 207–225). Berlin: Mouton de Gruyter.
Anthony, L. (2017). AntConc (Version 344) [Computer software]. Waseda University, Tokyo. Available from http://www.laurenceanthony.net. Accessed 23 May 2019.
Ashby, S., Bourban, S., Carlettea, J., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., Karaiskos, V., Kraaij, W., Kronenthal, M., Lathoud, G., Lincoln, M., Lisowska, A., McCowan, I., Post, W., Reidsma, D., and Wellner, P. (2005). The AMI Meeting Corpus, In: Proceedings of measuring behavior conference, Wageningen.
Baiat, G.E., Coler, M., Pullen, M., Tienkouw, S., & Hunyadi, L. (2013). Multimodal analysis of ‘well’ as a discourse marker in conversation: A pilot study. In: Proceedings of the 4th IEEE International Conference on Cognitive Infocommunications (CogInfoCom 2013), (pp. 283–287). Budapest, Hungary.
Baldry, A. (2004). Phase and transition, type and instance: Patterns in media texts as seen through a multimodal concordance. In K. O’Halloran (Ed.), Multimodal discourse analysis: Systemic functional perspectives (pp. 83–108). London/New York: Continuum.
Baldry, A. (2005). Multimodal corpus linguistics. In G. Thompson & S. Hunston (Eds.), System and Corpus: Exploring connections (pp. 164–183). London/New York: Equinox.
Baldry, A., & Thibault, P. J. (2001). Towards multimodal corpora. In G. Aston & L. Burnard (Eds.), Corpora in the description and teaching of English, papers from the 5th ESSE (pp. 87–102). Conference Bologna: Cooperativa Libraria Universitaria Editrice Bologna.
Bazzanella, C. (2002). The significance of context in comprehension: The ‘we case’. Foundations of Science, 7, 39–254.
Biber, D. (1993). Representativeness in corpus design. Literary and Linguistic Computing, 8(4), 243–257.
Boersma, P., & Weenink, D. (2009). Praat: doing phonetics by computer [Computer Program]. Available from http://www.praat.org. Accessed 23 May 2019.
Brezina, V., McEnery, T., & Wattam, S. (2015). Collocations in context: A new perspective on collocation networks. International Journal of Corpus Linguistics, 20(2), 139–173.
Brundell, P., Tennent, P., Greenhalgh, C., Knight, D., Crabtree, A., O’Malley, C., Ainsworth, S., Clarke, D., Carter, R., and Adolphs, S. (2008). Digital Replay system (DRS): A tool for interaction analysis. In: Paper delivered at the International Conference for the Learning Sciences 2008 (ICLS), Utrecht, The Netherlands June–July 2008.
Burnard, L., & Aston, G. (1998). The BNC handbook: Exploring the British National Corpus. Edinburgh: Edinburgh University Press.
Carletta, J. (2006). Announcing the AMI meeting corpus. The ELRA Newsletter, 11(1), 3–5.
Chen, R. L., Rose, T., Qiao, Y., Kimbara, I., Parrill, F., Welji, H., Han, X., Tu, J., Huang, Z., Harper, M., Quek, F., Xiong, Y., McNeill, D., Tuttle, R., & Huang, T. (2005). VACE multimodal meeting corpus. In S. Renals & S. Bengio (Eds.), Proceedings of Machine Learning for Multimodal Interaction (MLMI 2005), Lecture Notes in Computer Science (Vol. 3869, pp. 40–45). Berlin/Heidelberg: Springer.
Conrad, S. (2002). Corpus linguistic approaches for discourse analysis. Annual Review of Applied Linguistics, 22, 75–95.
Crabtree, A., & Rodden, T. (2009). Understanding interaction in hybrid ubiquitous computing environments. Proceedings of the 8th International Conference on Mobile and Ubiquitous Media, Article No1, 1–10. Cambridge: ACM.
Cu, J., Suarez, M., & Sta, M.A. (2010). Filipino Multimodal Emotion Database. In: Proceedings of 7th Conference on Language Resources and Evaluation (LREC-2010) (pp. 37–42). Malta, May 2010.
Davies, M. (2008). The Corpus of Contemporary American English (COCA): 560 million words, 1990-present. Available online at https://corpusbyuedu/coca/. Accessed 23 May 2019.
Dittman, A., & Llewellyn, L. (1968). Relationships between vocalizations and head nods as listener responses. Journal of Personality and Social Psychology, 9, 79–84.
Duncan, S. (1972). Some signals and rules for taking speaking turns in conversation. Journal of Personality and Social Psychology, 23(2), 283–292.
Dybkjær, L., & Ole Bernsen, N. (2004). Recommendations for natural interactivity and multimodal annotation schemes. In: Proceedings of the 4th Language Resources and Evaluation Conference (LREC) 2004, Lisbon, Portugal.
Fanelli, G., Gall, J., Romsdorfer, H., Weise, T., & Van Gool, L. (2010). 3D vision technology for capturing multimodal corpora: chances and challenges. In: Proceedings of the LREC Workshop on Multimodal Corpora, Mediterranean Conference Centre (pp. 70–73). Malta, 18 May 2010.
Fisher, D., Williams, M., & Andriacchi, T. (2003). The therapeutic potential for changing patterns of locomotion: an application to the acl-deficient knee. In: Proceedings of the ASME Bioengineering Conference (pp. 849–50). Miami, Florida, June 2003.
Foster, M. E., & Oberlander, J. (2007). Corpus-based generation of head and eyebrow motion for an embodied conversational agent. Language Resources and Evaluation (LREC), 41(3/4), 305–323.
Garofolo, J.S., Laprun, C.D., Michel, M., Stanford, V.M., & Tabassi, E. (2004). The NIST Meeting Room Pilot Corpus. In: Proceedings of the 4th Language Resources and Evaluation Conference (LREC) 2004, Lisbon, Portugal.
Goodwin, C. (1994). Professional vision. American Anthropologist, 96(3), 606–633.
Gu, Y. (2006). Multimodal text analysis: A corpus linguistic approach to situated discourse. Text and Talk, 26(2), 127–167.
Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Edward Arnold.
Herrera, D., Novick, D., Jan, D., & Traum, D. (2010). The UTEP-ICT Cross-Cultural Multiparty Multimodal Dialog Corpus. In: Proceedings of LREC Workshop on Multimodal Corpora. May 2010, Malta.
Johnston, T., & Schembri, A. (2006). Issues in the creation of a digital archive or a signed language. In L. Barwick & N. Thieberger (Eds.), Sustainable data from digital fieldwork (pp. 7–16). Sydney: Sydney University Press.
Jokinen, K., & Allwood, J. (2010). Hesitation in Intercultural Communication: Some observations on Interpreting Shoulder Shrugging. In: Proceedings of the international workshop on agents in cultural context, (pp. 25–37). The First International Conference on Culture and Computing 2010 Kyoto, Japan.
Kilgarriff, A., Baisa, V., Bušta, J., Jakubíček, M., Kovvář, V., Michelfeit, J., Rychlý, P., & Suchomel, V. (2014). The sketch engine: Ten years on. Lexicography, 1, 7–36.
Kipp, M. (2001). ANVIL – A generic annotation tool for multimodal dialogue, 1367–1370. In: Proceedings of 7th European Conference on Speech Communication and Technology 2nd INTERSPEECH Event, Aalborg, Denmark.
Knight, D. (2011a). Multimodality and active listenership: A corpus approach. London: Bloomsbury.
Knight, D. (2011b). The future of multimodal corpora. Brazilian Journal of Applied Linguistics (BJAS), 11(2), 391–415.
Knight, D., Evans, D., Carter, R. A., & Adolphs, S. (2009). Redrafting corpus development methodologies: Blueprints for 3rd generation multimodal, multimedia corpora. Corpora, 4(1), 1–32.
Knight, D., Tennent, P., Adolphs, S., & Carter, R. (2010). Developing heterogeneous corpora using the Digital Replay System (DRS). In: Proceedings of the LREC 2010 (Language Resources Evaluation Conference) Workshop on Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality. Malta, May 2010.
Kress, G. (2010). Multimodality–A social semiotic approach to contemporary communication. London: Routledge.
Kress, G. R., & Van Leeuwen, T. (2001). Multimodal discourse: The modes and Media of Contemporary Communication. London: Arnold.
Ladewig, S. H. (2014a). The cyclic gesture. In C. Müller, A. Cienki, E. Fricke, S. Ladewig, D. McNeil, & J. Bressem (Eds.), Body language communication volume 2: An international handbook on multimodality in human interaction (pp. 1605–1618). Berlin: De Gruyter.
Ladewig, S. H. (2014b). Recurrent gestures. In C. Müller, A. Cienki, E. Fricke, S. Ladewig, D. McNeil, & J. Bressem (Eds.), Body language communication volume 2: An international handbook on multimodality in human interaction (pp. 1558–1574). Berlin: De Gruyter.
Lemke, J. L. (2002). Travels in hypermodality. Visual Communication, 1(3), 299–325.
Lücking A., Bergman, K., Hahn, F., Kopp, S., & Rieser, H. (2010). The Bielefeld Speech and Gesture Alignment Corpus (SaGA). In: Proceedings of LREC Workshop on Multimodal Corpora, 2010. Malta, May 2010.
MacWhinney, B. (2000). The CHILDES project: Tools for analyzing talk (3rd ed.). Mahwah: Lawrence Erlbaum Associates.
MacWhinney, B. (2007). The TalkBank project. In J. Beal & K. Moisl (Eds.), Creating and digitizing language corpora: Synchronic databases (Vol. 1). Basingstoke/Hampshire: Palgrave Macmillan.
Malisz, Z., Wlodarczak, M., Buschmeier, H., Skubisz, J., Kopp, S., & Wagner, P. (2016). The ALICO corpus: Analyzing the active listener. Language Resources and Evaluation, 50(2), 411–442.
Mana, N., Lepri, B., Chippendale, P., Cappelletti, A., Pianesi, F., Svaizer, P., & Zancanaro, M. (2007). Multimodal Corpus of Multi-Party Meetings for Automatic Social Behavior Analysis and Personality Traits Detection. In: Proceeding of Workshop on Tagging, Mining and Retrieval of Human-Related Activity Information at ICMI’07 (pp. 9–14). Nagoya, Japan.
Maynard, S. K. (1987). International functions of a nonverbal sign head movement in Japanese dyadic casual conversation. Journal of Pragmatics, 11, 589–606.
McCowan, I., Bengio, S., Gatica-Perez, D., Lathoud, G., Monay, F., Moore, D., Wellner, P., & Bourlard, H. (2003). Modelling Human Interaction in Meetings. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 748–751). Hong Kong, April 2003.
McEnery, T., Xiao, R., & Tono, Y. (2006). Corpus-based language studies: An advanced resource book. London: Routledge.
Mlakar, I., Kačič, Z., & Rojc, M. (2017). A Corpus for investigating the multimodal nature of multi-speaker spontaneous conversations – EVA Corpus. WSEAS Transactions on Information Science and Applications, 14, 213–226.
Müller, C., Bressem, J., & Ladewig, S. H. (2013). Towards a grammar of gestures: A form-based view. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf (Eds.), Body-language-communication: An international handbook on multimodality in human interaction (Vol. 1, pp. 707–733). De Gruyter Mouton: Berlin.
Navarretta, C., & Paggio, P. (2013). Classifying multimodal turn management in Danish dyadic first encounters. In: Proceedings of the 19th Nordic Conference of Computational Linguistics (NODALIDA 2013) (pp. 133–146). Oslo, Norway.
O’Halloran, K. L. (2006). Multimodal discourse analysis – Systemic functional perspectives. London: Continuum.
O’Halloran, K. L. (2011). Multimodal discourse analysis. In K. Hyland & B. Paltridge (Eds.), Companion to discourse (pp. 120–137). London/New York: Bloomsbury.
Oertel, C., Cummins, F., Campbell, N., Edlund, J., & Wagner, P. (2010). D64: a corpus of richly recorded conversational interaction. In: Proceedings of the LREC Workshop on Multimodal Corpora, Mediterranean Conference Centre (pp. 27–30). Malta, 18 May 2010.
Paggio, P., & Navarretta, C. (2010). Feedback in Head Gesture and Speech. In: Proceedings of 7th Conference on Language Resources and Evaluation (LREC-2010) (pp. 1–4). Malta, May.
Paggio, P., & Navarretta, C. (2017). The Danish NOMCO corpus: Multimodal interaction in first acquaintance conversations. Language Resources and Evaluation, 51(2), 463–494.
Paggio, P., Allwood, J., Ahlsén, E., Jokinen, K., & Navarretta, C. (2010). The NOMCO Multimodal Nordic Resource - Goals and Characteristics. In: Calzolari, N., Choukri, K., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S., Rosner, M., & Tapias, D. (Eds.), In: Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC’10). Valletta, Malta, May 19–21, European Language Resources Association (ELRA).
Rayson, P. 2009. Wmatrix: A web-based corpus processing environment 7 [computer software] Computing Department, Lancaster University, Lancaster. Available from: http://ucrellancsacuk/wmatrix/. Accessed 23 May 2019.
Rehm, M., André, E., Bee, N., Endrass, B., Wissner, M., Nakano, Y., Lipi, A. A., Nishida, T., & Huang, H.-H. (2008). Creating standardized video recordings of multimodal interactions across cultures. In M. Kipp, J.-C. Martin, P. Paggio, & D. Heylen (Eds.), Multimodal corpora from models of natural interaction to systems and applications, LNAI 5509 (pp. 138–159). New York: Springer.
Rossini, N. (2012). Reinterpreting Gesture as Language. Amsterdam: IOS Press.
Royce, T. D., & Bowcher, W. (2006). New directions in the analysis of multimodal discourse. London: Routledge.
Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organisation of turn taking for conversation. Language, 50(4–1), 696–735.
Saferstein, B. (2004). Digital technology and methodological adaptation: Text on video as a resource for analytical reflexivity. Journal of Applied Linguistics, 1(2), 197–223.
Scollon, R., & Levine, P. (2004). Multimodal discourse analysis as the confluence of discourse and technology. Hillsdale: Lawrence Erlbaum Associates.
Scott, M. 2017. WordSmith Tools version 7 [Computer software]. Lexical Analysis Software, Stroud. Available from: http://www.lexically.net/wordsmith/. Accessed 23 May 2019.
Sinclair, J. (2005). Corpus and text – Basic principles. In M. Wynne (Ed.), Developing linguistic corpora: A guide to good practice (pp. 1–16). Oxford: Oxbow Books.
Steen, F., & Turner, M. B. (2013). Multimodal construction grammar. In M. Borknet, B. Dancygier, & J. Hinnell (Eds.), Language and the creative mind. Stanford: CSLI Publications.
Stubbs, M. (1996). Text and corpus analysis: Computer-assisted studies of language and culture. Oxford: Blackwell.
Sweetser, E., & Fauconnier, G. (1996). Cognitive links and domains: Basic aspects of mental space theory. In G. Fauconnier & E. Sweetser (Eds.), Spaces, worlds and grammars (pp. 1–28). Chicago: University of Chicago Press.
Tannen, D. (1982). Ethnic style in male-female conversation. In J. J. Gumperz (Ed.), Language and social identity (pp. 217–231). Cambridge: Cambridge University Press.
Tennent, P., Crabtree, A., & Greenhalgh, C. (2008). Ethno-goggles: supporting field capture of qualitative material. In: Proceedings of the 4th Annual International e-Social Science Conference [online], University of Manchester, Manchester.
van Leeuwen, T. (2005). Introducing social semiotics. London: Routledge.
Ventola, E., Cassily, C., & Kaltenbacher, M. (Eds.). (2004). Perspectives on multimodality. John Benjamins: Amsterdam and Philadelphia.
Wittenburg, P., Broeder, D., & Sloman, B. (2000). Meta-description for language resources EAGLES/ISLE White paper [online report], available at: http://www.mpi.nl/isle/documents/papers/white_paper_11.pdf. Accessed 23 May 2019.
Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., & Sloetjes, H. (2006). ELAN: a professional framework for multimodality research. In: Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006), 1556–1559. ELRA, Genoa, Italy.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Further Reading
Further Reading
-
Allwood, J. 2008. Multimodal Corpora. In Corpus Linguistics. An International Handbook, eds. Lüdeling, A. and Kytö, M., 207–225. Berlin: Mouton de Gruyter.
Allwood’s (2008) chapter includes a particularly detailed focus on the definition and annotation of gestures and the steps towards developing a standardised approach to such within the field of multimodal corpus linguistics. Extensive examples of what may be analysed using multimodal corpora are also offered, from the examination of communication and power, emotion, consciousness and awareness to the analysis of multimodality within and across specific types of media. Allwood also provides some more practical reflections on the possible applications of multimodal corpus research, from the development of tools to support human-human and human-machine communication to the construction of multimodal translation and interpretation systems.
-
Knight, D. 2011a. Multimodality and Active Listenership: A Corpus Approach. London: Continuum.
Building on some of the discussions outlined by Allwood (2008), Knight’s monograph takes readers through every step of multimodal corpus design and construction and provides a worked example of multimodal corpus analysis. This analysis focuses on the examination of the use of, and the relationship between, spoken and non-verbal forms of backchanneling behaviour, and a particular sub-set of gestural forms: head nods. This investigation is undertaken by means of analysing the patterned use of specific forms and functions of backchannels within and across sentence boundaries, in 5 h of dyadic (supervision) data taken from the NMMC (Nottingham Multi-Modal Corpus). Knight discusses the validity and applicability of different existing categorisations of backchannels to multi-modal corpus data, and examines the requirements for a redefinition of these considering the findings resulting from the analyses.
-
Gu, Y. 2006. Multimodal text analysis: A corpus linguistic approach to situated discourse. Text and Talk 26(2): 127–167.
In a similar way to Knight, Gu concentrates on providing some guidelines for an approach to multimodal corpus analysis. This study is situated mainly within the MDA paradigm, so it focuses on small-scale corpora of situated discourse, utilizing discourse analytic methods as the initial point of departure, but discusses how such an approach can be extended with the integration of corpus methods. Gu focuses on different modalities within the analysis, beyond what the majority of multimodal corpus studies typically afford, making this work of particular relevance to the final section of this chapter: projections for the future directions of this field.
-
Kipp, M., Martin, J-C., Paggio, P. and Heylen, D. (Eds). 2009. Multimodal Corpora: from models of natural interaction to systems and applications. Lecture Notes in Artificial Intelligence 5509. Berlin: Springer-Verlag.
This edited collection provides a state-of-the-art survey of research into multimodal corpus construction and analysis, taken from the ‘Multimodal Corpora: From Models of Natural Interaction to Systems and Applications’ workshop held in conjunction with the 6th LREC (International Conference for Language Resources and Evaluation) conference, held in Marrakech in 2008. The volume presents research from different fields including computational linguistics, human-human interaction, signal processing, artificial intelligence, psychology and robotics, comprising six contributions from the workshop and additional invited research articles from key multimodal corpora projects, including AMI. The volume discusses a range of topics from corpus construction to analysis, so is invaluable for researchers who are perhaps new to the field and need some advice on carrying out their own work in this area.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Knight, D., Adolphs, S. (2020). Multimodal Corpora. In: Paquot, M., Gries, S.T. (eds) A Practical Handbook of Corpus Linguistics. Springer, Cham. https://doi.org/10.1007/978-3-030-46216-1_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-46216-1_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-46215-4
Online ISBN: 978-3-030-46216-1
eBook Packages: Religion and PhilosophyPhilosophy and Religion (R0)