Multimodal Resources in Face-to-Face Interviews

  • Patrick Kiernan


Beginning with a quote from one of the study participants who emphasised the importance of communicating multimodally in study abroad contexts, this chapter makes the case for a multimodal approach. It introduces systemic functional linguistics (SFL) and multimodality and provides an overall rationale for the methodology which is expounded over Part I of the book. The multimodal account of spoken discourse treats the verbal language of speech and the semiotics of gesture as the overarching semiotic modes. Each of these modes is explored in turn through the three complimentary SFL perspectives of ideational, interpersonal and textual meaning. This chapter includes a brief introduction to multimodality and an overview of relevant work in gesture studies.


  1. Adolphs, Svenja, and Ronald Carter. 2013. Spoken Corpus Linguistics: From Monomodal to Multimodal. London: Routledge.Google Scholar
  2. Adolphs, S., D. Knight, and R. Carter. 2015. Beyond Modal Spoken Corpora: A Dynamic Approach to Tracking Language in Context. Corpora and Discourse Studies: Integrating Discourse and Corpora: 41–62.
  3. Bamberg, Michael G.W. 1997a. A Constructivist Approach to Narrative Development. In Narrative Development, ed. Michael Bamberg, 89–132. Mahwah: Lawrence Erlbaum Associates.Google Scholar
  4. ———., ed. 1997b. Oral Versions of Personal Experience: Three Decades of Narrative Analysis (Special Issue of Journal of Narrative and Life History: Volume 7, Numbers 1–4). Mahwah: Lawrence Erlbaum Associates.Google Scholar
  5. Bateman, John A. 2011. The Decomposability of Semiotic Modes. In Multimodal Studies: Exploring Issues and Domains, ed. Kay L. O’Halloran and Bradley A. Smith. New York: Routledge.Google Scholar
  6. Bateman, John A., Janina Wildfeuer, and Tuomo Hiippala. 2017. Multimodality. Berlin: Walter de Gruyter.CrossRefGoogle Scholar
  7. Blache, Philippe, Roxane Bertrand, and Gaëlle Ferré. 2009. Creating and Exploiting Multimodal Annoted Corpora: The TOMA Project. In Multimodal Copora: From Models of Natural Interaction to Systems and Applications, ed. Michael Kipp, Jean-Claude Martin, Patrizia Paggio, and Dirk Heylen, 38–53. Berlin: Springer.CrossRefGoogle Scholar
  8. Borg, James. 2013. Body Language: How to Know What’s REALLY Being Said. London: Pearson.Google Scholar
  9. Brazil, David. 1994. Pronunciation for Advanced Learners of English. Cambridge: Cambridge University Press.Google Scholar
  10. ———. 1995. A Grammar of Speech. In Describing English Conversation, ed. John Sinclair and Ronald Carter. Oxford: Oxford University Press.Google Scholar
  11. ———. 1997. The Communicative Value of Intonation in English. Cambridge: Cambridge University Press.Google Scholar
  12. Brown, Amanda. 2010. Gesture and Viewpoint in Japanese and English: Cross-Linguistic Interactions Between Languages in One Speaker. In Gestures in Language Development, ed. M. Gullberg and K. d’e Bot, 111–133. Amsterdam: John Benjamins.Google Scholar
  13. Butterworth, B., and G. Beattie. 1978. Gestures and Silence as Indicators of Planning in Speech. In Recent Advances in the Psychology of Language: Formal and Experimental Approaches, ed. R. Campbell and P.T. Smith. London: Plenum.Google Scholar
  14. Carneigie, Dale. 2006. Public Speaking for Success. New York: Jeremy P. Tarcher.Google Scholar
  15. Cassell, Justine, and David McNeill. 1991. Gesture and the Poetics of Prose. Poetics Today 12 (3): 375–404.CrossRefGoogle Scholar
  16. Chomsky, Noam. 1965. Aspects of the Theory of Syntax. Cambridge, MA: MIT Press.Google Scholar
  17. Colletta, Jean-Marc, and Michèle Guidetti, eds. 2012. Gesture and Multimodal Development. Amsterdam: John Benjamins.Google Scholar
  18. Cook, Vivian, and Mark Newson. 2007. Comsky’s Universal Grammar. 3rd ed. Hoboken: Wiley.Google Scholar
  19. Costandi, Moheb. 2016. Neuroplasticity. Cambridge, MA: MIT Press.Google Scholar
  20. Coulthard, Malcolm, ed. 1992. Advances in Spoken Discourse Analysis. London: Routledge.Google Scholar
  21. Eggins, Suzanne, and Diana Slade. 1997. Analysing Casual Conversation. London: Cassell.Google Scholar
  22. Georgakopoulou, Alexandra. 1998. Conversational Stories as Performances: The Case of Greek. Narrative Inquiry 8 (2): 319–350.CrossRefGoogle Scholar
  23. Goffman, Erving. 1959. The Presentation of Self in Everyday Life. New York: Doubleday Anchor Books.Google Scholar
  24. ———. 1967. Interaction Ritual: Essays in Face-to-Face Behaviour. Chicago: Aldine Publishing.Google Scholar
  25. Goldin-Meadow, Susan. 2003. Hearing Gesture: How Our Hands Help Us to Think. Cambridge, MA: Harvard University Press.Google Scholar
  26. Goto Butler, Yuko. 2005. English Language Education in Japanese Elementary Schools: Analysis and Suggestions Based on East Asian Perspectives. Tokyo: Sanseido.Google Scholar
  27. Gullberg, M. 1998. Gesture as a Communication Strategy in Second Language Discourse. Lund: Lund University Press.Google Scholar
  28. Gullberg, Marianne, and Kees de Bot, eds. 2010. Gesture in Language Development. Amsterdam: John Benjamins.Google Scholar
  29. Gullberg, Marianne, Kees de Bot, and Virginia Volterra. 2010. Gesture and Some Key Issues in the Study of Language Development. In Gesture in Language Development, ed. Marianne Gullberg and Kees de Bot, 3–33. Amsterdam: John Benjamins.CrossRefGoogle Scholar
  30. Hall, Edward T. 1969. The Hidden Dimension. New York: Anchor Books.Google Scholar
  31. Halliday, M.A.K. 1978. Meaning and the Construction of Reality in Early Childhood. In Modes of Perceiving and Processing Information, ed. Herbert L. Pick and Elliot Saltzman, 67–96. Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  32. ———. 1979. Language as a Social Semiotic: The Social Interpretation of Language and Meaning. London: Edward Arnold.Google Scholar
  33. ———. 2002a. Linguistic Studies of Text and Discourse: Volume 2 in the Collected Works of M. A. K. Halliday. Ed. Jonathan Webster. London: Continuum.Google Scholar
  34. ———. 2002b. On Grammar: Volume 1 in the Collected Works of M. A. K. Halliday. Ed. Jonathan Webster. London: Continuum.Google Scholar
  35. ———. 2004. The Language of Early Childhood: Volume 4 in the Collected Works of M. A. K. Halliday. Ed. Jonathan Webster. London: Continuum.Google Scholar
  36. ———. 2006. The Language of Science: Volume 5 in the Collected Works of M. A. K. Halliday. Ed. Jonathan J. Webster. London: Continuum.Google Scholar
  37. ———. 2009a. Language and Society: Volume 10 in the Collected Works of M. A. K. Halliday. Ed. Jonathon J. Webster. Paperback ed., Collected Works of M. A. K. Halliday. London: Continuum.Google Scholar
  38. ———. 2009b. Studies in Chinese Language: Volume 8 in the Collected Works of M. A. K. Halliday. Ed. Jonathon J. Webster. London: Continuum.Google Scholar
  39. ———. 2013. Halliday in the 21st Century: Volume 11 in the Collected Works of M. A. K Halliday. Ed. Jonathan Webster. London: Bloomsbury.Google Scholar
  40. Halliday, M.A.K., and Christian Matthiessen. 2013. Halliday’s Introduction to Functional Grammar. 4th ed. London: Routledge.Google Scholar
  41. Hasan, Ruqaiya. 2009. Semantic Variation: Meaning in Society in and in Sociolinguistics. London: Equinox.Google Scholar
  42. ———. 2015. Systemic Functional Linguistics: Halliday and the Evolution of the Social Semiotic. In The Bloomsbury Companion to M.A.K. Halliday, ed. Jonathan J. Webster, 101–136. London: Bloomsbury.Google Scholar
  43. Herman, David. 2010. Word-Image/Utterance-Gesture: Case Studies in Multimodal Storytelling. In New Perspectives on Narrative and Multimodality, ed. Ruth E. Page. New York: Routledge.Google Scholar
  44. Hofstede, Geert. 1986. Cultural Differences in Teaching and Learning. International Journal of Intercultural Relations 10: 301–320.CrossRefGoogle Scholar
  45. Hutchby, Ian, and Robin Wooffitt. 1998. Conversation Analysis: Principles, Practices and Applications. Cambridge: Polity Press.Google Scholar
  46. Jones, Rodney H. 2016. Spoken Discourse. London: Bloomsbury Academic.Google Scholar
  47. Kendon, Adam. 2004. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  48. Kolbert, Elizabeth. 2014. The Sixth Extinction: An Unnatural History. London: Bloomsbury.Google Scholar
  49. Kress, Gunther, and Theo van Leeuwen. 2006. Reading Images: The Grammar of Visual Design. 2nd ed. Abingdon: Routledge.Google Scholar
  50. Kress, Gunther, Carey Jewitt, Jon Ogborn, and Tsatsarelis Charalampos. 2001. Multimodal Teaching and Learning: The Rhetorics of the Science Classroom. London: Bloomsbury.Google Scholar
  51. Labov, Wiliam. 1972. Language in the Inner City: Studies in the Black English Vernacular. Philadelphia: University of Pennsylvania Press.Google Scholar
  52. ———. 1997. Some Further Steps in Narrative Analysis. Journal of Narrative and Life History 7 (1–4): 395–415.CrossRefGoogle Scholar
  53. Leary, Mark R. 1996. Self-Presentation: Impression Management and Interpersonal Behavior. Boulder: Westview Press.Google Scholar
  54. Leavens, David A., and William D. Hopkins. 1998. Intentional Communication by Chimpanzees: A Cross-Sectional Study of the Use of Referential Gestures. Developmental Psychology 34 (5): 813–822.CrossRefGoogle Scholar
  55. Luelsdorff, Philip A., ed. 1994. The Prague School of Structural and Functional Linguistics. Amsterdam: John Benjamins.Google Scholar
  56. Markee, Numa. 2000. Conversation Analysis. Mahwah: Lawrence Erlbaum Associates.Google Scholar
  57. Martin, J.R., and David Rose. 2003. Working with Discourse: Meaning Beyond the Clause. London: Continuum.Google Scholar
  58. ———. 2008. Genre Relations: Mapping Culture. London: Equinox.Google Scholar
  59. Martin, J.R., and Peter R.R. White. 2005. The Language of Evaluation: Appraisal in English. Basingstoke: Palgrave Macmillan.CrossRefGoogle Scholar
  60. McNeil, D. 1992. Hand and Mind: What Gestures Reveal About Thought. Chicago: University of Chicago Press.Google Scholar
  61. McNeill, David. 2000. Language and Gesture: Window into Thought and Action. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  62. ———. 2012. How Language Began: Gesture and Speech in Human Evolution. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  63. ———. 2014. Gesture: A Psycholinguistic Approach. Accessed 28 March.
  64. Melrose, Robin. 2015. The Communicative Syllabus: A Systemic-Functional Approach to Language Teaching. London: Pinter Publishing.Google Scholar
  65. Morris, Desmond. 2002. People Watching: The Desmond Morris Guide to Body Language. New York: Vintage.Google Scholar
  66. Norrick, Neal R. 2000. Conversational Narrative: Storytelling in Everyday Talk. In Current Issues in Linguistic Theory, ed. E.F. Konrad Koerner. Amsterdam: John Benjamins.Google Scholar
  67. O’Halloren, Kay L. 2016. A Multimodal Approach to Classroom Discourse. London: Equinox.Google Scholar
  68. Ochs, Elinor, and Lisa Capps. 2001. Living Narrative: Creating Lives in Everyday Storytelling. Cambridge, MA: Harvard University Press.Google Scholar
  69. Page, Ruth E., ed. 2010. New Perspectives on Narrative and Multimodality. New York: Routledge.Google Scholar
  70. ———. 2012. The Linguistics of Self-Branding and Micro-Celebrity in Twitter: The Role of Hashtags. Discourse & Communication 6 (2): 181–201.CrossRefGoogle Scholar
  71. Palmer, F.R., ed. 1968. Selected Papers of J.R. Firth 1952–59. London: Longmans.Google Scholar
  72. Pavlenko, Aneta. 2014. The Bilingual Mind: And What It Tells Us About Language and Thought. Cambridge: Cambridge University Press.Google Scholar
  73. Pinker, Steven. 1995. The Language Instinct: How the Mind Creates Language: The New Science of Language and Mind. London: Penguin.Google Scholar
  74. Plum, Guenter A. 1988. Text and Contextual Conditioning in Spoken English: A Genre-Based Approach. PhD, Department of Linguistics, University of Sydney.Google Scholar
  75. Reynolds, Garr. 2008. Presentation Zen: Simple Ideas on Presentation Design and Delivery. Berkeley: New Riders.Google Scholar
  76. ———. 2014. Presentation Zen Design: A Simple Visual Approach to Presenting in Today’s World. 2nd ed. Berkeley: New Riders.Google Scholar
  77. Sacks, Harvey. 1995. Lectures on Conversation: Volumes I & II. Oxford: Blackwell Publishers.CrossRefGoogle Scholar
  78. Saussure, Ferdinand. 1986. Course in General Linguistics. Trans. Roy Harris. La Salle: Open Court.Google Scholar
  79. Savignon, Sandra J., ed. 2002. Interpreting Communicative Language Teaching: Contexts and Concerns in Teacher Education. New Haven: Yale University Press.Google Scholar
  80. Scollon, Ron, and Suzanne Wong Scollon. 2001. Intercultural Communication: A Discourse Approach. 2nd ed. Malden: Blackwell Publishing.Google Scholar
  81. Streeck, Jürgen. 2009. Gesturecraft: The Manufacturing of Meaning. Amsterdam: John Benjamins.CrossRefGoogle Scholar
  82. Stubbs, Michael. 1983. Discourse Analysis: The Sociolinguistic Analysis of natural Language. Chicago: The University of Chicago Press.Google Scholar
  83. Tomasello, Michael. 2003. Constructing a Language: A Usage-Based Theory of Language Acquisition. Cambridge, MA: Harvard University Press.Google Scholar
  84. ———. 2014. A Natural History of Human Thinking. Cambridge, MA: Harvard University Press.CrossRefGoogle Scholar
  85. van Leeuwen, Theo. 2012. Rhythm and Multimodal Semiosis. In Semiotic Margins: Meaning in Multimodalities, ed. Shoshana Dreyfus, Susan Hood, and Maree Stenglin, 168–177. London: Continuum.Google Scholar
  86. Wolfson, Nessa. 1978. A Feature of Performed Narrative: The Conversational Historical Present. Language in Society 1 (7): 215–237.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2019

Authors and Affiliations

  • Patrick Kiernan
    • 1
  1. 1.KenkyutoMeiji UniversityTokyoJapan

Personalised recommendations