Skip to main content
Log in

The importance of gaze and gesture in interactive multimodal explanation

  • Published:
Language Resources and Evaluation Aims and scope Submit manuscript

Abstract

The objective of this research is twofold. Firstly, we argue that gaze and gesture play an essential part in interactive explanation and that it is thus a multimodal phenomenon. Two corpora are analyzed: (1) a group of teacher novices and experts and (2) a student teacher dyad, both of whom construct explanations of students’ reasoning after viewing videos of student dyads who are solving physics problems. We illustrate roles of gaze in explanations constructed within a group and roles of gesture in explanation constructed within a dyad. Secondly, we show how the analysis of such knowledge-rich empirical data pinpoints particular difficulties in designing human–computer interfaces that can support explanation between humans, or a fortiori, that can support explanation between a human and a computer.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. By “interaction role”, we mean either a role that is taken on successively by a participant during a particular interaction, sufficiently so that the role characterizes the participant’s behavior (e.g. animator, evaluator, questioner) or a role that is taken on in a punctual manner to meet a local goal.

  2. Our approach was to define a new speech turn at each participant’s verbalization, regardless of whether the turn was purely regulatory (Kerbrat-Orecchioni 1998). Overlapping speech was recorded.

  3. Automated studies of gaze using eye-tracking could systematize and render verifiable and replicable such data, but this author knows of no eye-tracking tools capable of recording gazes of multiple participants around a table in the way done here.

  4. Transcription conventions include square brackets signifying overlapping speech, the equal sign (“=”), signifying rapidly following speech, the symbol (.) signifying a micro-pause, the symbol (^) signifying a rising tone and underlined words signifying speaker insistence. Finally, gestures and other actions are marked within parentheses.

References

  • Achinstein, P. (1988). The illocutionary theory of explanation. In J. C. Pitt (Ed.), Theories of explanation (pp. 74–94). New York: Oxford University Press.

    Google Scholar 

  • Allwood, J. (1998). Cooperation and flexibility in multimodal communication. Lecture Notes in Computer Science, 2155, 113–124.

    Article  Google Scholar 

  • Antaki, C. (1994). Explaining and arguing. The social organisation of accounts. London: Sage Publications.

    Google Scholar 

  • Argyle, M., & Cook, M. (1976). Gaze and mutual gaze. Cambridge: Cambridge University Press.

  • Baker, M., Joab, M., Safar, B., & Schlienger, D. (2000). Introduction: Analyses et modélisations d’explications dans un corpus de dialogues finalisés. Psychologie de l’interaction, 9–10, 7–22.

    Google Scholar 

  • Bavelas, J. B., Coates, L., & Johnson, T. (2002). Listener responses as a collaborative process: The role of gaze. Journal of Communication, 52, 566–580.

    Article  Google Scholar 

  • Beattie, G., & Shovelton, H. (2002). An experimental investigation of some properties of individual iconic gestures that mediate their communicative power. British Journal of Psychology, 93, 179–192.

    Article  Google Scholar 

  • Bolden, G. B. (2003). Multiple modalities in collaborative turn sequences. Gesture, 3(2), 187–212.

    Article  Google Scholar 

  • Cassell, J., Torres, O. E., & Prevost, S. (1999). Turn taking vs. discourse structure: How best to model multimodal conversation. In Y. Wilks (Ed.), Machine conversations (pp. 143–154). The Hague: Kluwer.

    Google Scholar 

  • Cohen, P. R., Coulston, R., & Krout, K. (2002). Multimodal interaction during multiparty dialogues: Initial results. In Proceedings of the Fourth IEEE International Conference on Multimodal Interfaces, pp. 448–453.

  • Cosnier, J. (2000). Le non-verbal dans la relation duelle. Paper presented at the Communication & Organisation GRED/O, Bordeaux.

  • de Gaulmyn, M.-M. (1991). Expliquer des explications. In E. G. U. Dausendschön-Gay & U. Krafft (Eds.), Linguistische Arbeiten, Linguistische Interaktionsanalysen (pp. 279–314). Tübingen: Max Niemeyer Verlag.

    Google Scholar 

  • Dicks, B., Soyinka, B., & Coffey, A. (2006). Multimodal ethnography. Qualitative Research, 6, 77–96.

    Article  Google Scholar 

  • Edwards, D., & Potter, J. (1992). Discursive psychology. London: Sage.

    Google Scholar 

  • Fussell, S. R., Setlock, L. D., Yang, J., Ou, J., Mauer, E., & Kramer, A. D. I. (2004). Gestures over video streams to support remote collaboration on physical tasks. Human–Computer Interaction, 19, 273–309.

    Article  Google Scholar 

  • Hollerand, J., & Beattie, G. (2003). Pragmatic aspects of representational gestures. Do speakers use them to clarify verbal ambiguity for the listener? Gesture, 3(2), 127–154.

    Article  Google Scholar 

  • Kendon, A. (2004a). Gesture: Visible action as utterance. Cambridge: Cambridge University Press.

    Google Scholar 

  • Kendon, A. (2004b). Review of Susan Goldin-Meadow’s book “Hearing gesture: How our hands help us think”. Gesture, 4(1), 91–107.

    Article  Google Scholar 

  • Kerbrat-Orecchioni, C. (1998). Les interactions verbales, Approche interactionnelle et structure des conversations Tome 1. Paris: Armand Colin.

    Google Scholar 

  • Kress, G., & Van Leeuwen, T. (2001) Multi-modal discourse. London: Arnold.

    Google Scholar 

  • Lund, K. (2003). Analyse de l’activité explicative en interaction: étude de dialogues d’enseignants de physique en formation interprétant les interactions entre élèves. Dissertation, Université J. Fourier, Grenoble, France.

  • McNeil, D. (Ed.) (2000). Language and gesture. Cambridge: Cambridge University Press.

    Google Scholar 

  • Mondada, L. (2006). Participants’ online analysis and multimodal practices: Projecting the end of the turn and the closing of the sequence. Discourse Studies, 8(1), 117–129.

    Article  Google Scholar 

  • Monk, A. F., & Gale, C. (2002). A look is worth a thousand words: Full gaze awareness in video-mediated conversation. Discourse Processes, 33(3), 257–278.

    Article  Google Scholar 

  • Norris, S. (2006). Multiparty interaction: A multimodal perspective on relevance. Discourse Studies, 8(3), 401–421.

    Article  Google Scholar 

  • Pallotta, V., Ghorbel, H., Ruch, P., & Coray, G. (2004). An argumentative annotation schema for meeting discussions. In Procedings of the LREC 2004 International Conference, 26–28 May 2004, Lisbon, Portugal.

  • Pineda, L., & Garza, G. (2000). A model for multimodal reference resolution. Computational Linguistics, 26(2), 139–193.

    Article  Google Scholar 

  • Rehm, M., & André, E. (2005). Informing the design of embodied conversational agents by analyzing multimodal politeness behaviors in human–human communication. Paper presented at the AISB Symposium for Conversational Informatics.

  • Roth, W.-M., & Lawless, D. (2002). Scientific investigations, metaphorical gestures, and the emergence of abstract scientific concepts. Learning and Instruction, 12, 285–304.

    Article  Google Scholar 

  • Ruben, D.-H. (1993). Introduction. In D.-H. Ruben (Ed.), Explanation (pp. 1–16). Oxford: Oxford University Press.

    Google Scholar 

  • Salvucci, D. D. (1999). Inferring intent in eye-based interfaces: Tracing eye movements with process models. In Human Factors in Computing Systems: CHI 99 Conference Proceedings (pp. 254–261). New York: ACM Press.

  • Schank, R. C. (1986). Explanation: A first pass. In J. L. Kolodner & C. K. Rieskoede (Eds.), Experience, meaning & reasoning. Mahwah: Lawrence Erlbaum Associates.

    Google Scholar 

  • Stern, D. N. (1985). The interpersonal world of the infant: A view from psychoanalysis and developmental psychology. New York: Basic Books.

    Google Scholar 

Download references

Acknowledgements

My thanks go to Michael Baker, who directed my Ph.D. in which this data was originally collected and to Catherine Kerbrat-Orecchioni, who shared notions of gaze.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Kristine Lund.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lund, K. The importance of gaze and gesture in interactive multimodal explanation. Lang Resources & Evaluation 41, 289–303 (2007). https://doi.org/10.1007/s10579-007-9058-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10579-007-9058-0

Keywords

Navigation