Abstract
The objective of this research is twofold. Firstly, we argue that gaze and gesture play an essential part in interactive explanation and that it is thus a multimodal phenomenon. Two corpora are analyzed: (1) a group of teacher novices and experts and (2) a student teacher dyad, both of whom construct explanations of students’ reasoning after viewing videos of student dyads who are solving physics problems. We illustrate roles of gaze in explanations constructed within a group and roles of gesture in explanation constructed within a dyad. Secondly, we show how the analysis of such knowledge-rich empirical data pinpoints particular difficulties in designing human–computer interfaces that can support explanation between humans, or a fortiori, that can support explanation between a human and a computer.
Similar content being viewed by others
Notes
By “interaction role”, we mean either a role that is taken on successively by a participant during a particular interaction, sufficiently so that the role characterizes the participant’s behavior (e.g. animator, evaluator, questioner) or a role that is taken on in a punctual manner to meet a local goal.
Our approach was to define a new speech turn at each participant’s verbalization, regardless of whether the turn was purely regulatory (Kerbrat-Orecchioni 1998). Overlapping speech was recorded.
Automated studies of gaze using eye-tracking could systematize and render verifiable and replicable such data, but this author knows of no eye-tracking tools capable of recording gazes of multiple participants around a table in the way done here.
Transcription conventions include square brackets signifying overlapping speech, the equal sign (“=”), signifying rapidly following speech, the symbol (.) signifying a micro-pause, the symbol (^) signifying a rising tone and underlined words signifying speaker insistence. Finally, gestures and other actions are marked within parentheses.
References
Achinstein, P. (1988). The illocutionary theory of explanation. In J. C. Pitt (Ed.), Theories of explanation (pp. 74–94). New York: Oxford University Press.
Allwood, J. (1998). Cooperation and flexibility in multimodal communication. Lecture Notes in Computer Science, 2155, 113–124.
Antaki, C. (1994). Explaining and arguing. The social organisation of accounts. London: Sage Publications.
Argyle, M., & Cook, M. (1976). Gaze and mutual gaze. Cambridge: Cambridge University Press.
Baker, M., Joab, M., Safar, B., & Schlienger, D. (2000). Introduction: Analyses et modélisations d’explications dans un corpus de dialogues finalisés. Psychologie de l’interaction, 9–10, 7–22.
Bavelas, J. B., Coates, L., & Johnson, T. (2002). Listener responses as a collaborative process: The role of gaze. Journal of Communication, 52, 566–580.
Beattie, G., & Shovelton, H. (2002). An experimental investigation of some properties of individual iconic gestures that mediate their communicative power. British Journal of Psychology, 93, 179–192.
Bolden, G. B. (2003). Multiple modalities in collaborative turn sequences. Gesture, 3(2), 187–212.
Cassell, J., Torres, O. E., & Prevost, S. (1999). Turn taking vs. discourse structure: How best to model multimodal conversation. In Y. Wilks (Ed.), Machine conversations (pp. 143–154). The Hague: Kluwer.
Cohen, P. R., Coulston, R., & Krout, K. (2002). Multimodal interaction during multiparty dialogues: Initial results. In Proceedings of the Fourth IEEE International Conference on Multimodal Interfaces, pp. 448–453.
Cosnier, J. (2000). Le non-verbal dans la relation duelle. Paper presented at the Communication & Organisation GRED/O, Bordeaux.
de Gaulmyn, M.-M. (1991). Expliquer des explications. In E. G. U. Dausendschön-Gay & U. Krafft (Eds.), Linguistische Arbeiten, Linguistische Interaktionsanalysen (pp. 279–314). Tübingen: Max Niemeyer Verlag.
Dicks, B., Soyinka, B., & Coffey, A. (2006). Multimodal ethnography. Qualitative Research, 6, 77–96.
Edwards, D., & Potter, J. (1992). Discursive psychology. London: Sage.
Fussell, S. R., Setlock, L. D., Yang, J., Ou, J., Mauer, E., & Kramer, A. D. I. (2004). Gestures over video streams to support remote collaboration on physical tasks. Human–Computer Interaction, 19, 273–309.
Hollerand, J., & Beattie, G. (2003). Pragmatic aspects of representational gestures. Do speakers use them to clarify verbal ambiguity for the listener? Gesture, 3(2), 127–154.
Kendon, A. (2004a). Gesture: Visible action as utterance. Cambridge: Cambridge University Press.
Kendon, A. (2004b). Review of Susan Goldin-Meadow’s book “Hearing gesture: How our hands help us think”. Gesture, 4(1), 91–107.
Kerbrat-Orecchioni, C. (1998). Les interactions verbales, Approche interactionnelle et structure des conversations Tome 1. Paris: Armand Colin.
Kress, G., & Van Leeuwen, T. (2001) Multi-modal discourse. London: Arnold.
Lund, K. (2003). Analyse de l’activité explicative en interaction: étude de dialogues d’enseignants de physique en formation interprétant les interactions entre élèves. Dissertation, Université J. Fourier, Grenoble, France.
McNeil, D. (Ed.) (2000). Language and gesture. Cambridge: Cambridge University Press.
Mondada, L. (2006). Participants’ online analysis and multimodal practices: Projecting the end of the turn and the closing of the sequence. Discourse Studies, 8(1), 117–129.
Monk, A. F., & Gale, C. (2002). A look is worth a thousand words: Full gaze awareness in video-mediated conversation. Discourse Processes, 33(3), 257–278.
Norris, S. (2006). Multiparty interaction: A multimodal perspective on relevance. Discourse Studies, 8(3), 401–421.
Pallotta, V., Ghorbel, H., Ruch, P., & Coray, G. (2004). An argumentative annotation schema for meeting discussions. In Procedings of the LREC 2004 International Conference, 26–28 May 2004, Lisbon, Portugal.
Pineda, L., & Garza, G. (2000). A model for multimodal reference resolution. Computational Linguistics, 26(2), 139–193.
Rehm, M., & André, E. (2005). Informing the design of embodied conversational agents by analyzing multimodal politeness behaviors in human–human communication. Paper presented at the AISB Symposium for Conversational Informatics.
Roth, W.-M., & Lawless, D. (2002). Scientific investigations, metaphorical gestures, and the emergence of abstract scientific concepts. Learning and Instruction, 12, 285–304.
Ruben, D.-H. (1993). Introduction. In D.-H. Ruben (Ed.), Explanation (pp. 1–16). Oxford: Oxford University Press.
Salvucci, D. D. (1999). Inferring intent in eye-based interfaces: Tracing eye movements with process models. In Human Factors in Computing Systems: CHI 99 Conference Proceedings (pp. 254–261). New York: ACM Press.
Schank, R. C. (1986). Explanation: A first pass. In J. L. Kolodner & C. K. Rieskoede (Eds.), Experience, meaning & reasoning. Mahwah: Lawrence Erlbaum Associates.
Stern, D. N. (1985). The interpersonal world of the infant: A view from psychoanalysis and developmental psychology. New York: Basic Books.
Acknowledgements
My thanks go to Michael Baker, who directed my Ph.D. in which this data was originally collected and to Catherine Kerbrat-Orecchioni, who shared notions of gaze.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lund, K. The importance of gaze and gesture in interactive multimodal explanation. Lang Resources & Evaluation 41, 289–303 (2007). https://doi.org/10.1007/s10579-007-9058-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10579-007-9058-0