Multimodal Feedback in First Encounter Interactions

  • Kristiina Jokinen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8007)


Human interactions are predominantly conducted via verbal communication which allows presentation of sophisticated propositional content. However, much of the interpretation of the utterances and the speaker’s attitudes are conveyed using multimodal cues such as facial expressions, hand gestures, head movements and body posture. This paper reports some observations on multimodal communication and feedback giving activity in first encounter interactions, and discusses how head, hand, and body movements are used in conversational interactions as means ofvisual interaction management, i.e. unobtrusive ways to control the interaction and construct shared understanding among the interlocutors. The observations and results contribute to the models for coordinating communication in human-human conversations as well as in interactions between humans and intelligent situated agents.


multimodal interaction feedback nodding head movements 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Allwood, J.: An Activity Based Approach to Pragmatics. In: Gothenburg Papers In Theoretical Linguistics 76. Department of Linguistics. Göteborg University (2000)Google Scholar
  2. 2.
    Allwood, J., Cerrato, L., Jokinen, K., Navarretta, C., Paggio, P.: The MUMIN Coding Scheme for the Annotation of Feedback, Turn Management and Sequencing Phenomena. Multimodal Corpora for Modelling Human Multimodal Behaviour. Special Issue of the International Journal of Language Resources and Evaluation 41(3-4), 273–287 (2007)Google Scholar
  3. 3.
    André, E., Pelachaud, C.: Interacting with Embodied Conversational Agents. In: Jokinen, K., Cheng, F. (eds.) Speech-based Interactive Systems: Theory and Applications. Springer (2009)Google Scholar
  4. 4.
    Argyle, M., Cook, M.: Gaze and Mutual Gaze. Cambridge University Press (1976)Google Scholar
  5. 5.
    Battersby, S.: Moving Together: the organization of Non-verbal cues during multiparty conversation. PhD Thesis, Queen Mary, University of London (2011)Google Scholar
  6. 6.
    Clark, H.H., Schaefer, E.F.: Contributing to Discourse. Cognitive Science 13, 259–294 (1989)CrossRefGoogle Scholar
  7. 7.
    Feldman, R.S., Rim, B.: Fundamentals of Nonverbal Behavior. Cambridge University Press (1991)Google Scholar
  8. 8.
    Hart, C.L., Fillmore, D.G., Griffith, J.D.: Indirect Detection of Deception: Looking for Change. Current Research in Social Psychology 1(9), 134–142 (2009)Google Scholar
  9. 9.
    Jokinen, K.: Rational communication and affordable natural language interaction for ambient environments. In: Lee, G.G., Mariani, J., Minker, W., Nakamura, S. (eds.) IWSDS 2010. LNCS, vol. 6392, pp. 163–168. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  10. 10.
    Jokinen, K.: Pointing gestures and synchronous communication management. In: Esposito, A., Campbell, N., Vogel, C., Hussain, A., Nijholt, A. (eds.) COST 2102 Int. Training School 2009. LNCS, vol. 5967, pp. 33–49. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Jokinen, K., Furukawa, H., Nishida, M., Yamamoto, S.: Gaze and Turn-taking behaviour in Casual Conversational Interactions. ACM Trans. Interactive Intelligent Systems (2013)Google Scholar
  12. 12.
    Jokinen, K., Pärkson, S.: Synchrony and copying in conversational interactions. In: Proceedings of the 3rd Nordic Symposium on Multimodal Communication, vol. 15, pp. 18–24 (2011)Google Scholar
  13. 13.
    Jokinen, K., Wilcock, G.: Multimodal Signals and Holistic Interaction Structuring. In: Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012), Mumbai, India (2012)Google Scholar
  14. 14.
    Kendon, A.: Gesture. Visible Action as Utterance. Cambridge University Press (2005)Google Scholar
  15. 15.
    Kipp, M.: Anvil - A Generic Annotation Tool for Multimodal Dialogue. In: Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech), pp. 1367–1370 (2001)Google Scholar
  16. 16.
    Lee, J., Marsella, S.C., Traum, D.R., Gratch, J., Lance, B.: The Rickel Gaze Model: A Window on the Mind of a Virtual Human. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 296–303. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  17. 17.
    Levitan, R., Gravano, A., Hirschberg, J.: Entrainment in Speech Preceding Back-channels. In: Proceedings of ACL 2011, pp. 113–117 (2011)Google Scholar
  18. 18.
    Mancini, M., Castellano, G., Bevacqua, E., Peters, C.: Copying Behaviour of Expressive Motion. In: Gagalowicz, A., Philips, W. (eds.) MIRAGE 2007. LNCS, vol. 4418, pp. 180–191. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  19. 19.
    Nakano, Y., Nishida, T.: Attentional Behaviours as Nonverbal Communicative Signals in Situated Interactions with Conversational Agents. In: Nishida, T. (ed.) Engineering Approaches to Conversational Informatics. John Wiley (2007)Google Scholar
  20. 20.
    Navarretta, C., Ahlsén, E., Allwood, J., Jokinen, K., Paggio, P.: Feedback in Nordic First-Encounters: a Comparative Study. In: Proceedings of the Language Resources and Evaluation Conference (LREC-2012), Istanbul, Turkey (2012)Google Scholar
  21. 21.
    Norman, D.A.: The Psychology of Everyday Things. Basic Books, New York (1988)Google Scholar
  22. 22.
    Pickering, M., Garrod, S.: Towards a mechanistic psychology of dialogue. Behavioral and Brain Sciences 27, 169–226 (2004)Google Scholar
  23. 23.
    Toivio, E., Jokinen, K.: Multimodal Feedback Signaling in Finnish. In: Proceedings of the Human Language Technologies – The Baltic Perspective (2012); Published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial LicenseGoogle Scholar
  24. 24.
    Tomasello, M.: First verbs: A case study of early grammatical development. Cambridge University Press, Cambridge (1992)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Kristiina Jokinen
    • 1
  1. 1.University of HelsinkiFinland

Personalised recommendations