Nonverbal Feedback in Interactions

  • Kristiina Jokinen


Understanding how nonverbal aspects of communication support, complement, and in some cases, override verbal communication is necessary for human interactions. It is also crucial for designing and implementing interactive systems that aim at supporting flexible interaction management using natural language with users. In particular, the need for more comprehensive communication has become obvious in theubiquitous computing context where context-aware applications and automatic services require sophisticated knowledge management and adaptation to various user needs. Interactions with smart objects, services, and environments need to address challenges concerning natural, intuitive, easy, and friendly interaction. For instance, the Roadmap for Smart Human Environments (Plomp et al., 2002) envisages that smart environments will be populated by several context-aware devices which communicate with each other and with the users. The systems will identify their current context of use, adapt their behaviour accordingly, and also allow natural interaction.

In this chapter, I discuss verbal and nonverbal communication especially for the purposes of designing and developing interactive systems. I report on machinelearning experiments conducted on annotated gesture and facial expression data, and focus on the feedback and turn-taking processes that are important in building shared understanding of the semantics and flow of interaction.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Allwood, J. (2001). The structure of dialog. In M. Taylor, D. Bouwhuis, & F. Nel (Eds.), The structure of multimodal dialogue II (pp. 3–24). Amsterdam: Benjamins.Google Scholar
  2. 2.
    Allwood, J, Cerrato, L., Jokinen, K., Navarretta, K., & Paggio, P. (2007). The MUMIN coding scheme for the annotation of feedback, turn management and sequencing phenomena. In J.C. Martin, P. Paggio, P. Kuenlein, R. Stiefelhagen, & F. Pianesi (Eds), Multimodal corpora for modelling human multimodal behaviour. Special issue of the International Journal of Language Resources and Evaluation, 41(3–4), 273–287.
  3. 3.
    Allwood, J., Traum, D., & Jokinen, K. (2000). Cooperation, dialogue and ethics. Special issue on collaboration, cooperation and conflict in dialogue systems, International Journal of Human— Computer Studies, 53(6), 871–914.MATHCrossRefGoogle Scholar
  4. 4.
    André, E., & Pelachaud, C. (forthcoming). Interacting with embodied conversational agents. In K. Jokinen & F. Cheng (Eds.), New trends in speech-based interactive systems. New York: Springer.Google Scholar
  5. 5.
    Arbib, M. (2003). The evolving mirror system: A neural basis for language readiness. In M. Christiansen and S. Kirby (Eds.), Language evolution (pp. 182–200).Oxford: Oxford University Press,Google Scholar
  6. 6.
    Barsalou, L. W. (1999) Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577–660.Google Scholar
  7. 7.
    Beira, R., Lopes, M., Praca, M., Santos-Victor, J., Bernardino, A., Mettay, G., Becchiz, F., & Saltar, R. (2006). Design of the robot-cub (iCub) head. In Proceedings of the IEEE International Conference on Robotics and Automation, Orlando, FL (pp. 94–100).Google Scholar
  8. 8.
    Campbell, N. (2007). On the use of nonverbal speech sounds in human communication. In N. Campbell (Ed.), Verbal and Nonverbal Communication Behaviors (LNAI 4775, pp.117– 128). New York: Springer.CrossRefGoogle Scholar
  9. 9.
    Campbell, N., & Jokinen, K. (2008). Non-verbal information resources for constructive dialogue management. Tutorial at the LREC 2008. Marrakech, Marocco.Google Scholar
  10. 10.
    Campbell, N., & Ohara, R. (2005). How far can non-verbal information help us follow a conversation? Preliminary experiments with speech-style and gesture tracking. In Proceedings of the ATR Symposium on the Cross-Modal Processing of Faces & Voices. No laughing matter.Google Scholar
  11. 11.
    Cassell, J., Sullivan J., Prevost, S., & Churchill, E. (Eds.)(2000). Embodied conversational agents. Cambridge, MA: MIT Press.Google Scholar
  12. 12.
    Clark, H., & Wilkes-Gibbs, D. (1986). Referring as a collaborative process. Cognition 22, 1–39.CrossRefGoogle Scholar
  13. 13.
    Douglas, C. E., Campbell, N., Cowie, R., & Roach, P. (2003). Emotional speech: Towards a new generation of databases. Speech Communication, 40, 33–60.MATHCrossRefGoogle Scholar
  14. 14.
    Duncan, S., Jr., & Fiske, D.W. (1977). Face-to-face interaction: Research, methods and theory. Hillsdale, NJ: Lawrence Erlbaum. Distributed by John Wiley & Sons.Google Scholar
  15. 15.
    Garrod, S., & Doherty, G. (1994). Conversation, co-ordination and Convention: An empirical investigation of how groups establish linguistic conventions. Cognition, 53,181–215.CrossRefGoogle Scholar
  16. 16.
    Gibson, J. J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.Google Scholar
  17. 17.
    Harnard, S. (1990). The symbol grounding problem. Physica D, 42, 335–346.CrossRefGoogle Scholar
  18. 18.
    Jokinen, K. (2000). Learning dialogue systems. In Proceedings of the LREC Workshop from Spoken Dialogue to Full Natural Interactive Dialogue. Athens (pp. 13–17).Google Scholar
  19. 19.
    Jokinen, K. (2007). Interaction and mobile route navigation application. In L. Meng, A. Zipf, & S. Winter (Eds.), Map-based mobile services—Usage context, interaction and application. New York: Springer Series on Geoinformatics.Google Scholar
  20. 20.
    Jokinen, K. (2008). Constructive Dialogue Management — Speech interaction and rational agents. Hoboken, NJ: John Wiley & Sons.Google Scholar
  21. 21.
    Jokinen, K., & Hurtig, T. (2006). User expectations and real experience on a multimodal interactive system. In Proceedings of the Interspeech 2006, Pittsburgh, PA.Google Scholar
  22. 22.
    Jokinen, K., Paggio, P., & Navarretta, C. (2008). Distinguishing the communicative functions of gestures — An experiment with annotated gesture data. In Proceedings of the Conference on Machine-Learning in Multimodal Interaction.Google Scholar
  23. 23.
    Jokinen, K., & Ragni, A. (2007). On the annotation and analysis of multimodal corpus. In Proceedings of the 3rd Baltic Conference on Human Technology. Kaunas, Lithuania.Google Scholar
  24. 24.
    Katagiri, Y. (2005). Interactional alignment in collaborative problem solving dialogues, In Proceedings of the 9th International Pragmatics Conference, Riva del Garda Italy.Google Scholar
  25. 25.
    Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge: Cambridge University Press.Google Scholar
  26. 26.
    Kipp, M. (2001). Anvil — A generic annotation tool for multimodal dialogue. In Proceedings of the Seventh European Conference on Speech Communication and Technology (pp. 1367–1370).Google Scholar
  27. 27.
    Maes, P. (Ed.) (1990). Designing autonomous agents: Theory and practice from biology to engineering and back. Cambridge, MA: MIT Press.Google Scholar
  28. 28.
    Mandler, J. (2004). The foundations of mind: Origins of conceptual thought. Oxford: Oxford University Press.Google Scholar
  29. 29.
    Mavridis, N., & Roy, D. (2006). Grounded situation models for robots: where words and percepts meet. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).Google Scholar
  30. 30.
    McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.Google Scholar
  31. 31.
    Norman, D. A. (1988). The psychology of everyday things. New York: Basic Books.Google Scholar
  32. 32.
    Norros, L., Kaasinen, E., Plomp, J., & Rämä, P. (2003) Human—technology interaction research and design. VTT roadmaps. VTT Research Notes 2220. Espoo: VTT Industrial Systems,Google Scholar
  33. 33.
    Peirce, C. S. (1931). Elements of logic. Collected papers of Charles Sanders Peirce. C. Hartshorne and P. Weiss (Eds.) (vol. 2). Cambridge, MA: Harvard University Press.Google Scholar
  34. 34.
    Pickering, M. & Garrod, S. (2004). Towards a mechanistic psychology of dialogue, Behavioral and Brain Sciences 27, 169–226.Google Scholar
  35. 35.
    Plomp, J., Ahola, J., Alahuhta, P., Kaasinen, E., Korhonen, I., Laikari, A., Lappalainen, V., Pakanen, J., Rentto, K., & Virtanen, A. (2002). Smart human environments. In: Sipilä, M. (Ed.). Communications Technologies. The VTT Roadmaps. Espoo: Technical Research Centre of Finland, VTT Research Notes 2146, pp. 61–81.Google Scholar
  36. 36.
    Steels, L. (2003). Evolving grounded communication for robots. Trends in Cognitive Science, 7(7), 308–312.CrossRefGoogle Scholar
  37. 37.
    Swerts, M., & Krahmer, E. (2005). Audiovisual prosody and feeling of knowing. Journal of Memory and Language, 53, 81–94.CrossRefGoogle Scholar
  38. 38.
    Tao, J. (forthcoming). Multimodal information processing for affective computing. In K. Jokinen and F. Cheng (Eds.), New trends in speech-based interactive systems. New York: Springer.Google Scholar
  39. 39.
    Thompson, E. (2001). Empathy and consciousness. Journal of Consciousness Studies, 8, 1–32.Google Scholar
  40. 40.
    Tomasello, M. (1992). First verbs: A case study of early grammatical development. Cambridge: Cambridge University Press.Google Scholar
  41. 41.
    Traum, D. (1999). Computational models of grounding in collaborative systems. In Working Papers of the AAAI}Fall Symposium on Psychological Models of Communication in Collaborative Systems, AAAI, Menlo Park, CA (pp. 124–131).Google Scholar
  42. 42.
    Weiser, M. (1991). The computer for the twenty-first century. Scientific American, 265(3): 94–10.CrossRefGoogle Scholar
  43. 43.
    Witten, I. H., & Frank, E. (2005). Data mining: Practical machine learning tools and techniques (2nd ed.). San Francisco: Morgan Kaufmann.MATHGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2009

Authors and Affiliations

  • Kristiina Jokinen
    • 1
  1. 1.University of Helsinki and University of TampereFinland

Personalised recommendations