Skip to main content
SpringerLink
Log in

Generation and Evaluation of Communicative Robot Gesture

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

How is communicative gesture behavior in robots perceived by humans? Although gesture is crucial in social interaction, this research question is still largely unexplored in the field of social robotics. Thus, the main objective of the present work is to investigate how gestural machine behaviors can be used to design more natural communication in social robots. The chosen approach is twofold. Firstly, the technical challenges encountered when implementing a speech-gesture generation model on a robotic platform are tackled. We present a framework that enables the humanoid robot to flexibly produce synthetic speech and co-verbal hand and arm gestures at run-time, while not being limited to a predefined repertoire of motor actions. Secondly, the achieved flexibility in robot gesture is exploited in controlled experiments. To gain a deeper understanding of how communicative robot gesture might impact and shape human perception and evaluation of human-robot interaction, we conducted a between-subjects experimental study using the humanoid robot in a joint task scenario. We manipulated the non-verbal behaviors of the robot in three experimental conditions, so that it would refer to objects by utilizing either (1) unimodal (i.e., speech only) utterances, (2) congruent multimodal (i.e., semantically matching speech and gesture) or (3) incongruent multimodal (i.e., semantically non-matching speech and gesture) utterances. Our findings reveal that the robot is evaluated more positively when non-verbal behaviors such as hand and arm gestures are displayed along with speech, even if they do not semantically match the spoken utterance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Bennewitz M, Faber F, Joho D, Behnke S (2007) Fritz—a humanoid communication robot. In: RO-MAN 07: Proc of the 16th IEEE international symposium on robot and human interactive communication

    Google Scholar 

  2. Bergmann K, Kopp S, Eyssel F (2010) Individualized gesturing outperforms average gesturing—evaluating gesture production in virtual humans. In: Proceedings of the 10th conference on intelligent virtual agents. Springer, Berlin, pp 104–117

    Chapter  Google Scholar 

  3. Billard A, Calinon S, Dillmann R, Schaal S (2008) Robot programming by demonstration. In: Siciliano B, Khatib O (eds) Handbook of robotics. Springer, New York, pp 1371–1394

    Chapter  Google Scholar 

  4. Calinon S, Billard A (2007) Learning of gestures by imitation in a humanoid robot. In: Dautenhahn K, Nehaniv C (eds) Imitation and social learning in robots, humans and animals: behavioural, social and communicative dimensions. Cambridge University Press, Cambridge, pp 153–177

    Chapter  Google Scholar 

  5. Cassell J, Bickmore T, Campbell L, Vilhjálmsson H, Yan H (2000) Human conversation as a system framework: designing embodied conversational agents. In: Embodied conversational agents. MIT Press, Cambridge, pp 29–63

    Google Scholar 

  6. Gienger M, Janßen H, Goerick S (2005) Task-oriented whole body motion for humanoid robots. In: Proceedings of the IEEE-RAS international conference on humanoid robots, Tsukuba, Japan

    Google Scholar 

  7. Goldin-Meadow S (1999) The role of gesture in communication and thinking. Trends Cogn Sci 3:419–429

    Article  Google Scholar 

  8. Gorostiza J, Barber R, Khamis A, Malfaz M, Pacheco R, Rivas R, Corrales A, Delgado E, Salichs M (2006) Multimodal human-robot interaction framework for a personal robot. In: RO-MAN 06: Proc of the 15th IEEE international symposium on robot and human interactive communication

    Google Scholar 

  9. Hartmann B, Mancini M, Pelachaud C (2005) Implementing expressive gesture synthesis for embodied conversational agents. In: Gesture in human-computer interaction and simulation

    Google Scholar 

  10. Honda Motor Co, L (2000) The Honda humanoid robot Asimo, year 2000 model. http://world.honda.com/ASIMO/technology/spec.html

  11. Hostetter AB (2011) When do gestures communicate? A meta-analysis. Psychol Bull 137(2):297–315

    Article  Google Scholar 

  12. Hostetter AB, Alibali MW (2008) Visible embodiment: gestures as simulated action. Psychon Bull Rev 15(3):495–514

    Article  Google Scholar 

  13. Itoh K, Matsumoto H, Zecca M, Takanobu H, Roccella S, Carrozza M, Dario P, Takanishi A (2004) Various emotional expressions with emotion expression humanoid robot we-4rii. In: Proceedings of the 1st IEEE technical exhibition based conference on robotics and automation proceedings TExCRA 2004, pp 35–36

    Chapter  Google Scholar 

  14. Kendon A (1980) Gesticulation and speech: two aspects of the process of utterance. In: The relationship of verbal and non-verbal communication, pp 207–227

    Chapter  Google Scholar 

  15. Kendon A (2004) Gesture: visible action as utterance. Gesture 6(1):119–144

    Google Scholar 

  16. Kopp S, Wachsmuth I (2000) A knowledge-based approach for lifelike gesture animation. In: Horn W (ed) ECAI 2000—Proceedings of the 14th European conference on artificial intelligence. IOS Press, Amsterdam, pp 663–667

    Google Scholar 

  17. Kopp S, Wachsmuth I (2004) Synthesizing multimodal utterances for conversational agents. Comput Animat Virtual Worlds 15(1):39–52

    Article  Google Scholar 

  18. Kopp S, Bergmann K, Wachsmuth I (2008) Multimodal communication from multimodal thinking—towards an integrated model of speech and gesture production. Semant Comput 2(1):115–136

    Article  Google Scholar 

  19. Kramer N, Simons N, Kopp S (2007) The effects of an embodied conversational agent’s nonverbal behavior on user’s evaluation and behavioral mimicry. In: Proc of intelligent virtual agents (IVA 2007), vol 4722. Springer, Berlin, pp 238–251

    Chapter  Google Scholar 

  20. Kranstedt A, Kopp S, Wachsmuth I (2002) MURML: a multimodal utterance representation markup language for conversational agents. In: Proceedings of the AAMAS02 workshop on embodied conversational agents—let’s specify and evaluate them, Bologna, Italy

    Google Scholar 

  21. Levelt W (1989) Speaking. MIT Press, Cambridge

    Google Scholar 

  22. Macdorman K, Ishiguro H (2006) The uncanny advantage of using androids in cognitive and social science research. Interact Stud 7(3):297–337

    Article  Google Scholar 

  23. Mataric MJ, Pomplun M (1998) Fixation behavior in observation and imitation of human movement. Cogn Brain Res 7(2):191–202

    Article  Google Scholar 

  24. McNeill D (1992) Hand and mind: what gestures reveal about thought. University of Chicago Press, Chicago

    Google Scholar 

  25. McNeill D (2005) Gesture and thought. University of Chicago Press, Chicago

    Google Scholar 

  26. Minato T, Shimada M, Ishiguro H, Itakura S (2004) Development of an android robot for studying human-robot interaction. In: Innovations in applied artificial intelligence, pp 424–434

    Chapter  Google Scholar 

  27. Miyashita T, Shinozawa K, Hagita N (2006) Gesture translation for heterogeneous robots. In: Proceedings of 6th IEEE-RAS international conference on humanoid robots, pp 462–467

    Chapter  Google Scholar 

  28. Mori M (1970) The uncanny valley. Energy 7(4):33–35 (KF MacDorman and T Minato, Trans)

    Google Scholar 

  29. Nehaniv CL, Dautenhahn K (1998) The correspondence problem

  30. Ng-Thow-Hing V, Luo P, Okita S (2010) Synchronized gesture and speech production for humanoid robots. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, pp 4617–4624

    Google Scholar 

  31. Niewiadomski R, Bevacqua E, Mancini M, Pelachaud C (2009) Greta: an interactive expressive ECA system. In: Proceedings of 8th int conf on autonomous agents and multiagent systems (AAMAS2009), pp 1399–1400

    Google Scholar 

  32. Okuno Y, Kanda T, Imai M, Ishiguro H, Hagita N (2009) Providing route directions: design of robot’s utterance, gesture, and timing. In: Proceedings of the 4th ACM/IEEE international conference on human robot interaction, HRI’09. ACM, New York, pp 53–60

    Chapter  Google Scholar 

  33. Pollard N, Hodgins J, Riley M, Atkeson C (2002) Adapting human motion for the control of a humanoid robot. In: Proceedings of international conference on robotics and automation, pp 1390–1397

    Google Scholar 

  34. Reiter E, Dale R (2000) Building natural language generation systems. Cambridge Univ Press, Cambridge

    Book  Google Scholar 

  35. Salem M, Kopp S, Wachsmuth I, Joublin F (2010) Generating robot gesture using a virtual agent framework. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems

    Google Scholar 

  36. Salem M, Kopp S, Wachsmuth I, Joublin F (2010) Towards an integrated model of speech and gesture production for multi-modal robot behavior. In: Proceedings of the IEEE international symposium on robot and human interactive communication

    Google Scholar 

  37. Salem M, Rohlfing K, Kopp S, Joublin F (2011) A friendly gesture: investigating the effect of multimodal robot behavior in human-robot interaction. In: Proceedings of the IEEE international symposium on robot and human interactive communication

    Google Scholar 

  38. Saygin A, Chaminade T, Ishiguro H, Driver J, Frith C (2011) The thing that should not be: predictive coding and the uncanny valley in perceiving human and humanoid robot actions. Soc Cogn Affect Neurosci. doi:10.1093/scan/nsr025

    Google Scholar 

  39. Schröder M, Trouvain J (2003) The German text-to-speech synthesis system MARY: a tool for research, development and teaching. Int J Speech Technol 6:365–377

    Article  Google Scholar 

  40. Sidner C, Lee C, Lesh N (2003) The role of dialog in human robot interaction. In: International workshop on language understanding and agents for real world interaction

    Google Scholar 

  41. Sugiyama O, Kanda T, Imai M, Ishiguro H, Hagita N (2007) Natural deictic communication with humanoid robots. In: Proceedings of the IEEE international conference on intelligent robots and systems, pp 1441–1448

    Google Scholar 

  42. Wachsmuth I, Kopp S (2002) Lifelike gesture synthesis and timing for conversational agents. In: Wachsmuth I, Sowa T (eds) Gesture and sign language in human-computer interaction. LNAI, vol 2298. Springer, Berlin, pp 120–133

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Research Institute for Cognition and Robotics, Bielefeld, Germany

    Maha Salem

  2. Sociable Agents Group, Bielefeld University, Bielefeld, Germany

    Stefan Kopp

  3. Artificial Intelligence Group, Bielefeld University, Bielefeld, Germany

    Ipke Wachsmuth

  4. Emergentist Semantics Group, Bielefeld University, Bielefeld, Germany

    Katharina Rohlfing

  5. Honda Research Institute Europe, Offenbach, Germany

    Frank Joublin

Authors
  1. Maha Salem
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefan Kopp
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ipke Wachsmuth
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Katharina Rohlfing
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Frank Joublin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maha Salem.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Salem, M., Kopp, S., Wachsmuth, I. et al. Generation and Evaluation of Communicative Robot Gesture. Int J of Soc Robotics 4, 201–217 (2012). https://doi.org/10.1007/s12369-011-0124-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-011-0124-9

Keywords

Search

Navigation