Skip to main content
SpringerLink
Log in

Towards an intelligent system for generating an adapted verbal and nonverbal combined behavior in human–robot interaction

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

In human–robot interaction scenarios, an intelligent robot should be able to synthesize an appropriate behavior adapted to human profile (i.e., personality). Recent research studies discussed the effect of personality traits on human verbal and nonverbal behaviors. The dynamic characteristics of the generated gestures and postures during the nonverbal communication can differ according to personality traits, which similarly can influence the verbal content of human speech. This research tries to map human verbal behavior to a corresponding verbal and nonverbal combined robot behavior based on the extraversion–introversion personality dimension. We explore the human–robot personality matching aspect and the similarity attraction principle, in addition to the different effects of the adapted combined robot behavior expressed through speech and gestures, and the adapted speech-only robot behavior, on interaction. Experiments with the humanoid NAO robot are reported.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. http://www.aldebaran-robotics.com/.

References

  • Aly, A. (2014). Towards an interactive human-robot relationship: Developing a customized robot’s behavior to human’s profile. PhD thesis, ENSTA ParisTech, France.

  • Aly, A., & Tapus, A. (2011). Speech to head gesture mapping in multimodal human-robot interaction. In Proceedings of the European conference on mobile robotics (ECMR), Orebro, Sweden.

  • Aly, A., & Tapus, A. (2012a). An integrated model of speech to arm gestures mapping in human-robot interaction. In Proceedings of the 14th IFAC symposium on information control problems in manufacturing (INCOM), Bucharest, Romania.

  • Aly, A., & Tapus, A. (2012b). Prosody-driven robot arm gestures generation in human-robot interaction. In Proceedings of the 7th ACM/IEEE international conference on human-robot interaction (HRI), Boston, MA.

  • Aly, A., & Tapus, A. (2013a). A model for synthesizing a combined verbal and nonverbal behavior based on personality traits in human-robot interaction. In Proceedings of the 8th ACM/IEEE international conference on human-robot interaction (HRI), Tokyo, Japan (pp. 325–332).

  • Aly, A., & Tapus, A. (2013b). Prosody-based adaptive metaphoric head and arm gestures synthesis in human robot interaction. In Proceedings of the 16th IEEE international conference on advanced robotics (ICAR), Montevideo, Uruguay (pp. 1–8).

  • Andre, E., Rist, T., Mulken, S., Klesen, M., & Baldes, S. (2000). The automated design of believable dialogues for animated presentation teams. In S. Prevost, J. S. Cassell, & E. Churchill (Eds.), Embodied conversational agents (pp. 220–255). Cambridge, MA: MIT Press.

    Google Scholar 

  • Bailenson, J., & Yee, N. (2005). Digital chameleons: Automatic assimilation of nonverbal gestures in immersive virtual environments. Psychological Science, 16, 814–819.

    Article  Google Scholar 

  • Bargh, J., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Personality and Social Psychology, 71, 230–244.

    Article  Google Scholar 

  • Barrick, M., & Mount, M. (1991). The big five personality dimensions and job performance: A meta-analysis. Personnel Pshychology, 44, 1–26.

    Article  Google Scholar 

  • Beattie, G., & Sale, L. (2012). Do metaphoric gestures influence how a message is perceived? The effects of metaphoric gesture-speech matches and mismatches on semantic communication and social judgment. Semiotica, 192, 77–98.

    Google Scholar 

  • Bernieri, F. (1988). Coordinated movement and rapport in teacher-student interactions. Nonverbal Behavior, 12, 120–138.

    Article  Google Scholar 

  • Bevacqua, E., Mancini, M., & Pelachaud, C. (2004). Speaking with emotions. In AISB convention: Motion, emotion and cognition, Leeds: University of Leeds.

  • Byrne, D., & Griffit, W. (1969). Similarity and awareness of similarity of personality characteristic determinants of attraction. Experimental Research in Personality, 3, 179–186.

    Google Scholar 

  • Cappella, J., & Planalp, S. (1981). Talk and silence sequence in informal conversations III: Interspeaker influence. Human Communication Research, 7(2), 117–132.

    Article  Google Scholar 

  • Cassell, J., & Bickmore, T. (2003). Negotiated collusion: Modeling social language and its relationship effects in intelligent agents. In User modeling and user-adapted interaction (vol 13, pp. 89–132).

  • Cassell, J., Bickmore, T., Campbell, L., Vilhjálmsson, H., & Yan, H. (2000). Human conversation as a system framework: Designing embodied conversational agents. In J. Cassell, J. Sullivan, S. Prevost, & E. Churchill (Eds.), Embodied conversational agents (pp. 29–63). Cambridge, MA: MIT Press.

    Google Scholar 

  • Cassell, J., Vilhjálmsson, H., & Bickmore, T. (2001). BEAT: The behavior expression animation toolkit. In Proceedings of the SIGGRAPH (pp. 477–486).

  • Chartrand, T., & Bargh, J. (1999). The chameleon effect: The perception-behavior link and social interaction. Personality and Social Psychology, 76(6), 893–910.

    Article  Google Scholar 

  • Chklovski, T., & Pantel, P. (2004). VERBOCEAN: Mining the web for fine-grained semantic verb relations. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), Barcelona, Spain.

  • Coltheart, M. (1981). The MRC psycholinguistic database. Quarterly Journal of Experimental Psychology, 33, 497–505.

    Article  Google Scholar 

  • Dewaele, J., & Furnham, A. (1999). Extraversion: The unloved variable in applied linguistic research. Language Learning, 49(3), 509–544.

    Article  Google Scholar 

  • Dicaprio, N. (1983). Personality theories: A guide to human nature. New York: Holt, Rinehart and Wilson.

    Google Scholar 

  • Dijkstra, P., & Barelds, D. (2008). Do people know what they want: A similar or complementary partner? Evolutionary Psychology, 6(4), 595–602.

    Article  Google Scholar 

  • Eriksson, J., Matarić, M., & Winstein, C. (2005). Hands-off assistive robotics for post-stroke arm rehabilitation. In Proceedings of the IEEE international conference on rehabilitation robotics (ICORR), IL (pp. 21–24).

  • Eysenck, H. (1953). The structure of human personality. London: Methuen.

    Google Scholar 

  • Eysenck, H. (1991). Dimensions of personality: 16, 5 or 3? Criteria for a taxonomic paradigm. Personality and Individual Differences, 12, 773–790.

    Article  Google Scholar 

  • Eysenck, H., & Eysenck, S. (1968). Manual: Eysenck personality inventory. San Diego, CA: Educational and Industrial Testing Service.

    Google Scholar 

  • Fellbaum, C. (1998). WordNet: An electronic lexical database. Cambridge, MA: MIT Press.

    MATH  Google Scholar 

  • Finin, T., Joshi, A., & Webber, B. (1986). Natural language interactions with artificial experts. Proceedings of the IEEE, 10(2), 921–938.

    Article  Google Scholar 

  • Forbes-Riley, K., & Litman, D. (2007). Investigating human tutor responses to student uncertainty for adaptive system development. In Lecture Notes in Computer Science (Vol 4738, pp. 678–689).

  • Forbes-Riley, K., Litman, D., & Rotaru, M. (2008). Responding to student uncertainty during computer tutoring: An experimental evaluation. In Lecture Notes in Computer Science (Vol. 5091, pp. 60–69).

  • Furnham, A. (1990). Language and personality. In H. Giles & W. Robinson (Eds.), Handbook of language and social psychology. Chichester: Wiley.

    Google Scholar 

  • Giles, H., & Powesland, P. (1978). Speech style and social evaluation. In F. Erickson (Ed.), Language in society (pp. 428–433). Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Gill, A., & Oberlander, J. (2002). Taking care of the linguistic features of extraversion. In Proceedings of the 24th annual conference of the cognitive science society (pp. 363–368).

  • Goldberg, L. (1990). An alternative description of personality: The Big-Five factor structure. Personality and Social Psychology, 59, 1216–1229.

    Article  Google Scholar 

  • Goldberg, L. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality Psychology in Europe, 7, 7–28.

    Google Scholar 

  • Grosz, B. (1983). TEAM: A transportable natural language interface system. In Proceedings of the conference on applied natural language processing, Santa Monica, CA (pp. 39–45).

  • Gueguen, N. (2007). 100 petites experiences en psychologie de la seduction. Paris: Dunod.

    Google Scholar 

  • Gump, B., & Kulik, J. (1997). Stress, affiliation, and emotional contagion. Personality and Social Psychology, 72, 305–319.

    Article  Google Scholar 

  • Hall, E. (1966). The hidden dimension. New York: Doubleday.

    Google Scholar 

  • Hartmann, B., Mancini, M., & Pelachaud, C. (2002). Formational parameters and adaptive prototype instantiation for MPEG-4 compliant gesture synthesis. In Proceedings of the computer animations. Geneva: IEEE Computer Society Press.

  • Hassin, R., & Trope, Y. (2000). Facing faces: Studies on the cognitive aspects of physiognomy. Personality and Social Psychology, 78, 837–852.

    Article  Google Scholar 

  • Huang, C., & Mutlu, B. (2014). Learning-based modeling of multimodal behaviors for humanlike robots. In Proceedings of the 9th ACM/IEEE human-robot interaction conference (HRI), Germany.

  • Isbister, K., & Nass, C. (2000). Consistency of personality in interactive characters: Verbal cues, non-verbal cues, and user characteristics. Human-Computer Studies, 53, 251–267.

    Article  Google Scholar 

  • J-Campbell, L., Gleason, K., Adams, R., & Malcolm, K. (2003). Interpersonal conflict, agreeableness, and personality development. Personality, 71(6), 1059–1085.

    Article  Google Scholar 

  • Jung, C., Hull, R., & Baynes, H. (1976). Psychological types. Princeton, NJ: Princeton University Press.

    Google Scholar 

  • Kendon, A. (1980). Gesticulation and speech: Two aspects of the process of utterance. In M. Key (Ed.), The relationship of verbal and nonverbal communication (pp. 207–227). The Hague: Mouton Publishers.

    Google Scholar 

  • Kopp, S., Krenn, B., Marsella, S., Marshall, A., Pelachaud, C., Pirker, H., et al. (2006). Towards a common framework for multimodal generation: The behavior markup language. Intelligent Virtual Agents, 4133, 205–217.

    Article  Google Scholar 

  • Kopp, S., Bergmann, K., & Wachsmuth, I. (2008). Multimodal communication from multimodal thinking—Towards an integrated model of speech and gesture production. Semantic Computing, 2(1), 115–136.

    Article  Google Scholar 

  • Lafrance, M. (1982). Posture mirroring and rapport. In M. Davis (Ed.), Interaction rhythms: Periodicity in commutative behavior (pp. 279–298). New York: Human Sciences Press.

    Google Scholar 

  • Lavoie, B., & Rambow, O. (1997). A fast and portable realizer for text generation. In Proceedings of the 5th conference on applied natural language processing (ANLP).

  • Le, Q., & Pelachaud, C. (2012). Generating co-speech gestures for the humanoid robot NAO through BML. In Proceedings of the 9th international conference on gesture and sign language in human-computer interaction and embodied communication (pp. 228–237).

  • Leary, T. (1957). Interpersonal diagnosis of personality. New York: Ronald Press.

    Google Scholar 

  • Lee, K., Peng, W., Jin, S. A., & Yan, C. (2006). Can robots manifest personality? An empirical test of personality recognition, social responses, and social presence in human robot interaction. Communication, 56, 754–772.

    Google Scholar 

  • Leuwerink, K. (2012). A robot with personality: Interacting with a group of humans. In Proceedings of the 16th twente student conference on IT, Enschede, The Netherlands.

  • Lippa, R., & Dietz, J. (2000). The relation of gender, personality, and intelligence to judges’ accuracy in judging strangers’ personality from brief video segments. Nonverbal Behavior, 24, 25–43.

    Article  Google Scholar 

  • Lucignano, L., Cutugno, F., Rossi, S., & Finzi, A. (2013). A dialogue system for multimodal human-robot interaction. In Proceedings of the 15th ACM international conference on multimodal interaction (ICMI), Australia.

  • Luo, P., Ng-Thow-Hing, V., & Neff, M. (2013). An examination of whether people prefer agents whose gestures mimic their own. In Intelligent Virtual Agents: Lecture Notes in Computer Science (Vol. 8108, pp. 229–238).

  • Mairesse, F., & Walker, M. (2011). Controlling user perceptions of linguistic style: Trainable generation of personality traits. Computational Linguistics, 37, 455–488.

    Article  Google Scholar 

  • Mairesse, F., Walker, M., Mehl, M., & Moore, R. (2007). Using linguistic cues for the automatic recognition of personality in conversation and text. Journal of Artificial Intelligence Research (JAIR), 30, 457–500.

    MATH  Google Scholar 

  • Mancini, M., & Pelachaud, C. (2008). Distinctiveness in multimodal behaviors. In Proceedings of the 7th international joint conference on autonomous agents and multiagent systems (pp. 159–166).

  • Maurer, R., & Tindall, J. (1983). Effects of postural congruence on client’s perception of counselor empathy. Counseling Psychology, 30, 158–163.

    Article  Google Scholar 

  • McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.

    Google Scholar 

  • McNeill, D. (2000). Language and gesture. Cambridge, UK: Cambridge University Press.

    Book  Google Scholar 

  • McNeill, D. (2005). Gesture and thought. Chicago: University of Chicago Press.

    Book  Google Scholar 

  • Mehl, M., Gosling, S., & Pennebaker, J. (2006). Personality in its natural habitat: Manifestations and implicit folk theories of personality in daily life. Personality and Social Psychology, 90, 862–877.

    Article  Google Scholar 

  • Moon, Y., & Nass, C. (1996). How real are computer personalities? Psychological responses to personality types in human-computer interaction. Communication Research, 23, 651–674.

    Article  Google Scholar 

  • Morris, L. (1979). Extraversion and introversion: An interactional perspective. New York: Hemisphere Publishing Corporation.

    Google Scholar 

  • Murray, J. (1990). Review of research on the Myers-Briggs type indicator. Perceptual and Motor Skills, 70, 1187–1202.

    Article  Google Scholar 

  • Myers-Briggs, I., & Myers, P. (1980). Gifts differing: Understanding personality type. Mountain View, CA: Davies-Black Publishing.

    Google Scholar 

  • Nakajima, H., Nass, S., Yamada, R., Morishima, Y., & Kawaji, S. (2003). The functionality of human-machine collaboration systems-mind model and social behavior. In Proceedings of the IEEE conference on systems, man, and cybernetics, Charlottesville, VA (pp. 2381–2387).

  • Nakajima, H., Morishima, Y., Yamada, R., Brave, S., Maldonado, H., Nass, C., et al. (2004). Social intelligence in a human-machine collaboration system: Social responses to agents with mind model and personality. Japanese Society for Artificial Intelligence, 19(3), 184–196.

    Article  Google Scholar 

  • Nass, C., & Lee, M. (2001). Does computer-synthesized speech manifest personality? Experimental tests of recognition, similarity-attraction, and consistency attraction. Experimental Psychology: Applied, 7, 171–181.

    Google Scholar 

  • Ng-Thow-Hing, V., Luo, P., & Okita, S. (2010). Synchronized gesture and speech production for humanoid robots. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS), Taipei, Taiwan.

  • Niewiadomski, R., Hyniewska, S., & Pelachaud, C. (2009). Modeling emotional expressions as sequences of behaviors. In Proceedings of the 9th international conference on intelligent virtual agents, Amsterdam, The Netherlands (pp. 316–322).

  • Park, E., Jin, D., & Del-Pobil, A. (2012). The law of attraction in human-robot interaction. Advanced Robotic Systems, 9, 35.

    Google Scholar 

  • Pennebaker, J., & King, L. (1999). Linguistic styles: Language use as an individual difference. Personality and Social Psychology, 77, 1296–1312.

    Article  Google Scholar 

  • Pittam, J. (1994). Voice in social interaction: An interdisciplinary approach. Hemet, CA: Sage.

    Book  Google Scholar 

  • Rabiner, L. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77, 257–286.

    Article  Google Scholar 

  • Reeves, B., & Nass, C. (1996). The media equation. Chicago: University of Chicago Press.

    Google Scholar 

  • Reiter, E., & Dale, R. (2000). Building natural language generation systems. Cambridge, UK: Cambridge University Press.

    Book  Google Scholar 

  • Rezek, I., & Roberts, S. (2000). Estimation of coupled hidden Markov models with application to biosignal interaction modeling. In Proceedings of the IEEE international workshop on neural networks for signal processing (NNSP), Sydney, Australia.

  • Rezek, I., Sykacek, P., & Roberts, S. (2000). Coupled hidden Markov models for biosignal interaction modeling. In Proceedings of the 1st international conference on advances in medical signal and information processing (MEDSIP), UK (pp. 54–59).

  • Riggio, R., & Friedman, H. (1986). Impression formation: The role of expressive behavior. Personality and Social Psychology, 50, 421–427.

    Article  Google Scholar 

  • Scherer, K. (1979). Language and personality. In K. Scherer & H. Giles (Eds.), Social markers in speech (pp. 147–209). Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Selfhout, M., Burk, W., Branje, S., Denissen, J., Aken, M., & Meeus, W. (2010). Emerging late adolescent friendship networks and Big Five personality traits: A social network approach. Personality, 78(2), 509–538.

    Article  Google Scholar 

  • Stent, A., Prasad, R., & Walker, M. (2004). Trainable sentence planning for complex information presentation in spoken dialog systems. In Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL), Morristown, NJ (pp. 79–86).

  • Sullivan, H. (1953). The interpersonal theory of psychiatry. Rochester, NY: Norton.

    Google Scholar 

  • Tapus, A., & Matarić, M. (2008). Socially assistive robots: The link between personality, empathy, physiological signals, and task performance. In Proceedings of the AAAI spring symposium on emotion, personality and social behavior, Menlo Park, CA.

  • Tapus, A., Tapus, C., & Matarić, J. (2008). User-robot personality matching and robot behavior adaptation for post-stroke rehabilitation therapy. Intelligent Service Robotics, 1(2), 169–183.

    Article  Google Scholar 

  • Van-Baaren, R., Holland, R., Steenaert, B., & Van-Knippenberg, A. (2003). Mimicry for money: Behavioral consequences of imitation. Experimental Social Psychology, 39, 393–398.

    Article  Google Scholar 

  • Vinacke, W., Shannon, K., Palazzo, V., Balsavage, L., et al. (1988). Similarity and complementarity in intimate couples. Genetic, Social, and General Psychology Monographs, 114(1), 51–76.

    Google Scholar 

  • Vogel, K., & Vogel, S. (1986). L’interlangue et la personnalite de l’apprenant. Applied Linguistics, 24(1), 48–68.

    Google Scholar 

  • Wahlster, W., & Kobsa, A. (1989). User models in dialog systems. Berlin: Springer.

    Book  MATH  Google Scholar 

  • Webb, J. (1972). Interview synchrony: An investigation of two speech rate measures. In A. Siegman & B. Pope (Eds.), Studies in dyadic communication (pp. 115–133). New York: Pergamon Press.

    Chapter  Google Scholar 

  • Williams, J. (1971). Personal space and its relation to extraversion-introversion. Canadian Journal of Behavioural Science, 3(2), 156–160.

    Article  Google Scholar 

  • Windhouwer, D. (2012). The effects of the task context on the perceived personality of a Nao robot. In Proceedings of the 16th twente student conference on IT, Enschede, The Netherlands.

  • Woods, S., Dautenhahn, K., Kaouri, C., Boekhorst, R., & Koay, K. (2005). Is this robot like me? Links between human and robot personality traits. In Proceedings of the 5th IEEE-RAS international conference on humanoid robots (Humanoids), Tsukuba, Japan.

  • Woods, S., Dautenhahn, K., Kaouri, C., Boekhorst, R., Koay, K., & Walters, M. (2007). Are robots like people? Relationships between participant and robot personality traits in human-robot interaction studies. Interaction Studies, 8(2), 281–305.

    Article  Google Scholar 

  • Young, K. (1927). Source book for social psychology. New York: A.A. Knopf.

    Book  Google Scholar 

  • Zukerman, I., & Litman, D. (2001). Natural language processing and user modeling: Synergies and limitations. User Modeling and User-Adapted Interaction, 11, 129–158.

    Article  MATH  Google Scholar 

Download references

Acknowledgments

This work was supported by the French National Research Agency (ANR) through Chaire d’Excellence program 2009 (Human–Robot Interaction for Assistive Applications). The project’s website is accessible at: http://www.ensta-paristech.fr/~tapus/HRIAA/.

Author information

Authors and Affiliations

  1. Robotics and Computer Vision Lab, ENSTA ParisTech, 828 Boulevard des Maréchaux, 91120, Palaiseau, France

    Amir Aly & Adriana Tapus

Authors
  1. Amir Aly
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adriana Tapus
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amir Aly.

Appendix: General metaphoric gesture generation

Appendix: General metaphoric gesture generation

Our proposed system for synthesizing metaphoric gestures is integrated through 3 stages, as illustrated in Fig. 14 (Aly and Tapus 2013b; Aly 2014). Stage 1 constitutes the training phase of the system, through which the raw speech and gesture training inputs get processed in order to extract relevant features (e.g., the pitch–intensity curves for speech and the motion curves for gesture). Afterwards, the calculated characteristic curves undergo both of the segmentation phase (which is concerned with segmenting a continuous sequence of gestures into independent gestures using the kinetic features of body segments, and with segmenting speech into corresponding syllables to the segmented gestures, for which their prosodic cues will be calculated), and the Coupled Hidden Markov Models (CHMM) training phase. The segmented patterns of prosody and gestures are modeled separately into two parallel HMM constituting the CHMM (Rabiner 1989; Rezek and Roberts 2000; Rezek et al. 2000), through which new metaphoric head and arm gestures are generated (i.e., stage 2) based on the prosodic cues of a new speech-test signal, which will follow the same previously illustrated phases of the training stage.

The main purpose of stage 3 is to setup for a successful long-term human–robot interaction (a future concern for our research), for which the robot should be able to extend incrementally the constructed learning database by acquiring more raw speech and gesture data elements from the nearby humans. Therefore, a Kinect sensor should be continuously employed in parallel with the robot in order to precisely calculate the motion curves of articulations, in addition to a microphone to receive the speech signal of a human user. Afterwards, both of the captured prosody and gestures data will undergo the previously explained phases of the training stage 1 so as to increase the robot ability to synthesize more appropriate gestures.

Fig. 14
figure 14

Overview of the metaphoric gesture generator. More details are available in Aly and Tapus (2013b)

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aly, A., Tapus, A. Towards an intelligent system for generating an adapted verbal and nonverbal combined behavior in human–robot interaction. Auton Robot 40, 193–209 (2016). https://doi.org/10.1007/s10514-015-9444-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-015-9444-1

Keywords

Search

Navigation