Skip to main content

“I Know That You Know How I Feel”: Behavioral and Physiological Signals Demonstrate Emotional Attunement While Interacting with a Computer Simulating Emotional Intelligence

Abstract

Human–human communication studies have suggested that within communicative interactions, individuals acknowledge each other as intentional agents and adjust their emotion nonverbal behavior according to the other. This process has been defined as emotional attunement. In this study, we examine the emotional attunement process in the context of affective human–computer interactions. To this purpose, participants were exposed to one of two conditions. In one case, they played with a computer that simulated understanding of their emotional reactions while guiding them across four different game-like activities; in the other, the computer guided participants across the activities without mentioning any ability to understand emotional responses. Face movements, gaze direction, posture, vocal behavior, electrocardiogram and electrodermal activity were simultaneously recorded during the experimental sessions. Results showed that if participants were aware of interacting with an agent able to recognize their emotions, they reported that the computer was able to “understand” them and showed a higher number of nonverbal behaviors during the most interactive activity. The implications are discussed.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2

References

  1. Axelrode, L., & Hone, K. (2005). Uncharted passions: User displays of positive affect with an adaptive affective system. Lecture Notes in Computer Science, 3784, 890–897.

    Article  Google Scholar 

  2. Bailenson, J. N., Pontikakis, E. D., Mauss, I. B., Gross, J. J., Jabon, M. E., Hutcherson, C. A., et al. (2008). Real-time classification of evoked emotions using facial feature tracking and physiological responses. International Journal of Human Machine Studies, 66, 303–317.

    Article  Google Scholar 

  3. Beebe, B., Knoblauch, S., Rustin, J., Sorter, D., Jacobs, T., & Pally, R. (2005). Forms of intersubjectivity in infant research and adult treatment. New York: Other Press.

    Google Scholar 

  4. Boone, R. T., & Buck, R. (2003). Emotional expressivity and trustworthiness: The role of nonverbal behavior in the evolution of cooperation. Journal of Nonverbal Behavior, 27, 163–182.

    Article  Google Scholar 

  5. Brennan, S., & Ohaeri, J. (1994). Effects of message style on users’ attributions toward agents. Proceedings of Conference Companion on Human Factors in Computing Systems (pp. 281–282). New York, NY: ACM.

    Chapter  Google Scholar 

  6. Buck, R. (1985). Prime theory: An integrated view of motivation and emotion. Psychological Review, 92, 389–413.

    Article  Google Scholar 

  7. Cappella, N. J., & Pelachaud, C. (2001). Rules for responsive robots: Using human interactions to build virtual interactions. In A. L. Vangelisti, H. T. Reis, & M. A. Fitzpatrick (Eds.), Stability and change in relationships. New York: Cambridge University Press.

    Google Scholar 

  8. Caridakis, G., Raouzaiou, A., Karpouzis, K., & Kollias, S. (2006). Synthesizing gesture expressivity based on real sequences. In: Workshop on multimodal corpora: From multimodal behavior theories to usable models, LREC 2006 Conference, Genoa, Italy.

  9. Cassel, J., & Bickmore, T. (2003). Negotiated collusion: Modeling social language and its relationship effects in intelligent agents. User Modeling and User-Adapted Interaction, 13, 89–132.

    Article  Google Scholar 

  10. Cassell, J., & Tartaro, A. (2007). Intersubjectivity in human-agent interaction. Interaction Studies, 8, 391–410.

    Article  Google Scholar 

  11. Chovil, N. (1991). Social determinants of facial displays. Journal of Nonverbal Behavior, 15, 141–154.

    Article  Google Scholar 

  12. Ciceri, R., & Balzarotti, S. (2008). From signals to emotions: Applying emotion models to HM affective interactions. In J. Or (Ed.), Affective computing: Emotion modeling, synthesis and recognition. Vienna, Austria: I-Tech Education and Publishing.

    Google Scholar 

  13. Ciceri, R., & Biassoni, F. (2006). Zooming on multimodality and attuning: A multilayer model for the analysis of the vocal act in conversational interactions. In G. Riva, M. T. Anguerra, B. K. Wiederhold, & F. Mantovani (Eds.), From communication to presence (pp. 145–165). Amsterdam: IOS Press.

    Google Scholar 

  14. Clark, H. H. (1985). Language use and language users. In G. Lindzey & E. Aronson (Eds.), Handbook of social psychology (3rd ed., pp. 179–231). New York: Harper and RowClark.

    Google Scholar 

  15. Clark, H. H., & Wilkes-Gibbs, D. (1986). Referring as a collaborative process. Cognition, 22, 1–39.

    PubMed  Article  Google Scholar 

  16. Darves, C., & Oviatt, S. (2002). Adaptation of users’ spoken dialogue patterns in a conversational interface. In: J. Hansen & B. Pellom (Eds.), Proceedings of the 7th International Conference on Spoken Language Processing. Paper presented at the 7th International Conference on Spoken Language Processing, Denver, Colorado, 16–20 September (pp. 561–564).Denver, CO: Casual Prod. Ltd.

  17. de Melo, C., Carnevale, P., & Gratch, J. (2010). The influence of emotions in embodied agents on human decision-making. Proceedings of intelligent virtual agents (pp. 357–370). Berlin Heidelberg, Germany: Springer Verlag.

    Chapter  Google Scholar 

  18. Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6, 169–200.

    Article  Google Scholar 

  19. Ekman, P., & Frisen, W. (1978). Facial action coding system (FACS): A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychology Press.

    Google Scholar 

  20. Fridlund, A. J. (1991). Sociality of solitary smiling: Potentiation by an implicit audience. Journal of Personality and Social Psychology, 60, 229–240.

    Article  Google Scholar 

  21. Frijda, N. H., & Mesquita, B. (1994). The social roles and functions of emotions. In S. Kitayama & H. S. Markus (Eds.), Emotion and culture: Empirical studies of mutual influence (pp. 51–87). Washington, DC: American Psychological Association.

    Google Scholar 

  22. Garrod, S., & Anderson, A. (1987). Saying what you mean in dialogue: A study in conceptual and semantic co-ordination. Cognition, 27, 181–218.

    PubMed  Article  Google Scholar 

  23. Giles, H., Coupland, N., & Coupland, J. (1992). Accommodation theory: Communication, context and consequences. In H. Giles, J. Coupland, & N. Coupland (Eds.), Contexts of accommodation (pp. 1–68). Cambridge: Cambridge University Press.

    Google Scholar 

  24. Giles, H., Shepard, C. A., & Le Poire, B. A. (2001). Communication accommodation theory. In W. P. Robinson & H. Giles (Eds.), The new handbook of language and social psychology (pp. 33–56). Chichester, UK: Wiley.

    Google Scholar 

  25. Gilleade, K.M., Dix, A., & Allanson, J. (2005). Affective videogames and modes of affective gaming: Assist me, challenge me, emote me. Paper presented at the DiGRA 2005 Conference: Changing Views–Worlds in Play.

  26. Goffman, E. (1967). Interaction ritual: Essays on face-to-face behavior. New York, NY: Anchor.

    Google Scholar 

  27. Gratch, J., & Marsella, S. (2007). The architectural role of emotion in cognitive systems. In W. Gray (Ed.), Integrated models of cognitive systems. Oxford: Oxford University Press.

    Google Scholar 

  28. Grice, H. P. (1989). Studies in the way of words. Cambridge, MA: Harvard University Press.

    Google Scholar 

  29. Hall, L., Woods, S., Aylett, R., Newall, L., & Paiva, A. (2005). Achieving empathic engagement through affective interaction with synthetic characters. Lecture Notes in Computer Science, 3784, 731–738.

    Article  Google Scholar 

  30. Hess, U., Banse, R., & Kappas, A. (1995). The intensity of facial expression is determined by underlying affective state and social situation. Journal of Personality and Social Psychology, 69, 280–288.

    Article  Google Scholar 

  31. Höök, K. (1998). Evaluating the utility and usability of an adaptive hypermedia system. Journal of Knowledge-Based Systems, 10, 311–319.

    Article  Google Scholar 

  32. Höök, K. (2004). User-centred design and evaluation of affective interfaces. In Z. Ruttkay & C. Pelachaud (Eds.), From brows to trust: Evaluating embodied conversational agents (pp. 127–160). Dordrecht, The Netherlands: Kluwer Academic Publisher.

    Google Scholar 

  33. Izard, C. E. (1977). Human emotions. New York, NY: Plenum Press.

    Book  Google Scholar 

  34. Kaiser, S., & Wehrle, T. (2001). Facial expressions as indicators of appraisal processes. In K. R. Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal processes in emotions: Theory, methods, research (pp. 285–300). New York, NY: Oxford University Press.

    Google Scholar 

  35. Kappas, A., & Pecchinenda, A. (2000). Rules of disengagement: Cardiovascular changes as a function of appraisal and nine levels of difficulty of an interactive video game task. Psychophysiology, 37, S53, Abstract.

  36. Keltner, D., & Haidt, J. (1999). Social functions of emotions at four levels of analysis. Cognition and Emotion, 13, 505–521.

    Article  Google Scholar 

  37. King, W. J., & Ohya, J. (1995). The representation of agents: A study of phenomena in virtual environments. Proceedings of the IEEE International Workshop in Robot and Human Communication (pp. 199–204). Piscataway, NJ: IEEE Press.

    Chapter  Google Scholar 

  38. Klein, J., Moon, Y., & Picard, R. W. (2002). This computer responds to user frustration. Interacting with Computers, 14, 119–140.

    Article  Google Scholar 

  39. Kort, B., Reilly, R., & Picard, R.W. (2001). An affective model of interplay between emotions and learning: Reengineering educational pedagogy—building a learning companion. Paper presented at the international conference on advanced learning technologies, Madison, Wisconsin.

  40. Krumhuber, E., Manstead, A. S. R., Cosker, D., Marshall, D., Rosin, P. L., & Kappas, A. (2007). Facial dynamics as indicators of trustworthiness and cooperative behavior. Emotion, 7, 730–735.

    PubMed  Article  Google Scholar 

  41. Levenson, R. W. (2003). Blood, sweat, and fears: The autonomic architecture of emotion. In P. Ekman, J. J. Campos, R. J. Davidson, & F. B. M. de Waal (Eds.), Emotions inside out (pp. 348–366). New York: The New York Academy of Sciences.

    Google Scholar 

  42. Lisetti, C. L. (2002). Personality, affect and emotion taxonomy for socially intelligent agents. In proceedings of the 15th international florida artificial intelligence research society conference (FLAIRS’02). Pensacola, FL: AAAI Press.

  43. Lisetti, C. L., Nasoz, F., LeRouge, C., Ozyer, O., & Alverez, K. (2003). Intelligent affective interfaces: A patient-modelling assessment for tele-home health care. International Journal of Human-Computer Studies, 59, 245–255.

    Article  Google Scholar 

  44. Koda T., & Maes, P. (1996). Agents with faces: The effects of personification of agents. Paper presented at the proceedings of the fifth IEEE international workshop on robot and human communication (RO-MAN’96).

  45. Mauss, I. B., & Robinson, M. D. (2009). Measures of emotion: A review. Cognition and Emotion, 23, 209–237.

    PubMed Central  PubMed  Article  Google Scholar 

  46. Nass, C., & Lee, K. L. (2000). Does computer-generated speech manifest personality? An experimental test of similarity-attraction. Proceedings of the conference on human factors in computing systems (pp. 329–336). New York, NY: ACM Press.

    Chapter  Google Scholar 

  47. Oviatt, S., & Adams, B. (2000). Designing and evaluating conversational interfaces with animated characters. In J. Cassell, J. Sullivan, S. Prevost, & E. Churchill (Eds.), Embodied conversational agents (pp. 319–343). Cambridge, MA: MIT Press.

    Google Scholar 

  48. Oviatt, S., Darves, C., & Coulston, R. (2005). Toward adaptive conversational interfaces: Modeling speech convergence with animated personas. Transactions on Computer-Human Interaction, 11, 300–328.

    Google Scholar 

  49. Picard, R. W. (1997). Affective computing. Cambridge, MA: MIT Press.

    Google Scholar 

  50. Picard, R. W., Vyzas, E., & Healey, J. (2001). Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions Pattern Analysis and Machine Intelligence, 23, 1185–1191.

    Article  Google Scholar 

  51. Prendinger, H., Mayer, S., Mori, J., & Ishizuka, M. (2003). Using bio-signals to measure and reflect the impact of character-based interfaces. Paper presented at the fourth international working conference on intelligent virtual agents (IVA-03).

  52. Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people and places. New York: Cambridge University Press.

    Google Scholar 

  53. Schug, J., Matsumoto, D., Horita, Y., Yamagishi, T., & Bonnet, K. (2010). Emotional expressivity as a signal of cooperation. Evolution and Human Behavior, 31, 87–94.

    Article  Google Scholar 

  54. Searle, J. R. (1998). Mind, language and society. New York: Basic Books.

    Google Scholar 

  55. Siegman, A. W., & Feldstein, S. (1979). Of speech and time. Hillsdale: Erlbaum.

    Google Scholar 

  56. Stern, D. N. (1985). The interpersonal world of the infant. New York, NY: Basic.

    Google Scholar 

  57. Surakka, V., & Vanhala, T. (2011). Emotions in human-computer interaction. In A. Kappas & N. Krämer (Eds.), Face-to-face communication over the internet: Emotions in a web of culture, language, and technology (pp. 213–236). Cambridge, MA: Cambridge University Press.

    Chapter  Google Scholar 

  58. Trevarthen, C. (1993). The self born in intersubjectivity: An infant communicating. In U. Neisser (Ed.), The perceived self: Ecological and interpersonal sources of self-knowledge (pp. 121–173). New York: Cambridge University Press.

    Google Scholar 

  59. van Bakel, H. J. A., & Riksen-Walraven, J. M. (2008). Adrenocortical and behavioral attunement in parents with 1-year-old infants. Developmental Psychobiology, 50, 196–201.

    PubMed  Article  Google Scholar 

  60. Van Kleef, G. A., De Dreu, C. K. W., & Manstead, A. S. R. (2010). An interpersonal approach to emotion in social decision making: The emotions as social information model. Advances in Experimental Social Psychology, 42, 45–96.

    Article  Google Scholar 

  61. Van Reekum, C. M., Johnstone, T., Banse, R., Etter, A., Wehrle, T., & Scherer, K. R. (2004). Psychophysiological responses to appraisal dimensions in a computer game. Cognition Emotion, 18, 663–688.

    Article  Google Scholar 

  62. Zerubavel, E. (1981). Hidden rhythms: Schedules and calendars in social life. Chicago, IL: University of Chicago Press.

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Rita Ciceri.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Balzarotti, S., Piccini, L., Andreoni, G. et al. “I Know That You Know How I Feel”: Behavioral and Physiological Signals Demonstrate Emotional Attunement While Interacting with a Computer Simulating Emotional Intelligence. J Nonverbal Behav 38, 283–299 (2014). https://doi.org/10.1007/s10919-014-0180-6

Download citation

Keywords

  • Attunement
  • Emotion
  • Nonverbal behavior
  • Physiology
  • Human–computer interaction