Abstract
Human–human communication studies have suggested that within communicative interactions, individuals acknowledge each other as intentional agents and adjust their emotion nonverbal behavior according to the other. This process has been defined as emotional attunement. In this study, we examine the emotional attunement process in the context of affective human–computer interactions. To this purpose, participants were exposed to one of two conditions. In one case, they played with a computer that simulated understanding of their emotional reactions while guiding them across four different game-like activities; in the other, the computer guided participants across the activities without mentioning any ability to understand emotional responses. Face movements, gaze direction, posture, vocal behavior, electrocardiogram and electrodermal activity were simultaneously recorded during the experimental sessions. Results showed that if participants were aware of interacting with an agent able to recognize their emotions, they reported that the computer was able to “understand” them and showed a higher number of nonverbal behaviors during the most interactive activity. The implications are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Axelrode, L., & Hone, K. (2005). Uncharted passions: User displays of positive affect with an adaptive affective system. Lecture Notes in Computer Science, 3784, 890–897.
Bailenson, J. N., Pontikakis, E. D., Mauss, I. B., Gross, J. J., Jabon, M. E., Hutcherson, C. A., et al. (2008). Real-time classification of evoked emotions using facial feature tracking and physiological responses. International Journal of Human Machine Studies, 66, 303–317.
Beebe, B., Knoblauch, S., Rustin, J., Sorter, D., Jacobs, T., & Pally, R. (2005). Forms of intersubjectivity in infant research and adult treatment. New York: Other Press.
Boone, R. T., & Buck, R. (2003). Emotional expressivity and trustworthiness: The role of nonverbal behavior in the evolution of cooperation. Journal of Nonverbal Behavior, 27, 163–182.
Brennan, S., & Ohaeri, J. (1994). Effects of message style on users’ attributions toward agents. Proceedings of Conference Companion on Human Factors in Computing Systems (pp. 281–282). New York, NY: ACM.
Buck, R. (1985). Prime theory: An integrated view of motivation and emotion. Psychological Review, 92, 389–413.
Cappella, N. J., & Pelachaud, C. (2001). Rules for responsive robots: Using human interactions to build virtual interactions. In A. L. Vangelisti, H. T. Reis, & M. A. Fitzpatrick (Eds.), Stability and change in relationships. New York: Cambridge University Press.
Caridakis, G., Raouzaiou, A., Karpouzis, K., & Kollias, S. (2006). Synthesizing gesture expressivity based on real sequences. In: Workshop on multimodal corpora: From multimodal behavior theories to usable models, LREC 2006 Conference, Genoa, Italy.
Cassel, J., & Bickmore, T. (2003). Negotiated collusion: Modeling social language and its relationship effects in intelligent agents. User Modeling and User-Adapted Interaction, 13, 89–132.
Cassell, J., & Tartaro, A. (2007). Intersubjectivity in human-agent interaction. Interaction Studies, 8, 391–410.
Chovil, N. (1991). Social determinants of facial displays. Journal of Nonverbal Behavior, 15, 141–154.
Ciceri, R., & Balzarotti, S. (2008). From signals to emotions: Applying emotion models to HM affective interactions. In J. Or (Ed.), Affective computing: Emotion modeling, synthesis and recognition. Vienna, Austria: I-Tech Education and Publishing.
Ciceri, R., & Biassoni, F. (2006). Zooming on multimodality and attuning: A multilayer model for the analysis of the vocal act in conversational interactions. In G. Riva, M. T. Anguerra, B. K. Wiederhold, & F. Mantovani (Eds.), From communication to presence (pp. 145–165). Amsterdam: IOS Press.
Clark, H. H. (1985). Language use and language users. In G. Lindzey & E. Aronson (Eds.), Handbook of social psychology (3rd ed., pp. 179–231). New York: Harper and RowClark.
Clark, H. H., & Wilkes-Gibbs, D. (1986). Referring as a collaborative process. Cognition, 22, 1–39.
Darves, C., & Oviatt, S. (2002). Adaptation of users’ spoken dialogue patterns in a conversational interface. In: J. Hansen & B. Pellom (Eds.), Proceedings of the 7th International Conference on Spoken Language Processing. Paper presented at the 7th International Conference on Spoken Language Processing, Denver, Colorado, 16–20 September (pp. 561–564).Denver, CO: Casual Prod. Ltd.
de Melo, C., Carnevale, P., & Gratch, J. (2010). The influence of emotions in embodied agents on human decision-making. Proceedings of intelligent virtual agents (pp. 357–370). Berlin Heidelberg, Germany: Springer Verlag.
Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6, 169–200.
Ekman, P., & Frisen, W. (1978). Facial action coding system (FACS): A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychology Press.
Fridlund, A. J. (1991). Sociality of solitary smiling: Potentiation by an implicit audience. Journal of Personality and Social Psychology, 60, 229–240.
Frijda, N. H., & Mesquita, B. (1994). The social roles and functions of emotions. In S. Kitayama & H. S. Markus (Eds.), Emotion and culture: Empirical studies of mutual influence (pp. 51–87). Washington, DC: American Psychological Association.
Garrod, S., & Anderson, A. (1987). Saying what you mean in dialogue: A study in conceptual and semantic co-ordination. Cognition, 27, 181–218.
Giles, H., Coupland, N., & Coupland, J. (1992). Accommodation theory: Communication, context and consequences. In H. Giles, J. Coupland, & N. Coupland (Eds.), Contexts of accommodation (pp. 1–68). Cambridge: Cambridge University Press.
Giles, H., Shepard, C. A., & Le Poire, B. A. (2001). Communication accommodation theory. In W. P. Robinson & H. Giles (Eds.), The new handbook of language and social psychology (pp. 33–56). Chichester, UK: Wiley.
Gilleade, K.M., Dix, A., & Allanson, J. (2005). Affective videogames and modes of affective gaming: Assist me, challenge me, emote me. Paper presented at the DiGRA 2005 Conference: Changing Views–Worlds in Play.
Goffman, E. (1967). Interaction ritual: Essays on face-to-face behavior. New York, NY: Anchor.
Gratch, J., & Marsella, S. (2007). The architectural role of emotion in cognitive systems. In W. Gray (Ed.), Integrated models of cognitive systems. Oxford: Oxford University Press.
Grice, H. P. (1989). Studies in the way of words. Cambridge, MA: Harvard University Press.
Hall, L., Woods, S., Aylett, R., Newall, L., & Paiva, A. (2005). Achieving empathic engagement through affective interaction with synthetic characters. Lecture Notes in Computer Science, 3784, 731–738.
Hess, U., Banse, R., & Kappas, A. (1995). The intensity of facial expression is determined by underlying affective state and social situation. Journal of Personality and Social Psychology, 69, 280–288.
Höök, K. (1998). Evaluating the utility and usability of an adaptive hypermedia system. Journal of Knowledge-Based Systems, 10, 311–319.
Höök, K. (2004). User-centred design and evaluation of affective interfaces. In Z. Ruttkay & C. Pelachaud (Eds.), From brows to trust: Evaluating embodied conversational agents (pp. 127–160). Dordrecht, The Netherlands: Kluwer Academic Publisher.
Izard, C. E. (1977). Human emotions. New York, NY: Plenum Press.
Kaiser, S., & Wehrle, T. (2001). Facial expressions as indicators of appraisal processes. In K. R. Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal processes in emotions: Theory, methods, research (pp. 285–300). New York, NY: Oxford University Press.
Kappas, A., & Pecchinenda, A. (2000). Rules of disengagement: Cardiovascular changes as a function of appraisal and nine levels of difficulty of an interactive video game task. Psychophysiology, 37, S53, Abstract.
Keltner, D., & Haidt, J. (1999). Social functions of emotions at four levels of analysis. Cognition and Emotion, 13, 505–521.
King, W. J., & Ohya, J. (1995). The representation of agents: A study of phenomena in virtual environments. Proceedings of the IEEE International Workshop in Robot and Human Communication (pp. 199–204). Piscataway, NJ: IEEE Press.
Klein, J., Moon, Y., & Picard, R. W. (2002). This computer responds to user frustration. Interacting with Computers, 14, 119–140.
Kort, B., Reilly, R., & Picard, R.W. (2001). An affective model of interplay between emotions and learning: Reengineering educational pedagogy—building a learning companion. Paper presented at the international conference on advanced learning technologies, Madison, Wisconsin.
Krumhuber, E., Manstead, A. S. R., Cosker, D., Marshall, D., Rosin, P. L., & Kappas, A. (2007). Facial dynamics as indicators of trustworthiness and cooperative behavior. Emotion, 7, 730–735.
Levenson, R. W. (2003). Blood, sweat, and fears: The autonomic architecture of emotion. In P. Ekman, J. J. Campos, R. J. Davidson, & F. B. M. de Waal (Eds.), Emotions inside out (pp. 348–366). New York: The New York Academy of Sciences.
Lisetti, C. L. (2002). Personality, affect and emotion taxonomy for socially intelligent agents. In proceedings of the 15th international florida artificial intelligence research society conference (FLAIRS’02). Pensacola, FL: AAAI Press.
Lisetti, C. L., Nasoz, F., LeRouge, C., Ozyer, O., & Alverez, K. (2003). Intelligent affective interfaces: A patient-modelling assessment for tele-home health care. International Journal of Human-Computer Studies, 59, 245–255.
Koda T., & Maes, P. (1996). Agents with faces: The effects of personification of agents. Paper presented at the proceedings of the fifth IEEE international workshop on robot and human communication (RO-MAN’96).
Mauss, I. B., & Robinson, M. D. (2009). Measures of emotion: A review. Cognition and Emotion, 23, 209–237.
Nass, C., & Lee, K. L. (2000). Does computer-generated speech manifest personality? An experimental test of similarity-attraction. Proceedings of the conference on human factors in computing systems (pp. 329–336). New York, NY: ACM Press.
Oviatt, S., & Adams, B. (2000). Designing and evaluating conversational interfaces with animated characters. In J. Cassell, J. Sullivan, S. Prevost, & E. Churchill (Eds.), Embodied conversational agents (pp. 319–343). Cambridge, MA: MIT Press.
Oviatt, S., Darves, C., & Coulston, R. (2005). Toward adaptive conversational interfaces: Modeling speech convergence with animated personas. Transactions on Computer-Human Interaction, 11, 300–328.
Picard, R. W. (1997). Affective computing. Cambridge, MA: MIT Press.
Picard, R. W., Vyzas, E., & Healey, J. (2001). Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions Pattern Analysis and Machine Intelligence, 23, 1185–1191.
Prendinger, H., Mayer, S., Mori, J., & Ishizuka, M. (2003). Using bio-signals to measure and reflect the impact of character-based interfaces. Paper presented at the fourth international working conference on intelligent virtual agents (IVA-03).
Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people and places. New York: Cambridge University Press.
Schug, J., Matsumoto, D., Horita, Y., Yamagishi, T., & Bonnet, K. (2010). Emotional expressivity as a signal of cooperation. Evolution and Human Behavior, 31, 87–94.
Searle, J. R. (1998). Mind, language and society. New York: Basic Books.
Siegman, A. W., & Feldstein, S. (1979). Of speech and time. Hillsdale: Erlbaum.
Stern, D. N. (1985). The interpersonal world of the infant. New York, NY: Basic.
Surakka, V., & Vanhala, T. (2011). Emotions in human-computer interaction. In A. Kappas & N. Krämer (Eds.), Face-to-face communication over the internet: Emotions in a web of culture, language, and technology (pp. 213–236). Cambridge, MA: Cambridge University Press.
Trevarthen, C. (1993). The self born in intersubjectivity: An infant communicating. In U. Neisser (Ed.), The perceived self: Ecological and interpersonal sources of self-knowledge (pp. 121–173). New York: Cambridge University Press.
van Bakel, H. J. A., & Riksen-Walraven, J. M. (2008). Adrenocortical and behavioral attunement in parents with 1-year-old infants. Developmental Psychobiology, 50, 196–201.
Van Kleef, G. A., De Dreu, C. K. W., & Manstead, A. S. R. (2010). An interpersonal approach to emotion in social decision making: The emotions as social information model. Advances in Experimental Social Psychology, 42, 45–96.
Van Reekum, C. M., Johnstone, T., Banse, R., Etter, A., Wehrle, T., & Scherer, K. R. (2004). Psychophysiological responses to appraisal dimensions in a computer game. Cognition Emotion, 18, 663–688.
Zerubavel, E. (1981). Hidden rhythms: Schedules and calendars in social life. Chicago, IL: University of Chicago Press.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Balzarotti, S., Piccini, L., Andreoni, G. et al. “I Know That You Know How I Feel”: Behavioral and Physiological Signals Demonstrate Emotional Attunement While Interacting with a Computer Simulating Emotional Intelligence. J Nonverbal Behav 38, 283–299 (2014). https://doi.org/10.1007/s10919-014-0180-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10919-014-0180-6