Inferring Intentions from Emotion Expressions in Social Decision Making



In the last decade we have seen increasing experimental evidence that people make important inferences from emotion expressions about others’ intentions in situations of interdependent decision making. Reverse appraisal has been proposed as one mechanism whereby people retrieve, from emotion displays, information about how others are appraising the ongoing interaction (e.g., does my counterpart find the current outcome to be goal conducive? Does s/he blame me for it?); in turn, from these appraisal attributions, people make inferences about the others’ goals (e.g., is my counterpart likely to cooperate?) that shape their decision making. Here we review experimental evidence and progress that has been done in understanding this inferential mechanism and its relationship to other mechanisms for the interpersonal effects of emotion (e.g., emotional contagion and social appraisal). We discuss theoretical implications for our understanding of the role of emotion expression on human decision making, but also practical implications for the growing industry of socially intelligent machines (e.g., personal digital assistants and social robots).


Emotional expression Reverse appraisal Theory-of-mind Automatic expression recognition 


  1. Alfonso, B., Pynadath, D., Lhommet, M., & Marsella, S. (2013). Emotional perception for updating agents’ beliefs. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII).
  2. Al-Fuqaha, A., Guizani, M., Mohammadi, M., Aledhari, M., & Ayyash, M. (2015). Internet of things: A survey on enabling technologies, protocols, and applications. IEEE Communication Surveys & Tutorials, 17, 2347–2376. Scholar
  3. Algoe, S. B., Buswell, B. N., & DeLamater, J. D. (2000). Gender and job status as contextual cues for the interpretation of facial expression of emotion. Sex Roles, 42, 183–208.CrossRefGoogle Scholar
  4. Amodio, D., & Frith, C. (2006). Meeting of minds: The medial frontal cortex and social cognition. Nature Reviews Neuroscience, 7, 268–277. Scholar
  5. Andrade, E., & Ho, T.-H. (2009). Gaming emotions in social interactions. Journal of Consumer Research, 36, 539–551. Scholar
  6. Aviezer, H., Hassin, R., Ryan, J., Grady, C., Susskind, J., Anderson, A., et al. (2008). Angry, disgusted, or afraid? Studies on the malleability of emotion perception. Psychological Science, 19, 724–732. Scholar
  7. Baltrušaitis, T., Ahuja, C., & Morency, L. P. (2018). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence. Scholar
  8. Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., & Plumb, I. (2001). The “Reading the Mind in the Eyes” test revised version: A study with normal adults, and adults with Asperger syndrome or high-functioning autism. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 42, 241–251.CrossRefGoogle Scholar
  9. Boone, R., & Buck, R. (2003). Emotional expressivity and trustworthiness: The role of nonverbal behavior in the evolution of cooperation. Journal of Nonverbal Behavior, 27, 163–182.CrossRefGoogle Scholar
  10. Breazeal, C. (2003). Toward sociable robots. Robotics and Autonomous Systems, 42, 167–175. Scholar
  11. Brown, W., Palameta, B., & Moore, C. (2003). Are there nonverbal cues to commitment? An exploratory study using the zero-acquaintance video presentation paradigm. Evolutionary Psychology, 1, 42–69. Scholar
  12. Calvo, R., & D’Mello, S. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1, 16–37. Scholar
  13. Camerer, C. (2003). Psychology and economics. Strategizing in the brain. Science, 300, 1673–1675. Scholar
  14. Côté, S., Hideg, I., & van Kleef, G. (2013). The consequences of faking anger in negotiations. Journal of Experimental Social Psychology, 49, 453–463. Scholar
  15. D’Mello, S., & Kory, J. (2015). A review and meta-analysis of multimodal affect detection systems. ACM Computing Surveys, 47. Scholar
  16. Dally, M., Medasani, S., Behringer, R., & Trivedl, M. (2017). Self-driving cars. IEEE Computer, 50, 18–23. Scholar
  17. Damasio, A. (1994). Descartes’ error: Emotion, reason and the human brain. New York: Putnam.Google Scholar
  18. Dawes, R. (1980). Social dilemmas. Annual Review of Psychology, 31, 169–193. Scholar
  19. de Melo, C., Carnevale, P., Read, S., Antos, D., & Gratch, J. (2012). Bayesian model of the social effects of emotion in decision-making in multiagent systems. In: Proceedings of Autonomous Agents and Multiagent Systems (AAMAS 12).Google Scholar
  20. de Melo, C., Carnevale, P., & Gratch, J. (2014a). Using virtual confederates to research intergroup bias and conflict. Academy of Management Proceedings, 2014(1).Google Scholar
  21. de Melo, C., Carnevale, P., Read, S., & Gratch, J. (2014b). Reading people’s minds from emotion expressions in interdependent decision making. Journal of Personality and Social Psychology, 106, 73–88. Scholar
  22. de Melo, C., Marsella, S., & Gratch, J. (2019). Human cooperation when acting through autonomous machines. Proceedings of the National Academy of Sciences U.S.A., 116, 3482–3487. Scholar
  23. DeSteno, D., Breazeal, C., Frank, R., Pizarro, D., Baumann, J., Dickens, L., et al. (2012). Detecting the trustworthiness of novel partners in economic exchange. Psychological Science, 23, 1549–1556. Scholar
  24. Ekman, P. (1993). Facial expression and emotion. American Psychologist, 48, 384–392. Scholar
  25. Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17, 124–129. Scholar
  26. Ellsworth, P., & Scherer, K. (2003). Appraisal processes in emotion. In R. Davidson, K. Scherer, & H. Goldsmith (Eds.), Handbook of affective sciences (pp. 572–595). New York: Oxford University Press.Google Scholar
  27. Farrell, J., & Rabin, M. (1996). Cheap talk. Journal of Economics Perspectives, 10, 103–118.CrossRefGoogle Scholar
  28. Fischer, A., Rotteveel, M., Evers, C., & Manstead, A. (2004). Emotional assimilation: How we are influenced by others’ emotions. Cahiers de Psychologie Cognitive, 22, 223–245.Google Scholar
  29. Fitness, J. (2000). Anger in the workplace: An emotion script approach to anger episodes between workers and their superiors, co-workers and subordinates. Journal of Organizational Behavior, 21, 147–162.;2-T.CrossRefGoogle Scholar
  30. Floreano, D., & Wood, R. (2015). Science, technology and the future of small autonomous drones. Nature, 521, 460–466.CrossRefGoogle Scholar
  31. Frank, R. (1988). Passions within reason: The strategic role of the emotions. New York: Norton.Google Scholar
  32. Frank, R. (2004). Introducing moral emotions into models of rational choice. In A. Manstead, N. Frijda, & A. Fischer (Eds.), Feelings and emotions (pp. 422–440). New York: Cambridge University Press. Scholar
  33. Frijda, N. (1986). The emotions. Cambridge: Cambridge University Press.Google Scholar
  34. Frijda, N., & Mesquita, B. (1994). The social roles and functions of emotions. In S. Kitayama & H. Markus (Eds.), Emotion and culture: Empirical studies of mutual influence (pp. 51–87). Washington: American Psychological Association. Scholar
  35. Frijda, N., Kuipers, P., & ter Schure, E. (1989). Relations among emotion, appraisal, and emotional action readiness. Journal of Personality and Social Psychology, 57, 212–228. Scholar
  36. Gallese, V. (2007). Before and below ‘theory of mind’: Embodied simulation and the neural correlates of social cognition. Philosophical Transactions Royal Social London B Biological Science, 362, 659–669. Scholar
  37. Gratch, J., Rickel, J., Andre, E., Badler, N., Cassell, J., & Petajan, E. (2002). Creating interactive virtual humans: Some assembly required. IEEE Intelligent Systems, 17, 54–63. Scholar
  38. Gunnery, S., Hall, J., & Ruben, M. (2013). The deliberate Duchenne smile: Individual differences in expressive control. Journal of Nonverbal Behavior, 37, 29–41.CrossRefGoogle Scholar
  39. Haidt, J. (2003). Autonomic specificity and emotion. In R. J. Davidson, K. R. Scherer, & H. H. Goldsmith (Eds.), Handbook of affective sciences (pp. 852–870). New York: Oxford University Press.Google Scholar
  40. Hareli, S. (2014). Making sense of the social world and influencing it by using a naïve attribution theory of emotions. Emotion Review, 6, 336–343. Scholar
  41. Hareli, S., & Hess, U. (2010). What emotional reactions can tell us about the nature of others: An appraisal perspective on person perception. Cognition and Emotion, 24, 128–140. Scholar
  42. Hareli, S., Elkabetz, S., & Hess, U. (2019). Drawing inferences from emotion expressions: The role of situative informativeness and context. Emotion. Advance online publication. Scholar
  43. Hatfield, E., Cacioppo, J., & Rapson, R. (1994). Emotional contagion. New York: Cambridge University Press. Scholar
  44. Hess, U., & Hareli, S. (2016). The impact of context on the perception of emotions. In C. Abell & J. Smith (Eds.), The expression of emotion: Philosophical, psychological, and legal perspectives (pp. 199–218). Cambridge: Cambridge University Press. Scholar
  45. Hess, U., & Hareli, S. (2018). On the malleability of the meaning of contexts: The influence of another person’s emotion expressions on situation perception. Cognition and Emotion, 32, 185–191. Scholar
  46. Hess, U., Blaison, C., & Kafetsios, K. (2016). Judging facial emotion expressions in context: The influence of culture and self-construal orientation. Journal of Nonverbal Behavior, 40, 50–64.CrossRefGoogle Scholar
  47. Ickes, W., & Simpson, J. A. (2004). Motivational aspects of empathic accuracy. In M. B. Brewer & M. Hewstone (Eds.), Emotion and motivation. Perspectives on social psychology (pp. 225–246). Malden: Blackwell Publishing.Google Scholar
  48. Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M.-T. (2014). Socially assistive robots in elderly care: A mixed-method systematic literature review. International Journal of Human-Computer Interaction, 30, 369–393. Scholar
  49. Keltner, D., & Ekman, P. (2000). Facial expression of emotion. In M. Lewis & J. Haviland-Jones (Eds.), Handbook of emotion (pp. 236–249). New York: Guilford Press.Google Scholar
  50. Keltner, D., & Haidt, J. (1999). Social functions of emotions at four levels of analysis. Cognition and Emotion, 13, 505–521. Scholar
  51. Keltner, D., & Kring, A. M. (1998). Emotion, social function, and psychopathology. Review of General Psychology, 2, 320–342. Scholar
  52. Kott, A., Swami, A., & West, B. (2016). The internet of battle things. IEEE Computer, 49, 70–75. Scholar
  53. Kreibig, S. (2010). Autonomic nervous system activity in emotion: A review. Biological Psychology, 84, 394–421. Scholar
  54. Krumhuber, E. G., & Manstead, A. S. (2009). Can Duchenne smiles be feigned? New evidence on felt and false smiles. Emotion, 9(6), 807. Scholar
  55. Krumhuber, E., Manstead, A., & Kappas, A. (2007). Facial dynamics as indicators of trustworthiness and cooperative behavior. Emotion, 7, 730–735. Scholar
  56. Krumhuber, E., Kappas, A., & Manstead, A. (2013). Effects of dynamic aspects of facial expressions: A review. Emotion Review, 5, 41–46. Scholar
  57. Lanzetta, J., & Englis, B. (1989). Expectations of cooperation and competition and their effects on observer’s vicarious emotional responses. Journal of Personality and Social Psychology, 36, 543–554. Scholar
  58. Lazarus, R. (1991). Progress on a cognitive-motivational-relational theory of emotion. American Psychologist, 46, 819–834.CrossRefGoogle Scholar
  59. Lerner, J., Li, Y., Valdesolo, P., & Kassam, K. (2015). Emotion and decision making. Annual Review of Psychology, 66, 799–823. Scholar
  60. Levenson, R. (2003). Autonomic specificity and emotion. In R. J. Davidson, K. R. Scherer, & H. H. Goldsmith (Eds.), Handbook of affective sciences (pp. 212–224). New York: Oxford University Press.Google Scholar
  61. Leventhal, H., & Scherer, K. (1987). The relationship of emotion to cognition: A functional approach to a semantic controversy. Cognition and Emotion, 1, 3–28. Scholar
  62. Lewis, M. (2001). Personal pathways in the development of appraisal: A complex systems/stage theory perspective. In K. Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal processes in emotion: Theory, methods, research (pp. 205–220). New York: Oxford University Press.Google Scholar
  63. Lin, R., & Kraus, S. (2010). Can automated agents proficiently negotiate with humans? Communications of the ACM, 53, 78–88. Scholar
  64. Loewenstein, G., & Lerner, J. (2003). The role of affect in decision making. In R. Davidson, K. Scherer, & H. Goldsmith (Eds.), Handbook of affective sciences (pp. 619–642). New York: Oxford University Press.Google Scholar
  65. López, G., Quesada, L., & Guerrero, L. (2017). Alexa vs. Siri vs. Cortana vs. Google Assistant: A comparison of speech-based natural user interfaces. In Proceedings of International Conference on Applied Human Factors and Ergonomics (AHFE 2017).Google Scholar
  66. Manstead, A., & Fischer, A. (2001). Social appraisal: The social world as object of and influence on appraisal processes. In K. Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal processes in emotion: Theory, methods, research (pp. 221–232). Oxford: Oxford University Press.Google Scholar
  67. Mehu, M., Grammer, K., & Dunbar, R. (2007). Smiles when sharing. Evolution and Human Behavior, 28, 415–422. Scholar
  68. Mesquita, B., & Ellsworth, P. (2001). The role of culture in appraisal. In K. Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal processes in emotion: Theory, methods, research (pp. 233–248). New York: Oxford University Press.Google Scholar
  69. Morris, M., & Keltner, D. (2000). How emotions work: An analysis of the social functions of emotional expression in negotiations. Research in Organizational Behavior, 22, 1–50. Scholar
  70. Niedenthal, P., Mermillod, M., Maringer, M., & Hess, U. (2010). The simulation of smiles (SIMS) model: Embodied simulation and the meaning of facial expression. Behavioral and Brain Sciences, 33, 417–480. Scholar
  71. Ortony, A., Clore, G., & Collins, A. (1988). The cognitive structure of emotions. New York: Cambridge University Press.CrossRefGoogle Scholar
  72. Parkinson, B. (2019). Calibrating emotional orientations: Social appraisal and other kinds of relation alignment. In D. Dukes & F. Clément (Eds.), Foundations of affective social learning: Conceptualising the transmission of social value. Cambridge: Cambridge University Press.Google Scholar
  73. Parkinson, B., & Simons, G. (2009). Affecting others: Social appraisal and emotion contagion in everyday decision making. Personality and Social Psychology Bulletin, 35, 1071–1084. Scholar
  74. Petta, P., Marsella, S., & Gratch, J. (2010). Computational models of emotion. In K. Scherer, T. Bänziger, & E. Roesch (Eds.), A blueprint for affective computing: A sourcebook and manual (pp. 21–40). New York: Oxford University Press.Google Scholar
  75. Picard, R. (1997). Affective computing. Cambridge: The MIT Press.Google Scholar
  76. Poletti, M., Enrici, I., & Adenzato, M. (2012). Cognitive and affective theory of mind in neurodegenerative diseases: Neuropsychological, neuroanatomical and neurochemical levels. Neuroscience and Biobehavioral Reviews, 26, 2147–2164. Scholar
  77. Preacher, K., & Hayes, A. (2008). Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behavior Research Methods, 40, 879–891.CrossRefGoogle Scholar
  78. Reisenzein, R. (2009). Emotions as metarepresentational states of mind: Naturalizing the belief–desire theory of emotion. Cognitive Systems Research, 10, 6–20. Scholar
  79. Rens, H., Stratou, G., & Gratch, J. (2017). Incorporating emotion perception into opponent modeling for social dilemmas. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS).Google Scholar
  80. Rilling, J., & Sanfey, A. (2011). The neuroscience of social decision-making. Annual Review of Psychology, 62, 23–48. Scholar
  81. Roseman, I. (2001). A model of appraisal in the emotion system: Integrating theory, research, and applications. In K. Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal processes in emotion: Theory, methods, research (pp. 68–91). New York: Oxford University Press.Google Scholar
  82. Sariyanidi, E., Gunes, H., & Cavallaro, A. (2015). Automatic analysis of facial affect: A survey of registration, representation, and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37, 1113–1133. Scholar
  83. Scharlemann, J., Eckel, C., Kacelnik, A., & Wilson, R. (2001). The value of a smile: Game theory with a human face. Journal of Economic Psychology, 22, 617–640. Scholar
  84. Scherer, K. (2001). Appraisal considered as a process of multi-level sequential checking. In K. Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal processes in emotion: Theory, methods, research (pp. 92–120). New York: Oxford University Press.Google Scholar
  85. Scherer, K. (2010). The component process model: Architecture for a comprehensive computational model of emergent emotion. In K. Scherer, T. Bänziger, & E. Roesch (Eds.), A blueprint for affective computing: A sourcebook and manual (pp. 47–70). Oxford University Press.Google Scholar
  86. Scherer, K., & Grandjean, D. (2008). Facial expressions allow inference of both emotions and their components. Cognition and Emotion, 22, 789–801. Scholar
  87. Scherer, K., & Moors, A. (2019). The emotion process: Event appraisal and component differentiation. Annual Review of Psychology, 70. Scholar
  88. Scherer, K., Johnstone, T., & Klasmeyer, G. (2003). Vocal expression of emotion. In R. J. Davidson, K. R. Scherer, & H. H. Goldsmith (Eds.), Handbook of affective sciences (pp. 433–456). New York: Oxford University Press.Google Scholar
  89. Scherer, K., Mortillaro, M., Rotondi, I., Sergi, I., & Trznadel, S. (2018). Appraisal-driven facial actions as building blocks for emotion inference. Journal of Personality and Social Psychology, 114, 358–379. Scholar
  90. Schug, J., Matsumoto, D., Horita, Y., Yamagishi, T., & Bonnet, K. (2010). Emotional expressivity as a signal of cooperation. Evolution and Human Behavior, 31, 87–94. Scholar
  91. Shamay-Tsoory, S. (2011). The neural bases for empathy. The Neuroscientist, 17, 18–24. Scholar
  92. Shamay-Tsoory, S., & Aharon-Peretz, J. (2007). Dissociable prefrontal networks for cognitive and affective theory of mind: A lesion study. Neuropsychologia, 45, 3054–3067. Scholar
  93. Shamay-Tsoory, S., Aharon-Peretz, J., & Perry, D. (2009). Two systems for empathy: A double dissociation between emotional and cognitive empathy in inferior frontal gyrus versus ventromedial prefrontal lesions. Brain, 132, 617–627. Scholar
  94. Shore, D., & Parkinson, B. (2018). Interpersonal effects of strategic and spontaneous guilt communication in trust games. Cognition and Emotion, 32, 1382–1390. Scholar
  95. Sorce, J. F., Emde, R. N., Campos, J., & Klinnert, M. D. (1985). Maternal emotional signaling: Its effect on the visual cliff behavior of 1 year olds. Developmental Psychology, 21, 195–200. Scholar
  96. Spencer, S., Zanna, M., & Fong, G. (2005). Establishing a causal chain: Why experiments are often more effective than mediational analyses in examining psychological processes. Journal of Personality and Social Psychology, 89, 845–851. Scholar
  97. Stone, R., & Lavine, M. (2014). The social life of robots. Science, 346, 178–179.CrossRefGoogle Scholar
  98. Stratou, G., Hoegen, R., Lucas, G., & Gratch, J. (2015). Emotional signaling in a social dilemma: An automatic analysis. In: Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII).Google Scholar
  99. Sutton, R. (1991). Maintaining norms about expressed emotions: The case of bill collectors. Administrative Science Quarterly, 36, 245–268. Scholar
  100. Tamir, M., Mitchell, C., & Gross, J. (2008). Hedonic and instrumental motives in anger regulation. Psychological Science, 19, 324–328. Scholar
  101. Terada, K., & Takeuchi, C. (2017). Emotional expression in simple line drawings of a robot’s face leads to higher offers in the ultimatum game. Frontiers of Psychology, 8, 724. Scholar
  102. van Dijk, E., van Kleef, G. A., Steinel, W., & van Beest, I. (2008). A social functional approach to emotions in bargaining: When communicating anger pays and when it backfires. Journal of Personality and Social Psychology, 94, 600–614. Scholar
  103. van Kleef, G. (2016). The interpersonal dynamics of emotion: Toward an integrative theory of emotions as social information. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  104. van Kleef, G., De Dreu, C., & Manstead, A. (2004). The interpersonal effects of anger and happiness in negotiations. Journal of Personality and Social Psychology, 86, 57–76. Scholar
  105. van Kleef, G., De Dreu, C., & Manstead, A. (2006). Supplication and appeasement in negotiation: The interpersonal effects of disappointment, worry, guilt, and regret. Journal of Personality and Social Psychology, 91, 124–142. Scholar
  106. van Kleef, G., De Dreu, C., & Manstead, A. (2010). An interpersonal approach to emotion in social decision making: The emotions as social information model. Advances in Experimental Social Psychology, 42, 45–96. Scholar
  107. van Kleef, G., Homan, A., & Cheshin, A. (2012). Emotional influence at work: Take it EASI. Organizational Psychology Review, 2, 311–339. Scholar
  108. Waldrop, M. (2015). No drivers required. Nature, 518, 20–23.CrossRefGoogle Scholar
  109. Weiner, B. (1987). The social psychology of emotion: Applications of a naive psychology. Journal of Social and Clinical Psychology, 5, 405–419. Scholar
  110. Wu, Y., Baker, C., Tenenbaum, J., & Schulz, L. (2018). Rational inference of beliefs and desires from emotional expressions. Cognitive Science, 42, 850–884. Scholar
  111. Zeng, Z., Pantic, M., Roisman, G., & Huang, T. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31, 39–58. Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Institute for Creative TechnologiesUniversity of Southern CaliforniaPlaya VistaUSA
  2. 2.US Army Research LaboratoryPlaya VistaUSA

Personalised recommendations