Modeling empathy: building a link between affective and cognitive processes

  • Özge Nilay YalçınEmail author
  • Steve DiPaola


Computational modeling of empathy has recently become an increasingly popular way of studying human relations. It provides a way to increase our understanding of the link between affective and cognitive processes and enhance our interaction with artificial agents. However, the variety of fields contributing to empathy research has resulted in isolated approaches to modeling empathy, and this has led to various definitions of empathy and an absence of common ground regarding underlying empathic processes. Although this diversity may be useful in that it allows for an in-depth examination of various processes linked to empathy, it also may not yet provide a coherent theoretical picture of empathy. We argue that a clear theoretical positioning is required for collective progress. The aim of this article is, therefore, to call for a holistic and multilayered view of a model of empathy, taken from the rich background research from various disciplines. To achieve this, we present a comprehensive background on the theoretical foundations, followed by the working definitions, components, and models of empathy that are proposed by various fields. Following this introduction, we provide a detailed review of the existing techniques used in AI research to model empathy in interactive agents, focusing on the strengths and weaknesses of each approach. We conclude with a discussion of future directions in this emerging field.


Empathy Affective computing Cognitive modeling Artificial agents 



The authors would like to thank the reviewers for their valuable comments and suggestions to improve this article. This work was partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) [RGPIN-2019-06767] and the Social Sciences and Humanities Research Council of Canada (SSHRC) [435-2017-0625].


  1. Asada M (2015) Towards artificial empathy. Int J Soc Robot 7(1):19–33Google Scholar
  2. Baron-Cohen S, Wheelwright S (2003) The friendship questionnaire: an investigation of adults with asperger syndrome or high-functioning autism, and normal sex differences. J Autism Dev Disord 33(5):509–517Google Scholar
  3. Baron-Cohen S, Wheelwright S (2004) The empathy quotient: an investigation of adults with asperger syndrome or high functioning autism, and normal sex differences. J Autism Dev Disord 34(2):163–175Google Scholar
  4. Baron-Cohen S, Leslie AM, Frith U (1986) Mechanical, behavioural and intentional understanding of picture stories in autistic children. Br J Dev Psychol 4(2):113–125Google Scholar
  5. Baron-Cohen S, Wheelwright S, Hill J, Raste Y, Plumb I (2001a) The “reading the mind in the eyes” test revised version: a study with normal adults, and adults with asperger syndrome or high-functioning autism. J Child Psychol Psychiatry Allied Discip 42(2):241–251Google Scholar
  6. Baron-Cohen S, Wheelwright S, Spong A, Scahill V, Lawson J et al (2001b) Are intuitive physics and intuitive psychology independent? A test with children with asperger syndrome. J Dev Learn Disord 5(1):47–78Google Scholar
  7. Baron-Cohen S, Richler J, Bisarya D, Gurunathan N, Wheelwright S (2003) The systemizing quotient: an investigation of adults with asperger syndrome or high-functioning autism, and normal sex differences. Philos Trans R Soc B Biol Sci 358(1430):361–374Google Scholar
  8. Baron-Cohen S, Tager-Flusberg H, Lombardo M (2013) Understanding other minds: perspectives from developmental social neuroscience. Oxford University Press, OxfordGoogle Scholar
  9. Barros P (forthcoming) OMG empathy challenge. In: IEEE international conference on automatic face and gesture recognitionGoogle Scholar
  10. Bates J et al (1994) The role of emotion in believable agents. Commun ACM 37(7):122–125Google Scholar
  11. Batson CD (2009) These things called empathy: eight related but distinct phenomenaGoogle Scholar
  12. Batson CD (2012) The empathy-altruism hypothesis: issues and implications. From bench to bedside, Empathy, pp 41–54Google Scholar
  13. Boukricha H, Wachsmuth I, Carminati MN, Knoeferle P (2013) A computational model of empathy: Empirical evaluation. In: Humaine association conference on affective computing and intelligent interaction (ACII). IEEE, pp 1–6Google Scholar
  14. Bratman M (1987) Intention, plans, and practical reasonGoogle Scholar
  15. Brave S, Nass C, Hutchinson K (2005) Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent. Int J Hum Comput Stud 62(2):161–178Google Scholar
  16. Broekens J, Degroot D, Kosters WA (2008) Formal models of appraisal: theory, specification, and computational model. Cognit Syst Res 9(3):173–197Google Scholar
  17. Burleson W, Picard RW (2007) Gender-specific approaches to developing emotionally intelligent learning companions. IEEE Intell Syst 22(4):62–69Google Scholar
  18. Calvo RA, D’Mello S (2010) Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput 1(1):18–37Google Scholar
  19. Cambria E (2016) Affective computing and sentiment analysis. IEEE Intell Syst 31(2):102–107Google Scholar
  20. Clark AJ (2014) Empathy in counseling and psychotherapy: perspectives and practices. Routledge, Abingdon-on-ThamesGoogle Scholar
  21. Cooper B, Brna P, Martins A (2000) Effective affective in intelligent systems-building on evidence of empathy in teaching and learning. In: Affective interactions, Springer, pp 21–34Google Scholar
  22. Coplan A (2011) Understanding empathy: its features and effects. In: Goldie P, Coplan A (eds) Philosophical and psychological perspectives. Oxford University Press, Oxford, pp 3–18Google Scholar
  23. Coplan A, Goldie P (2011) Empathy: philosophical and psychological perspectives. Oxford University Press, OxfordGoogle Scholar
  24. Davis MH (1983) Measuring individual differences in empathy: evidence for a multidimensional approach. J Person Soc Psychol 44(1):113Google Scholar
  25. Davis MH (1994) Empathy: a social psychological approach. Brown and Benchmark Publishers, New YorkGoogle Scholar
  26. De Vignemont F, Singer T (2006) The empathic brain: How, when and why? Trends Cognit Sci 10(10):435–441Google Scholar
  27. De Waal FB (2007) The ‘russian doll’model of empathy and imitation. From mirror neurons to empathy, On being moved, pp 35–48Google Scholar
  28. De Waal F (2010) The age of empathy: nature’s lessons for a kinder society. Broadway Books, New YorkGoogle Scholar
  29. de Waal FB, Preston SD (2017) Mammalian empathy: behavioural manifestations and neural basis. Nat Rev Neurosci 18(8):498Google Scholar
  30. Decety J, Cowell JM (2014) The complex relation between morality and empathy. Trends Cognit Sci 18(7):337–339Google Scholar
  31. D’mello S, Graesser A (2012) Autotutor and affective autotutor: learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Trans Interact Intell Syst 2(4):23Google Scholar
  32. Gibson J, Malandrakis N, Romero F, Atkins DC, Narayanan SS (2015) Predicting therapist empathy in motivational interviews using language features inspired by psycholinguistic norms. In: Sixteenth annual conference of the international speech communication associationGoogle Scholar
  33. Golan O, Baron-Cohen S, Hill JJ, Golan Y (2006) “The reading the mind in films” task: complex emotion recognition in adults with and without autism spectrum conditions. Soc Neurosci 1(2):111–123Google Scholar
  34. Golan O, Baron-Cohen S, Hill JJ, Rutherford M (2007) The “reading the mind in the voice’test-revised: a study of complex emotion recognition in adults with and without autism spectrum conditions. J Autism Dev Disord 37(6):1096–1106Google Scholar
  35. Goldman AI (2006) Simulating minds: the philosophy, psychology, and neuroscience of mindreading. Oxford University Press, OxfordGoogle Scholar
  36. Goldman AI (2011) Two routes to empathy. In: Goldie P, Coplan A (eds) Empathy: philosophical and psychological perspectives. Oxford University Press, Oxford, pp 31–44Google Scholar
  37. Gonsior B, Sosnowski S, Mayer C, Blume J, Radig B, Wollherr D, Kühnlenz K (2011) Improving aspects of empathy and subjective performance for HRI through mirroring facial expressions. In: RO-MAN. IEEE, pp 350–356Google Scholar
  38. Hagendoorn I (2004) Some speculative hypotheses about the nature and perception of dance and choreography. J Consciousness Stud 11(F0020003):79–110Google Scholar
  39. Hegel F, Spexard T, Wrede B, Horstmann G, Vogt T (2006) Playing a different imitation game: interaction with an empathic android robot. IEEE, pp 56–61Google Scholar
  40. Hoffman ML (2000) Empathy and moral development: implications for caring and justice. Cambridge University Press, Cambridge. Google Scholar
  41. Hume D (1739) A treatise of human nature: being an attempt to introduce the experimental method of reasoning into moral subjects. printed for John Noon, LondonGoogle Scholar
  42. Iacoboni M (2011) Within each other: neural mechanisms for empathy in the primate brain. In: Coplan A, Goldie P (eds) Empathy: philosophical and psychological perspectives. Oxford University Press, Oxford, p 45Google Scholar
  43. Jaimes A, Sebe N (2005) Multimodal human computer interaction: a survey. In: Sebe N, Lew M, Huang TS (eds) Computer vision in human-computer interaction. Springer, Berlin, pp 1–15Google Scholar
  44. Kohut H (2011) The search for the self: volume 1: selected writings of Heinz Kohut 1950–1978. Karnac Books, LondonGoogle Scholar
  45. Kumano S, Otsuka K, Mikami D, Matsuda M, Yamato J (2015) Analyzing interpersonal empathy via collective impressions. IEEE Trans Affect Comput 6(4):324–336Google Scholar
  46. Lawrence EJ, Shaw P, Baker D, Baron-Cohen S, David AS (2004) Measuring empathy: reliability and validity of the empathy quotient. Psychol Med 34(5):911–920Google Scholar
  47. Lawson J, Baron-Cohen S, Wheelwright S (2004) Empathising and systemising in adults with and without asperger syndrome. J Autism Dev Disord 34(3):301–310Google Scholar
  48. Lazarus RS (1966) Psychological stress and the coping process/[by] Richard S. Lazarus. McGraw-Hill series in psychologyGoogle Scholar
  49. Lefimann N, Kopp S, Wachsmuth I (2006) Situated interaction with a virtual human perception, action, and cognition. Situat Commun 166:287Google Scholar
  50. Leiberg S, Anders S (2006) The multiple facets of empathy: a survey of theory and evidence. In: Anders S, Ende G, Junghofer M, Kissler J, Wildgruber D (eds) Understanding emotions, progress in brain research, vol 156. Elsevier, Amsterdam, pp 419–440. Google Scholar
  51. Leite I, Castellano G, Pereira A, Martinho C, Paiva A (2014) Empathic robots for long-term interaction. Int J Soc Robot 6(3):329–341Google Scholar
  52. Lim A, Okuno HG (2015) A recipe for empathy. Int J Soc Robot 7(1):35–49Google Scholar
  53. Lisetti C, Amini R, Yasavur U, Rishe N (2013) I can help you change! An empathic virtual agent delivers behavior change health interventions. ACM Trans Manag Inf Syst 4(4):19Google Scholar
  54. Marsella SC, Gratch J (2009) EMA: a process model of appraisal dynamics. Cognit Syst Res 10(1):70–90Google Scholar
  55. Marsella S, Gratch J (2014) Computationally modeling human emotion. Commun ACM 57(12):56–67. Google Scholar
  56. McQuiggan SW, Robison JL, Phillips R, Lester JC (2008) Modeling parallel and reactive empathy in virtual agents: an inductive approach. In: Proceedings of the 7th international joint conference on autonomous agents and multiagent systems, vol 1. International Foundation for Autonomous Agents and Multiagent Systems, pp 167–174Google Scholar
  57. McQuiggan SW, Lester JC (2007) Modeling and evaluating empathy in embodied companion agents. Int J Hum Compur Stud 65(4):348–360Google Scholar
  58. Moridis CN, Economides AA (2012) Affective learning: empathetic agents with emotional facial and tone of voice expressions. IEEE Trans Affect Comput 3(3):260–272Google Scholar
  59. Moyers TB, Rowell LN, Manuel JK, Ernst D, Houck JM (2016) The motivational interviewing treatment integrity code (MITI 4): rationale, preliminary reliability and validity. J Subst Abuse Treat 65:36–42Google Scholar
  60. Nowak M (2011) The complicated history of einfühlung. Argum Biannu Philos J 1(2):301–326Google Scholar
  61. Ochs M, Sadek D, Pelachaud C (2012) A formal model of emotions for an empathic rational dialog agent. Auton Agents Multi-Agent Syst 24(3):410–440Google Scholar
  62. Omdahl BL (1995) Cognitive appraisal, emotion, and empathyGoogle Scholar
  63. Ortony A, Clore GL, Collins A (1990) The cognitive structure of emotions. Cambridge University Press, CambridgeGoogle Scholar
  64. Paiva A, Leite I, Boukricha H, Wachsmuth I (2017) Empathy in virtual agents and robots: a survey. ACM Trans Interact Intell Syst 7(3):11Google Scholar
  65. Picard RW (2014) The promise of affective computing. The Oxford handbook of affective computing 11Google Scholar
  66. Poria S, Cambria E, Howard N, Huang GB, Hussain A (2016) Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing 174:50–59Google Scholar
  67. Poria S, Cambria E, Bajpai R, Hussain A (2017) A review of affective computing: from unimodal analysis to multimodal fusion. Inf Fusion 37:98–125Google Scholar
  68. Premack D, Woodruff G (1978) Does the chimpanzee have a theory of mind? Behav Brain Sci 1(4):515–526Google Scholar
  69. Prendinger H, Ishizuka M (2005) The empathic companion: a character-based interface that addresses users’affective states. Appl Artif Intell 19(3–4):267–285Google Scholar
  70. Preston SD, De Waal FB (2002) Empathy: its ultimate and proximate bases. Behav Brain Sci 25(1):1–20Google Scholar
  71. Prinz J (2011) Is empathy necessary for morality. Empathy Philos Psychol Perspect 1:211–229Google Scholar
  72. Rao AS, Georgeff MP (1991) Modeling rational agents within a BDI-architecture. KR 91:473–484MathSciNetzbMATHGoogle Scholar
  73. Rashkin H, Smith EM, Li M, Boureau YL (2018) I know the feeling: learning to converse with empathy. arXiv preprint arXiv:1811.00207
  74. Reeves B, Nass CI (1996) The media equation: how people treat computers, television, and new media like real people and places. Cambridge University Press, CambridgeGoogle Scholar
  75. Riek LD, Rabinowitch TC, Chakrabarti B, Robinson P (2009) How anthropomorphism affects empathy toward robots. In: Proceedings of the 4th ACM/IEEE international conference on Human robot interaction. ACM, pp 245–246Google Scholar
  76. Riek LD, Paul PC, Robinson P (2010) When my robot smiles at me: enabling human-robot rapport via real-time head gesture mimicry. J Multimodal User Interfaces 3(1–2):99–108Google Scholar
  77. Rizzolatti G, Fabbri-Destro M (2010) Mirror neurons: from discovery to autism. Exp Brain Res 200(3–4):223–237Google Scholar
  78. Rodrigues SH, Mascarenhas S, Dias J, Paiva A (2014) A process model of empathy for virtual agents. Interact Comput 27(4):371–391Google Scholar
  79. Roseman IJ, Smith CA (2001) Appraisal theory. Appraisal processes in emotion: theory, methods, research, pp 3–19Google Scholar
  80. Rosenthal-Von Der Pütten AM, Schulte FP, Eimler SC, Sobieraj S, Hoffmann L, Maderwald S, Brand M, Krämer NC (2014) Investigations on empathy towards humans and robots using FMRI. Comput Hum Behav 33:201–212Google Scholar
  81. Russell JA (1980) A circumplex model of affect. J Person Soc Psychol 39(6):1161Google Scholar
  82. Salovey P, Mayer JD (1990) Emotional intelligence. Imagin Cogn Person 9(3):185–211Google Scholar
  83. Scherer KR (2010a) Emotion and emotional competence: conceptual and theoretical issues for modelling agents. Blueprint for affective computing, pp 3–20Google Scholar
  84. Scherer KR (2010b) The component process model: architecture for a comprehensive computational model of emergent emotion. A sourcebook, Blueprint for affective computing, pp 47–70Google Scholar
  85. Scherer KR (2001) Appraisal considered as a process of multilevel sequential checking. Oxford University Press, OxfordGoogle Scholar
  86. Scherer KR (2005) What are emotions? And how can they be measured? Soc Sci Inf 44(4):695–729Google Scholar
  87. Searle JR (1969) Speech acts: an essay in the philosophy of language, vol 626. Cambridge University Press, CambridgeGoogle Scholar
  88. Singer T, Lamm C (2009) The social neuroscience of empathy. Ann N Y Acad Sci 11561(1):81–96Google Scholar
  89. Slote M (2007) The ethics of care and empathy. Routledge, LondonGoogle Scholar
  90. Smith M (2011) Empathy, expansionism, and the extended mind. Philosophical and psychological perspectives, empathy, pp 99–117Google Scholar
  91. Smith A (1959) The theory of moral sentiments. A. Millar and A. Kincaid and J. Bell, LondonGoogle Scholar
  92. Soleymani M, Garcia D, Jou B, Schuller B, Chang SF, Pantic M (2017) A survey of multimodal sentiment analysis. Image Vis Comput 65:3–14Google Scholar
  93. Stueber KR (2006) Rediscovering empathy : agency, folk psychology, and the human sciencesGoogle Scholar
  94. Tan ZX, Goel A, Nguyen TS, Ong DC (2018) A multimodal lSTM for predicting listener empathic responses over time. arXiv e-prints arXiv:1812.04891
  95. Tavassoli T, Hoekstra RA, Baron-Cohen S (2014) The sensory perception quotient (SPQ): development and validation of a new sensory questionnaire for adults with and without autism. Mol Autism 5(1):29Google Scholar
  96. Titchener EB (1909) Lectures on the experimental psychology of the thought-processes. Macmillan, OttawaGoogle Scholar
  97. Wisp L (1987) History of the concept of empathy. Empathy and its development, pp 17–37Google Scholar
  98. Wood K (2016) Kinesthetic empathy: conditions for viewing. Oxford University Press, OxfordGoogle Scholar
  99. Xiao B, Bone D, Segbroeck MV, Imel ZE, Atkins DC, Georgiou PG, Narayanan SS (2014) Modeling therapist empathy through prosody in drug addiction counseling. In: Fifteenth annual conference of the international speech communication associationGoogle Scholar
  100. Yalçın ÖN, DiPaola S (2019) Evaluating levels of emotional contagion with an embodied conversational agent. In: Proceedings of the 41st annual conference of the cognitive science societyGoogle Scholar
  101. Yalçın ÖN (2019) Evaluating empathy in artificial agents. In: 2019 International conference on affective computing and intelligent interaction (ACII). IEEEGoogle Scholar
  102. Yalçın ÖN (in press) Empathy framework for embodied conversational agents. Cogn Syst Res JGoogle Scholar
  103. Yalçın ÖN, DiPaola S (2018) A computational model of empathy for interactive agents. Biol Inspir Cogn Archit 26:20–25Google Scholar
  104. Yalçın ÖN, DiPaola S (2019) M-path: a conversational system for the empathic virtual agent. In: Samsonovich AV (ed) Biologically inspired cognitive architectures. Springer, Cham, pp 597–607Google Scholar
  105. Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 31(1):39–58Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.School of Interactive Arts and TechnologySimon Fraser UniversitySurreyCanada

Personalised recommendations