Skip to main content
Log in

Modeling empathy: building a link between affective and cognitive processes

Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Computational modeling of empathy has recently become an increasingly popular way of studying human relations. It provides a way to increase our understanding of the link between affective and cognitive processes and enhance our interaction with artificial agents. However, the variety of fields contributing to empathy research has resulted in isolated approaches to modeling empathy, and this has led to various definitions of empathy and an absence of common ground regarding underlying empathic processes. Although this diversity may be useful in that it allows for an in-depth examination of various processes linked to empathy, it also may not yet provide a coherent theoretical picture of empathy. We argue that a clear theoretical positioning is required for collective progress. The aim of this article is, therefore, to call for a holistic and multilayered view of a model of empathy, taken from the rich background research from various disciplines. To achieve this, we present a comprehensive background on the theoretical foundations, followed by the working definitions, components, and models of empathy that are proposed by various fields. Following this introduction, we provide a detailed review of the existing techniques used in AI research to model empathy in interactive agents, focusing on the strengths and weaknesses of each approach. We conclude with a discussion of future directions in this emerging field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

References

  • Asada M (2015) Towards artificial empathy. Int J Soc Robot 7(1):19–33

    Google Scholar 

  • Baron-Cohen S, Wheelwright S (2003) The friendship questionnaire: an investigation of adults with asperger syndrome or high-functioning autism, and normal sex differences. J Autism Dev Disord 33(5):509–517

    Google Scholar 

  • Baron-Cohen S, Wheelwright S (2004) The empathy quotient: an investigation of adults with asperger syndrome or high functioning autism, and normal sex differences. J Autism Dev Disord 34(2):163–175

    Google Scholar 

  • Baron-Cohen S, Leslie AM, Frith U (1986) Mechanical, behavioural and intentional understanding of picture stories in autistic children. Br J Dev Psychol 4(2):113–125

    Google Scholar 

  • Baron-Cohen S, Wheelwright S, Hill J, Raste Y, Plumb I (2001a) The “reading the mind in the eyes” test revised version: a study with normal adults, and adults with asperger syndrome or high-functioning autism. J Child Psychol Psychiatry Allied Discip 42(2):241–251

    Google Scholar 

  • Baron-Cohen S, Wheelwright S, Spong A, Scahill V, Lawson J et al (2001b) Are intuitive physics and intuitive psychology independent? A test with children with asperger syndrome. J Dev Learn Disord 5(1):47–78

    Google Scholar 

  • Baron-Cohen S, Richler J, Bisarya D, Gurunathan N, Wheelwright S (2003) The systemizing quotient: an investigation of adults with asperger syndrome or high-functioning autism, and normal sex differences. Philos Trans R Soc B Biol Sci 358(1430):361–374

    Google Scholar 

  • Baron-Cohen S, Tager-Flusberg H, Lombardo M (2013) Understanding other minds: perspectives from developmental social neuroscience. Oxford University Press, Oxford

    Google Scholar 

  • Barros P (forthcoming) OMG empathy challenge. In: IEEE international conference on automatic face and gesture recognition

  • Bates J et al (1994) The role of emotion in believable agents. Commun ACM 37(7):122–125

    Google Scholar 

  • Batson CD (2009) These things called empathy: eight related but distinct phenomena

  • Batson CD (2012) The empathy-altruism hypothesis: issues and implications. From bench to bedside, Empathy, pp 41–54

    Google Scholar 

  • Boukricha H, Wachsmuth I, Carminati MN, Knoeferle P (2013) A computational model of empathy: Empirical evaluation. In: Humaine association conference on affective computing and intelligent interaction (ACII). IEEE, pp 1–6

  • Bratman M (1987) Intention, plans, and practical reason

  • Brave S, Nass C, Hutchinson K (2005) Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent. Int J Hum Comput Stud 62(2):161–178

    Google Scholar 

  • Broekens J, Degroot D, Kosters WA (2008) Formal models of appraisal: theory, specification, and computational model. Cognit Syst Res 9(3):173–197

    Google Scholar 

  • Burleson W, Picard RW (2007) Gender-specific approaches to developing emotionally intelligent learning companions. IEEE Intell Syst 22(4):62–69

    Google Scholar 

  • Calvo RA, D’Mello S (2010) Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput 1(1):18–37

    Google Scholar 

  • Cambria E (2016) Affective computing and sentiment analysis. IEEE Intell Syst 31(2):102–107

    Google Scholar 

  • Clark AJ (2014) Empathy in counseling and psychotherapy: perspectives and practices. Routledge, Abingdon-on-Thames

    Google Scholar 

  • Cooper B, Brna P, Martins A (2000) Effective affective in intelligent systems-building on evidence of empathy in teaching and learning. In: Affective interactions, Springer, pp 21–34

  • Coplan A (2011) Understanding empathy: its features and effects. In: Goldie P, Coplan A (eds) Philosophical and psychological perspectives. Oxford University Press, Oxford, pp 3–18

    Google Scholar 

  • Coplan A, Goldie P (2011) Empathy: philosophical and psychological perspectives. Oxford University Press, Oxford

    Google Scholar 

  • Davis MH (1983) Measuring individual differences in empathy: evidence for a multidimensional approach. J Person Soc Psychol 44(1):113

    Google Scholar 

  • Davis MH (1994) Empathy: a social psychological approach. Brown and Benchmark Publishers, New York

    Google Scholar 

  • De Vignemont F, Singer T (2006) The empathic brain: How, when and why? Trends Cognit Sci 10(10):435–441

    Google Scholar 

  • De Waal FB (2007) The ‘russian doll’model of empathy and imitation. From mirror neurons to empathy, On being moved, pp 35–48

    Google Scholar 

  • De Waal F (2010) The age of empathy: nature’s lessons for a kinder society. Broadway Books, New York

    Google Scholar 

  • de Waal FB, Preston SD (2017) Mammalian empathy: behavioural manifestations and neural basis. Nat Rev Neurosci 18(8):498

    Google Scholar 

  • Decety J, Cowell JM (2014) The complex relation between morality and empathy. Trends Cognit Sci 18(7):337–339

    Google Scholar 

  • D’mello S, Graesser A (2012) Autotutor and affective autotutor: learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Trans Interact Intell Syst 2(4):23

    Google Scholar 

  • Gibson J, Malandrakis N, Romero F, Atkins DC, Narayanan SS (2015) Predicting therapist empathy in motivational interviews using language features inspired by psycholinguistic norms. In: Sixteenth annual conference of the international speech communication association

  • Golan O, Baron-Cohen S, Hill JJ, Golan Y (2006) “The reading the mind in films” task: complex emotion recognition in adults with and without autism spectrum conditions. Soc Neurosci 1(2):111–123

    Google Scholar 

  • Golan O, Baron-Cohen S, Hill JJ, Rutherford M (2007) The “reading the mind in the voice’test-revised: a study of complex emotion recognition in adults with and without autism spectrum conditions. J Autism Dev Disord 37(6):1096–1106

    Google Scholar 

  • Goldman AI (2006) Simulating minds: the philosophy, psychology, and neuroscience of mindreading. Oxford University Press, Oxford

    Google Scholar 

  • Goldman AI (2011) Two routes to empathy. In: Goldie P, Coplan A (eds) Empathy: philosophical and psychological perspectives. Oxford University Press, Oxford, pp 31–44

    Google Scholar 

  • Gonsior B, Sosnowski S, Mayer C, Blume J, Radig B, Wollherr D, Kühnlenz K (2011) Improving aspects of empathy and subjective performance for HRI through mirroring facial expressions. In: RO-MAN. IEEE, pp 350–356

  • Hagendoorn I (2004) Some speculative hypotheses about the nature and perception of dance and choreography. J Consciousness Stud 11(F0020003):79–110

    Google Scholar 

  • Hegel F, Spexard T, Wrede B, Horstmann G, Vogt T (2006) Playing a different imitation game: interaction with an empathic android robot. IEEE, pp 56–61

  • Hoffman ML (2000) Empathy and moral development: implications for caring and justice. Cambridge University Press, Cambridge. https://doi.org/10.1017/CBO9780511805851

    Article  Google Scholar 

  • Hume D (1739) A treatise of human nature: being an attempt to introduce the experimental method of reasoning into moral subjects. printed for John Noon, London

  • Iacoboni M (2011) Within each other: neural mechanisms for empathy in the primate brain. In: Coplan A, Goldie P (eds) Empathy: philosophical and psychological perspectives. Oxford University Press, Oxford, p 45

    Google Scholar 

  • Jaimes A, Sebe N (2005) Multimodal human computer interaction: a survey. In: Sebe N, Lew M, Huang TS (eds) Computer vision in human-computer interaction. Springer, Berlin, pp 1–15

    Google Scholar 

  • Kohut H (2011) The search for the self: volume 1: selected writings of Heinz Kohut 1950–1978. Karnac Books, London

  • Kumano S, Otsuka K, Mikami D, Matsuda M, Yamato J (2015) Analyzing interpersonal empathy via collective impressions. IEEE Trans Affect Comput 6(4):324–336

    Google Scholar 

  • Lawrence EJ, Shaw P, Baker D, Baron-Cohen S, David AS (2004) Measuring empathy: reliability and validity of the empathy quotient. Psychol Med 34(5):911–920

    Google Scholar 

  • Lawson J, Baron-Cohen S, Wheelwright S (2004) Empathising and systemising in adults with and without asperger syndrome. J Autism Dev Disord 34(3):301–310

    Google Scholar 

  • Lazarus RS (1966) Psychological stress and the coping process/[by] Richard S. Lazarus. McGraw-Hill series in psychology

  • Lefimann N, Kopp S, Wachsmuth I (2006) Situated interaction with a virtual human perception, action, and cognition. Situat Commun 166:287

    Google Scholar 

  • Leiberg S, Anders S (2006) The multiple facets of empathy: a survey of theory and evidence. In: Anders S, Ende G, Junghofer M, Kissler J, Wildgruber D (eds) Understanding emotions, progress in brain research, vol 156. Elsevier, Amsterdam, pp 419–440. https://doi.org/10.1016/S0079-6123(06)56023-6

    Chapter  Google Scholar 

  • Leite I, Castellano G, Pereira A, Martinho C, Paiva A (2014) Empathic robots for long-term interaction. Int J Soc Robot 6(3):329–341

    Google Scholar 

  • Lim A, Okuno HG (2015) A recipe for empathy. Int J Soc Robot 7(1):35–49

    Google Scholar 

  • Lisetti C, Amini R, Yasavur U, Rishe N (2013) I can help you change! An empathic virtual agent delivers behavior change health interventions. ACM Trans Manag Inf Syst 4(4):19

    Google Scholar 

  • Marsella SC, Gratch J (2009) EMA: a process model of appraisal dynamics. Cognit Syst Res 10(1):70–90

    Google Scholar 

  • Marsella S, Gratch J (2014) Computationally modeling human emotion. Commun ACM 57(12):56–67. https://doi.org/10.1145/2631912

    Article  Google Scholar 

  • McQuiggan SW, Robison JL, Phillips R, Lester JC (2008) Modeling parallel and reactive empathy in virtual agents: an inductive approach. In: Proceedings of the 7th international joint conference on autonomous agents and multiagent systems, vol 1. International Foundation for Autonomous Agents and Multiagent Systems, pp 167–174

  • McQuiggan SW, Lester JC (2007) Modeling and evaluating empathy in embodied companion agents. Int J Hum Compur Stud 65(4):348–360

    Google Scholar 

  • Moridis CN, Economides AA (2012) Affective learning: empathetic agents with emotional facial and tone of voice expressions. IEEE Trans Affect Comput 3(3):260–272

    Google Scholar 

  • Moyers TB, Rowell LN, Manuel JK, Ernst D, Houck JM (2016) The motivational interviewing treatment integrity code (MITI 4): rationale, preliminary reliability and validity. J Subst Abuse Treat 65:36–42

    Google Scholar 

  • Nowak M (2011) The complicated history of einfühlung. Argum Biannu Philos J 1(2):301–326

    Google Scholar 

  • Ochs M, Sadek D, Pelachaud C (2012) A formal model of emotions for an empathic rational dialog agent. Auton Agents Multi-Agent Syst 24(3):410–440

    Google Scholar 

  • Omdahl BL (1995) Cognitive appraisal, emotion, and empathy

  • Ortony A, Clore GL, Collins A (1990) The cognitive structure of emotions. Cambridge University Press, Cambridge

    Google Scholar 

  • Paiva A, Leite I, Boukricha H, Wachsmuth I (2017) Empathy in virtual agents and robots: a survey. ACM Trans Interact Intell Syst 7(3):11

    Google Scholar 

  • Picard RW (2014) The promise of affective computing. The Oxford handbook of affective computing 11

  • Poria S, Cambria E, Howard N, Huang GB, Hussain A (2016) Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing 174:50–59

    Google Scholar 

  • Poria S, Cambria E, Bajpai R, Hussain A (2017) A review of affective computing: from unimodal analysis to multimodal fusion. Inf Fusion 37:98–125

    Google Scholar 

  • Premack D, Woodruff G (1978) Does the chimpanzee have a theory of mind? Behav Brain Sci 1(4):515–526

    Google Scholar 

  • Prendinger H, Ishizuka M (2005) The empathic companion: a character-based interface that addresses users’affective states. Appl Artif Intell 19(3–4):267–285

    Google Scholar 

  • Preston SD, De Waal FB (2002) Empathy: its ultimate and proximate bases. Behav Brain Sci 25(1):1–20

    Google Scholar 

  • Prinz J (2011) Is empathy necessary for morality. Empathy Philos Psychol Perspect 1:211–229

    Google Scholar 

  • Rao AS, Georgeff MP (1991) Modeling rational agents within a BDI-architecture. KR 91:473–484

    MathSciNet  MATH  Google Scholar 

  • Rashkin H, Smith EM, Li M, Boureau YL (2018) I know the feeling: learning to converse with empathy. arXiv preprint arXiv:1811.00207

  • Reeves B, Nass CI (1996) The media equation: how people treat computers, television, and new media like real people and places. Cambridge University Press, Cambridge

    Google Scholar 

  • Riek LD, Rabinowitch TC, Chakrabarti B, Robinson P (2009) How anthropomorphism affects empathy toward robots. In: Proceedings of the 4th ACM/IEEE international conference on Human robot interaction. ACM, pp 245–246

  • Riek LD, Paul PC, Robinson P (2010) When my robot smiles at me: enabling human-robot rapport via real-time head gesture mimicry. J Multimodal User Interfaces 3(1–2):99–108

    Google Scholar 

  • Rizzolatti G, Fabbri-Destro M (2010) Mirror neurons: from discovery to autism. Exp Brain Res 200(3–4):223–237

    Google Scholar 

  • Rodrigues SH, Mascarenhas S, Dias J, Paiva A (2014) A process model of empathy for virtual agents. Interact Comput 27(4):371–391

    Google Scholar 

  • Roseman IJ, Smith CA (2001) Appraisal theory. Appraisal processes in emotion: theory, methods, research, pp 3–19

  • Rosenthal-Von Der Pütten AM, Schulte FP, Eimler SC, Sobieraj S, Hoffmann L, Maderwald S, Brand M, Krämer NC (2014) Investigations on empathy towards humans and robots using FMRI. Comput Hum Behav 33:201–212

    Google Scholar 

  • Russell JA (1980) A circumplex model of affect. J Person Soc Psychol 39(6):1161

    Google Scholar 

  • Salovey P, Mayer JD (1990) Emotional intelligence. Imagin Cogn Person 9(3):185–211

    Google Scholar 

  • Scherer KR (2010a) Emotion and emotional competence: conceptual and theoretical issues for modelling agents. Blueprint for affective computing, pp 3–20

  • Scherer KR (2010b) The component process model: architecture for a comprehensive computational model of emergent emotion. A sourcebook, Blueprint for affective computing, pp 47–70

  • Scherer KR (2001) Appraisal considered as a process of multilevel sequential checking. Oxford University Press, Oxford

    Google Scholar 

  • Scherer KR (2005) What are emotions? And how can they be measured? Soc Sci Inf 44(4):695–729

    Google Scholar 

  • Searle JR (1969) Speech acts: an essay in the philosophy of language, vol 626. Cambridge University Press, Cambridge

    Google Scholar 

  • Singer T, Lamm C (2009) The social neuroscience of empathy. Ann N Y Acad Sci 11561(1):81–96

    Google Scholar 

  • Slote M (2007) The ethics of care and empathy. Routledge, London

    Google Scholar 

  • Smith M (2011) Empathy, expansionism, and the extended mind. Philosophical and psychological perspectives, empathy, pp 99–117

  • Smith A (1959) The theory of moral sentiments. A. Millar and A. Kincaid and J. Bell, London

    Google Scholar 

  • Soleymani M, Garcia D, Jou B, Schuller B, Chang SF, Pantic M (2017) A survey of multimodal sentiment analysis. Image Vis Comput 65:3–14

    Google Scholar 

  • Stueber KR (2006) Rediscovering empathy : agency, folk psychology, and the human sciences

  • Tan ZX, Goel A, Nguyen TS, Ong DC (2018) A multimodal lSTM for predicting listener empathic responses over time. arXiv e-prints arXiv:1812.04891

  • Tavassoli T, Hoekstra RA, Baron-Cohen S (2014) The sensory perception quotient (SPQ): development and validation of a new sensory questionnaire for adults with and without autism. Mol Autism 5(1):29

    Google Scholar 

  • Titchener EB (1909) Lectures on the experimental psychology of the thought-processes. Macmillan, Ottawa

    Google Scholar 

  • Wisp L (1987) History of the concept of empathy. Empathy and its development, pp 17–37

  • Wood K (2016) Kinesthetic empathy: conditions for viewing. Oxford University Press, Oxford

    Google Scholar 

  • Xiao B, Bone D, Segbroeck MV, Imel ZE, Atkins DC, Georgiou PG, Narayanan SS (2014) Modeling therapist empathy through prosody in drug addiction counseling. In: Fifteenth annual conference of the international speech communication association

  • Yalçın ÖN, DiPaola S (2019) Evaluating levels of emotional contagion with an embodied conversational agent. In: Proceedings of the 41st annual conference of the cognitive science society

  • Yalçın ÖN (2019) Evaluating empathy in artificial agents. In: 2019 International conference on affective computing and intelligent interaction (ACII). IEEE

  • Yalçın ÖN (in press) Empathy framework for embodied conversational agents. Cogn Syst Res J

  • Yalçın ÖN, DiPaola S (2018) A computational model of empathy for interactive agents. Biol Inspir Cogn Archit 26:20–25

    Google Scholar 

  • Yalçın ÖN, DiPaola S (2019) M-path: a conversational system for the empathic virtual agent. In: Samsonovich AV (ed) Biologically inspired cognitive architectures. Springer, Cham, pp 597–607

    Google Scholar 

  • Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 31(1):39–58

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the reviewers for their valuable comments and suggestions to improve this article. This work was partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) [RGPIN-2019-06767] and the Social Sciences and Humanities Research Council of Canada (SSHRC) [435-2017-0625].

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Özge Nilay Yalçın.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Social Sciences and Humanities Research Council of Canada (SSHRC).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yalçın, Ö.N., DiPaola, S. Modeling empathy: building a link between affective and cognitive processes. Artif Intell Rev 53, 2983–3006 (2020). https://doi.org/10.1007/s10462-019-09753-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-019-09753-0

Keywords

Navigation