Abstract
Computational modeling of empathy has recently become an increasingly popular way of studying human relations. It provides a way to increase our understanding of the link between affective and cognitive processes and enhance our interaction with artificial agents. However, the variety of fields contributing to empathy research has resulted in isolated approaches to modeling empathy, and this has led to various definitions of empathy and an absence of common ground regarding underlying empathic processes. Although this diversity may be useful in that it allows for an in-depth examination of various processes linked to empathy, it also may not yet provide a coherent theoretical picture of empathy. We argue that a clear theoretical positioning is required for collective progress. The aim of this article is, therefore, to call for a holistic and multilayered view of a model of empathy, taken from the rich background research from various disciplines. To achieve this, we present a comprehensive background on the theoretical foundations, followed by the working definitions, components, and models of empathy that are proposed by various fields. Following this introduction, we provide a detailed review of the existing techniques used in AI research to model empathy in interactive agents, focusing on the strengths and weaknesses of each approach. We conclude with a discussion of future directions in this emerging field.
References
Asada M (2015) Towards artificial empathy. Int J Soc Robot 7(1):19–33
Baron-Cohen S, Wheelwright S (2003) The friendship questionnaire: an investigation of adults with asperger syndrome or high-functioning autism, and normal sex differences. J Autism Dev Disord 33(5):509–517
Baron-Cohen S, Wheelwright S (2004) The empathy quotient: an investigation of adults with asperger syndrome or high functioning autism, and normal sex differences. J Autism Dev Disord 34(2):163–175
Baron-Cohen S, Leslie AM, Frith U (1986) Mechanical, behavioural and intentional understanding of picture stories in autistic children. Br J Dev Psychol 4(2):113–125
Baron-Cohen S, Wheelwright S, Hill J, Raste Y, Plumb I (2001a) The “reading the mind in the eyes” test revised version: a study with normal adults, and adults with asperger syndrome or high-functioning autism. J Child Psychol Psychiatry Allied Discip 42(2):241–251
Baron-Cohen S, Wheelwright S, Spong A, Scahill V, Lawson J et al (2001b) Are intuitive physics and intuitive psychology independent? A test with children with asperger syndrome. J Dev Learn Disord 5(1):47–78
Baron-Cohen S, Richler J, Bisarya D, Gurunathan N, Wheelwright S (2003) The systemizing quotient: an investigation of adults with asperger syndrome or high-functioning autism, and normal sex differences. Philos Trans R Soc B Biol Sci 358(1430):361–374
Baron-Cohen S, Tager-Flusberg H, Lombardo M (2013) Understanding other minds: perspectives from developmental social neuroscience. Oxford University Press, Oxford
Barros P (forthcoming) OMG empathy challenge. In: IEEE international conference on automatic face and gesture recognition
Bates J et al (1994) The role of emotion in believable agents. Commun ACM 37(7):122–125
Batson CD (2009) These things called empathy: eight related but distinct phenomena
Batson CD (2012) The empathy-altruism hypothesis: issues and implications. From bench to bedside, Empathy, pp 41–54
Boukricha H, Wachsmuth I, Carminati MN, Knoeferle P (2013) A computational model of empathy: Empirical evaluation. In: Humaine association conference on affective computing and intelligent interaction (ACII). IEEE, pp 1–6
Bratman M (1987) Intention, plans, and practical reason
Brave S, Nass C, Hutchinson K (2005) Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent. Int J Hum Comput Stud 62(2):161–178
Broekens J, Degroot D, Kosters WA (2008) Formal models of appraisal: theory, specification, and computational model. Cognit Syst Res 9(3):173–197
Burleson W, Picard RW (2007) Gender-specific approaches to developing emotionally intelligent learning companions. IEEE Intell Syst 22(4):62–69
Calvo RA, D’Mello S (2010) Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput 1(1):18–37
Cambria E (2016) Affective computing and sentiment analysis. IEEE Intell Syst 31(2):102–107
Clark AJ (2014) Empathy in counseling and psychotherapy: perspectives and practices. Routledge, Abingdon-on-Thames
Cooper B, Brna P, Martins A (2000) Effective affective in intelligent systems-building on evidence of empathy in teaching and learning. In: Affective interactions, Springer, pp 21–34
Coplan A (2011) Understanding empathy: its features and effects. In: Goldie P, Coplan A (eds) Philosophical and psychological perspectives. Oxford University Press, Oxford, pp 3–18
Coplan A, Goldie P (2011) Empathy: philosophical and psychological perspectives. Oxford University Press, Oxford
Davis MH (1983) Measuring individual differences in empathy: evidence for a multidimensional approach. J Person Soc Psychol 44(1):113
Davis MH (1994) Empathy: a social psychological approach. Brown and Benchmark Publishers, New York
De Vignemont F, Singer T (2006) The empathic brain: How, when and why? Trends Cognit Sci 10(10):435–441
De Waal FB (2007) The ‘russian doll’model of empathy and imitation. From mirror neurons to empathy, On being moved, pp 35–48
De Waal F (2010) The age of empathy: nature’s lessons for a kinder society. Broadway Books, New York
de Waal FB, Preston SD (2017) Mammalian empathy: behavioural manifestations and neural basis. Nat Rev Neurosci 18(8):498
Decety J, Cowell JM (2014) The complex relation between morality and empathy. Trends Cognit Sci 18(7):337–339
D’mello S, Graesser A (2012) Autotutor and affective autotutor: learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Trans Interact Intell Syst 2(4):23
Gibson J, Malandrakis N, Romero F, Atkins DC, Narayanan SS (2015) Predicting therapist empathy in motivational interviews using language features inspired by psycholinguistic norms. In: Sixteenth annual conference of the international speech communication association
Golan O, Baron-Cohen S, Hill JJ, Golan Y (2006) “The reading the mind in films” task: complex emotion recognition in adults with and without autism spectrum conditions. Soc Neurosci 1(2):111–123
Golan O, Baron-Cohen S, Hill JJ, Rutherford M (2007) The “reading the mind in the voice’test-revised: a study of complex emotion recognition in adults with and without autism spectrum conditions. J Autism Dev Disord 37(6):1096–1106
Goldman AI (2006) Simulating minds: the philosophy, psychology, and neuroscience of mindreading. Oxford University Press, Oxford
Goldman AI (2011) Two routes to empathy. In: Goldie P, Coplan A (eds) Empathy: philosophical and psychological perspectives. Oxford University Press, Oxford, pp 31–44
Gonsior B, Sosnowski S, Mayer C, Blume J, Radig B, Wollherr D, Kühnlenz K (2011) Improving aspects of empathy and subjective performance for HRI through mirroring facial expressions. In: RO-MAN. IEEE, pp 350–356
Hagendoorn I (2004) Some speculative hypotheses about the nature and perception of dance and choreography. J Consciousness Stud 11(F0020003):79–110
Hegel F, Spexard T, Wrede B, Horstmann G, Vogt T (2006) Playing a different imitation game: interaction with an empathic android robot. IEEE, pp 56–61
Hoffman ML (2000) Empathy and moral development: implications for caring and justice. Cambridge University Press, Cambridge. https://doi.org/10.1017/CBO9780511805851
Hume D (1739) A treatise of human nature: being an attempt to introduce the experimental method of reasoning into moral subjects. printed for John Noon, London
Iacoboni M (2011) Within each other: neural mechanisms for empathy in the primate brain. In: Coplan A, Goldie P (eds) Empathy: philosophical and psychological perspectives. Oxford University Press, Oxford, p 45
Jaimes A, Sebe N (2005) Multimodal human computer interaction: a survey. In: Sebe N, Lew M, Huang TS (eds) Computer vision in human-computer interaction. Springer, Berlin, pp 1–15
Kohut H (2011) The search for the self: volume 1: selected writings of Heinz Kohut 1950–1978. Karnac Books, London
Kumano S, Otsuka K, Mikami D, Matsuda M, Yamato J (2015) Analyzing interpersonal empathy via collective impressions. IEEE Trans Affect Comput 6(4):324–336
Lawrence EJ, Shaw P, Baker D, Baron-Cohen S, David AS (2004) Measuring empathy: reliability and validity of the empathy quotient. Psychol Med 34(5):911–920
Lawson J, Baron-Cohen S, Wheelwright S (2004) Empathising and systemising in adults with and without asperger syndrome. J Autism Dev Disord 34(3):301–310
Lazarus RS (1966) Psychological stress and the coping process/[by] Richard S. Lazarus. McGraw-Hill series in psychology
Lefimann N, Kopp S, Wachsmuth I (2006) Situated interaction with a virtual human perception, action, and cognition. Situat Commun 166:287
Leiberg S, Anders S (2006) The multiple facets of empathy: a survey of theory and evidence. In: Anders S, Ende G, Junghofer M, Kissler J, Wildgruber D (eds) Understanding emotions, progress in brain research, vol 156. Elsevier, Amsterdam, pp 419–440. https://doi.org/10.1016/S0079-6123(06)56023-6
Leite I, Castellano G, Pereira A, Martinho C, Paiva A (2014) Empathic robots for long-term interaction. Int J Soc Robot 6(3):329–341
Lim A, Okuno HG (2015) A recipe for empathy. Int J Soc Robot 7(1):35–49
Lisetti C, Amini R, Yasavur U, Rishe N (2013) I can help you change! An empathic virtual agent delivers behavior change health interventions. ACM Trans Manag Inf Syst 4(4):19
Marsella SC, Gratch J (2009) EMA: a process model of appraisal dynamics. Cognit Syst Res 10(1):70–90
Marsella S, Gratch J (2014) Computationally modeling human emotion. Commun ACM 57(12):56–67. https://doi.org/10.1145/2631912
McQuiggan SW, Robison JL, Phillips R, Lester JC (2008) Modeling parallel and reactive empathy in virtual agents: an inductive approach. In: Proceedings of the 7th international joint conference on autonomous agents and multiagent systems, vol 1. International Foundation for Autonomous Agents and Multiagent Systems, pp 167–174
McQuiggan SW, Lester JC (2007) Modeling and evaluating empathy in embodied companion agents. Int J Hum Compur Stud 65(4):348–360
Moridis CN, Economides AA (2012) Affective learning: empathetic agents with emotional facial and tone of voice expressions. IEEE Trans Affect Comput 3(3):260–272
Moyers TB, Rowell LN, Manuel JK, Ernst D, Houck JM (2016) The motivational interviewing treatment integrity code (MITI 4): rationale, preliminary reliability and validity. J Subst Abuse Treat 65:36–42
Nowak M (2011) The complicated history of einfühlung. Argum Biannu Philos J 1(2):301–326
Ochs M, Sadek D, Pelachaud C (2012) A formal model of emotions for an empathic rational dialog agent. Auton Agents Multi-Agent Syst 24(3):410–440
Omdahl BL (1995) Cognitive appraisal, emotion, and empathy
Ortony A, Clore GL, Collins A (1990) The cognitive structure of emotions. Cambridge University Press, Cambridge
Paiva A, Leite I, Boukricha H, Wachsmuth I (2017) Empathy in virtual agents and robots: a survey. ACM Trans Interact Intell Syst 7(3):11
Picard RW (2014) The promise of affective computing. The Oxford handbook of affective computing 11
Poria S, Cambria E, Howard N, Huang GB, Hussain A (2016) Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing 174:50–59
Poria S, Cambria E, Bajpai R, Hussain A (2017) A review of affective computing: from unimodal analysis to multimodal fusion. Inf Fusion 37:98–125
Premack D, Woodruff G (1978) Does the chimpanzee have a theory of mind? Behav Brain Sci 1(4):515–526
Prendinger H, Ishizuka M (2005) The empathic companion: a character-based interface that addresses users’affective states. Appl Artif Intell 19(3–4):267–285
Preston SD, De Waal FB (2002) Empathy: its ultimate and proximate bases. Behav Brain Sci 25(1):1–20
Prinz J (2011) Is empathy necessary for morality. Empathy Philos Psychol Perspect 1:211–229
Rao AS, Georgeff MP (1991) Modeling rational agents within a BDI-architecture. KR 91:473–484
Rashkin H, Smith EM, Li M, Boureau YL (2018) I know the feeling: learning to converse with empathy. arXiv preprint arXiv:1811.00207
Reeves B, Nass CI (1996) The media equation: how people treat computers, television, and new media like real people and places. Cambridge University Press, Cambridge
Riek LD, Rabinowitch TC, Chakrabarti B, Robinson P (2009) How anthropomorphism affects empathy toward robots. In: Proceedings of the 4th ACM/IEEE international conference on Human robot interaction. ACM, pp 245–246
Riek LD, Paul PC, Robinson P (2010) When my robot smiles at me: enabling human-robot rapport via real-time head gesture mimicry. J Multimodal User Interfaces 3(1–2):99–108
Rizzolatti G, Fabbri-Destro M (2010) Mirror neurons: from discovery to autism. Exp Brain Res 200(3–4):223–237
Rodrigues SH, Mascarenhas S, Dias J, Paiva A (2014) A process model of empathy for virtual agents. Interact Comput 27(4):371–391
Roseman IJ, Smith CA (2001) Appraisal theory. Appraisal processes in emotion: theory, methods, research, pp 3–19
Rosenthal-Von Der Pütten AM, Schulte FP, Eimler SC, Sobieraj S, Hoffmann L, Maderwald S, Brand M, Krämer NC (2014) Investigations on empathy towards humans and robots using FMRI. Comput Hum Behav 33:201–212
Russell JA (1980) A circumplex model of affect. J Person Soc Psychol 39(6):1161
Salovey P, Mayer JD (1990) Emotional intelligence. Imagin Cogn Person 9(3):185–211
Scherer KR (2010a) Emotion and emotional competence: conceptual and theoretical issues for modelling agents. Blueprint for affective computing, pp 3–20
Scherer KR (2010b) The component process model: architecture for a comprehensive computational model of emergent emotion. A sourcebook, Blueprint for affective computing, pp 47–70
Scherer KR (2001) Appraisal considered as a process of multilevel sequential checking. Oxford University Press, Oxford
Scherer KR (2005) What are emotions? And how can they be measured? Soc Sci Inf 44(4):695–729
Searle JR (1969) Speech acts: an essay in the philosophy of language, vol 626. Cambridge University Press, Cambridge
Singer T, Lamm C (2009) The social neuroscience of empathy. Ann N Y Acad Sci 11561(1):81–96
Slote M (2007) The ethics of care and empathy. Routledge, London
Smith M (2011) Empathy, expansionism, and the extended mind. Philosophical and psychological perspectives, empathy, pp 99–117
Smith A (1959) The theory of moral sentiments. A. Millar and A. Kincaid and J. Bell, London
Soleymani M, Garcia D, Jou B, Schuller B, Chang SF, Pantic M (2017) A survey of multimodal sentiment analysis. Image Vis Comput 65:3–14
Stueber KR (2006) Rediscovering empathy : agency, folk psychology, and the human sciences
Tan ZX, Goel A, Nguyen TS, Ong DC (2018) A multimodal lSTM for predicting listener empathic responses over time. arXiv e-prints arXiv:1812.04891
Tavassoli T, Hoekstra RA, Baron-Cohen S (2014) The sensory perception quotient (SPQ): development and validation of a new sensory questionnaire for adults with and without autism. Mol Autism 5(1):29
Titchener EB (1909) Lectures on the experimental psychology of the thought-processes. Macmillan, Ottawa
Wisp L (1987) History of the concept of empathy. Empathy and its development, pp 17–37
Wood K (2016) Kinesthetic empathy: conditions for viewing. Oxford University Press, Oxford
Xiao B, Bone D, Segbroeck MV, Imel ZE, Atkins DC, Georgiou PG, Narayanan SS (2014) Modeling therapist empathy through prosody in drug addiction counseling. In: Fifteenth annual conference of the international speech communication association
Yalçın ÖN, DiPaola S (2019) Evaluating levels of emotional contagion with an embodied conversational agent. In: Proceedings of the 41st annual conference of the cognitive science society
Yalçın ÖN (2019) Evaluating empathy in artificial agents. In: 2019 International conference on affective computing and intelligent interaction (ACII). IEEE
Yalçın ÖN (in press) Empathy framework for embodied conversational agents. Cogn Syst Res J
Yalçın ÖN, DiPaola S (2018) A computational model of empathy for interactive agents. Biol Inspir Cogn Archit 26:20–25
Yalçın ÖN, DiPaola S (2019) M-path: a conversational system for the empathic virtual agent. In: Samsonovich AV (ed) Biologically inspired cognitive architectures. Springer, Cham, pp 597–607
Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 31(1):39–58
Acknowledgements
The authors would like to thank the reviewers for their valuable comments and suggestions to improve this article. This work was partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) [RGPIN-2019-06767] and the Social Sciences and Humanities Research Council of Canada (SSHRC) [435-2017-0625].
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Social Sciences and Humanities Research Council of Canada (SSHRC).
Rights and permissions
About this article
Cite this article
Yalçın, Ö.N., DiPaola, S. Modeling empathy: building a link between affective and cognitive processes. Artif Intell Rev 53, 2983–3006 (2020). https://doi.org/10.1007/s10462-019-09753-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-019-09753-0