Computational Analysis and Simulation of Empathic Behaviors: a Survey of Empathy Modeling with Behavioral Signal Processing Framework

  • Bo Xiao
  • Zac E. Imel
  • Panayiotis Georgiou
  • David C. Atkins
  • Shrikanth S. Narayanan
Psychiatry in the Digital Age (JS Luo, Section Editor)
Part of the following topical collections:
  1. Topical Collection on Psychiatry in the Digital Age


Empathy is an important psychological process that facilitates human communication and interaction. Enhancement of empathy has profound significance in a range of applications. In this paper, we review emerging directions of research on computational analysis of empathy expression and perception as well as empathic interactions, including their simulation. We summarize the work on empathic expression analysis by the targeted signal modalities (e.g., text, audio, and facial expressions). We categorize empathy simulation studies into theory-based emotion space modeling or application-driven user and context modeling. We summarize challenges in computational study of empathy including conceptual framing and understanding of empathy, data availability, appropriate use and validation of machine learning techniques, and behavior signal processing. Finally, we propose a unified view of empathy computation and offer a series of open problems for future research.


Empathy Computational modeling Analysis Simulation Behavioral signal processing 


Compliance with Ethical Standards

Conflict of Interest

Bo Xiao, Zac E. Imel, Panayiotis Georgiou, and Shrikanth S. Narayanan declare that they have no conflict of interest.

David C. Atkins reports grants from NIAAA and NIDA.

Human and Animal Rights and Informed Consent

This article does not contain any studies with human or animal subjects performed by any of the authors.


Papers of particular interest, published recently, have been highlighted as: • Of importance •• Of major importance

  1. 1.
    Titchener EB. Lectures on the experimental psychology of the thought-processes. Macmillan; 1909.Google Scholar
  2. 2.
    Hoffman ML. Empathy and moral development: implications for caring and justice. Cambridge University Press; 2001.Google Scholar
  3. 3.
    Batson CD. In: Decety J, editor. These things called empathy: eight related but distinct phenomena. CRP press; 2009. p. 16–28.Google Scholar
  4. 4.
    Cuff BM, Brown SJ, Taylor L, Howat DJ. Empathy: a review of the concept. Emot Rev. 2014.Google Scholar
  5. 5.
    Decety J, Jackson P. The functional architecture of human empathy. Behav Cogn Neurosci Rev. 2004;3(2):71–100.CrossRefPubMedGoogle Scholar
  6. 6.
    Elliott R, Bohart AC, Watson JC, Greenberg LS. Empathy. Psychotherapy. 2011;48(1):43–9.CrossRefPubMedGoogle Scholar
  7. 7.
    Preston SD, De Waal F. Empathy: its ultimate and proximate bases. Behav Brain Sci. 2002;25(01):1–20.PubMedGoogle Scholar
  8. 8.
    De Vignemont F, Singer T. The empathic brain: how, when and why? Trends Cogn Sci. 2006;10(10):435–41.CrossRefPubMedGoogle Scholar
  9. 9.
    Iacoboni M. Imitation, empathy, and mirror neurons. Annu Rev Psychol. 2009;60:653–70.CrossRefPubMedGoogle Scholar
  10. 10.
    Derksen F, Bensing J, Lagro-Janssen A. Effectiveness of empathy in general practice: a systematic review. Br J Gen Pract. 2013;63(606):e76–84.CrossRefPubMedPubMedCentralGoogle Scholar
  11. 11.
    Lelorain S, Brédart A, Dolbeault S, Sultan S. A systematic review of the associations between empathy measures and patient outcomes in cancer care. Psychooncology. 2012;21(12):1255–64.CrossRefPubMedGoogle Scholar
  12. 12.
    Moyers TB, Miller WR. Is low therapist empathy toxic? Psychol Addict Behav. 2013;27(3):878.CrossRefPubMedPubMedCentralGoogle Scholar
  13. 13.
    van Berkhout ET, Malouff JM. The efficacy of empathy training: a meta-analysis of randomized controlled trials. J Couns Psychol. 2015.Google Scholar
  14. 14.•
    Atkins DC, Steyvers M, Imel ZE, Smyth P. Scaling up the evaluation of psychotherapy: evaluating motivational interviewing fidelity via statistical text classification. Implement Sci. 2014;9(1):49. This work used topic models to analyze provider fidelity based on text of the psychotherapy interaction, illuminating the possibility of scaling up the evaluation through computational means. CrossRefPubMedPubMedCentralGoogle Scholar
  15. 15.
    Riess H. Biomarkers in the psychotherapeutic relationship: the role of physiology, neurobiology, and biological correlates of E.M.P.A.T.H.Y. Harv Rev Psychiatry. 2011;19(3):162–74.CrossRefPubMedGoogle Scholar
  16. 16.
    Regenbogen C, Schneider DA, Finkelmeyer A, Kohn N, Derntl B, Kellermann T, et al. The differential contribution of facial expressions, prosody, and speech content to empathy. Cognition & emotion. 2012;26(6):995–1014.CrossRefGoogle Scholar
  17. 17.
    Duda RO, Hart PE, Stork DG. Pattern classification. Wiley; 2012.Google Scholar
  18. 18.
    Miller WR, Rollnick S. Motivational interviewing: helping people change. Guilford Press; 2012.Google Scholar
  19. 19.
    Miller WR, Rose GS. Toward a theory of motivational interviewing. Am Psychol. 2009;64(6):527.CrossRefPubMedPubMedCentralGoogle Scholar
  20. 20.
    Moyers, T., Martin, T., Manuel, J., Miller, W., Ernst, D.: Revised global scales: Motivational Interviewing Treatment Integrity 3.0 (2007)Google Scholar
  21. 21.
    Kumano, S., Otsuka, K., Matsuda, M., Yamato, J.: Analyzing perceived empathy/antipathy based on reaction time in behavioral coordination. In: Automatic Face and Gesture Recognition, pp. 1–8. IEEE (2013)Google Scholar
  22. 22.
    Delaherche E, Chetouani M, Mahdhaoui A, Saint-Georges C, Viaux S, Cohen D. Interpersonal synchrony: a survey of evaluation methods across disciplines. IEEE Trans Affect Comput. 2012;3(3):349–65.CrossRefGoogle Scholar
  23. 23.
    Coulehan JL, Platt FW, Egener B, Frankel R, Lin CT, Lown B, et al. “Let me see if I have this right”: words that help build empathy. Ann Intern Med. 2001;135(3):221–7.CrossRefPubMedGoogle Scholar
  24. 24.
    Kneser R, Ney H. Improved backing-off for m-gram language modeling. In: International conference on acoustics, speech, and signal processing, vol. 1. IEEE; 1995. p. 181–4.Google Scholar
  25. 25.•
    Xiao B, Can D, Georgiou PG, Atkins DC, Narayanan SS. Analyzing the language of therapist empathy in motivational interview based psychotherapy. Proc. APSIPA ASC; 2012. This work was the first to find empathy correlates based on N-gram language models.Google Scholar
  26. 26.
    Rabiner LR, Juang BH. An introduction to hidden Markov models. IEEE ASSP Mag. 1986;3(1):4–16.CrossRefGoogle Scholar
  27. 27.•
    Chakravarthula SN, Xiao B, Imel ZE, Atkins DC, Georgiou P. Assessing empathy using static and dynamic behavior models based on therapist’s language in addiction counseling. Dresden: Proc. Interspeech; 2015. This work improved text based empathy analysis through modeling the empathy states of the therapist in a dynamic fashion changing along time. Google Scholar
  28. 28.
    Pennebaker, J.W., Booth, R.J., Francis, M.E.: Linguistic Inquiry and Word Count (LIWC) (2007). URL
  29. 29.
    Malandrakis N, Potamianos A, Iosif E, Narayanan S. Distributional semantic models for affective text analysis. IEEE Trans Audio Speech Lang Process. 2013;21(11):2379–92.CrossRefGoogle Scholar
  30. 30.••
    Gibson J, Malandrakis N, Romero F, Atkins DC, Narayanan S. Predicting therapist empathy in motivational interviews using language features inspired by psycholinguistic norms. Dresden: Proc. Interspeech; 2015. This work extended lexical analysis of empathy to include linguistic and paralinguistic features. Google Scholar
  31. 31.•
    Lord SP, Sheng E, Imel ZE, Baer J, Atkins DC. More than reflections: empathy in motivational interviewing includes language style synchrony between therapist and client. Behav Ther. 2015;46(3):296–303. This work investigated the entrainment aspect of language style in order to model empathy from the spoken language. CrossRefPubMedGoogle Scholar
  32. 32.
    Aziz-Zadeh L, Sheng T, Gheytanchi A. Common premotor regions for the perception and production of prosody and correlations with empathy and prosodic ability. PLoS One. 2010;5(1):1–8.CrossRefGoogle Scholar
  33. 33.
    Weiste E, Peräkylä A. Prosody and empathic communication in psychotherapy interaction. Psychother Res. 2014;1–15.Google Scholar
  34. 34.••
    Xiao B, Bone D, Van Segbroeck M, Imel ZE, Atkins D, Georgiou P, et al. Modeling therapist empathy through prosody in drug addiction counseling. Proc. Interspeech; 2014. This work investigated prosodic correlates of empathy through quantizing the prosodic cues and finding the joint distribution of these cues.Google Scholar
  35. 35.
    Rabiner L, Schafer R. Theory and applications of digital speech processing. 1st ed. Prentice hall; 2010.Google Scholar
  36. 36.•
    Imel ZE, Barco JS, Brown HJ, Baucom BR, Baer JS, Kircher JC, et al. The association of therapist empathy and synchrony in vocally encoded arousal. J Couns Psychol. 2014;61(1):146. This work showed the correlation between empathy ratings and the synchrony of therapist-client in fundamental frequency of speech. CrossRefPubMedPubMedCentralGoogle Scholar
  37. 37.•
    Xiao B, Georgiou PG, Imel ZE, Atkins DC, Narayanan SS. Modeling therapist empathy and vocal entrainment in drug addiction counseling. Proc. Interspeech; 2013. This work showed computational vocal similarity measures that correlate with empathy ratings.Google Scholar
  38. 38.
    Bishop CM. Pattern recognition and machine learning. Springer; 2007.Google Scholar
  39. 39.
    Kullback S, Leibler RA. On information and sufficiency. Ann Math Stat. 1951;22(1):79–86.CrossRefGoogle Scholar
  40. 40.••
    Xiao B, Imel ZE, Atkins D, Georgiou P, Narayanan SS. Analyzing speech rate entrainment and its relation to therapist empathy in drug addiction counseling. Dresden: Proc. Interspeech; 2015. This work demonstrated entrainment of speech rate among interlocutors, and found that averaged absolute difference of speech rates between the therapist and client correlates to therapist empathy ratings. Google Scholar
  41. 41.
    Valstar, M., Girard, J., Almaev, T., McKeown, G., Mehu, M., Yin, L., et al.: FERA 2015-second facial expression recognition and analysis challenge. Proc. IEEE ICFG (2015)Google Scholar
  42. 42.
    Kumano, S., Otsuka, K., Mikami, D., Yamato, J.: Analyzing empathetic interactions based on the probabilistic modeling of the co-occurrence patterns of facial expressions in group meetings. In: Automatic Face and Gesture Recognition, pp. 43–50. IEEE (2011)Google Scholar
  43. 43.
    Murphy KP. Dynamic Bayesian networks: representation, inference and learning, Ph.D. thesis. Berkeley: University of California; 2002.Google Scholar
  44. 44.•
    Kumano S, Otsuka K, Matsuda M, Yamato J. Analyzing perceived empathy based on reaction time in behavioral mimicry. IEICE Trans Inf Syst. 2014;97(8):2008–20. This work investigated the function of reaction timing in relation to empathy perception, in addition to facial expression and gaze cues. CrossRefGoogle Scholar
  45. 45.•
    Kumano S, Otsuka K, Mikami D, Matsuda M, Yamato J. Analyzing interpersonal empathy via collective impressions. IEEE Trans Affect Comput. 2015;99. This work analyzed empathy based on an ensemble of evaluators and the distribution of perceived empathy levels.Google Scholar
  46. 46.
    Tapus, A., Mataric, M.J.: Emulating empathy in socially assistive robotics. In: AAAI Spring Symposium: Multidisciplinary Collaboration for Socially Assistive Robotics, pp. 93–96 (2007)Google Scholar
  47. 47.
    Riek LD, Robinson P. Real-time empathy: facial mimicry on a robot. In: Workshop on affective interaction in natural environments (AFFINE) at the international ACM conference on multimodal interfaces (ICMI 08). Citeseer: ACM; 2008.Google Scholar
  48. 48.
    Gonsior, B., Sosnowski, S., Mayer, C., Blume, J., Radig, B., Wollherr, D., et al.: Improving aspects of empathy and subjective performance for HRI through mirroring facial expressions. In: RO-MAN, pp. 350–356. IEEE (2011)Google Scholar
  49. 49.
    Becker C, Prendinger H, Ishizuka M, Wachsmuth I. Evaluating affective feedback of the 3d agent max in a competitive cards game. In: Affective computing and intelligent interaction. Springer; 2005. p. 466–73.Google Scholar
  50. 50.
    Boukricha, H., Wachsmuth, I.: Mechanism, modulation, and expression of empathy in a virtual human. In: IEEE Workshop on Affective Computational Intelligence (WACI), pp. 1–8 (2011)Google Scholar
  51. 51.•
    Boukricha H, Wachsmuth I, Carminati MN, Knoeferle P. A computational model of empathy: empirical evaluation. In: Proc. ACII. IEEE; 2013. p. 1–6. This work proposed a 3-dim emotion representation space and a three step empathy simulation approach, including empathy mechanism, modulation, and expression.Google Scholar
  52. 52.•
    Ochs M, Sadek D, Pelachaud C. A formal model of emotions for an empathic rational dialog agent. Auton Agent Multi-Agent Syst. 2012;24(3):410–40. This work proposed an approach of modeling empathy simulation through formal language and logic on emotion and intention of the user and the virtual agent. CrossRefGoogle Scholar
  53. 53.
    Rodrigues S, Mascarenhas S, Dias J, Paiva A. “I can feel it too!”: emergent empathic reactions between synthetic characters. Proc. ACII; 2009. p. 1–7.Google Scholar
  54. 54.••
    Rodrigues SH, Mascarenhas S, Dias JA, Paiva A. A process model of empathy for virtual agents. Interacting with Computers. 2014. This work improves previous work by composing emotion modeling, elicitation, modulation and expression.Google Scholar
  55. 55.
    McQuiggan SW, Lester JC. Learning empathy: a data-driven framework for modeling empathetic companion agents. In: Proceedings of the fifth international joint conference on autonomous agents and multiagent systems. New York: ACM; 2006. p. 961–8.CrossRefGoogle Scholar
  56. 56.
    McQuiggan SW, Lester JC. Modeling and evaluating empathy in embodied companion agents. Int J Hum Comput Stud. 2007;65(4):348–60.CrossRefGoogle Scholar
  57. 57.•
    Leite I, Pereira A, Castellano G, Mascarenhas S, Martinho C, Paiva A. Modelling empathy in social robotic companions. In: Ardissono L, Kuflik T, editors. Advances in user modeling, lecture notes in computer science, vol. 7138. Heidelberg: Springer Berlin; 2012. p. 135–47. This work simulated empathic behavior on the robot in a chess game by monitoring the user’s emotional and contextual states. Google Scholar
  58. 58.
    Auer P, Cesa-Bianchi N, Fischer P. Finite-time analysis of the multiarmed bandit problem. Mach Learn. 2002;47(2–3):235–56.CrossRefGoogle Scholar
  59. 59.•
    Leite I, Pereira A, Mascarenhas S, Martinho C, Prada R, Paiva A. The influence of empathy in human-robot relations. Int J Hum Comput Stud. 2013;71(3):250–60. This work compared human perception toward an empathy simulating robot and a neutral behaving one, where the former monitors user’s contextual states in a chess game so as to react in a way that may be perceived as empathic. CrossRefGoogle Scholar
  60. 60.•
    D’mello S, Graesser A. AutoTutor and affective autotutor: learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Trans Interact Intell Syst. 2013;2(4):23:1–23:39. This work introduced a virtual agent that monitors the user’s behavioral states and react with empathic expressions in a teaching environment. Google Scholar
  61. 61.
    Barrett-Lennard GT. The empathy cycle: refinement of a nuclear concept. J Couns Psychol. 1981;28(2):91.CrossRefGoogle Scholar
  62. 62.
    Baer JS, Wells EA, Rosengren DB, Hartzler B, Beadnell B, Dunn C. Agency context and tailored training in technology transfer: a pilot evaluation of motivational interviewing training for community counselors. J Subst Abus Treat. 2009;37(2):191.CrossRefGoogle Scholar
  63. 63.
    Lee CM, Kilmer JR, Neighbors C, Atkins DC, Zheng C, Walker DD, et al. Indicated prevention for college student marijuana use: a randomized controlled trial. J Consult Clin Psychol. 2013;81(4):702.CrossRefPubMedPubMedCentralGoogle Scholar
  64. 64.
    Lee CM, Neighbors C, Lewis MA, Kaysen D, Mittmann A, Geisner IM, et al. Randomized controlled trial of a spring break intervention to reduce high-risk drinking. J Consult Clin Psychol. 2014;82(2):189.CrossRefPubMedPubMedCentralGoogle Scholar
  65. 65.
    Neighbors C, Lee CM, Atkins DC, Lewis MA, Kaysen D, Mittmann A, et al. A randomized controlled trial of event-specific prevention strategies for reducing problematic drinking associated with 21st birthday celebrations. J Consult Clin Psychol. 2012;80(5):850.CrossRefPubMedPubMedCentralGoogle Scholar
  66. 66.
    Roy-Byrne P, Bumgardner K, Krupski A, Dunn C, Ries R, Donovan D, et al. Brief intervention for problem drug use in safety-net primary care settings: a randomized clinical trial. JAMA. 2014;312(5):492–501.CrossRefPubMedPubMedCentralGoogle Scholar
  67. 67.
    Tollison SJ, Lee CM, Neighbors C, Neil TA, Olson ND, Larimer ME. Questions and reflections: the use of motivational interviewing microskills in a peer-led brief alcohol intervention for college students. Behav Ther. 2008;39(2):183–94.CrossRefPubMedGoogle Scholar
  68. 68.
    Suykens JA, Vandewalle J. Least squares support vector machine classifiers. Neural Process Lett. 1999;9(3):293–300.CrossRefGoogle Scholar
  69. 69.
    Bone D, Goodwin MS, Black MP, Lee CC, Audhkhasi K, Narayanan S. Applying machine learning to facilitate autism diagnostics: pitfalls and promises. J Autism Dev Disord. 2015;45(5):1121–36.CrossRefPubMedGoogle Scholar
  70. 70.
    Can D, Gibson J, Vaz C, Georgiou P, Narayanan S. Barista: a framework for concurrent speech processing by USC-SAIL. Florence, Italy: Proc. ICASSP; 2014.Google Scholar
  71. 71.
    Van Segbroeck M, Tsiartas A, Narayanan SS. A robust frontend for VAD: exploiting contextual, discriminative and spectral cues of human voice. Lyon, France: Proc. InterSpeech; 2013.Google Scholar
  72. 72.
    Huang CW, Xiao B, Georgiou P, Narayanan S. Unsupervised speaker diarization using Riemannian manifold clustering. Proc. Interspeech; 2014.Google Scholar
  73. 73.
    Miro XA, Bozonnet S, Evans N, Fredouille C, Friedland G, Vinyals O. Speaker diarization: a review of recent research. IEEE Trans Audio Speech Lang Process. 2012;20(2):356–70.CrossRefGoogle Scholar
  74. 74.
    Hinton G, Deng L, Yu D, Dahl GE, Mohamed AR, Jaitly N, et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag. 2012;29(6):82–97.CrossRefGoogle Scholar
  75. 75.
    Povey D, Ghoshal A, Boulianne G, Burget L, Glembek O, Goel N, et al. The Kaldi speech recognition toolkit. Proc. ASRU; 2011.Google Scholar
  76. 76.
    Cole R, Halpern A, Ramig L, Van Vuuren S, Ngampatipatpong N, Yan J. A virtual speech therapist for individuals with Parkinson’s disease. Educ Technol. 2007;47(1):51.Google Scholar
  77. 77.
    Kleinsmith A, Rivera-Gutierrez D, Finney G, Cendan J, Lok B. Understanding empathy training with virtual patients. Comput Hum Behav. 2015;52:151–8.CrossRefGoogle Scholar
  78. 78.
    Tettegah S, Taylor K, Whang EW, Meistninkas S, Chamot R. Can virtual reality simulations be used as a research tool to study empathy, problems solving and perspective taking of educators?: theory, method and application. In: ACM SIGGRAPH 2006 educators program. New York: ACM; 2006.Google Scholar
  79. 79.
    van Vuuren S, Cherney LR. A virtual therapist for speech and language therapy. In: Intelligent virtual agents. Springer; 2014. p. 438–48.Google Scholar
  80. 80.••
    Narayanan S, Georgiou P. Behavioral signal processing: deriving human behavioral informatics from speech and language. Proc IEEE. 2013;101(5):1203–33. This paper is a survey of the Behavioral Signal Processing work that includes introduction of the concepts, example studies, and future research directions. CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Bo Xiao
    • 1
  • Zac E. Imel
    • 2
  • Panayiotis Georgiou
    • 1
  • David C. Atkins
    • 3
  • Shrikanth S. Narayanan
    • 1
  1. 1.SAIL, Department of Electrical EngineeringUniversity of Southern CaliforniaLos AngelesUSA
  2. 2.Department of Educational PsychologyUniversity of UtahSalt Lake CityUSA
  3. 3.Department of Psychiatry and Behavioral SciencesUniversity of WashingtonSeattleUSA

Personalised recommendations