Skip to main content
SpringerLink
Log in

Computational Analysis and Simulation of Empathic Behaviors: a Survey of Empathy Modeling with Behavioral Signal Processing Framework

  • Psychiatry in the Digital Age (JS Luo, Section Editor)
  • Published:
Current Psychiatry Reports Aims and scope Submit manuscript

Abstract

Empathy is an important psychological process that facilitates human communication and interaction. Enhancement of empathy has profound significance in a range of applications. In this paper, we review emerging directions of research on computational analysis of empathy expression and perception as well as empathic interactions, including their simulation. We summarize the work on empathic expression analysis by the targeted signal modalities (e.g., text, audio, and facial expressions). We categorize empathy simulation studies into theory-based emotion space modeling or application-driven user and context modeling. We summarize challenges in computational study of empathy including conceptual framing and understanding of empathy, data availability, appropriate use and validation of machine learning techniques, and behavior signal processing. Finally, we propose a unified view of empathy computation and offer a series of open problems for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

Papers of particular interest, published recently, have been highlighted as: • Of importance •• Of major importance

  1. Titchener EB. Lectures on the experimental psychology of the thought-processes. Macmillan; 1909.

  2. Hoffman ML. Empathy and moral development: implications for caring and justice. Cambridge University Press; 2001.

  3. Batson CD. In: Decety J, editor. These things called empathy: eight related but distinct phenomena. CRP press; 2009. p. 16–28.

  4. Cuff BM, Brown SJ, Taylor L, Howat DJ. Empathy: a review of the concept. Emot Rev. 2014.

  5. Decety J, Jackson P. The functional architecture of human empathy. Behav Cogn Neurosci Rev. 2004;3(2):71–100.

    Article  PubMed  Google Scholar 

  6. Elliott R, Bohart AC, Watson JC, Greenberg LS. Empathy. Psychotherapy. 2011;48(1):43–9.

    Article  PubMed  Google Scholar 

  7. Preston SD, De Waal F. Empathy: its ultimate and proximate bases. Behav Brain Sci. 2002;25(01):1–20.

    PubMed  Google Scholar 

  8. De Vignemont F, Singer T. The empathic brain: how, when and why? Trends Cogn Sci. 2006;10(10):435–41.

    Article  PubMed  Google Scholar 

  9. Iacoboni M. Imitation, empathy, and mirror neurons. Annu Rev Psychol. 2009;60:653–70.

    Article  PubMed  Google Scholar 

  10. Derksen F, Bensing J, Lagro-Janssen A. Effectiveness of empathy in general practice: a systematic review. Br J Gen Pract. 2013;63(606):e76–84.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Lelorain S, Brédart A, Dolbeault S, Sultan S. A systematic review of the associations between empathy measures and patient outcomes in cancer care. Psychooncology. 2012;21(12):1255–64.

    Article  PubMed  Google Scholar 

  12. Moyers TB, Miller WR. Is low therapist empathy toxic? Psychol Addict Behav. 2013;27(3):878.

    Article  PubMed  PubMed Central  Google Scholar 

  13. van Berkhout ET, Malouff JM. The efficacy of empathy training: a meta-analysis of randomized controlled trials. J Couns Psychol. 2015.

  14. Atkins DC, Steyvers M, Imel ZE, Smyth P. Scaling up the evaluation of psychotherapy: evaluating motivational interviewing fidelity via statistical text classification. Implement Sci. 2014;9(1):49. This work used topic models to analyze provider fidelity based on text of the psychotherapy interaction, illuminating the possibility of scaling up the evaluation through computational means.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Riess H. Biomarkers in the psychotherapeutic relationship: the role of physiology, neurobiology, and biological correlates of E.M.P.A.T.H.Y. Harv Rev Psychiatry. 2011;19(3):162–74.

    Article  PubMed  Google Scholar 

  16. Regenbogen C, Schneider DA, Finkelmeyer A, Kohn N, Derntl B, Kellermann T, et al. The differential contribution of facial expressions, prosody, and speech content to empathy. Cognition & emotion. 2012;26(6):995–1014.

    Article  Google Scholar 

  17. Duda RO, Hart PE, Stork DG. Pattern classification. Wiley; 2012.

  18. Miller WR, Rollnick S. Motivational interviewing: helping people change. Guilford Press; 2012.

  19. Miller WR, Rose GS. Toward a theory of motivational interviewing. Am Psychol. 2009;64(6):527.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Moyers, T., Martin, T., Manuel, J., Miller, W., Ernst, D.: Revised global scales: Motivational Interviewing Treatment Integrity 3.0 (2007)

  21. Kumano, S., Otsuka, K., Matsuda, M., Yamato, J.: Analyzing perceived empathy/antipathy based on reaction time in behavioral coordination. In: Automatic Face and Gesture Recognition, pp. 1–8. IEEE (2013)

  22. Delaherche E, Chetouani M, Mahdhaoui A, Saint-Georges C, Viaux S, Cohen D. Interpersonal synchrony: a survey of evaluation methods across disciplines. IEEE Trans Affect Comput. 2012;3(3):349–65.

    Article  Google Scholar 

  23. Coulehan JL, Platt FW, Egener B, Frankel R, Lin CT, Lown B, et al. “Let me see if I have this right”: words that help build empathy. Ann Intern Med. 2001;135(3):221–7.

    Article  CAS  PubMed  Google Scholar 

  24. Kneser R, Ney H. Improved backing-off for m-gram language modeling. In: International conference on acoustics, speech, and signal processing, vol. 1. IEEE; 1995. p. 181–4.

  25. Xiao B, Can D, Georgiou PG, Atkins DC, Narayanan SS. Analyzing the language of therapist empathy in motivational interview based psychotherapy. Proc. APSIPA ASC; 2012. This work was the first to find empathy correlates based on N-gram language models.

  26. Rabiner LR, Juang BH. An introduction to hidden Markov models. IEEE ASSP Mag. 1986;3(1):4–16.

    Article  Google Scholar 

  27. Chakravarthula SN, Xiao B, Imel ZE, Atkins DC, Georgiou P. Assessing empathy using static and dynamic behavior models based on therapist’s language in addiction counseling. Dresden: Proc. Interspeech; 2015. This work improved text based empathy analysis through modeling the empathy states of the therapist in a dynamic fashion changing along time.

    Google Scholar 

  28. Pennebaker, J.W., Booth, R.J., Francis, M.E.: Linguistic Inquiry and Word Count (LIWC) (2007). URL http://www.liwc.net/

  29. Malandrakis N, Potamianos A, Iosif E, Narayanan S. Distributional semantic models for affective text analysis. IEEE Trans Audio Speech Lang Process. 2013;21(11):2379–92.

    Article  Google Scholar 

  30. Gibson J, Malandrakis N, Romero F, Atkins DC, Narayanan S. Predicting therapist empathy in motivational interviews using language features inspired by psycholinguistic norms. Dresden: Proc. Interspeech; 2015. This work extended lexical analysis of empathy to include linguistic and paralinguistic features.

    Google Scholar 

  31. Lord SP, Sheng E, Imel ZE, Baer J, Atkins DC. More than reflections: empathy in motivational interviewing includes language style synchrony between therapist and client. Behav Ther. 2015;46(3):296–303. This work investigated the entrainment aspect of language style in order to model empathy from the spoken language.

    Article  PubMed  Google Scholar 

  32. Aziz-Zadeh L, Sheng T, Gheytanchi A. Common premotor regions for the perception and production of prosody and correlations with empathy and prosodic ability. PLoS One. 2010;5(1):1–8.

    Article  Google Scholar 

  33. Weiste E, Peräkylä A. Prosody and empathic communication in psychotherapy interaction. Psychother Res. 2014;1–15.

  34. Xiao B, Bone D, Van Segbroeck M, Imel ZE, Atkins D, Georgiou P, et al. Modeling therapist empathy through prosody in drug addiction counseling. Proc. Interspeech; 2014. This work investigated prosodic correlates of empathy through quantizing the prosodic cues and finding the joint distribution of these cues.

  35. Rabiner L, Schafer R. Theory and applications of digital speech processing. 1st ed. Prentice hall; 2010.

  36. Imel ZE, Barco JS, Brown HJ, Baucom BR, Baer JS, Kircher JC, et al. The association of therapist empathy and synchrony in vocally encoded arousal. J Couns Psychol. 2014;61(1):146. This work showed the correlation between empathy ratings and the synchrony of therapist-client in fundamental frequency of speech.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Xiao B, Georgiou PG, Imel ZE, Atkins DC, Narayanan SS. Modeling therapist empathy and vocal entrainment in drug addiction counseling. Proc. Interspeech; 2013. This work showed computational vocal similarity measures that correlate with empathy ratings.

  38. Bishop CM. Pattern recognition and machine learning. Springer; 2007.

  39. Kullback S, Leibler RA. On information and sufficiency. Ann Math Stat. 1951;22(1):79–86.

    Article  Google Scholar 

  40. Xiao B, Imel ZE, Atkins D, Georgiou P, Narayanan SS. Analyzing speech rate entrainment and its relation to therapist empathy in drug addiction counseling. Dresden: Proc. Interspeech; 2015. This work demonstrated entrainment of speech rate among interlocutors, and found that averaged absolute difference of speech rates between the therapist and client correlates to therapist empathy ratings.

    Google Scholar 

  41. Valstar, M., Girard, J., Almaev, T., McKeown, G., Mehu, M., Yin, L., et al.: FERA 2015-second facial expression recognition and analysis challenge. Proc. IEEE ICFG (2015)

  42. Kumano, S., Otsuka, K., Mikami, D., Yamato, J.: Analyzing empathetic interactions based on the probabilistic modeling of the co-occurrence patterns of facial expressions in group meetings. In: Automatic Face and Gesture Recognition, pp. 43–50. IEEE (2011)

  43. Murphy KP. Dynamic Bayesian networks: representation, inference and learning, Ph.D. thesis. Berkeley: University of California; 2002.

    Google Scholar 

  44. Kumano S, Otsuka K, Matsuda M, Yamato J. Analyzing perceived empathy based on reaction time in behavioral mimicry. IEICE Trans Inf Syst. 2014;97(8):2008–20. This work investigated the function of reaction timing in relation to empathy perception, in addition to facial expression and gaze cues.

    Article  Google Scholar 

  45. Kumano S, Otsuka K, Mikami D, Matsuda M, Yamato J. Analyzing interpersonal empathy via collective impressions. IEEE Trans Affect Comput. 2015;99. This work analyzed empathy based on an ensemble of evaluators and the distribution of perceived empathy levels.

  46. Tapus, A., Mataric, M.J.: Emulating empathy in socially assistive robotics. In: AAAI Spring Symposium: Multidisciplinary Collaboration for Socially Assistive Robotics, pp. 93–96 (2007)

  47. Riek LD, Robinson P. Real-time empathy: facial mimicry on a robot. In: Workshop on affective interaction in natural environments (AFFINE) at the international ACM conference on multimodal interfaces (ICMI 08). Citeseer: ACM; 2008.

    Google Scholar 

  48. Gonsior, B., Sosnowski, S., Mayer, C., Blume, J., Radig, B., Wollherr, D., et al.: Improving aspects of empathy and subjective performance for HRI through mirroring facial expressions. In: RO-MAN, pp. 350–356. IEEE (2011)

  49. Becker C, Prendinger H, Ishizuka M, Wachsmuth I. Evaluating affective feedback of the 3d agent max in a competitive cards game. In: Affective computing and intelligent interaction. Springer; 2005. p. 466–73.

  50. Boukricha, H., Wachsmuth, I.: Mechanism, modulation, and expression of empathy in a virtual human. In: IEEE Workshop on Affective Computational Intelligence (WACI), pp. 1–8 (2011)

  51. Boukricha H, Wachsmuth I, Carminati MN, Knoeferle P. A computational model of empathy: empirical evaluation. In: Proc. ACII. IEEE; 2013. p. 1–6. This work proposed a 3-dim emotion representation space and a three step empathy simulation approach, including empathy mechanism, modulation, and expression.

  52. Ochs M, Sadek D, Pelachaud C. A formal model of emotions for an empathic rational dialog agent. Auton Agent Multi-Agent Syst. 2012;24(3):410–40. This work proposed an approach of modeling empathy simulation through formal language and logic on emotion and intention of the user and the virtual agent.

    Article  Google Scholar 

  53. Rodrigues S, Mascarenhas S, Dias J, Paiva A. “I can feel it too!”: emergent empathic reactions between synthetic characters. Proc. ACII; 2009. p. 1–7.

  54. Rodrigues SH, Mascarenhas S, Dias JA, Paiva A. A process model of empathy for virtual agents. Interacting with Computers. 2014. This work improves previous work by composing emotion modeling, elicitation, modulation and expression.

  55. McQuiggan SW, Lester JC. Learning empathy: a data-driven framework for modeling empathetic companion agents. In: Proceedings of the fifth international joint conference on autonomous agents and multiagent systems. New York: ACM; 2006. p. 961–8.

    Chapter  Google Scholar 

  56. McQuiggan SW, Lester JC. Modeling and evaluating empathy in embodied companion agents. Int J Hum Comput Stud. 2007;65(4):348–60.

    Article  Google Scholar 

  57. Leite I, Pereira A, Castellano G, Mascarenhas S, Martinho C, Paiva A. Modelling empathy in social robotic companions. In: Ardissono L, Kuflik T, editors. Advances in user modeling, lecture notes in computer science, vol. 7138. Heidelberg: Springer Berlin; 2012. p. 135–47. This work simulated empathic behavior on the robot in a chess game by monitoring the user’s emotional and contextual states.

    Google Scholar 

  58. Auer P, Cesa-Bianchi N, Fischer P. Finite-time analysis of the multiarmed bandit problem. Mach Learn. 2002;47(2–3):235–56.

    Article  Google Scholar 

  59. Leite I, Pereira A, Mascarenhas S, Martinho C, Prada R, Paiva A. The influence of empathy in human-robot relations. Int J Hum Comput Stud. 2013;71(3):250–60. This work compared human perception toward an empathy simulating robot and a neutral behaving one, where the former monitors user’s contextual states in a chess game so as to react in a way that may be perceived as empathic.

    Article  Google Scholar 

  60. D’mello S, Graesser A. AutoTutor and affective autotutor: learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Trans Interact Intell Syst. 2013;2(4):23:1–23:39. This work introduced a virtual agent that monitors the user’s behavioral states and react with empathic expressions in a teaching environment.

    Google Scholar 

  61. Barrett-Lennard GT. The empathy cycle: refinement of a nuclear concept. J Couns Psychol. 1981;28(2):91.

    Article  Google Scholar 

  62. Baer JS, Wells EA, Rosengren DB, Hartzler B, Beadnell B, Dunn C. Agency context and tailored training in technology transfer: a pilot evaluation of motivational interviewing training for community counselors. J Subst Abus Treat. 2009;37(2):191.

    Article  Google Scholar 

  63. Lee CM, Kilmer JR, Neighbors C, Atkins DC, Zheng C, Walker DD, et al. Indicated prevention for college student marijuana use: a randomized controlled trial. J Consult Clin Psychol. 2013;81(4):702.

    Article  PubMed  PubMed Central  Google Scholar 

  64. Lee CM, Neighbors C, Lewis MA, Kaysen D, Mittmann A, Geisner IM, et al. Randomized controlled trial of a spring break intervention to reduce high-risk drinking. J Consult Clin Psychol. 2014;82(2):189.

    Article  PubMed  PubMed Central  Google Scholar 

  65. Neighbors C, Lee CM, Atkins DC, Lewis MA, Kaysen D, Mittmann A, et al. A randomized controlled trial of event-specific prevention strategies for reducing problematic drinking associated with 21st birthday celebrations. J Consult Clin Psychol. 2012;80(5):850.

    Article  PubMed  PubMed Central  Google Scholar 

  66. Roy-Byrne P, Bumgardner K, Krupski A, Dunn C, Ries R, Donovan D, et al. Brief intervention for problem drug use in safety-net primary care settings: a randomized clinical trial. JAMA. 2014;312(5):492–501.

    Article  PubMed  PubMed Central  Google Scholar 

  67. Tollison SJ, Lee CM, Neighbors C, Neil TA, Olson ND, Larimer ME. Questions and reflections: the use of motivational interviewing microskills in a peer-led brief alcohol intervention for college students. Behav Ther. 2008;39(2):183–94.

    Article  PubMed  Google Scholar 

  68. Suykens JA, Vandewalle J. Least squares support vector machine classifiers. Neural Process Lett. 1999;9(3):293–300.

    Article  Google Scholar 

  69. Bone D, Goodwin MS, Black MP, Lee CC, Audhkhasi K, Narayanan S. Applying machine learning to facilitate autism diagnostics: pitfalls and promises. J Autism Dev Disord. 2015;45(5):1121–36.

    Article  PubMed  Google Scholar 

  70. Can D, Gibson J, Vaz C, Georgiou P, Narayanan S. Barista: a framework for concurrent speech processing by USC-SAIL. Florence, Italy: Proc. ICASSP; 2014.

    Google Scholar 

  71. Van Segbroeck M, Tsiartas A, Narayanan SS. A robust frontend for VAD: exploiting contextual, discriminative and spectral cues of human voice. Lyon, France: Proc. InterSpeech; 2013.

    Google Scholar 

  72. Huang CW, Xiao B, Georgiou P, Narayanan S. Unsupervised speaker diarization using Riemannian manifold clustering. Proc. Interspeech; 2014.

  73. Miro XA, Bozonnet S, Evans N, Fredouille C, Friedland G, Vinyals O. Speaker diarization: a review of recent research. IEEE Trans Audio Speech Lang Process. 2012;20(2):356–70.

    Article  Google Scholar 

  74. Hinton G, Deng L, Yu D, Dahl GE, Mohamed AR, Jaitly N, et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag. 2012;29(6):82–97.

    Article  Google Scholar 

  75. Povey D, Ghoshal A, Boulianne G, Burget L, Glembek O, Goel N, et al. The Kaldi speech recognition toolkit. Proc. ASRU; 2011.

  76. Cole R, Halpern A, Ramig L, Van Vuuren S, Ngampatipatpong N, Yan J. A virtual speech therapist for individuals with Parkinson’s disease. Educ Technol. 2007;47(1):51.

    Google Scholar 

  77. Kleinsmith A, Rivera-Gutierrez D, Finney G, Cendan J, Lok B. Understanding empathy training with virtual patients. Comput Hum Behav. 2015;52:151–8.

    Article  Google Scholar 

  78. Tettegah S, Taylor K, Whang EW, Meistninkas S, Chamot R. Can virtual reality simulations be used as a research tool to study empathy, problems solving and perspective taking of educators?: theory, method and application. In: ACM SIGGRAPH 2006 educators program. New York: ACM; 2006.

    Google Scholar 

  79. van Vuuren S, Cherney LR. A virtual therapist for speech and language therapy. In: Intelligent virtual agents. Springer; 2014. p. 438–48.

  80. Narayanan S, Georgiou P. Behavioral signal processing: deriving human behavioral informatics from speech and language. Proc IEEE. 2013;101(5):1203–33. This paper is a survey of the Behavioral Signal Processing work that includes introduction of the concepts, example studies, and future research directions.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. SAIL, Department of Electrical Engineering, University of Southern California, Los Angeles, CA, USA

    Bo Xiao, Panayiotis Georgiou & Shrikanth S. Narayanan

  2. Department of Educational Psychology, University of Utah, Salt Lake City, UT, USA

    Zac E. Imel

  3. Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA

    David C. Atkins

Authors
  1. Bo Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zac E. Imel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Panayiotis Georgiou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David C. Atkins
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shrikanth S. Narayanan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shrikanth S. Narayanan.

Ethics declarations

Conflict of Interest

Bo Xiao, Zac E. Imel, Panayiotis Georgiou, and Shrikanth S. Narayanan declare that they have no conflict of interest.

David C. Atkins reports grants from NIAAA and NIDA.

Human and Animal Rights and Informed Consent

This article does not contain any studies with human or animal subjects performed by any of the authors.

Additional information

This article is part of the Topical Collection on Psychiatry in the Digital Age

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiao, B., Imel, Z.E., Georgiou, P. et al. Computational Analysis and Simulation of Empathic Behaviors: a Survey of Empathy Modeling with Behavioral Signal Processing Framework. Curr Psychiatry Rep 18, 49 (2016). https://doi.org/10.1007/s11920-016-0682-5

Download citation

  • Published:

  • DOI: https://doi.org/10.1007/s11920-016-0682-5

Keywords

Search

Navigation