International Journal of Social Robotics

, Volume 7, Issue 1, pp 35–49 | Cite as

A Recipe for Empathy

Integrating the Mirror System, Insula, Somatosensory Cortex and Motherese
  • Angelica LimEmail author
  • Hiroshi G. Okuno


Could a robot feel authentic empathy? What exactly is empathy, and why do most humans have it? We present a model which suggests that empathy is an emergent behavior with four main elements: a mirror neuron system, somatosensory cortices, an insula, and infant-directed “baby talk” or motherese. To test our hypothesis, we implemented a robot called MEI (multimodal emotional intelligence) with these functions, and allowed it to interact with human caregivers using comfort and approval motherese, the first kinds of vocalizations heard by infants at 3 and 6 months of age. The robot synchronized in real-time to the humans through voice and movement dynamics, while training statistical models associated with its low level gut feeling (“flourishing” or “distress”, based on battery or temperature). Experiments show that the post-interaction robot associates novel happy voices with physical flourishing 90 % of the time, sad voices with distress 84 % of the time. Our results also show that a robot trained with infant-directed “attention bids” can recognize adult fear voices. Importantly, this is the first emotion system to recognize adult emotional voices after training only with motherese, suggesting that this specific parental behavior may help build emotional intelligence.


Robot empathy Emotional contagion based on SIRE model MEI robot Developmental robotics 


  1. 1.
    de Waal FB (2007) Putting the altruism back into altruism: the evolution of empathy. Ann Rev Psychol 59:1–22Google Scholar
  2. 2.
    Shamay-Tsoory SG, Aharon-Peretz J, Perry D (2009) Two systems for empathy: a double dissociation between emotional and cognitive empathy in inferior frontal gyrus versus ventromedial prefrontal lesions. Brain 132(3):617–627CrossRefGoogle Scholar
  3. 3.
    Baron-Cohen S, Wheelwright S (2004) The empathy quotient: an investigation of adults with Asperger syndrome or high functioning autism, and normal sex differences. J Autism Develop Disord 34(2):163–175CrossRefGoogle Scholar
  4. 4.
    Juslin PN, Sloboda JA (2010) Handbook of music and emotion: theory, research, applications. Oxford University Press, Oxford, UKGoogle Scholar
  5. 5.
    Clore GL, Palmer J (2009) Affective guidance of intelligent agents: how emotion controls cognition. Cognit Syst Res 10(1):21–30CrossRefGoogle Scholar
  6. 6.
    Gonsior B, Sosnowski S, Buß M, Wollherr D, Kuhnlenz K (2012) An emotional adaption approach to increase helpfulness towards a robot. In: 2012 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 2429–2436. IEEEGoogle Scholar
  7. 7.
    Cramer H, Goddijn J, Wielinga B, Evers V (2010) Effects of (in) accurate empathy and situational valence on attitudes towards robots. In: Proceedings of the 5th ACM/IEEE international conference on human–robot interaction, pp 141–142. IEEE, Piscataway, NJ, USAGoogle Scholar
  8. 8.
    Leite I, Pereira A, Castellano G, Mascarenhas S, Martinho C, Paiva A (2012) Modelling empathy in social robotic companions. In: Ardissono L, Kuflik T (eds) Advances in user modeling. Springer, Heidelberg, pp 135–147Google Scholar
  9. 9.
    Pereira A, Leite I, Mascarenhas S, Martinho C, Paiva A (2011) Using empathy to improve human–robot relationships. In: Human–robot personal relationships. Springer, Berlin, pp 130–138Google Scholar
  10. 10.
    Riek LD, Rabinowitch TC, Chakrabarti B, Robinson P (2009) Empathizing with robots: fellow feeling along the anthropomorphic spectrum. IEEE affective computing and intelligent interaction and workshops, pp 1–6Google Scholar
  11. 11.
    Kwak SS, Kim Y, Kim E, Shin C, Cho K (2013) What makes people empathize with an emotional robot? The impact of agency and physical embodiment on human empathy for a robot. In: RO-MAN, 2013 IEEE, pp 180–185. IEEEGoogle Scholar
  12. 12.
    Turkle S (2012) Alone together: why we expect more from technology and less from each other. Basic Books, New YorkGoogle Scholar
  13. 13.
    Vesquez JD (1997) Modeling emotions and other motivations in synthetic agents. In: Proceedings of AAAI-97. AAAI, Providence, RI, USA, pp 1015Google Scholar
  14. 14.
    Yamada T, Hashimoto H, Tosa N (1995) Pattern recognition of emotion with neural network. In Proceedings of IECON, pp 183–187Google Scholar
  15. 15.
    Breazeal CL (2004) Designing sociable robots with CDROM. MIT press, Cambridge, MAGoogle Scholar
  16. 16.
    Mehrabian A (1996) Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in temperament. Curr Psychol Develop Learn Person Soc 14:261–292CrossRefMathSciNetGoogle Scholar
  17. 17.
    Nagai Y, Rohlfing J (2009) Computational analysis of motionese toward scaffolding robot action learning. IEEE Trans Auton Mental Develop 1(1):44–54CrossRefGoogle Scholar
  18. 18.
    Ishihara H, Yoshikawa Y, Asada M (2011) Realistic child robot Affetto for understanding the caregiver–child attachment relationship that guides the child development. Develop Learn (ICDL) 2:1–5Google Scholar
  19. 19.
    Asada M, Hosoda K, Kuniyoshi Y, Ishiguro H, Inui T, Yoshikawa Y, Ogino M, Yoshida C (2009) Cognitive developmental robotics: a Survey. IEEE Trans Auton Mental Dev 11:12–34Google Scholar
  20. 20.
    Kozima H, Nakagawa C, Yano H (2004) Can a robot empathize with people? Artific Life Robot 8(1):83–88CrossRefGoogle Scholar
  21. 21.
    Shic F, Scassellati B (2006) How not to evaluate a developmental system. International joint conference on neural networks (IJCNN), Vancouver, 2006, pp 5218–5225Google Scholar
  22. 22.
    Hatfield E, Cacioppo JT (1994) Emotional contagion. Cambridge University Press, Cambridge, UKGoogle Scholar
  23. 23.
    Baron-Cohen S (2011) Zero degrees of empathy: a new theory of human cruelty. Penguin, LondonGoogle Scholar
  24. 24.
    Edele A, Dziobek I, Keller M (2013) Explaining altruistic sharing in the dictator game: the role of affective empathy, cognitive empathy, and justice sensitivity. Learn Individ Differ 24:96–102CrossRefGoogle Scholar
  25. 25.
    Preston SD, de Waal F (2002) Empathy: its ultimate and proximate bases. Behav Brain Sci 25(01):1–20Google Scholar
  26. 26.
    Neumann R, Strack F (2000) Mood contagion: the automatic transfer of mood between persons. J Personal Soc Psychol 79(2):211–223Google Scholar
  27. 27.
    Juslin PN, Simon L, Daniel V, Lars-Olov L (2010) How does music evoke emotions? Exploring the underlying mechanisms. In: Juslin PN, Sloboda JA (eds) Handbook of music and emotion: theory, research, applications. Oxford University Press, Oxford, UKGoogle Scholar
  28. 28.
    Iacoboni M (2009) Imitation, empathy, and mirror neurons. Ann Rev Psychol 60(1):653–670CrossRefGoogle Scholar
  29. 29.
    Iacoboni M, Dapretto M (2006) The mirror neuron system and the consequences of its dysfunction. Nat Rev Neurosci 7(12):942–951CrossRefGoogle Scholar
  30. 30.
    Nummenmaa L, Hirvonen J, Parkkola R, Hietanen JK (2008) Is emotional contagion special? An fMRI study on neural systems for affective and cognitive empathy. NeuroImage 43(3):571–580CrossRefGoogle Scholar
  31. 31.
    di Pellegrino G, Fadiga L, Fogassi L, Gallese V, Rizzolatti G (1992) Understanding motor events: a neurophysiological study. Exper Brain Res 91(1):176–180CrossRefGoogle Scholar
  32. 32.
    Carr L, Iacoboni M, Dubeau MC, Mazziotta JC, Lenzi GL (2003) Neural mechanisms of empathy in humans: a relay from neural systems for imitation to limbic areas. Proc Nat Acad Sci 100(9):5497–5502Google Scholar
  33. 33.
    Rizzolatti G, Craighero L (2004) The mirror neuron system. Ann Rev Neurosci 27(1):169–192CrossRefGoogle Scholar
  34. 34.
    Kohler E (2002) Hearing sounds, understanding actions: action representation in mirror neurons. Science 297(5582):846–848CrossRefGoogle Scholar
  35. 35.
    Koelsch S, Fritz T, von Cramon DY, Müller K, Friederici AD (2006) Investigating emotion with music: an fMRI study. Hum Brain Mapp 27(3):239–250CrossRefGoogle Scholar
  36. 36.
    Damasio AR (2004) Looking for Spinoza: joy, sorrow and the feeling brain. Random House, New YorkGoogle Scholar
  37. 37.
    Dinstein I (2008) Human cortex: reflections of mirror neurons. Curr Biol 18(20):R956–R959CrossRefGoogle Scholar
  38. 38.
    Butti C, Hof PR (2010) The insular cortex: a comparative perspective. Brain Struct Funct 214(5–6):477–493CrossRefGoogle Scholar
  39. 39.
    Craig AD (2009) How do you feel—now? The anterior insula and human awareness. Nat Rev Neurosci 10:59–70CrossRefGoogle Scholar
  40. 40.
    Mutschler I, Schulze-Bonhage A, Glauche V, Demandt E, Speck O, Ball T (2007) A rapid sound–action association effect in human insular cortex. PLoS ONE 2(2):e259CrossRefGoogle Scholar
  41. 41.
    Adolphs R, Tranel D, Damasio AR (2003) Dissociable neural systems for recognizing emotions. Brain Cognit 52(1):61–69CrossRefGoogle Scholar
  42. 42.
    Calder AJ, Keane J, Manes F, Antoun N, Young AW (2000) Impaired recognition and experience of disgust following brain injury. Nat Neurosci 3(11):1077–1078CrossRefGoogle Scholar
  43. 43.
    Morrison I, Bjornsdotter M, Olausson H (2011) Vicarious responses to social touch in posterior insular cortex are tuned to pleasant caressing speeds. J Neurosci 31(26):9554–9562CrossRefGoogle Scholar
  44. 44.
    Kim P, Feldman R, Mayes LC, Eicher V, Thompson N, Leckman JF, Swain JE (2011) Breastfeeding, brain activation to own infant cry, and maternal sensitivity. J Child Psychol Psychiatry 52(8):907–915CrossRefGoogle Scholar
  45. 45.
    Pohl A, Anders S, Schulte-Rüther M, Mathiak K, Kircher T (2013) Positive facial affect—an fMRI study on the involvement of insula and amygdala. PLoS ONE 8(8):e69886Google Scholar
  46. 46.
    Singer T (2004) Empathy for pain involves the affective but not sensory components of pain. Science 303(5661):1157–1162CrossRefGoogle Scholar
  47. 47.
    Bernhardt BC, Singer T (2012) The neural basis of empathy. Ann Rev Neurosci 35:1–23CrossRefGoogle Scholar
  48. 48.
    Adolphs R, Damasio H, Tranel D, Cooper G, Damasio AR (2000) A role for somatosensory cortices in the visual recognition of emotion as revealed by three-dimensional lesion mapping. J Neurosci 20(7):2683–2690Google Scholar
  49. 49.
    Damasio A (1994) Descartes’ error: emotion, reason and the human mind. Grossett/Putnam, New YorkGoogle Scholar
  50. 50.
    Davis M, Whalen PJ (2001) The amygdala: vigilance and emotion. Mol Psychiatry 6(1):13–34CrossRefGoogle Scholar
  51. 51.
    Hein G, Singer T (2008) I feel how you feel but not always: the empathic brain and its modulation. Curr Opin Neurobiol 18(2):153–158CrossRefGoogle Scholar
  52. 52.
    Fellous JM, Ledoux JE (2005) Toward basic principles for emotional processing: what the fearful brain tells the robot. In: Fellous JM, Arbib MA (eds) Who needs emotions? The brain meets the robot. Oxford University Press, Oxford, UKCrossRefGoogle Scholar
  53. 53.
    Walker-Andrews AS (1997) Infants’ perception of expressive behaviors: differentiation of multimodal information. Psychol Bull 121(3):437CrossRefGoogle Scholar
  54. 54.
    Grossmann T, Striano T, Friederici AD (2006) Crossmodal integration of emotional information from face and voice in the infant brain. Develop Sci 9(3):309–315CrossRefGoogle Scholar
  55. 55.
    Boucenna S, Gaussier P, Andry P, Hafemeister L (2010) Imitation as a communication tool for online facial expression learning and recognition. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), 2010, pp 5323–5328Google Scholar
  56. 56.
    Watanabe A, Ogino M, Asada M (2007) Mapping facial expression to internal states based on intuitive parenting. J Robot Mechatron 19(3):315Google Scholar
  57. 57.
    Saint-Georges C, Chetouani M, Cassel R, Apicella F, Mahdhaoui A, Muratori F, Laznik MC, Cohen D (2013) Motherese in interaction at the cross-road of emotion and cognition? (a systematic review). PloS ONE 8(10):e78–e103CrossRefGoogle Scholar
  58. 58.
    Soken NH, Pick AD (1992) Intermodal perception of happy and angry expressive behaviors by seven-month-old infants. Child Develop 63(4):787–795CrossRefGoogle Scholar
  59. 59.
    Fernald A (1989) Intonation and communicative intent in mothers’ speech to infants: is the melody the message? Child Dev 60(6):1497–1510Google Scholar
  60. 60.
    Trainor LJ, Austin CM, Desjardins RN (2000) Is infant-directed speech prosody a result of the vocal expression of emotion? Psychol Sci 11(3):188–195CrossRefGoogle Scholar
  61. 61.
    Gleason JB (1977) Talking to children: some notes on feedback. In: Snow CE, Ferguson CA (eds) Talking to children: language input and acquisition. Cambridge University Press, Cambridge, UK, pp 199–205Google Scholar
  62. 62.
    Fernald A, Simon T (1984) Expanded intonation contours in mothers’ speech to newborns. Develop Psychol 20(1):104CrossRefGoogle Scholar
  63. 63.
    Reissland N, Stephenson T (1999) Turn-taking in early vocal interaction: a comparison of premature and term infants vocal interaction with their mothers. Child: care health develop 25(6):447–456Google Scholar
  64. 64.
    Best CT (1997) Accommodation in mean f, during mother–infant and father–infant vocal interactions: a longitudinal case study. J Child Lang 4(7):9–736Google Scholar
  65. 65.
    Lim A, Ogata T, Okuno HG (2012) Towards expressive musical robots: a cross-modal framework for emotional gesture, voice and music. EURASIP J Audio Speech Music Process 2012(1):1–12CrossRefGoogle Scholar
  66. 66.
    Lim A, Ogata T, Okuno HG (2011) Converting emotional voice to motion for robot telepresence. In: 11th IEEE-RAS international conference on humanoid robots (humanoids), 2011, pp 472–479. IEEEGoogle Scholar
  67. 67.
    Sievers B, Polansky L, Casey M, Wheatley T (2013) Music and movement share a dynamic structure that supports universal expressions of emotion. Proc Nat Acad Sci 110(1):70–75CrossRefGoogle Scholar
  68. 68.
    Fellous JM, Arbib MA (2005) Who needs emotions? The brain meets the robot. Oxford University Press, Oxford, UKCrossRefGoogle Scholar
  69. 69.
    Lim A, Okuno HG (2014) The MEI robot: towards using Motherese to develop multimodal emotional intelligence. IEEE Trans Auton Mental Develop 6(2):126–138Google Scholar
  70. 70.
    Schwarz G (1978) Estimating the dimension of a model. Ann Statist 6(2):461–464CrossRefzbMATHMathSciNetGoogle Scholar
  71. 71.
    Schuller B, Batliner A (2013) Computational paralinguistics: emotion, affect and personality in speech and language processing. Wiley, Hoboken, NJ, USAGoogle Scholar
  72. 72.
    Kitamura C, Lam C (2009) Age-specific preferences for infant-directed affective intent. Infancy 14(1):77–100Google Scholar
  73. 73.
    Fernald A (1993) Approval and disapproval: infant responsiveness to vocal affect in familiar and unfamiliar languages. Child Dev 64(3):657–674CrossRefGoogle Scholar
  74. 74.
    Asada M, Nagai Y, Ishihara H (2012) Why not artificial sympathy? In: International conference on social robotics, pp 278–287Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  1. 1.Kyoto UniversityKyotoJapan
  2. 2.Waseda UniversityTokyoJapan

Personalised recommendations