, Volume 26, Issue 3, pp 301–315 | Cite as

Emotional empathy transition patterns from human brain responses in interactive communication situations

Brain–computer and machine interactive interfacing approach
  • Tomasz M. RutkowskiEmail author
  • Andrzej Cichocki
  • Danilo P. Mandic
  • Toyoaki Nishida
Open Forum


The paper reports our research aiming at utilization of human interactive communication modeling principles in application to a novel interaction paradigm designed for brain–computer/machine-interfacing (BCI/BMI) technologies as well as for socially aware intelligent environments or communication support systems. Automatic procedures for human affective responses or emotional states estimation are still a hot topic of contemporary research. We propose to utilize human brain and bodily physiological responses for affective/emotional as well as communicative interactivity estimation, which potentially could be used in the future for human–machine/environment interaction design. As a test platform for such an intelligent human–machine communication application, an emotional stimuli paradigm was chosen to evaluate brain responses to various affective stimuli in an emotional empathy mode. Videos with moving faces expressing various emotional displays as well as speech stimuli with similarly emotionally articulated sentences are presented to the subjects in order to further analyze different affective responses. From information processing point of view, several challenges with multimodal signal conditioning and stimuli dynamic response extraction in time frequency domain are addressed. Emotions play an important role in human daily life and human-to-human communication. This is why involvement of affective stimuli principles to human–machine communication or machine-mediated communication with utilization of multichannel neurophysiological and periphery physiological signals monitoring techniques, allowing real-time subjective brain responses evaluation, is discussed. We present our preliminary results and discuss potential applications of brain/body affective responses estimation for future interactive/smart environments.


Emotional stages from brain responses estimation Communication with emotional stages evaluation Socially aware intelligent environments design Brain–computer or machine interfacing paradigms 



Authors would like to thank Prof. Michihiko Minoh and Prof. Koh Kakusho of Kyoto University for their support and fruitful discussions in frame of the project “Intelligent Media Technology for Supporting Natural Communication between People”, which was partially supported by the Ministry of Education, Science, Sports and Culture in Japan, Grant-in-Aid for Creative Scientific Research, 13GS0003, where communicative interactivity approach was first developed. Also, we would like to thank for many discussions Prof. Victor V. Kryssanov of Ritsumeikan University in Kyoto at the beginning stages of presented research, which were very valuable to shape the final approach.


  1. Adler R, Rodman G (2003) Undestanding human communication. Oxford University Press, OxfordGoogle Scholar
  2. Baron-Cohen S (2004) Mind reading—the interactive guide to emotions. Jessica Kingsley Publishers, LondonGoogle Scholar
  3. Cichocki A, Washizawa Y, Rutkowski T, Bakardjian H, Phan AH, Choi S, Lee H, Zhao Q, Zhang L, Li Y (2008) Noninvasive BCIs: multiway signal-processing array decompositions. Computer 41(10):34–42CrossRefGoogle Scholar
  4. Guger C, Daban S, Sellers E, Holzner C, Krausz G, Carabalona R, Gramatica F, Edlinger G (2009) How many people are able to control a P300-based brain–computer interface (BCI)?. Neurosci Lett 462(1):94–98CrossRefGoogle Scholar
  5. Hyvarinen A, Karhunen J, Oja E (2001) Independent component analysis. Wiley, New YorkCrossRefGoogle Scholar
  6. Lécuyer A, Lotte F, Reilly RB, Leeb R, Hirose M, Slater M (2008) Brain–computer interfaces, virtual reality, and videogames. Computer 41(10):66–72CrossRefGoogle Scholar
  7. McFarland DJ, Wolpaw JR (2008) Brain-computer interface operation of robotic and prosthetic devices. Computer 41(10):52–56CrossRefGoogle Scholar
  8. Murdoch D, Chow ED (2007) Ellipse: functions for drawing ellipses and ellipse-like confidence regions. R package version 0.3–5 (porting to R by Jesus M. Frias Celayeta)Google Scholar
  9. Niedermeyer E, Da Silva FL (eds) (2004) Electroencephalography: basic principles, clinical applications, and related fields. 5th edn. Lippincott Williams & Wilkins, PhiladelphiaGoogle Scholar
  10. Pfurtscheller G, Müller-Putz GR, Scherer R, Neuper C (2008) Rehabilitation with brain-computer interface systems. Computer 41(10):58–65CrossRefGoogle Scholar
  11. R Development Core Team (2009) R: a language and environment for statistical computing. R foundation for statistical computing, Vienna, Austria, ISBN 3-900051-07-0Google Scholar
  12. Rutkowski T, Mandic D, Barros A (2007) A multimodal approach to communicative interactivity classification. J VLSI Signal Process 49(2):317–328CrossRefGoogle Scholar
  13. Rutkowski TM, Mandic D (2007) Artificial intelligence for human computing. In: Lecture notes in artificial intelligence, vol 4451, chap Modelling the communication atmosphere—a human centered multimedia approach to evaluate communicative situations. Springer, Berlin, pp 155–169Google Scholar
  14. Rutkowski TM, Mandic D (2008) Engineering approaches to conversational informatics, chap Modeling communication atmosphere. Wiley, pp 353–369 (Wiley Series in Agent Technology)Google Scholar
  15. Rutkowski TM, Yamakata SSY, Kakusho K, Minoh M (2003) Knowledge-based intelligent information and engineering systems. In: Lecture notes in computer science, vol 2774, chap Toward the human communication efficiency monitoring from captured audio and video media in real environments. Springer, Berlin, pp 1093–1100Google Scholar
  16. Rutkowski TM, Cichocki A, Ralescu AL, Mandic DP (2008) Emotional states estimation from multichannel EEG maps. In: Wang R, Gu F, Shen E (eds) Advances in cognitive neurodynamics ICCN 2007, Proceedings of the international conference on cognitive neurodynamics, Neuroscience. Springer, Berlin, pp 695–698Google Scholar
  17. Rutkowski TM, Cichocki A, Tanaka T, Mandic DP, Cao J, Ralescu AL (2009) Multichannel spectral pattern separation—an EEG processing application. In: Proceedings of the 2009 IEEE international conference on acoustics, speech, and signal processing (ICASSP2009), IEEE. pp 373–376Google Scholar
  18. Rutkowski TM, Mandic DP, Cichocki A, Przybyszewski AW (2010) EMD approach to multichannel EEG data—the amplitude and phase components clustering analysis. J Circuits Syst Comput (JCSC) 19(1):215–229CrossRefGoogle Scholar
  19. Schlögl A, Brunner C (2008) Biosig: a free and open source software library for BCI research. Computer 41(10):44–50CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2010

Authors and Affiliations

  • Tomasz M. Rutkowski
    • 1
    • 2
    Email author
  • Andrzej Cichocki
    • 1
  • Danilo P. Mandic
    • 3
  • Toyoaki Nishida
    • 4
  1. 1.Advanced Brain Signal Processing LabRIKEN Brain Science InstituteWako-shiJapan
  2. 2.BSI-TOYOTA Collaboration CenterRIKEN Brain Science InstituteWako-shiJapan
  3. 3.Department of Electrical and Electronic EngineeringImperial College LondonLondonUnited Kingdom
  4. 4.Department of Intelligence Science and Technology, Graduate School of InformaticsKyoto UniversitySakyo-kuJapan

Personalised recommendations