Journal of Bionic Engineering

, Volume 15, Issue 2, pp 185–203 | Cite as

Emotion Modelling for Social Robotics Applications: A Review

  • Filippo CavalloEmail author
  • Francesco Semeraro
  • Laura Fiorini
  • Gergely Magyar
  • Peter Sinčák
  • Paolo Dario


Robots of today are eager to leave constrained industrial environments and embrace unexplored and unstructured areas, for extensive applications in the real world as service and social robots. Hence, in addition to these new physical frontiers, they must face human ones, too. This implies the need to consider a human-robot interaction from the beginning of the design; the possibility for a robot to recognize users’ emotions and, in a certain way, to properly react and “behave”. This could play a fundamental role in their integration in society. However, this capability is still far from being achieved. Over the past decade, several attempts to implement automata for different applications, outside of the industry, have been pursued. But very few applications have tried to consider the emotional state of users in the behavioural model of the robot, since it raises questions such as: how should human emotions be modelled for a correct representation of their state of mind? Which sensing modalities and which classification methods could be the most feasible to obtain this desired knowledge? Furthermore, which applications are the most suitable for the robot to have such sensitivity? In this context, this paper aims to provide a general overview of recent attempts to enable robots to recognize human emotions and interact properly.


social robotics service robotics human-robot interaction emotion recognition robot learning robot behavioural model bionics 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    International Federation of Robotics. World Robotics Report 2016: European Union Occupies Top Position in the Global Automation Race, Report, Frankfurt, Germany, 2016.Google Scholar
  2. [2]
    Fiorini L, Esposito R, Bonaccorsi M, Petrazzuolo C, Saponara F, Giannantonio R, De Petris G, Dario P, Cavallo F. Enabling personalised medical support for chronic disease management through a hybrid robot-cloud approach. Autonomous Robots, 2017, 41, 1263–1276.CrossRefGoogle Scholar
  3. [3]
    Reppou S, Karagiannis G. Social inclusion with robots: A RAPP case study using NAO for technology illiterate elderly at ormylia foundation. In: Szewczyk R, Zielinski C, Kaliczynska M eds., Progress in Automation, Robotics and Measuring Techniques. Advances in Intelligent Systems and Computing, Springer, Cham, Switzerland, 2015, 233–241.Google Scholar
  4. [4]
    Fasola J, Matarić M J. A socially assistive robot exercise coach for the elderly. Journal of Human-Robot Interaction, 2013, 2, 3–32.CrossRefGoogle Scholar
  5. [5]
    García-Soler Á, Facal D, Díaz-Orueta U, Pigini L, Blasi L, Qiu R X. Inclusion of service robots in the daily lives of frail older users: A step-by-step definition procedure on users’ requirements. Archives of Gerontology and Geriatrics, 2018, 74, 191–196.CrossRefGoogle Scholar
  6. [6]
    Lisetti C L. Affective computing. Pattern Analysis and Applications, 1998, 1, 71–73.CrossRefGoogle Scholar
  7. [7]
    Castellano G, Leite I, Pereira A, Martinho C, Paiva A, McOwan P W. Affect recognition for interactive companions: Challenges and design in real world scenarios. Journal on Multimodal User Interfaces, 2010, 3, 89–98.CrossRefGoogle Scholar
  8. [8]
    Spaulding S, Breazeal C. Towards affect-awareness for social robots. 2015 AAAI Fall Symposium Series, USA, 2015, 128–130.Google Scholar
  9. [9]
    Gray K, Wegner D M. Feeling robots and human zombies: Mind perception and the uncanny valley. Cognition, 2012, 125, 125–130.CrossRefGoogle Scholar
  10. [10]
    Rau P L P, Li Y, Li D. A cross-cultural study: Effect of robot appearance and task. International Journal of Social Robotics, 2010, 2, 175–186.CrossRefGoogle Scholar
  11. [11]
    Ho C C, MacDorman K F. Revisiting the uncanny valley theory: Developing and validating an alternative to the Godspeed indices. Computers in Human Behavior, 2010, 26, 1508–1518.CrossRefGoogle Scholar
  12. [12]
    Mathur M B, Reichling D B. Navigating a social world with robot partners: A quantitative cartography of the Uncanny Valley. Cognition, 2016, 146, 22–32.CrossRefGoogle Scholar
  13. [13]
    Hassenzahl M, Tractinsky N. User experience–A research agenda. Behaviour & Information Technology, 2006, 25, 91–97.CrossRefGoogle Scholar
  14. [14]
    Chumkamon S, Hayashi E, Masato K. Intelligent emotion and behavior based on topological consciousness and adaptive resonance theory in a companion robot. Biologically Inspired Cognitive Architectures, 2016, 18, 51–67.CrossRefGoogle Scholar
  15. [15]
    Limbu D K, Anthony W C Y, Adrian T H J, Dung T A, Kee T Y, Dat T H, Alvin W H Y, Terence N W Z, Jiang R D, Li J. Affective social interaction with CuDDler robot. The Proceedings of 6th IEEE Conference on Robotics, Automation and Mechatronics (RAM), IEEE, Manila, Philippines, 2013, 179–184.Google Scholar
  16. [16]
    Ekman P, Friesen W V. Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 1971, 17, 124–129.CrossRefGoogle Scholar
  17. [17]
    Bennett C C, Sabanovic S. Deriving minimal features for human-like facial expressions in robotic faces. International Journal of Social Robotics, 2014, 6, 367–381.CrossRefGoogle Scholar
  18. [18]
    Salmeron J L. Fuzzy cognitive maps for artificial emotions forecasting. Applied Soft Computing, 2012, 12, 3704–3710CrossRefGoogle Scholar
  19. [19]
    Russell J A. A circumplex model of affect. Journal of Personality and Social Psychology, 1980, 39, 1161–1178.CrossRefGoogle Scholar
  20. [20]
    Koelstra S, Patras I. Fusion of facial expressions and EEG for implicit affective tagging. Image & Vision Computing, 2013, 31, 164–174.CrossRefGoogle Scholar
  21. [21]
    Banda N, Engelbrecht A, Robinson P. Feature reduction for dimensional emotion recognition in human-robot interaction. 2015 IEEE Symposium Series on Computational Intelligence, Cape Town, South Africa, 2015, 803–810.CrossRefGoogle Scholar
  22. [22]
    Han J, Xie L, Li D, He Z J, Wang Z L. Cognitive emotion model for eldercare robot in smart home. China Communications, 2015, 12, 32–41.Google Scholar
  23. [23]
    Cambria E, Livingstone A, Hussain A. The hourglass of emotions. In: Esposito A, Esposito A M, Vinciarelli A, Hoffmann R, Müller V C, eds, Cognitive Behavioural Systems. Lecture Notes in Computer Science, Springer, Berlin, Heidelberg, Germany, 2012.Google Scholar
  24. [24]
    Poria S, Peng H, Hussain A, Howard N, Cambria E. Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing, 2017, 261, 217–230.CrossRefGoogle Scholar
  25. [25]
    Terada K, Yamauchi A, Ito A. Artificial emotion expression for a robot by dynamic color change. The Proceedings of IEEE RO-MAN, Paris, France, 2012, 314–321.Google Scholar
  26. [26]
    Raymundo C R, Johnson C G, Vargas P A. An architecture for emotional and context-aware associative learning for robot companions. The Proceedings of 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Kobe, Japan, 2015, 8799–8804.Google Scholar
  27. [27]
    Shi X-F, Wang Z L, Ping A, Zhang L-K. Artificial emotion model based on reinforcement learning mechanism of neural network. The Journal of China Universities of Posts Telecommunications, 2011, 18, 105–109.CrossRefGoogle Scholar
  28. [28]
    Lewis M, Canamero L. Are discrete emotions useful in human-robot interaction? Feedback from motion capture analysis. Humaine Association Conference on Affective Computing and Intelligent Interaction, 2013, 97–102.Google Scholar
  29. [29]
    Henriques R, Paiva A, Antunes C. Accessing emotion patterns from affective interactions using electrodermal activity. Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), Geneva, Switzerland, 2013, 43–48.Google Scholar
  30. [30]
    Kirby R, Forlizzi J, Simmons R. Affective social robots. Robotics and Autonomous Systems, 2010, 58, 322–332.CrossRefGoogle Scholar
  31. [31]
    Picard R W. Affective computing. Pattern Recognition, 1995, 20.Google Scholar
  32. [32]
    Giorgana G, Ploeger P G. Facial Expression Recognition for Domestic Service Robots, Springer, Berlin, Heidelberg, Germany, 2012.CrossRefGoogle Scholar
  33. [33]
    Vitale J, Williams M A, Johnston B, Boccignone G. Affective facial expression processing via simulation: A probabilistic model. Biologically Inspired Cognitive Architectures, 2014, 10, 30–41.CrossRefGoogle Scholar
  34. [34]
    Lourens T, Van Berkel R, Barakova E. Communicating emotions and mental states to robots in a real time parallel framework using Laban movement analysis. Robotics & Autonomous Systems, 2010, 58, 1256–1265.CrossRefGoogle Scholar
  35. [35]
    Lim A, Okuno H G. The MEI robot: Towards using motherese to develop multimodal emotional intelligence. IEEE Transactions on Autonomous Mental Development, 2014, 6, 126–138.CrossRefGoogle Scholar
  36. [36]
    Liu H, Zhang W. Mandarin emotion recognition based on multifractal theory towards human-robot interaction. IEEE International Conference on Robotics and Biomimetics (ROBIO), Shenzhen, China, 2013, 593–598.CrossRefGoogle Scholar
  37. [37]
    Juszkiewicz L. Improving speech emotion recognition system for a social robot with speaker recognition. 19th International Conference on Methods and Models in Automation and Robotics (MMAR), Miedzyzdroje, Poland, 2014.Google Scholar
  38. [38]
    Le B V, Lee S. Adaptive hierarchical emotion recognition from speech signal for human-robot communication. 2014 Tenth International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IIH-MSP), Kitakyushu, Japan, 2014, 807–810.CrossRefGoogle Scholar
  39. [39]
    Perez-Gaspar L A, Caballero-Morales S O, Trujillo-Romero F. Multimodal emotion recognition with evolutionary computation for human-robot interaction. Expert Systems with Applications, 2016, 66, 42–61.CrossRefGoogle Scholar
  40. [40]
    Nardelli M, Valenza G, Greco A, Lanata A, Scilingo E P. Arousal recognition system based on heartbeat dynamics during auditory elicitation. 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 2015.Google Scholar
  41. [41]
    Rattanyu K, Ohkura M, Mizukawa M. Emotion monitoring from physiological signals for service robots in the living space. International Conference on Control Automation and Systems (ICCAS), Gyeonggi-do, South Korea, 2010, 580–583.Google Scholar
  42. [42]
    Ferreira J, Br As S, Silva C F, Soares S C. An automatic classifier of emotions built from entropy of noise. Psychophysiology, 2017, 54, 620–627.CrossRefGoogle Scholar
  43. [43]
    Khezri M, Firoozabadi M, Sharafat A R. Reliable emotion recognition system based on dynamic adaptive fusion of forehead biopotentials and physiological signals. Computer Methods and Programs in Biomedicine, 2015, 122, 149–164.CrossRefGoogle Scholar
  44. [44]
    Wang S, Du J, Xu R. Decision fusion for EEG-based emotion recognition. International Conference on Machine Learning and Cybernetics (ICMLC), Guangzhou, China, 2015, 883–889.CrossRefGoogle Scholar
  45. [45]
    Nhan B R, Chau T. Classifying affective states using thermal infrared imaging of the human face. IEEE Transactions on Bio-medical Engineering, 2010, 57, 979–987.CrossRefGoogle Scholar
  46. [46]
    Henriques R, Paiva A. Learning effective models of emotions from physiological signals: The seven principles. Lecture Notes in Computer Science, 2014, 8908, 137–155.CrossRefGoogle Scholar
  47. [47]
    Saldien J, Goris K, Vanderborght B, Vanderfaeillie J, Lefeber D. Expressing emotions with the social robot probo. International Journal of Social Robotics, 2010, 2, 377–389.CrossRefGoogle Scholar
  48. [48]
    Ghayoumi M, Bansal A K. Multimodal architecture for emotion in robots using deep learning. Future Technologies Conference (FTC), San Francisco, USA, 2016.Google Scholar
  49. [49]
    Esposito D, Cavallo F. Preliminary design issues for inertial rings in Ambient Assisted Living applications. IEEE Instrumentation and Measurement Technology Conference, Pisa, Italy, 2015, 250–255.Google Scholar
  50. [50]
    Park J-H, Jang D-G, Park J, Youm S-K. Wearable sensing of in-ear pressure for heart rate monitoring with a piezoelectric Sensor. Sensors, 2015, 15, 23402–23417.CrossRefGoogle Scholar
  51. [51]
    Leo M, Del Coco M, Carcagnì P, Distante C, Bernava M, Pioggia G, Palestra G. Automatic emotion recognition in robot-children interaction for ASD treatment. IEEE International Conference on Computer Vision Workshop (ICCVW), Santiago, Chile, 2015, 537–545.CrossRefGoogle Scholar
  52. [52]
    De A, Saha A, Pal M C. A human facial expression recognition model based on eigen face approach. Procedia Computer Science, 2015, 45, 282–289.CrossRefGoogle Scholar
  53. [53]
    Luo R C, Lin P H, Wu Y C, Huang C Y. Dynamic face recognition system in recognizing facial expressions for service robotics. IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), Kachsiung, Taiwan, 2012, 879–884.CrossRefGoogle Scholar
  54. [54]
    Blais C, Roy C, Fiset D, Arguin M, Gosselin F. The eyes are not the window to basic emotions. Neuropsychologia, 2012, 50, 2830–2838.CrossRefGoogle Scholar
  55. [55]
    Khan R A, Meyer A, Konik H, Bouakaz S. Framework for reliable, real-time facial expression recognition for low resolution images. Pattern Recognition Letters, 2013, 34, 1159–1168.CrossRefGoogle Scholar
  56. [56]
    Ali H, Hariharan M, Yaacob S, Adom A H. Facial emotion recognition using empirical mode decomposition. Expert Systems with Applications, 2015, 42, 1261–1277.CrossRefGoogle Scholar
  57. [57]
    Kushiro K, Harada Y, Takeno J. Robot uses emotions to detect and learn the unknown. Biologically Inspired Cognitive Architectures, 2013, 4, 69–78CrossRefGoogle Scholar
  58. [58]
    Jitviriya W, Koike M, Hayashi E. Behavior selection system based on emotional variations. 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Kobe, Japan, 2015, 462–467.Google Scholar
  59. [59]
    Mei Y, Liu Z T. An emotion-driven attention model for service robot. 12th World Congress on Intelligent Control and Automation (WCICA), Guilin, China, 2016, 1526–1531.Google Scholar
  60. [60]
    Yin Z, Zhao M, Wang Y, Yang J, Zhang J. Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Computer Methods & Programs in Biomedicine, 2017, 140, 93–110.CrossRefGoogle Scholar
  61. [61]
    Mayya V, Pai R M, Pai M. Automatic facial expression recognition using DCNN. Procedia Computer Science, 2016, 93, 453–461.CrossRefGoogle Scholar
  62. [62]
    Barros P, Jirak D, Weber C, Wermter S. Multimodal emotional state recognition using sequence-dependent deep hierarchical features. Neural Networks, 2015, 72, 140–151.CrossRefGoogle Scholar
  63. [63]
    Mower E, Matarić M J, Narayanan S. A framework for automatic human emotion classification using emotion profiles. IEEE Transactions on Audio, Speech, and Language Processing, 2011, 19, 1057–1070.CrossRefGoogle Scholar
  64. [64]
    Nho Y-H, Seo J-W, Seol W-J, Kwon D-S. Emotional interaction with a mobile robot using hand gestures. 11th International Conference on Ubiquitous Robots and Ambient Intelligence, Kuala Lumpur, Malaysia, 2014, 506–509.Google Scholar
  65. [65]
    Röning J, Holappa J, Kellokumpu V, Tikanmäki A, Pietikäinen M. Minotaurus: A system for affective human–robot interaction in smart environments. Cognitive Computation, 2014, 6, 940–953.CrossRefGoogle Scholar
  66. [66]
    Van Chien D, Sung K J, Trung P X, Kim J-W. Emotion expression of humanoid robot by modification of biped walking pattern. 15th International Conference on Control, Automation and Systems (ICCAS), Busan, South Korea, 2015, 741–743.CrossRefGoogle Scholar
  67. [67]
    Sincák P, Novotná E, Cádrik T Magyar G, Mach M, Cavallo F. Bonaccorsi M. Cloud-based Wizard of Oz as a service. IEEE 19th International Conference on Intelligent Engineering Systems (INES), Bratislava, Slovakia, 2015, 445–448.CrossRefGoogle Scholar
  68. [68]
    Mazzei D, Zaraki A, Lazzeri N, De Rossi D. Recognition and expression of emotions by a symbiotic android head. 14th IEEE-RAS International Conference on Humanoid Robots (Humanoids), Madrid, Spain, 2014, 134–139.Google Scholar
  69. [69]
    Boccanfuso L, Barney E, Foster C, Ahn Y A, Chawarska K, Scassellati B, Shic F. Emotional robot to examine differences in play patterns and affective response of Children with and without ASD. 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Christchurch, New Zealand, 2016, 19–26.Google Scholar
  70. [70]
    Cao H L, Esteban P G, De Beir A, Simut R, Van De Perre G, Lefeber D, Vanderborght B. ROBEE: A homeostatic-based social behavior controller for robots in Human-Robot Interaction experiments. IEEE International Conference on Robotics and Biomimetics (ROBIO), 2014, 516–521.Google Scholar
  71. [71]
    Cavallo F, Limosani R, Manzi A, Bonaccorsi M, Esposito R, Di Rocco M, Pecora F, Teti G, Saffiotti A, Dario P. Development of a socially believable multi-robot solution from town to home. Cognitive Computation, 2014, 6, 954–967.CrossRefGoogle Scholar
  72. [72]
    Kopacek P. Ethical and social aspects of robots. IFAC Proceedings Volumes, 2014, 47, 11425–11430.CrossRefGoogle Scholar
  73. [73]
    Mell P, Grance T, Grance T. The NIST Definition of Cloud Computing Recommendations of the National Institute of Standards and Technology, National Institute of Standards and Technology-Special Publication 800-145, 2011.Google Scholar
  74. [74]
    Goldberg K, Kehoe B. Cloud Robotics and Automation: A Survey of Related Work, Technical Report, USA, 2013.Google Scholar
  75. [75]
    Kehoe B, Patil S, Abbeel P. Goldberg K. A survey of research on cloud robotics and automation. IEEE Transactions on Automation Science & Engineering, 2015, 12, 398–409.Google Scholar
  76. [76]
    Chen M, Ma Y, Song J, Lai C-F, Hu B. Smart clothing: Connecting human with clouds and big data for sustainable health monitoring. Mobile Networks & Applications, 2016, 21, 825–845.CrossRefGoogle Scholar
  77. [77]
    Lim A, Okuno H G. The MEI robot: Towards using motherese to develop multimodal emotional intelligence. IEEE Transactions on Autonomous Mental Development, 2014, 6, 126–138.CrossRefGoogle Scholar
  78. [78]
    Yan H, Ang Jr M H, Neow Poo A. A survey on perception methods for human–robot interaction in social robots. International Journal of Social Robotics, 2014, 6, 85–119.CrossRefGoogle Scholar

Copyright information

© Jilin University 2018

Authors and Affiliations

  • Filippo Cavallo
    • 1
    Email author
  • Francesco Semeraro
    • 1
  • Laura Fiorini
    • 1
  • Gergely Magyar
    • 2
  • Peter Sinčák
    • 2
  • Paolo Dario
    • 1
  1. 1.BioRobotics InstituteScuola Superiore Sant’AnnaPontederaItaly
  2. 2.Department of Cybernetics and Artificial Intelligence, Faculty of Electrical Engineering and InformaticsTechnical University of KosiceKosiceSlovakia

Personalised recommendations