Emotional States Detection Approaches Based on Physiological Signals for Healthcare Applications: A Review

  • Diana Patricia Tobón Vallejo
  • Abdulmotaleb El SaddikEmail author


Mood disorders, anxiety, depression, and stress affect people’s quality of life and increase the vulnerability to diseases and infections. Depression, e.g., can carry undesirable consequences such as death. Hence, emotional states detection approaches using wearable technology are gaining interest in the last few years. Emerging wearable devices allow monitoring different physiological signals in order to extract useful information about people’s health status and provide feedback about their health condition. Wearable applications include e.g., patient monitoring, stress detection, fitness monitoring, wellness monitoring, and assisted living for elderly people, to name a few. This increased interests in wearable applications have allowed the development of new approaches to assist people in everyday activities and emergencies that can be incorporated into the smart city concept. Accurate emotional state detection approaches will allow an effective assistance, thus improving people’s quality of life and well-being. With these issues in mind, this chapter discusses existing emotional states’ approaches using machine and/or deep learning techniques, the most commonly used physiological signals in these approaches, existing physiological databases for emotion recognition, and highlights challenges and future research directions in this field.


Affective recognition Deep learning Emotional states Emotions Machine learning Physiological signals Quality of life Smart city Well-being 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    T. Taleb, D. Bottazzi, N. Nasser, A novel middleware solution to improve ubiquitous healthcare systems aided by affective information. IEEE Trans. Inf. Technol. Biomed. 14(2), 335–349 (2010)CrossRefGoogle Scholar
  2. 2.
    D. Tobón, T. Falk, M. Maier, Context awareness in WBANs: a survey on medical and non-medical applications. IEEE Wirel. Commun. 20(4), 30–37 (2013)CrossRefGoogle Scholar
  3. 3.
    P. Bellavista, D. Bottazzi, A. Corradi, R. Montanari, Challenges, opportunities and solutions for ubiquitous eldercare, in Web Mobile-Based Applications for Healthcare Management (IGI Global, 2007), pp. 142–165Google Scholar
  4. 4.
    B. Arnrich, C. Setz, R. La Marca, G. Tröster, U. Ehlert, What does your chair know about your stress level? IEEE Trans. Inf. Technol. Biomed. 14(2), 207–214 (2010)CrossRefGoogle Scholar
  5. 5.
    T.R. Bennett, J. Wu, N. Kehtarnavaz, R. Jafari, Inertial measurement unit-based wearable computers for assisted living applications: a signal processing perspective. IEEE Signal Process. Mag. 33(2), 28–35 (2016)CrossRefGoogle Scholar
  6. 6.
    S. Greene, H. Thapliyal, A. Caban-Holt, A survey of affective computing for stress detection: evaluating technologies in stress detection for better health. IEEE Consum. Electr. Mag. 5(4), 44–56 (2016)CrossRefGoogle Scholar
  7. 7.
    O.C. Santos, R. Uria-Rivas, M. Rodriguez-Sanchez, J.G. Boticario, An open sensing and acting platform for context-aware affective support in ambient intelligent educational settings. IEEE Sensors J. 16(10), 3865–3874 (2016)CrossRefGoogle Scholar
  8. 8.
    G. Rebolledo-Mendez, A. Reyes, S. Paszkowicz, M.C. Domingo, L. Skrypchuk, Developing a body sensor network to detect emotions during driving. IEEE Trans. Intell. Transp. Syst. 15(4), 1850–1854 (2014)CrossRefGoogle Scholar
  9. 9.
    Z. Zeng, M. Pantic, G.I. Roisman, T.S. Huang, A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009)CrossRefGoogle Scholar
  10. 10.
    A. Hariharan, M.T.P. Adam, Blended emotion detection for decision support. IEEE Trans. Hum. Mac. Syst. 45(4), 510–517 (2015)CrossRefGoogle Scholar
  11. 11.
    H. Al Osman, T.H. Falk, Multimodal affect recognition: current approaches and challenges, in Emotion and Attention Recognition Based on Biological Signals and Images (InTech, 2017), pp. 59–86Google Scholar
  12. 12.
    M.A. Hogg, D. Abrams, Social cognition and attitudes, in Psychology, 3rd edn. ed. by G.N. Martin, N.R. Carlson, W. Buskist (Pearson Education Limited, 2007), pp. 684–721Google Scholar
  13. 13.
    P. Ekman, About brows: emotional and conversational signals. Hum. Ethol. 163–202 (1979)Google Scholar
  14. 14.
    P. Ekman, W.V. Friesen, P. Ellsworth, Emotion in the Human Face: Guidelines for Research and an Integration of Findings (Elsevier, 2013)Google Scholar
  15. 15.
    H.-J. Go, K.-C. Kwak, D.-J. Lee, M.-G. Chun, Emotion recognition from the facial image and speech signal, in SICE 2003 Annual Conference, (2003)Google Scholar
  16. 16.
    D. Sander, D. Grandjean, K.R. Scherer, A systems approach to appraisal mechanisms in emotion. Neural Netw. 18(4), 317–352 (2005)CrossRefGoogle Scholar
  17. 17.
    J. Kim, E. André, Emotion recognition based on physiological changes in music listening. IEEE Trans. Pattern Anal. Mach. Intell. 30(12), 2067–2083 (2008)CrossRefGoogle Scholar
  18. 18.
    H. Schlosberg, Three dimensions of emotion. Psychol. Rev. 61(2), 81–88 (1954)CrossRefGoogle Scholar
  19. 19.
    B.L. Fredrickson, R.W. Levenson, Positive emotions speed recovery from the cardiovascular sequelae of negative emotions. Cognit. Emot. 12(2), 191–220 (1998)CrossRefGoogle Scholar
  20. 20.
    A. Greco, G. Valenza, L. Citi, E.P. Scilingo, Arousal and valence recognition of affective sounds based on electrodermal activity. IEEE Sensors J. 17(3), 716–725 (2017)CrossRefGoogle Scholar
  21. 21.
    G. Valenza, L. Citi, C. Gentili, A. Lanata, E.P. Scilingo, R. Barbieri, Characterization of depressive states in bipolar patients using wearable textile technology and instantaneous heart rate variability assessment. IEEE J. Biomed. Health Inform. 19(1), 263–274 (2015)CrossRefGoogle Scholar
  22. 22.
    J. Ressel, A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980)CrossRefGoogle Scholar
  23. 23.
    D. Ravi, C. Wong, F. Deligianni, M. Berthelot, J. Andreu-Perez, B. Lo, G.-Z. Yang, Deep learning for health informatics. IEEE J. Biomed. Health Inform. 21(1), 4–21 (2017)CrossRefGoogle Scholar
  24. 24.
    J.T. Cacioppo, Introduction: emotion and health, in Handbook of Affective Sciences, (Oxford University Press, New York, 2003), pp. 1047–1052Google Scholar
  25. 25.
    J.A. Healey, R.W. Picard, Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 6(2), 156–166 (2005)CrossRefGoogle Scholar
  26. 26.
    R.W. Picard, Affective medicine: technology with emotional intelligence, in Future of Health Technology (IOS Press, 2002), pp. 69–84Google Scholar
  27. 27.
    R. Gravina, G. Fortino, Automatic methods for the detection of accelerative cardiac defense response. IEEE Trans. Affect. Comput. 7(3), 286–298 (2016)CrossRefGoogle Scholar
  28. 28.
    M. Murugappan, M. Rizon, R. Nagarajan, S. Yaacob, I. Zunaidi, D. Hazry, EEG feature extraction for classifying emotions using FCM and FKM. Int. J. Comp. Commun. 1(2), 21–25 (2007)Google Scholar
  29. 29.
    Y.-P. Lin, C.-H. Wang, T.-P. Jung, T.-L. Wu, S.-K. Jeng, J.-R. Duann, J.-H. Chen, EEG-based emotion recognition in music listening. IEEE Trans. Biomed. Eng. 57(7), 1798–1806 (2010)CrossRefGoogle Scholar
  30. 30.
    Harvard Health Publications, Harvard medical school, 18 March 2016. [Online] Accessed 2 Aug 2017
  31. 31.
    D. Lincoln, Correlation of unit activity in the hypothalamus with EEG patterns associated with the sleep cycle. Exp. Neurol. 24(1), 1–18 (1969)CrossRefGoogle Scholar
  32. 32.
    R.J. Davidson, G.E. Schwartz, C. Saron, J. Bennett, D. Goleman, Frontal Versus Parietal EEG Asymmetry During Positive and Negative Affect (Cambridge University Press, New York, 1979)Google Scholar
  33. 33.
    R.H. Chowdhury, M.B. Reaz, M.A.B.M. Ali, A.A. Bakar, K. Chellappan, T.G. Chang, Surface electromyography signal processing and classification techniques. Sensors 13(9), 12431–12466 (2013)CrossRefGoogle Scholar
  34. 34.
    A. Nakasone, H. Prendinger, M. Ishizuka, Emotion recognition from electromyography and skin conductance, in Proceedings of the 5th International Workshop on Biosignal Interpretation (2005)Google Scholar
  35. 35.
    B.M. Appelhans, L.J. Luecken, Heart rate variability as an index of regulated emotional responding. Rev. Gen. Psychol. 10(3), 229–240 (2006)CrossRefGoogle Scholar
  36. 36.
    R.B. Singh, G. Cornélissen, A. Weydahl, O. Schwartzkopff, G. Katinas, K. Otsuka, Y. Watanabe, S. Yano, H. Mori, Y. Ichimaru, Circadian heart rate and blood pressure variability considered for research and patient care. Int. J. Cardiol. 87(1), 9–28 (2003)CrossRefGoogle Scholar
  37. 37.
    A.H. Kemp, A.R. Brunoni, I.S. Santos, M.A. Nunes, E.M. Dantas, R. Carvalho de Figueiredo, A.C. Pereira, A.L. Ribeiro, J.G. Mill, R.V. Andreao, Effects of depression, anxiety, comorbidity, and antidepressants on resting-state heart rate and its variability: an ELSA-Brasil cohort baseline study. Am. J. Psychiatr. 171(12), 1328–1334 (2014)CrossRefGoogle Scholar
  38. 38.
    R.W. Levenson, P. Ekman, W.V. Friesen, Voluntary facial action generates emotion-specific autonomic nervous system activity. Psychophysiology 27(4), 363–384 (1990)CrossRefGoogle Scholar
  39. 39.
    S.D. Kreibig, Autonomic nervous system activity in emotion: a review. Biol. Psychol. 84(3), 394–421 (2010)CrossRefGoogle Scholar
  40. 40.
    H. Al Osman, H. Dong, A. El Saddik, Ubiquitous biofeedback serious game for stress management. IEEE Access 4, 1274–1286 (2016)CrossRefGoogle Scholar
  41. 41.
    M.D. van der Zwaag, J.H. Janssen, J.H. Westerink, Directing physiology and mood through music: validation of an affective music player. IEEE Trans. Affect. Comput. 4(1), 57–68 (2013)CrossRefGoogle Scholar
  42. 42.
    S. Khalfa, P. Isabelle, B. Jean-Pierre, R. Manon, Event-related skin conductance responses to musical emotions in humans. Neurosci. Lett. 328(2), 145–149 (2002)CrossRefGoogle Scholar
  43. 43.
    R.W. Picard, S. Fedor, Y. Ayzenberg, Multiple arousal theory and daily-life electrodermal activity asymmetry. Emot. Rev. 8(1), 62–75 (2016)CrossRefGoogle Scholar
  44. 44.
    M.-Z. Poh, N.C. Swenson, R.W. Picard, A wearable sensor for unobtrusive, long-term assessment of electrodermal activity. IEEE Trans. Biomed. Eng. 57(5), 1243–1252 (2010)CrossRefGoogle Scholar
  45. 45.
    H. Fukushima, H. Kawanaka, M.S. Bhuiyan, K. Oguri, Estimating heart rate using wrist-type photoplethysmography and acceleration sensor while running, in Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (2012)Google Scholar
  46. 46.
    A.A. Alian, K.H. Shelley, Photoplethysmography. Best Pract. Res. Clin. Anaesthesiol. 28, 395–406 (2014)CrossRefGoogle Scholar
  47. 47.
    L. Peter, N. Noury, M. Cerny, A review of methods for non-invasive and continuous blood pressure monitoring: pulse transit time method is promising? Irbm 35(5), 271–282 (2014)CrossRefGoogle Scholar
  48. 48.
    B. Padasdao, E. Shahhaidar, C. Stickley, O. Boric-Lubecke, Electromagnetic biosensing of respiratory rate. IEEE Sensors J. 13(11), 4204–4211 (2013)CrossRefGoogle Scholar
  49. 49.
    E. Shahhaidar, B. Padasdao, R. Romine, O. Boric-Lubecke, Piezoelectric and electromagnetic respiratory effort energy harvesters, in 35th Annual International Conference of the IEEE EMBS, Osaka, Japan (2013)Google Scholar
  50. 50.
    C.J. Wientjes, Respiration in psychophysiology: methods and applications. Biol. Psychol. 34(2), 179–203 (1992)CrossRefGoogle Scholar
  51. 51.
    J. Johnson, Bilateral finger temperature and the low of initial value. Psychophysiology 24, 666–669 (1978)Google Scholar
  52. 52.
    P. Vos, P. De Cock, V. Munde, K. Petry, W. Van Den Noortgate, B. Maes, The tell-tale: what do heart rate; skin temperature and skin conductance reveal about emotions of people with severe and profound intellectual disabilities? Res. Dev. Disabil. 33(4), 1117–1127 (2012)CrossRefGoogle Scholar
  53. 53.
    A. Greco, G. Valenza, A. Lanata, G. Rota, E.P. Scilingo, Electrodermal activity in bipolar patients during affective elicitation. IEEE J. Biomed. Health Inform. 18(6), 1865–1873 (2014)CrossRefGoogle Scholar
  54. 54.
    M. Kumar, D. Arndt, S. Kreuzfeld, K. Thurow, N. Stoll, R. Stoll, Fuzzy techniques for subjective workload-score modeling under uncertainties. IEEE Trans. Syst. Man Cybern. B Cybern. 38(6), 1449–1464 (2008)CrossRefGoogle Scholar
  55. 55.
    M. Nardelli, G. Valenza, A. Greco, A. Lanata, E.P. Scilingo, Recognizing emotions induced by affective sounds through heart rate variability. IEEE Trans. Affect. Comput. 6(4), 385–394 (2015)CrossRefGoogle Scholar
  56. 56.
    E.-H. Jang, B.-J. Park, S.-H. Kim, J.-H. Sohn, Emotion classification based on physiological signals induced by negative emotions: discrimination of negative emotions by machine learning algorithm, in 9th IEEE International Conference on Networking, Sensing and Control (ICNSC) (2012)Google Scholar
  57. 57.
    H.P. Martinez, Y. Bengio, G.N. Yannakakis, Learning deep physiological models of affect. IEEE Comput. Intell. Mag. 8(2), 20–33 (2013)CrossRefGoogle Scholar
  58. 58.
    O. AlZoubi, S.K. D’Mello, R.A. Calvo, Detecting naturalistic expressions of nonbasic affect using physiological signals. IEEE Trans. Affect. Comput. 3(3), 298–310 (2012)CrossRefGoogle Scholar
  59. 59.
    G. Chanel, C. Rebetez, M. Bétrancourt, T. Pun, Emotion assessment from physiological signals for adaptation of game difficulty. IEEE Trans. Syst. Man Cybern. A Syst. Humans 41(6), 1052–1063 (2011)CrossRefGoogle Scholar
  60. 60.
    W. Wen, G. Liu, N. Cheng, J. Wei, P. Shangguan, W. Huang, Emotion recognition based on multi-variant correlation of physiological signals. IEEE Trans. Affect. Comput. 5(2), 126–140 (2014)CrossRefGoogle Scholar
  61. 61.
    M. Swangnetr, D.B. Kaber, Emotional state classification in patient–robot interaction using wavelet analysis and statistics-based feature selection. IEEE Trans. Hum. Mac. Syst. 43(1), 63–75 (2013)CrossRefGoogle Scholar
  62. 62.
    R. Jenke, A. Peer, M. Buss, Feature extraction and selection for emotion recognition from EEG. IEEE Trans. Affect. Comput. 5(3), 327–339 (2014)CrossRefGoogle Scholar
  63. 63.
    C.A. Frantzidis, C. Bratsas, M.A. Klados, E. Konstantinidis, C.D. Lithari, A.B. Vivas, C.L. Papadelis, E. Kaldoudi, C. Pappas, P.D. Bamidis, On the classification of emotional biosignals evoked while viewing affective pictures: an integrated data-mining-based approach for healthcare applications. IEEE Trans. Inf. Technol. Biomed. 14(2), 309–318 (2010)CrossRefGoogle Scholar
  64. 64.
    J. Fleureau, P. Guillotel, Q. Huynh-Thu, Physiological-based affect event detector for entertainment video applications. IEEE Trans. Affect. Comput. 3(3), 379–385 (2012)CrossRefGoogle Scholar
  65. 65.
    C.-K. Wu, P.-C. Chung, C.-J. Wang, Representative segment-based emotion analysis and classification with automatic respiration signal segmentation. IEEE Trans. Affect. Comput. 3(4), 482–495 (2012)CrossRefGoogle Scholar
  66. 66.
    G.E. Sakr, I.H. Elhajj, H.A.-S. Huijer, Support vector machines to define and detect agitation transition. IEEE Trans. Affect. Comput. 1(2), 98–108 (2010)CrossRefGoogle Scholar
  67. 67.
    G. Valenza, A. Lanata, E.P. Scilingo, The role of nonlinear dynamics in affective valence and arousal recognition. IEEE Trans. Affect. Comput. 3(2), 237–249 (2012)CrossRefGoogle Scholar
  68. 68.
    S.K. Hadjidimitriou, L.J. Hadjileontiadis, Toward an EEG-based recognition of music liking using time-frequency analysis. IEEE Trans. Biomed. Eng. 59(12), 3498–3510 (2012)CrossRefGoogle Scholar
  69. 69.
    S. Walter, J. Kim, D. Hrabal, S.C. Crawcour, H. Kessler, H.C. Traue, Transsituational individual-specific biopsychological classification of emotions. IEEE Trans. Syst. Man Cybern. Syst. 43(4), 988–995 (2013)CrossRefGoogle Scholar
  70. 70.
    Imotions, What is GSR (Galvanic Skin Response) and How Does It Work?, 12 May 2015. [Online] Accessed 2 Aug 2017
  71. 71.
    S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, I. Patras, Deap: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2012)CrossRefGoogle Scholar
  72. 72.
    M. Soleymani, J. Lichtenauer, T. Pun, M. Pantic, A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2012)CrossRefGoogle Scholar
  73. 73.
    R.W. Picard, E. Vyzas, J. Healey, Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23(10), 1175–1191 (2001)CrossRefGoogle Scholar
  74. 74.
    E. Douglas-Cowie, R. Cowie, I. Sneddon, C. Cox, O. Lowry, M. Mcrorie, J.-C. Martin, L. Devillers, S. Abrilian, A. Batliner, et al., The HUMAINE database: addressing the collection and annotation of naturalistic and induced emotional data, in Affective Computing and Intelligent Interaction, (Springer, Berlin, 2007), pp. 488–500CrossRefGoogle Scholar
  75. 75.
    F. Ringeval, A. Sonderegger, J. Sauer, D. Lalanne, Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions, in 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China (2013)Google Scholar
  76. 76.
    L. Zhang, S. Walter, X. Ma, “BioVid Emo DB”: a multimodal database for emotion analyses validated by subjective ratings, in IEEE Symposium Series on Computational Intelligence (SSCI), Athens, Greece (2016)Google Scholar
  77. 77.
    University of Trento, Italy, in Multimedia and Human Understanding Group (MHUG) [Online] Accessed 10 Aug 2017
  78. 78.
    G. Chanel, J. Kronegg, D. Grandjean, T. Pun, Emotion assessment: arousal evaluation using EEG’s and peripheral physiological signals, in Multimedia Content Representation, Classification and Security (2006), pp. 530–537Google Scholar
  79. 79.
    S. Afzal, P. Robinson, Natural affect data: collection and annotation, in New Perspectives on Affect and Learning Technologies (Springer, 2011), pp. 55–70Google Scholar
  80. 80.
    H. P. Martínez and G. N. Yannakakis, "Mining multimodal sequential patterns: a case study on affect detection," in Proceedings of the 13th international conference on multimodal interfaces, 2011.CrossRefGoogle Scholar
  81. 81.
    M.I.o. Technology, Affective Computing, MIT [Online] Accessed 5 June 2017
  82. 82.
    T. Hui, S.R. Simon, D.S. Daniel, Major requirements for building Smart Homes in Smart Cities based on Internet of Things technologies. Futur. Gener. Comput. Syst. 76, 358–369 (2017)CrossRefGoogle Scholar
  83. 83.
    U.R. Acharya, K.P. Joseph, N. Kannathal, C.M. Lim, J.S. Suri, Heart rate variability: a review. Med. Biol. Eng. Comput. 44(12), 1031–1051 (2006)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Diana Patricia Tobón Vallejo
    • 1
  • Abdulmotaleb El Saddik
    • 2
    Email author
  1. 1.Universidad de MedellinMedellinColombia
  2. 2.Multimedia Communications Research LaboratoryUniversity of OttawaOttawaCanada

Personalised recommendations