Skip to main content
SpringerLink
Log in

A review of Deep Learning based methods for Affect Analysis using Physiological Signals

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Emotions are distinct reactions to internal or external events with implications for the organism. Automatic emotion recognition is a demanding task for pattern recognition and a required information retrieval method for diagnosing the condition of emotions in the peripheral nervous system and psychotherapy. In recent years, scientists have extensively considered physiological signals since they can give a modest, inexpensive, convenient, and easy-to-utilize result for recognizing emotions. Deep Learning has recently demonstrated incredible guarantees in figuring out physiological signals because of its ability to extract useful features and achieve better emotion recognition performance. In this survey, we analyzed a review of the neuro-physiological exploration made from 2012 to 2022, giving a complete outline of the current works in feeling acknowledgment from physiological signals utilizing deep learning strategies. We center our examination on the fundamental viewpoints engaged with the acknowledgment procedure (e.g., stimulus, features extracted, architectures). Our investigation reveals that most researchers have used Convolutional Neural Networks over other deep networks for classifying physiological-based emotions, as deep learning permits automatic end-to-end learning of pre-processing, including extraction and classification components. We determine many challenges and practice suggestions to help the exploration network, especially for the individuals entering this research field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

  1. Akter L, Ferdib-Al-Islam, Islam MM et al (2021) Prediction of cervical Cancer from behavior risk using machine learning techniques. SN Comput Sci 2. https://doi.org/10.1007/s42979-021-00551-6

  2. Alarcao SM, Fonseca MJ (2017) Emotions recognition using EEG signals: a survey. IEEE Trans Affect Comput 10:374–393

  3. Algarni, M, Saeed F, Al-Hadhrami T, Ghabban F, Al-Sarem M (2022) Deep learning-based approach for emotion recognition using electroencephalography (EEG) signals using bi-directional long short-term memory (Bi-LSTM). Sensors 22(8):2976. https://doi.org/10.3390/s22082976

  4. Alhagry S, Fahmy AA, El-Khoribi RA (2017) Emotion recognition based on EEG using LSTM recurrent neural network. Emotion 8:355–358

  5. Al-Nafjan A, Hosny M, Al-Ohali Y, Al-Wabil A (2017) Review and classification of emotion recognition based on EEG brain-computer interface system research: A systematic review. Appl Sci (Switzerland) 7:1239

    Google Scholar 

  6. An Y, Xu N, Qu Z (2021) Leveraging spatial-temporal convolutional features for EEG-based emotion recognition. Biomed Signal Process Control 69. https://doi.org/10.1016/j.bspc.2021.102743

  7. Arel I, Rose D, Karnowski T (2010) Deep machine learning-a new frontier in artificial intelligence research. IEEE Comput Intell Mag 5:13–18. https://doi.org/10.1109/MCI.2010.938364

    Article  Google Scholar 

  8. Arjun RAS, Panicker MR (2022) Subject independent emotion recognition using EEG signals employing attention driven neural networks. Biomed Signal Process Control 75. https://doi.org/10.1016/j.bspc.2022.103547

  9. Asraf A, Islam MZ, Haque MR, Islam MM (2020) Deep learning applications to combat novel coronavirus (COVID-19) pandemic. SN Comput Sci 1:363

    Article  Google Scholar 

  10. Awais M, Raza M, Singh N et al (2021) LSTM-based emotion detection using physiological signals: IoT framework for healthcare and distance learning in COVID-19. IEEE Internet Things J 8:16863–16871. https://doi.org/10.1109/JIOT.2020.3044031

    Article  Google Scholar 

  11. Ayon SI, Islam MM, Hossain MR (2020) Coronary artery heart disease prediction: a comparative study of computational intelligence techniques. IETE J Res. https://doi.org/10.1080/03772063.2020.1713916

  12. Bagherzadeh S, Maghooli K, Shalbaf A, Maghsoudi A (2022) Recognition of emotional states using frequency effective connectivity maps through transfer learning approach from electroencephalogram signals. Biomed Signal Process Control 75. https://doi.org/10.1016/j.bspc.2022.103544

  13. Balconi M, Mazza G (2009) Brain oscillations and BIS/BAS (behavioral inhibition/activation system) effects on processing masked emotional cues. ERS/ERD and coherence measures of alpha band. Int J Psychophysiol 74:158–165. https://doi.org/10.1016/j.ijpsycho.2009.08.006

    Article  Google Scholar 

  14. Bao G, Yang K, Tong L et al (2022) Linking Multi-Layer Dynamical GCN With Style-Based Recalibration CNN for EEG-Based Emotion Recognition. Front Neurorobot 16. https://doi.org/10.3389/fnbot.2022.834952

  15. Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2:1–27. https://doi.org/10.1561/2200000006

    Article  MATH  Google Scholar 

  16. Bethel CL, Salomon K, Murphy RR, Burke JL (2007) Survey of psychophysiology measurements applied to human-robot interaction. In: Proceedings - IEEE International Workshop on Robot and Human Interactive Communication

  17. Bhardwaj H, Tomar P, Sakalle A, Ibrahim W (2021) EEG-based personality prediction using fast Fourier transform and DeepLSTM model. Comput Intell Neurosci 2021. https://doi.org/10.1155/2021/6524858

  18. Chanel G, Kierkels JJM, Soleymani M, Pun T (2009) Short-term emotion assessment in a recall paradigm. Int J Hum Comput Stud 67:607–627. https://doi.org/10.1016/j.ijhcs.2009.03.005

    Article  Google Scholar 

  19. Chang S, Dong W, Jun H (2020) Use of electroencephalogram and long short-term memory networks to recognize design preferences of users toward architectural design alternatives. J Comput Des Eng 7:551–562. https://doi.org/10.1093/jcde/qwaa045

    Article  Google Scholar 

  20. Chao H, Dong L (2021) Emotion recognition using three-dimensional feature and convolutional neural network from multichannel EEG signals. IEEE Sensors J 21:2024–2034. https://doi.org/10.1109/JSEN.2020.3020828

    Article  Google Scholar 

  21. Chao H, Liu Y (2020) Emotion recognition from Multi-Channel EEG signals by exploiting the deep belief-conditional random field framework. IEEE Access 8:33002–33012. https://doi.org/10.1109/ACCESS.2020.2974009

    Article  Google Scholar 

  22. Chao H, Zhi H, Dong L, Liu Y (2018) Recognition of emotions using multichannel EEG data and DBN-GC-based ensemble deep learning framework. Computational intelligence and neuroscience, 2018. https://doi.org/10.1155/2018/9750904

  23. Chen T, Ju S, Yuan X et al (2018) Emotion recognition using empirical mode decomposition and approximation entropy. Comput Electr Eng 72:383–392. https://doi.org/10.1016/j.compeleceng.2018.09.022

    Article  Google Scholar 

  24. Chen JX, Zhang PW, Mao ZJ et al (2019) Accurate EEG-based emotion recognition on combined features using deep convolutional neural networks. IEEE Access 7:44317–44328. https://doi.org/10.1109/ACCESS.2019.2908285

    Article  Google Scholar 

  25. Chen JX, Jiang DM, Zhang YN (2019) A hierarchical bidirectional GRU model with attention for EEG-based emotion classification. IEEE Access 7:118530–118540. https://doi.org/10.1109/ACCESS.2019.2936817

    Article  Google Scholar 

  26. Chen J, Jiang D, Zhang Y, Zhang P (2020) Emotion recognition from spatiotemporal EEG representations with hybrid convolutional recurrent neural networks via wearable multi-channel headset. Comput Commun 154:58–65. https://doi.org/10.1016/j.comcom.2020.02.051

    Article  Google Scholar 

  27. Chen Y, Chang R, Guo J (2021) Effects of data augmentation method borderline-SMOTE on emotion recognition of EEG signals based on convolutional neural network. IEEE Access 9:47491–47502. https://doi.org/10.1109/ACCESS.2021.3068316

    Article  Google Scholar 

  28. Dar MN, Akram MU, Yuvaraj R et al (2022) EEG-based emotion charting for Parkinson’s disease patients using Convolutional Recurrent Neural Networks and cross dataset learning. Comput Biol Med 144:105327. https://doi.org/10.1016/j.compbiomed.2022.105327

  29. Das S, Sadi MS, Haque MA, Islam MM (2019) A Machine Learning Approach to Protect Electronic Devices from Damage Using the Concept of Outlier. In: 1st International Conference on Advances in Science, Engineering and Robotics Technology 2019, ICASERT 2019

  30. Du X, Ma C, Zhang G et al (2020) An efficient LSTM network for emotion recognition from multichannel EEG signals. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2020.3013711

  31. Du G, Wang Z, Gao B et al (2021) A convolution bidirectional long short-term memory neural network for driver emotion recognition. IEEE Trans Intell Transp Syst 22:4570–4578. https://doi.org/10.1109/TITS.2020.3007357

    Article  Google Scholar 

  32. Duan RN, Zhu JY, Lu BL (2013) Differential entropy feature for EEG-based emotion classification. In: International IEEE/EMBS Conference on Neural Engineering, NER

  33. Ekman P (1999) Basic emotions. Handbook of cognition and emotion 98:16

  34. Fang WC, Wang KY, Fahier N, et al (2019) Development and validation of an EEG-based real-time emotion recognition system using edge AI computing platform with convolutional neural network system-on-Chip Design. In: IEEE Journal on Emerging and Selected Topics in Circuits and Systems. Institute of Electrical and Electronics Engineers Inc., pp. 645–657

  35. Feng L, Cheng C, Zhao M et al (2022) EEG-based emotion recognition using spatial-temporal graph convolutional LSTM with attention mechanism. IEEE J Biomed Health Inform. https://doi.org/10.1109/JBHI.2022.3198688

  36. Ferdib-Al-Islam, Akter L, Islam MM (2021) Hepatocellular carcinoma Patient’s survival prediction using oversampling and machine learning techniques. In: International Conference on Robotics, Electrical and Signal Processing Techniques

  37. Fox E (2008) Emotion science cognitive and neuroscientific approaches to understanding human emotions. Palgrave Macmillan

    Book  Google Scholar 

  38. Ganapathy N, Veeranki YR, Kumar H, Swaminathan R (2021) Emotion recognition using electrodermal activity signals and multiscale deep convolutional neural network. J Med Syst 45(4):1–10. https://doi.org/10.1007/s10916-020-01676-6

  39. Gao Z, Li Y, Yang Y et al (2020) A coincidence-filtering-based approach for CNNs in EEG-based recognition. IEEE Trans Industr Inform 16:7159–7167. https://doi.org/10.1109/TII.2019.2955447

    Article  Google Scholar 

  40. Gao Z, Wang X, Yang Y et al (2021) A channel-fused dense convolutional network for EEG-based emotion recognition. IEEE Trans Cogn Dev Syst 13:945–954. https://doi.org/10.1109/TCDS.2020.2976112

    Article  Google Scholar 

  41. Garg D, Verma GK (2020) Emotion recognition in valence-arousal space from multi-channel EEG data and wavelet based deep learning framework. Procedia Comput Sci 171:857–867

    Article  Google Scholar 

  42. Garg D, Verma GK (2021) An improved DCNN based facial Micro-expression recognition system

  43. Garg S, Patro RK, Behera S et al (2021) An overlapping sliding window and combined features based emotion recognition system for EEG signals. Appl Comput Inf. https://doi.org/10.1108/ACI-05-2021-0130

  44. Garg D, Verma GK, Singh AK (2022) Modelling and statistical analysis of emotions in 3D space. Eng Res Express 4:035062. https://doi.org/10.1088/2631-8695/ac93e8

  45. Ghosh L, Saha S, Konar A (2020) Bi-directional Long Short-Term Memory model to analyze psychological effects on gamers. Appl Soft Comput J 95. https://doi.org/10.1016/j.asoc.2020.106573

  46. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT press

    MATH  Google Scholar 

  47. Gunes H, Schuller B, Pantic M, Cowie R (2011) Emotion representation, analysis and synthesis in continuous space: a survey. In: 2011 IEEE international conference on automatic face and gesture recognition and workshops, FG 2011

  48. Haque MR, Islam MM, Iqbal H, et al (2018) Performance evaluation of random forests and artificial neural networks for the classification of liver disorder. In: international conference on computer, communication, chemical, material and electronic engineering, IC4ME2 2018

  49. Hassan MM, Alam MGR, Uddin MZ et al (2019) Human emotion recognition using deep belief network architecture. Information Fusion 51:10–18. https://doi.org/10.1016/j.inffus.2018.10.009

    Article  Google Scholar 

  50. Hinton GE, Osindero SA (n.d.) Fast Learning Algorithm for Deep Belief Nets Yee-Whye Teh

  51. Hinton GE, Krizhevsky A, Wang SD (2011) Transforming auto-encoders. In: International conference on artificial neural networks. pp. 44–51

  52. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9:1735–1780

    Article  Google Scholar 

  53. Hockenbury DH, Hockenbury SE (2010) Discovering psychology. Macmillan

    Google Scholar 

  54. Hssayeni MD, Ghoraani B (2021) Multi-modal physiological data fusion for affect estimation using deep learning. IEEE Access 9:21642–21652. https://doi.org/10.1109/ACCESS.2021.3055933

    Article  Google Scholar 

  55. Hu J, Wang C, Jia Q et al (2021) ScalingNet: extracting features from raw EEG data for emotion recognition. Neurocomputing 463:177–184. https://doi.org/10.1016/j.neucom.2021.08.018

    Article  Google Scholar 

  56. Huang D, Guan C, Ang KK, et al (2012) Asymmetric spatial pattern for EEG-based emotion detection. In: Proceedings of the International Joint Conference on Neural Networks

  57. Huang H, Hu Z, Wang W, Wu M (2019) Multimodal emotion recognition based on ensemble convolutional neural network. IEEE Access

  58. Hwang S, Hong K, Son G, Byun H (2019) Learning CNN features from DE features for EEG-based emotion recognition. Pattern Anal Applic 23:1323–1335

    Article  Google Scholar 

  59. Hwang S, Hong K, Son G, Byun H (2020) Learning CNN features from DE features for EEG-based emotion recognition. Pattern Anal Applic 23:1323–1335. https://doi.org/10.1007/s10044-019-00860-w

    Article  Google Scholar 

  60. Islam Ayon S, Md MI (2019) Diabetes prediction: a deep learning approach. Int J Inf Eng Electron Business 11:21–27. https://doi.org/10.5815/ijieeb.2019.02.03

    Article  Google Scholar 

  61. Islam M, Tayan O, Islam R et al (2020) Deep Learning Based Systems Developed for Fall Detection: A Review. IEEE Access 8:166117–166137

    Article  Google Scholar 

  62. Islam MM, Islam MZ, Asraf A, Ding W (2020) Diagnosis of COVID-19 from X-rays using combined CNN-RNN architecture with transfer learning. medRxiv

  63. Islam MZ, Islam MM, Asraf A (2020) A combined deep CNN-LSTM network for the detection of novel coronavirus (COVID-19) using X-ray images. Inform Med Unlocked 20. https://doi.org/10.1016/j.imu.2020.100412

  64. Islam MM, Haque MR, Iqbal H et al (2020) Breast Cancer prediction: a comparative study using machine learning techniques. SN Comput Sci 1. https://doi.org/10.1007/s42979-020-00305-w

  65. Islam MR, Islam MM, Rahman MM et al (2021) EEG Channel Correlation Based Model for Emotion Recognition. Comput Biol Med 136. https://doi.org/10.1016/j.compbiomed.2021.104757

  66. Jatupaiboon N, Pan-Ngum S, Israsena P (2013) Emotion classification using minimal EEG channels and frequency bands. In: Proceedings of the 2013 10th international joint conference on computer science and software engineering, JCSSE 2013. pp 21–24

  67. Jeong DK, Kim HG, Kim JY (2022) Automated video classification system driven by characteristics of emotional human brainwaves caused by audiovisual stimuli. IEEE Trans Cogn Dev Syst 8920:1–11. https://doi.org/10.1109/TCDS.2022.3179427

    Article  Google Scholar 

  68. Jirayucharoensak S, Pan-Ngum S, Israsena P (2014) EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. Sci World J 2014. https://doi.org/10.1155/2014/627892

  69. Joshi VM, Ghongade RB (2021) EEG based emotion detection using fourth order spectral moment and deep learning. Biomed Signal Process Control 68. https://doi.org/10.1016/j.bspc.2021.102755

  70. Katsigiannis S, Ramzan N (2018) DREAMER: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J Biomed Health Inform 22:98–107. https://doi.org/10.1109/JBHI.2017.2688239

    Article  Google Scholar 

  71. Kim BH, Jo S (2018) Deep physiological affect network for the recognition of human emotions. IEEE Trans Affect Comput 11:230–243

  72. Kim MK, Kim M, Oh E, Kim SP (2013) A review on the computational methods for emotional state estimation from the human EEG. Comput Math Methods Med 2013:573734

    Article  MathSciNet  MATH  Google Scholar 

  73. Koelstra S, Mühl C, Soleymani M et al (2012) DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3:18–31. https://doi.org/10.1109/T-AFFC.2011.15

    Article  Google Scholar 

  74. Kwon YH, Shin SB, Kim SD (2018) Electroencephalography based fusion two-dimensional (2D)-convolution neural networks (CNN) model for emotion recognition system. Sensors (Switzerland) 18. https://doi.org/10.3390/s18051383

  75. LeCun Y, Bengio Y, Hinton G (2015) Deep learning nature 521

  76. Lee TMC, Liu HL, Chan CCH et al (2005) Neural activities associated with emotion recognition observed in men and women. Mol Psychiatry 10:450–455. https://doi.org/10.1038/sj.mp.4001595

  77. Li R, Liu Z (2020) Stress detection using deep neural networks. BMC Med Inform Decis Mak 20. https://doi.org/10.1186/s12911-020-01299-4

  78. Li M, Lu B-L (2009) Emotion classification based on gamma-band EEG. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, pp 1223–1226. https://doi.org/10.1109/IEMBS.2009.5334139

  79. Li X, Zhang P, Song D, Hou Y (2015) Recognizing emotions based on multimodal neurophysiological signals. Advances in computational psychophysiology 28–30

  80. Li M-A, Zhang M, Sun Y-J (2016) A novel motor imagery EEG recognition method based on deep learning. In: 2016 international forum on management, education and information technology application

  81. Li Y, Huang J, Zhou H, Zhong N (2017) Human emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks. Appl Sci (Switzerland) 7. https://doi.org/10.3390/app7101060

  82. Li X, Song D, Zhang P, et al (2017) Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. In: Proceedings - 2016 IEEE international conference on bioinformatics and biomedicine, BIBM 2016. Institute of Electrical and Electronics Engineers Inc., pp 352–359

  83. Li J, Zhang Z, He H (2018) Hierarchical convolutional neural networks for EEG-based emotion recognition. Cogn Comput 10:368–380. https://doi.org/10.1007/s12559-017-9533-x

    Article  Google Scholar 

  84. Li C, Bao Z, Li L, Zhao Z (2020) Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition. Inf Process Manag 57. https://doi.org/10.1016/j.ipm.2019.102185

  85. Li X, Zhao Z, Song D et al (2020) Latent factor decoding of Multi-Channel EEG for emotion recognition through autoencoder-like neural networks. Front Neurosci 14. https://doi.org/10.3389/fnins.2020.00087

  86. Li Y, Wang L, Zheng W et al (2021) A novel bi-hemispheric discrepancy model for EEG emotion recognition. IEEE Trans Cogn Dev Syst 13:354–367. https://doi.org/10.1109/TCDS.2020.2999337

    Article  Google Scholar 

  87. Li C, Zhang Z, Song R et al (2021) EEG-based emotion recognition via neural architecture search. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2021.3130387

  88. Liang Z, Zhou R, Zhang L et al (2021) EEGFuseNet: hybrid unsupervised deep feature characterization and fusion for high-dimensional EEG with an application to emotion recognition. IEEE Trans Neural Syst Rehabilitation Eng 29:1913–1925. https://doi.org/10.1109/TNSRE.2021.3111689

    Article  Google Scholar 

  89. Lin Q, Ye S, Huang X, et al (2016) Classification of epileptic EEG signals with stacked sparse autoencoder based on deep learning. In: International Conference on Intelligent Computing. pp. 802–810

  90. Lin W, Li C, Sun S (2017) Deep convolutional neural network for emotion recognition using EEG and peripheral physiological signal

  91. Liu Y, Sourina O, Nguyen MK (2011) Real-time EEG-based emotion recognition and its applications. In: Transactions on computational science XII. Springer, pp. 256–277

  92. Liu M, Wu W, Gu Z et al (2018) Deep learning based on batch normalization for P300 signal detection. Neurocomputing 275:288–297. https://doi.org/10.1016/j.neucom.2017.08.039

    Article  Google Scholar 

  93. Liu J, Wu G, Luo Y et al (2020) EEG-Based Emotion Classification Using a Deep Neural Network and Sparse Autoencoder. Front Syst Neurosci 14. https://doi.org/10.3389/fnsys.2020.00043

  94. Liu S, Wang X, Zhao L et al (2021) 3DCANN: a Spatio-temporal convolution attention neural network for EEG emotion recognition. IEEE J Biomed Health Inform. https://doi.org/10.1109/JBHI.2021.3083525

  95. Marg E (1995) DESCARTES’ERROR: emotion, reason, and the human brain. Optom Vis Sci 72:847–848

    Article  Google Scholar 

  96. Mauss IB, Robinson MD (2009) Measures of emotion: a review. Cognit Emot 23:209–237

    Article  Google Scholar 

  97. Miranda-Correa JA, Patras I (2018) A multi-task cascaded network for prediction of affect, personality, mood and social context using EEG signals. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, pp 373–380. https://doi.org/10.1109/FG.2018.00060

  98. Miranda-Correa JA, Abadi MK, Sebe N, Patras I (2021) AMIGOS: a dataset for affect, personality and mood research on individuals and groups. IEEE Trans Affect Comput 12:479–493. https://doi.org/10.1109/TAFFC.2018.2884461

    Article  Google Scholar 

  99. Mishra A, Ranjan P, Ujlayan A (2020) Empirical analysis of deep learning networks for affective video tagging. Multimed Tools Appl 79:18611–18626. https://doi.org/10.1007/s11042-020-08714-y

    Article  Google Scholar 

  100. Mousavinasr S, Pourmohammad A, Saffari M (2019) Providing a four-layer method based on deep belief network to improve emotion recognition in electroencephalography in brain signals. J Med Signals Sens 9:77–87. https://doi.org/10.4103/jmss.JMSS_34_17

    Article  Google Scholar 

  101. Muhammad LJ, Islam MM, Usman SS, Ayon SI (2020) Predictive data mining models for novel coronavirus (COVID-19) infected patients’ recovery. SN Comput Sci 1. https://doi.org/10.1007/s42979-020-00216-w

  102. Nasr M, Islam MM, Shehata S et al (2021) Smart healthcare in the age of AI: recent advances, challenges, and future prospects. IEEE Access 9. https://doi.org/10.1109/ACCESS.2021.3118960

  103. Nathan K, Contreras-Vidal JL (2016) Negligible motion artifacts in scalp electroencephalography (EEG) during treadmill walking. Front Hum Neurosci 9:708

    Article  Google Scholar 

  104. Niedermeyer E, da Silva FHL (2005) Electroencephalography: basic principles, clinical applications, and related fields. Lippincott Williams & Wilkins

    Google Scholar 

  105. Ozdemir MA, Degirmenci M, Guren O, Akan A (2019) EEG based emotional state estimation using 2-D deep learning technique. In: TIPTEKNO 2019 - tip Teknolojileri Kongresi

  106. Ozdemir MA, Degirmenci M, Izci E, Akan A (2021) EEG-based emotion recognition with deep convolutional neural networks. Biomed Tech 66:43–57. https://doi.org/10.1515/bmt-2019-0306

    Article  Google Scholar 

  107. Park KS, Choi H, Lee KJ et al (2011) Emotion recognition based on the asymmetric left and right activation. Int J Med Med Sci 3:201–209

    Google Scholar 

  108. Plass D, Bos O, Bos DO (n.d.) EEG-based emotion recognition EEG-based Emotion Recognition The Influence of Visual and Auditory Stimuli

  109. Plutchik R (2001) The nature of emotions: human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. Am Sci 89

  110. Posner J, Russell JA, Peterson BS (2005) The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev Psychopathol 17:715–734. https://doi.org/10.1017/S0954579405050340

    Article  Google Scholar 

  111. Rahman MM, Manik MMH, Islam MM, et al (2020) An automated system to limit COVID-19 using facial mask detection in smart city network. In: IEMTRONICS 2020 - international IOT, electronics and mechatronics conference, proceedings

  112. Rahman MM, Islam MM, Manik MMH et al (2021) Machine learning approaches for tackling novel coronavirus (COVID-19) pandemic. SN Comput Sci 2:384

    Article  Google Scholar 

  113. Ramzan M, Dawn S (2021) Fused CNN-LSTM deep learning emotion recognition model using electroencephalography signals. Int J Neurosci. https://doi.org/10.1080/00207454.2021.1941947

  114. Ribas GC (2010) The cerebral sulci and gyri. Neurosurg Focus 28. https://doi.org/10.3171/2009.11.FOCUS09245

  115. Rosalind WP (2000) Affective_Computing. MIT Press

    Google Scholar 

  116. Rosenblatt F (1958) The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev 65:386

    Article  Google Scholar 

  117. Saha P, Sadi MS, Islam MM (2021) EMCNet: automated COVID-19 diagnosis from X-ray images using convolutional neural network and ensemble of machine learning classifiers. Inform Med Unlocked 22. https://doi.org/10.1016/j.imu.2020.100505

  118. Sakalle A, Tomar P, Bhardwaj H et al (2021) A LSTM based deep learning network for recognizing emotions using wireless brainwave driven system. Expert Syst Appl 173. https://doi.org/10.1016/j.eswa.2020.114516

  119. Salama ES, El-Khoribi RA, Shoman ME, Wahby Shalaby MA (2018) EEG-based emotion recognition using 3D convolutional neural networks. Int J Adv Comput Sci Appl 9:329–337. https://doi.org/10.14569/ijacsa.2018.090843

    Article  Google Scholar 

  120. Salama ES, El-Khoribi RA, Shoman ME, Wahby Shalaby MA (2021) A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition. Egypt Inform J 22:167–176. https://doi.org/10.1016/j.eij.2020.07.005

    Article  Google Scholar 

  121. Samavat A, Khalili E, Ayati B, Ayati M (2022) Deep learning model with adaptive regularization for EEG-based emotion recognition using temporal and frequency features. IEEE Access 10:24520–24527. https://doi.org/10.1109/ACCESS.2022.3155647

    Article  Google Scholar 

  122. Sarkar P, Etemad A (2020) Self-supervised ECG representation learning for emotion recognition. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2020.3014842

  123. Şengür D, Siuly S (2020) Efficient approach for EEG-based emotion recognition. Electron Lett 56. https://doi.org/10.1049/el.2020.2685

  124. Shu Y, Wang S (2017) Emotion recognition through integrating EEG and peripheral signals. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

  125. Siddharth S, Jung T-P, Sejnowski TJ (2019) Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing. IEEE Trans Affect Comput

  126. Soleymani M, Lichtenauer J, Pun T, Pantic M (2012) A multimodal database for affect recognition and implicit tagging. IEEE Trans Affect Comput 3:42–55. https://doi.org/10.1109/T-AFFC.2011.25

    Article  Google Scholar 

  127. Soleymani M, Asghari-Esfeden S, Fu Y, Pantic M (2015) Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Trans Affect Comput 7:17–28

    Article  Google Scholar 

  128. Song T, Zheng W, Lu C et al (2019) MPED: A multi-modal physiological emotion database for discrete emotion recognition. IEEE Access 7:12177–12191. https://doi.org/10.1109/ACCESS.2019.2891579

    Article  Google Scholar 

  129. Suhaimi NS, Mountstephens J, Teo J (2020) EEG-Based Emotion Recognition: A State-of-the-Art Review of Current Trends and Opportunities. Comput Intell Neurosci 2020:8875426

    Article  Google Scholar 

  130. Teplan M et al (2002) Fundamentals of EEG measurement. Meas Sci Rev 2:1–11

    Google Scholar 

  131. Topic A, Russo M, Stella M, Saric M (2022) Emotion Recognition Using a Reduced Set of EEG Channels Based on Holographic Feature Maps. Sensors 22. https://doi.org/10.3390/s22093248

  132. Torres PEP, Torres EA, Hernández-Álvarez M, Yoo SG (2020) EEG-based BCI emotion recognition: A survey. Sensors (Switzerland) 20. https://doi.org/10.3390/s20185083

  133. Tripathi S, Acharya S, Sharma RD, et al (2017) Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset. In: Twenty-Ninth IAAI Conference

  134. Wang D, Shang Y (2013) Modeling physiological data with deep belief networks. Int J Inf Educ Technol 3:505–511. https://doi.org/10.7763/IJIET.2013.V3.326

    Article  Google Scholar 

  135. Wang Y, Qiu S, Li J, et al (2019) EEG-Based Emotion Recognition with Similarity Learning Network; EEG-based emotion recognition with similarity learning network

  136. Wang Y, Zhang L, Xia P et al (2022) EEG-based emotion recognition using a 2D CNN with different kernels. Bioengineering 9. https://doi.org/10.3390/bioengineering9060231

  137. Wang H, Zhu X, Chen P et al (2022) A gradient-based automatic optimization CNN framework for EEG state recognition. J Neural Eng 19. https://doi.org/10.1088/1741-2552/ac41ac

  138. Wankhade SB, Doye DD (2020) Deep learning of empirical mean curve decomposition-wavelet decomposed EEG signal for emotion recognition. Int J Uncertain Fuzziness Knowl-Based Syst 28:153–177. https://doi.org/10.1142/S0218488520500075

    Article  Google Scholar 

  139. Wilaiprasitporn T, Ditthapron A, Matchaparn K et al (2020) Affective EEG-based person identification using the deep learning approach. IEEE Trans Cogn Dev Syst 12:486–496. https://doi.org/10.1109/TCDS.2019.2924648

  140. Wu Y, Xia M, Nie L et al (2022) Simultaneously exploring multi-scale and asymmetric EEG features for emotion recognition. Comput Biol Med 149. https://doi.org/10.1016/j.compbiomed.2022.106002

  141. Xiao G, Shi M, Ye M et al (2022) 4D attention-based neural network for EEG emotion recognition. Cogn Neurodyn 16:805–818. https://doi.org/10.1007/s11571-021-09751-5

  142. Xing X, Li Z, Xu T et al (2019) SAE+LSTM: a new framework for emotion recognition from multi-channel EEG. Front Neurorobot 13. https://doi.org/10.3389/fnbot.2019.00037

  143. Xu H, Plataniotis KN (2017) Affective states classification using EEG and semi-supervised deep learning approaches. In: 2016 IEEE 18th international workshop on multimedia signal processing, MMSP 2016

  144. Yanagimoto M, Sugimoto C (2016) Recognition of persisting emotional valence from EEG using convolutional neural networks. In: 2016 IEEE 9th International Workshop on Computational Intelligence and Applications (IWCIA). IEEE, pp 27–32

  145. Yang Y, Wu Q, Qiu M et al (2018) Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. In: 2018 international joint conference on neural networks (IJCNN). IEEE, pp 1–7

  146. Yang Y, Wu Q, Qiu M, et al (2018) Emotion recognition from Multi-Channel EEG through parallel convolutional recurrent neural network. In: Proceedings of the International Joint Conference on Neural Networks

  147. Yang H, Han J, Min K (2019) A multi-column CNN model for emotion recognition from EEG signals. Sensors (Switzerland) 19. https://doi.org/10.3390/s19214736

  148. Yang K, Wang C, Gu Y et al (2021) Behavioral and physiological signals-based deep multimodal approach for Mobile emotion recognition. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2021.3100868

  149. Yang Y, Gao Q, Song Y et al (2022) Investigating of deaf emotion cognition pattern by EEG and facial expression combination. IEEE J Biomed Health Inform 26:589–599. https://doi.org/10.1109/JBHI.2021.3092412

    Article  Google Scholar 

  150. Yin Y, Zheng X, Hu B et al (2021) EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM. Appl Soft Comput 100. https://doi.org/10.1016/j.asoc.2020.106954

  151. Zeng H, Wu Z, Zhang J et al (2019) EEG emotion classification using an improved sincnet-based deep learning model. Brain Sci 9. https://doi.org/10.3390/brainsci9110326

  152. Zhang J, Wu Y (2018) Complex-valued unsupervised convolutional neural networks for sleep stage classification. Comput Methods Prog Biomed 164:181–191. https://doi.org/10.1016/j.cmpb.2018.07.015

    Article  Google Scholar 

  153. Zhang T, Zheng W, Cui Z et al (2018) Spatial--temporal recurrent neural network for emotion recognition. IEEE Trans Cybern 49:839–847

    Article  Google Scholar 

  154. Zhang Y, Chen J, Tan JH et al (2020) An Investigation of Deep Learning Models for EEG-Based Emotion Recognition. Front Neurosci 14. https://doi.org/10.3389/fnins.2020.622759

  155. Zhang P, Min C, Zhang K et al (2021) Hierarchical Spatiotemporal Electroencephalogram Feature Learning and Emotion Recognition With Attention-Based Antagonism Neural Network. Front Neurosci 15. https://doi.org/10.3389/fnins.2021.738167

  156. Zhang Y, Cheng C, Zhang YD (2022) Multimodal emotion recognition based on manifold learning and convolution neural network. Multimed Tools Appl. https://doi.org/10.1007/s11042-022-13149-8

  157. Zheng WL, Lu BL (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev 7:162–175. https://doi.org/10.1109/TAMD.2015.2431497

    Article  Google Scholar 

  158. Zheng W-L, Zhu J-Y, Peng Y, Lu B-L (2014) EEG-based emotion classification using deep belief networks. In: 2014 IEEE international conference on multimedia and expo (ICME). pp 1–6

  159. Zheng X, Yu X, Yin Y et al (2021) Three-dimensional feature maps and convolutional neural network-based emotion recognition. Int J Intell Syst 36:6312–6336. https://doi.org/10.1002/int.22551

    Article  Google Scholar 

  160. Zhong Q, Zhu Y, Cai D et al (2020) Electroencephalogram Access for Emotion Recognition Based on a Deep Hybrid Network. Front Hum Neurosci 14. https://doi.org/10.3389/fnhum.2020.589001

  161. Zhou L (2021) Analysis of Psychological and Emotional Tendency Based on Brain Functional Imaging and Deep Learning. Discret Dyn Nat Soc 2021. https://doi.org/10.1155/2021/1272502

  162. Zhu J-Y, Zheng W-L, Lu B-L (2015) Cross-subject and cross-gender emotion classification from EEG. In: world congress on medical physics and biomedical engineering, June 7-12, 2015, Toronto, Canada. pp. 1188–1191

  163. Zhu M, Wang Q, Luo J (2022) Emotion Recognition Based on Dynamic Energy Features Using a Bi-LSTM Network. Front Comput Neurosci 15. https://doi.org/10.3389/fncom.2021.741086

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, National Institute of Technology Kurukshetra, Kurukshetra, Haryana, India

    Divya Garg & Awadhesh Kumar Singh

  2. Department of Information Technology, National Institute of Technology Raipur, Raipur, Chhattisgarh, India

    Gyanendra Kumar Verma

Authors
  1. Divya Garg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gyanendra Kumar Verma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Awadhesh Kumar Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Divya Garg.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Garg, D., Verma, G.K. & Singh, A.K. A review of Deep Learning based methods for Affect Analysis using Physiological Signals. Multimed Tools Appl 82, 26089–26134 (2023). https://doi.org/10.1007/s11042-023-14354-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14354-9

Keywords

Search

Navigation