Skip to main content

EEG-based emotion recognition via capsule network with channel-wise attention and LSTM models

Abstract

Emotion is a kind of psychological and physical state that people produce to objective things. Accurate recognition of emotions is very important in the field of human-machine interface. It is still challenging to extract discriminative features used for electroencephalogram (EEG) emotion recognition which contain the subtle spatial feature of EEG signal and the temporal representations of EEG signal. In order to overcome these challenges, the paper proposes a model based on the capsule network for multi-channel EEG emotion recognition, which combines the attention mechanism and the LSTM network. First, the channel-wise attention mechanism is used to adaptively assign different weights to each channel, then the CapsNet is used to extract the spatial features of the EEG channel, and LSTM is used to extract temporal features of the EEG sequences. The paper proposed method achieves average accuracy of 97.17%, 97.34% and 96.50% on the valence, arouse and dominance of the DEAP dataset, respectively. These results show that the proposed method is better than the current state-of-the-art method.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

References

  1. Alhagry, S., Fahmy, A.A., El-Khoribi, R.A.: Emotion recognition based on EEG using LSTM recurrent neural network. Emotion 8(10), 355–358 (2017)

    Google Scholar 

  2. Anuradha, R., Saranya, N., Priyadharsini, M., et al.: Assessment of extended MNIST (EMNIST) dataset using capsule networks. In: 2019 International Conference on Intelligent Sustainable Systems (ICISS), pp. 263–266 (2019)

  3. Catarina, C.K., Izelle, L., Sophie, C.A., et al.: Diminished facial EMG responses to disgusting scenes and happy and fearful faces in Huntington’s disease. Cortex 106, 185–99 (2018)

    Article  Google Scholar 

  4. Chen, J., Zhang, P., Mao, Z., et al.: Accurate EEG-based emotion recognition on combined features using deep convolutional neural networks. IEEE Access 7, 44317–44328 (2019)

    Article  Google Scholar 

  5. Christensen, L., Abdullah, M.: EEG Emotion Detection Review, pp. 1–7 (2018)

  6. Du, X., Ma, C., Zhang, G., et al.: An efficient LSTM network for emotion recognition from multi-channel EEG signals. IEEE Trans. Affect. Comput. 99, 1–1 (2020). https://doi.org/10.1109/TAFFC.2020.3013711

    Article  Google Scholar 

  7. Ekman, P.: An argument for basic emotions. Cognit. Emotion 6, 169–200 (1992)

    Article  Google Scholar 

  8. Fabiano, D., Canavan, S.: Emotion recognition using fused physiological signals. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), Cambridge, pp. 42–48. IEEE (2019)

  9. Goshvarpour, A.: Evaluation of novel entropy-based complex wavelet sub-bands measures of PPG in an emotion recognition system. J. Med. Biol. Eng. 40(3), 451–461 (2020)

    Article  Google Scholar 

  10. Goshvarpour, A., Goshvarpour, A.: The potential of photoplethysmogram and galvanic skin response in emotion recognition using nonlinear features. Australas. Phys. Eng. Sci. Med. 1, 43 (2020)

    Google Scholar 

  11. Hassan, M.M., Alam, M.G.R., Uddin, M.Z., et al.: Human emotion recognition using deep belief network architecture. Int. J. Inf. Fusion 51, 10–18 (2019)

    Article  Google Scholar 

  12. Hemanth, D.J.: EEG signal based modified Kohonen neural networks for classification of human mental emotions. J. Artif. Intell. Syst. 2, 1–13 (2020)

    Article  Google Scholar 

  13. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  14. Hollosi, J., Pozna, C.R.: Improve the accuracy of neural networks using capsule layers. In: 2018 IEEE 18th International Symposium on Computational Intelligence and Informatics (CINTI), pp. 15–18 (2018)

  15. Huang, H., Hu, Z., Wang, W., et al.: Multimodal emotion recognition based on ensemble convolutional neural network. IEEE Access 8, 3265–3271 (2020)

    Article  Google Scholar 

  16. Hwang, S., Ki, M., Hong, K., et al.: Subject-independent EEG-based emotion recognition using adversarial learning. In: 2020 8th International Winter Conference on Brain-Computer Interface (BCI), pp. 1–4 (2020)

  17. Jin, Y.X., Luo, Y., Yu, Y.: Research on EEG emotion recognition based on deep forest. Softw. Guide 18(07), 53–55 (2019)

    Google Scholar 

  18. Kan, W., Li, Y.: Emotion recognition from EEG signals by using LSTM recurrent neural networks. J. Nanjing Univ. (Natural Science) 55(01), 110–116 (2019)

    Google Scholar 

  19. Katsigiannis, S., Ramzan, N.: DREAMER: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J. Biomed. Health Inform. 22(1), 98–107 (2017)

    Article  Google Scholar 

  20. Keshari, T., Palaniswamy, S.: Emotion recognition using feature-level fusion of facial expressions and body gestures. In: 2019 International Conference on Communication and Electronics Systems (ICCES), pp. 1184–1189 (2019)

  21. Kim, B.H., Jo, S., et al.: Deep physiological affect network for the recognition of human emotions. IEEE Trans. Affect. Comput. 2, 230–243 (2020)

    Google Scholar 

  22. Kim, M.K., Kim, M.Y., Oh, E., et al.: A review on the computational methods for emotional state estimation from the human EEG. Comput. Math. Methods Med. 2013, 573734 (2013)

    MathSciNet  MATH  Google Scholar 

  23. Koelstra, S., Muhl, C., Soleymani, M.: Deap: a database for emotion analysis using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2011)

    Article  Google Scholar 

  24. Lee, M.S., Lee, Y.K., Pae, D.S., et al.: Fast emotion recognition based on single pulse PPG signal with convolutional neural network. Appl. Sci. 9(16), 3355 (2019)

    Article  Google Scholar 

  25. Li, X., Cai, E.J., Tian, Y.X.: An improved electroencephalogram feature extraction algorithm and its application in emotion recognition. J. Biomed. Eng. 34(4), 510–517 (2017)

    Google Scholar 

  26. Liao, J., Zhong, Q., Zhu, Y., et al.: Multimodal physiological signal emotion recognition based on convolutional recurrent neural network. MSE 782(3), 032005 (2020)

    Google Scholar 

  27. Liu, W., Qiu, J.L., Zheng, W.L., et al.: Multimodal emotion recognition using deep canonical correlation analysis (2019)

  28. Liu, Y., Ding, Y., Li, C., et al.: Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Comput. Biol. Med. 123, 103927 (2020)

    Article  Google Scholar 

  29. Luo, J.Y.: Overview of emotion recognition based on EEG signals. Mod. Comput. 08, 68–71 (2020)

    Google Scholar 

  30. Ma, J., Tang, H., Zheng, W.L., et al.: Emotion recognition using multimodal residual LSTM network. In: Proceedings of the 27th ACM International Conference on Multimedia, Nice, vol. 8, pp. 176–183. New York (2019)

  31. Ma, X., Zhong, H., Li, Y., et al.: Forecasting transportation network speed using deep capsule networks with nested LSTM models. IEEE Trans. Intell. Transport. Syst. 99, 1–12 (2020)

    Google Scholar 

  32. Mehrabian, A.: Comparison of the PAD and PANAS as models for describing emotions and for differentiating anxiety from depression. J. Psychopathol. Behav. Assess. 19(4), 331–357 (1997)

    Article  Google Scholar 

  33. Mert, A., Akan, A.: Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal. Appl. 21(1), 81–89 (2016)

    MathSciNet  Article  Google Scholar 

  34. Nie, D., Wang, X.W., Duan, R.N., et al.: A survey on EEG based emotion recognition. Chin. J. Biomed. Eng. 31(4), 595–606 (2012)

    Google Scholar 

  35. Pan, L., Yin, Z., She, S., et al.: Emotional state recognition from peripheral physiological signals using fused nonlinear features and team-collaboration identification strategy. Entropy 22(5), 511 (2020)

    Article  Google Scholar 

  36. Sabour S., Frosst N., Hinton G.E.: Dynamic routing between capsules. (2017). arXiv:1710.09829

  37. Samarth, T., Shrinivas, A., Ranti, D.S., et al.: Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI’ 17). AAAI Press, pp. 4746–4752 (2017)

  38. Sun, Z.G., Xue, Q.D., Wang, X.J., Huang, X.L.: A survey of emotion recognition method based on EEG signals. Beijing Biomed. Eng. 39(2), 186–194 (2020)

    Google Scholar 

  39. Tao, W., Li, C., Song, R., et al.: EEG-based emotion recognition via channel-wise attention and self attention[J]. IEEE Trans. Affect. Comput. Early Access (2020). https://doi.org/10.1109/TAFFC.2020.3025777

    Article  Google Scholar 

  40. Verma, G.K., Tiwary, U.S.: Affect representation and recognition in 3d continuous valence-arousal-dominance space. Multimed. Tools Appl. 76(2), 2159–2183 (2017)

    Article  Google Scholar 

  41. Wang, B.: A Research of Emotion Recognition Based on Respiratory Signals. M.A. dissertation, Southwest University (2010)

  42. Wang, Y.: Research and Application on Emotion Recognition Algorithm Based on Multi-model Eye Movement Information. M.A. dissertation, Anhui University, Anhui(2019)

  43. Wang, X.W., Nie, D., Lu, B.L.: Emotional state classification from EEG data using machine learning approach. Neurocomputing 129, 94–106 (2014)

    Article  Google Scholar 

  44. Wu, S.Y., Xu, X., Shu, L., et al.: Estimation Of valence of emotion using two frontal EEG channels. In: 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Kansas, Nov 13–16, 2017. IEEE, pp. 1127–1130 (2017)

  45. Yang, Y., Wu, Q., Qiu, M., et al.: Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. In: 2018 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 1–7 (2018)

  46. Zhang, X.N., Luo, P.C., Hu, X.W., et al.: Research on classification performance of small-scale dataset based on capsule network. In: Proceedings of the 2018 4th International Conference on Robotics and Artificial Intelligence, pp. 24–28 (2018)

  47. Zhang, D.,Yao, L., Zhang X., Wang S., et al.: Eeg-based intention recognition from spatio-temporal representations via cascade and parallel convolutional recurrent neural networks (2017). arXiv:1708.06578

  48. Zhang, G.H., Yu, M.J., Chen, G., et al.: A review of EEG features for emotion recognition (in Chinese). Sci. Sin. (Informationis) 49, 1097–1118 (2019)

    Google Scholar 

Download references

Acknowledgements

The financial support for this work provided by the Cooperative Education Fund of China Ministry of Education (201702113002, 201801193119) and the Scientific Research Fund of Hunan Provincial Education Department (20A191) are greatly appreciated by the authors.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Xiaoliang Wang.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Deng, L., Wang, X., Jiang, F. et al. EEG-based emotion recognition via capsule network with channel-wise attention and LSTM models. CCF Trans. Pervasive Comp. Interact. (2021). https://doi.org/10.1007/s42486-021-00078-y

Download citation

Keywords

  • EEG
  • Emotion recognition
  • Capsule network
  • LSTM
  • Channel-wise attention