Abstract
Electroencephalograph (EEG) emotion recognition is a significant task in the brain-computer interface field. Although many deep learning methods are proposed recently, it is still challenging to make full use of the information contained in different domains of EEG signals. In this paper, we present a novel method, called four-dimensional attention-based neural network (4D-aNN) for EEG emotion recognition. First, raw EEG signals are transformed into 4D spatial-spectral-temporal representations. Then, the proposed 4D-aNN adopts spectral and spatial attention mechanisms to adaptively assign the weights of different brain regions and frequency bands, and a convolutional neural network (CNN) is utilized to deal with the spectral and spatial information of the 4D representations. Moreover, a temporal attention mechanism is integrated into a bidirectional Long Short-Term Memory (LSTM) to explore temporal dependencies of the 4D representations. Our model achieves state-of-the-art performances on both DEAP, SEED and SEED-IV datasets under intra-subject splitting. The experimental results have shown the effectiveness of the attention mechanisms in different domains for EEG emotion recognition.
This is a preview of subscription content, log in via an institution to check access.
Availability of data and material
The dataset used is available on https://www.eecs.qmul.ac.uk/mmv/datasets/deap/ and http://bcmi.sjtu.edu.cn/home/~seed.
Code availability
The code of our model will be available after acceptance.
References
Alhagry S, Fahmy AA, El-Khoribi RA (2017) Emotion recognition based on EEG using LSTM recurrent neural network. Int J Adv Comput Sci Appl 8:335–358. https://doi.org/10.14569/IJACSA.2017.081046
Bamdad M, Zarshenas H, Auais MA (2015) Application of BCI systems in neurorehabilitation: a scoping review disability and rehabilitation. Assist Technol 10:355–364. https://doi.org/10.3109/17483107.2014.961569
Blankertz B et al (2016) The Berlin brain-computer interface: progress beyond communication and control. Front Neurosci 10:530. https://doi.org/10.3389/fnins.2016.00530
Brittona JC, Phan KL, Taylor SF, Welsh RC, Berridge KC, Liberzon I (2006) Neural correlates of social and nonsocial emotions: an fMRI study. Neuroimage 31:397–409. https://doi.org/10.1016/j.neuroimage.2005.11.027
Chattopadhay A, Sarkar A, Howlader P, Balasubramanian VN (2018) Grad-CAM++: generalized gradient-based visual explanations for deep convolutional networks. Paper presented at the 2018 IEEE winter conference on applications of computer vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018. doi:https://doi.org/10.1109/WACV.2018.00097
Craik A, He Y, Contreras-Vidal JL (2019) Deep learning for electroencephalogram (EEG) classification tasks: a review. J Neural Eng 16:031001. https://doi.org/10.1088/1741-2552/ab0ab5
Damasio AR, Grabowski TJ, Bechara A, Damasio H, Ponto LL, Parvizi J, Hichwa RD (2000) Subcortical and cortical brain activity during the feeling of self-generated emotions. Nat Neurosci 3:1049. https://doi.org/10.1038/79871
Dolan RJ (2002) Emotion, cognition, and behavior. Science 298:1191–1194. https://doi.org/10.1126/science.1076358
Duan R-N, Zhu J-Y, Lu B-L (2013) Differential entropy feature for eeg-based emotion classification. Paper presented at the 2013 6th international IEEE/EMBS conference on neural engineering (NER), San Diego, CA, USA. doi:https://doi.org/10.1109/NER.2013.6695876
Figueiredo GR, Ripka WL, Romaneli EFR, Ulbricht L (2019) Attentional bias for emotional faces in depressed and nondepressed individuals: an eye-tracking study. Paper presented at the 2019 41st annual international conference of the IEEE engineering in medicine and biology society (EMBC), Berlin, Germany, 23–27 July 2019. doi:https://doi.org/10.1109/EMBC.2019.8857878
Fiorinia L, Mancioppi G, Semeraro F, Fujita H, Cavallo F (2020) Unsupervised emotional state classification through physiological parameters for social robotics applications. Knowledge-Based Syst. https://doi.org/10.1016/j.knosys.2019.105217
Goh SK, Abbass HA, Tan KC, Al-Mamun A, Thakor N, Bezerianos A, Li J (2018) Spatio-spectral representation learning for electroencephalographic gait-pattern classification. IEEE Trans Neural Syst Rehabil Eng 26:1858–1867. https://doi.org/10.1109/TNSRE.2018.2864119
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. pp 770–778. doi:https://doi.org/10.1109/CVPR.2016.90
Jenke R, Peer A, Buss M (2014) Feature extraction and selection for emotion recognition from eeg. IEEE Trans Affect Comput 5:327–339. https://doi.org/10.1109/TAFFC.2014.2339834
Jia Z, Lin Y, Cai X, Chen H, Gou H, Wang J (2020) SST-EmotionNet: spatial-spectral-temporal based attention 3D Dense Network for EEG emotion recognition. In: Proceedings of the 28th ACM international conference on multimedia, Seattle, WA, USA, 2020. Association for Computing Machinery, pp 2909–2917. doi:https://doi.org/10.1145/3394171.3413724
Katsigiannis S, Ramzan N (2017) Dreamer: a database for emotion recognition through eeg and ecg signals from wireless low-cost off-the-shelf devices. IEEE J Biomed Health Inf 22:98–107. https://doi.org/10.1109/JBHI.2017.2688239
Kim M-K, Kim M, Oh E, Kim S-P (2013) A review on the computational methods for emotional state estimation from the human eeg. Comput Math Methods Med. https://doi.org/10.1155/2013/573734
Koelstra S, Muhl C, Soleymani M, Lee JS, Yazdani A, Ebrahimi T, Patras I (2011) Deap: a dataset for emotion analysis using physiological signals. IEEE Trans Affect Comput 3:18–31. https://doi.org/10.1109/T-AFFC.2011.15
Koven NS, Heller W, Banich MT, Miller GA (2003) Relationships of distinct affective dimensions to performance on an emotional stroop task. Cogn Ther Res 27:671–680. https://doi.org/10.1023/A:1026303828675
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, 2012. Curran Associates, Inc., pp 1097–1105. doi:https://doi.org/10.1145/3065386
Leon-Carrion J, Mcmanis MH, Castillo EM, Papanicolaou AC (2006) Time-locked brain activity associated with emotion: a pilot MEG study. Brain Injury: BI 20:857–865. https://doi.org/10.1080/02699050600832304
Li Y, Zheng W, Zong Y, Cui Z, Zhang T, Zhou X (2018) A bi-hemisphere domain adversarial neural network model for EEG emotion recognition. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2018.2885474
Li M, Lu B-L (2009) Emotion classification based on gamma-band EEG. Paper presented at the 2009 annual international conference of the IEEE engineering in medicine and biology society, Minneapolis, MN, USA. doi:https://doi.org/10.1109/IEMBS.2009.5334139
Li Y et al (2020) A novel bi-hemispheric discrepancy model for EEG emotion recognition. IEEE Trans Cogn Dev Syst. https://doi.org/10.1109/TCDS.2020.2999337
Lotfia E, Akbarzadeh-T M-R (2014) Practical emotional neural networks. Neural Netw 59:61–72. https://doi.org/10.1016/j.neunet.2014.06.012
Jiaxin Ma, Tang H, Zheng W-L, Lu B-L (2019) Emotion recognition using multimodal residual LSTM network. In: Proceedings of the 27th ACM international conference on multimedia, Nice, France. Association for computing machinery, New York, USA, pp 176–183. doi:https://doi.org/10.1145/3343031.3350871
Mühl C, Allison B, Nijholt A, Chanel G (2014) A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges. Brain-Comput Interfaces 1:66–84. https://doi.org/10.1080/2326263X.2014.912881
Pfurtscheller G et al (2010) The Hybrid BCI. Front Neurosci 4:3. https://doi.org/10.3389/fnpro.2010.00003
Shen F, Dai G, Lin G, Zhang J, Kong W, Zeng H (2020) EEG-based emotion recognition using 4D convolutional recurrent neural network. Cogn Neurodyn 14:815–828. https://doi.org/10.1007/s11571-020-09634-1
Shi L-C, Jiao Y-Y, Lu B-L (2013) Differential entropy feature for eeg-based vigilance estimation. Paper presented at the 2013 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC), Osaka, Japan, 3–7 July 2013. doi:https://doi.org/10.1109/EMBC.2013.6611075
Song T, Zheng W, Song P, Cui Z (2020) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput 11:532–541. https://doi.org/10.1109/TAFFC.2018.2817622
Tao W, Li C, Song R, Cheng J, Liu Y, Wan F, Chen X (2020) EEG-based emotion recognition via channel-wise attention and self attention. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2020.3025777
Vaswani A, et al (2017) Attention is all you need. Paper presented at the Advances in Neural Information Processing Systems. doi:https://doi.org/10.5555/3295222.3295349
Woo S, Park J, Lee J-Y, Kweon IS (2018) Cbam: convolutional block attention module. Computer Vision—ECCV 2018. Springer International Publishing, Cham. https://doi.org/10.1007/978-3-030-01234-2_1
Yan J, Zheng W, Xu Q, Lu G, Li H, Wang B (2016) Sparse kernel reduced-rank regression for bimodal emotion recognition from facial expression and speech. IEEE Trans Multimed 18:1319–1329. https://doi.org/10.1109/TMM.2016.2557721
Yang Y, Wu QMJ, Zheng W-L, Lu B-L (2018c) EEG-based emotion recognition using hierarchical network with subnetwork nodes. IEEE Trans Cogn Dev Syst 10:408–419. https://doi.org/10.1109/TCDS.2017.2685338
Yang Y, Wu Q, Fu Y, Chen X (2018a) Continuous convolutional neural network with 3D input for EEG-based emotion recognition. In: Cheng L, Leung ACS, Ozawa S (eds) Neural information processing. Springer International Publishing, pp 433–433. doi:https://doi.org/10.1007/978-3-030-04239-4_39
Yang Y, Wu Q, Qiu M, Wang Y, Chen X (2018b) Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. In: 2018 International joint conference on neural networks (IJCNN). IEEE, pp 1–7. doi:https://doi.org/10.1109/IJCNN.2018.8489331
Zheng W-L, Lu B-L (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev 7:162–175. https://doi.org/10.1109/TAMD.2015.2431497
Zheng W-L, Zhu J-Y, Lu B-L (2017) Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans Affect Comput 10:417–429. https://doi.org/10.1109/TAFFC.2017.2712143
Zheng W-L, Liu W, Lu Y, Lu B-L, Cichocki A (2018) Emotionmeter: A multimodal framework for recognizing human emotions. IEEE Trans Cybern 49:1110–1122. https://doi.org/10.1109/TCYB.2018.2797176
Zhong P, Wang D, Miao C (2020) EEG-based emotion recognition using regularized graph neural networks. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2020.2994159
Acknowledgements
Sponsored by Zhejiang Lab (No. 2021RD0AB01), and supported by High-performance Computing Platform of Peking University.
Funding
The work was sponsored by Zhejiang Lab (No. 2021RD0AB01).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Xiao, G., Shi, M., Ye, M. et al. 4D attention-based neural network for EEG emotion recognition. Cogn Neurodyn 16, 805–818 (2022). https://doi.org/10.1007/s11571-021-09751-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11571-021-09751-5