Skip to main content
SpringerLink
Log in

Feature hypergraph representation learning on spatial-temporal correlations for EEG emotion recognition

  • Research Article
  • Published:
Cognitive Neurodynamics Aims and scope Submit manuscript

Abstract

Electroencephalogram(EEG) becomes popular in emotion recognition for its capability of selectively reflecting the real emotional states. Existing graph-based methods have made primary progress in representing pairwise spatial relationships, but leaving higher-order relationships among EEG channels and higher-order relationships inside EEG series. Constructing a hypergraph is a general way of representing higher-order relations. In this paper, we propose a spatial-temporal hypergraph convolutional network(STHGCN) to capture higher-order relationships that existed in EEG recordings. STHGCN is a two-block hypergraph convolutional network, in which feature hypergraphs are constructed over the spectrum, space, and time domains, to explore spatial and temporal correlations under specific emotional states, namely the correlations of EEG channels and the dynamic relationships of temporal stamps. What’s more, a self-attention mechanism is combined with the hypergraph convolutional network to initialize and update the relationships of EEG series. The experimental results demonstrate that constructed feature hypergraphs can effectively capture the correlations among valuable EEG channels and the correlations inside valuable EEG series, leading to the best emotion recognition accuracy among the graph methods. In addition, compared with other competitive methods, the proposed method achieves state-of-art results on SEED and SEED-IV datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availibility Statement

The datasets analysed during the current study are available in the SEED repository,https://bcmi.sjtu.edu.cn/~seed/index.html.

References

  • Bagherzadeh S, Maghooli K, Shalbaf A, Maghsoudi A (2022) Emotion recognition using effective connectivity and pre-trained convolutional neural networks in EEG signals. Cogn Neurodyn 1–20

  • Bahari F, Janghorbani A (2013) EEG-based emotion recognition using recurrence plot analysis and k nearest neighbor classififier, Paper presented at 20th Iranian conference on biomedical engineering (ICBME). https://doi.org/10.1109/ICBME.2013.6782224

  • Bai S, Zhang F, Torr PH (2021) Hypergraph convolution and hypergraph attention. Pattern Recognit 110:107637

    Article  Google Scholar 

  • Deng L, Wang X, Jiang F, Doss R (2021) EEG-based emotion recognition via capsule network with channel-wise attention and lstm models. CCF Trans Pervasive Comput Interact 3(4):425–435

    Article  Google Scholar 

  • Deng X, Zhu J, Yang S (2021) Sfe-net: EEG-based emotion recognition with symmetrical spatial feature extraction. In: Proceedings of the 29th ACM international conference on multimedia, pp. 2391–2400

  • Ding Y, Robinson N, Zhang S, Zeng Q, Guan C (2021) Tsception: capturing temporal dynamics and spatial asymmetry from EEG for emotion recognition. arXiv preprint arXiv:2104.02935

  • Feng Y, You, H, Zhang, Z (2019) Hypergraph neural networks. Biomedical engineering, Paper presented at the Proceedings of the AAAI conference on artificial intelligence, 7(3), 162–175

  • Huang J, Yang J (2021) Unignn: a unified framework for graph and hypergraph neural networks. arXiv preprint arXiv:2105.00956

  • Jia Z, Lin Y, Cai X, Chen H, Gou H, Wang J (2020) Sst-emotionnet: spatial-spectral-temporal based attention 3d dense network for EEG emotion recognition. In: Proceedings of the 28th ACM international conference on multimedia, pp. 2909–2917

  • Jiang J, Wei Y, Feng Y, Cao J, Gao Y (2019) Dynamic hypergraph neural networks. In: IJCAI, pp. 2635–2641

  • Li Y, Wang L, Zheng W, Zong Y, Qi L, Cui Z, Zhang T, Song T (2020) A novel bi-hemispheric discrepancy model for EEG emotion recognition. IEEE Trans Cogn Dev Syst 13(2):354–367

    Article  Google Scholar 

  • Li Y, Zheng W, Cui Z, Zhang T, Zong Y (2018) A novel neural network model based on cerebral hemispheric asymmetry for EEG emotion recognition. In: IJCAI, pp. 1561–1567

  • Li Y, Zheng W, Wang L, Zong Y, Cui Z (2019) From regional to global brain: a novel hierarchical spatial-temporal neural network model for EEG emotion recognition. IEEE Trans Affect Comput

  • Lotfi E, Akbarzadeh-T M-R (2014) Practical emotional neural networks. Neural Netw 59:61–72

    Article  PubMed  Google Scholar 

  • Lugo-Martinez J, Radivojac P (2017) Classification in biological networks with hypergraphlet kernels. arXiv preprint arXiv:1703.04823

  • Sawhney R, Agarwal S, Wadhwa A, Derr T, Shah RR (2021) Stock selection via spatiotemporal hypergraph attention network: a learning to rank approach. In: Proceedding of AAAI, 497–504

  • Shen F, Dai G, Lin G, Zhang J, Kong W, Zeng H (2020) EEG-based emotion recognition using 4D convolutional recurrent neural network. Cogn Neurodyn 14(6):815–828

    Article  PubMed  PubMed Central  Google Scholar 

  • Song T, Zheng W, Song P, Cui Z (2018) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput 11(3):532–541

    Article  Google Scholar 

  • Tuncer T, Dogan S, Subasi A (2021) Ledpatnet19: automated emotion recognition model based on nonlinear led pattern feature extraction function using EEG signals. Cogn Neurodyn 1–12

  • Wang XW, Nie D, Lu BL (2011) Eeg-based emotion recognition using frequency domain features and support vector machines (2011). Paper presented at international conference on neural information processing

  • Wang Z, Wang Y, Hu C, Yin Z, Song Y (2022) Transformers for EEG-based emotion recognition: a hierarchical spatial information learning model. IEEE Sens J

  • Xiao G, Shi M, Ye M, Xu B, Chen Z, Ren Q (2022) 4D attention-based neural network for EEG emotion recognition. Cogn Neurodyn 1–14

  • Xia X, Yin H, Yu J, Wang Q, Cui L, Zhang X (2021) Self-supervised hypergraph convolutional networks for session-based recommendation. In: Proceedings of the AAAI conference on artificial intelligence, vol. 35, pp. 4503–4511

  • Yadati N, Nimishakavi M, Yadav P (2019) Hypergcn: a new method for training graph convolutional networks on hypergraphs. Adv Neural Inf process syst 32

  • Zhang D, Yao L, Chen K, Wang S, Haghighi PD, Sullivan C (2019) A graph-based hierarchical attention model for movement intention detection from EEG signals. IEEE Trans Neural Syst Rehabil Eng 27(11):2247–2253. https://doi.org/10.1109/TNSRE.2019.2943362

    Article  PubMed  Google Scholar 

  • Zheng W-L, Lu B-L (2015) Investigating critical frequency bands and channels for eeg-based emotion recognition with deep neural networks. IEEE Trans. Auton. Ment. Dev. 7(3):162–175

    Article  Google Scholar 

  • Zheng W-L, Liu W, Lu Y, Lu B-L, Cichocki A (2018) Emotionmeter: a multimodal framework for recognizing human emotions. IEEE Trans Cybern 49(3):1110–1122

    Article  PubMed  Google Scholar 

  • Zhong P, Wang, D, Miao C (2020) EEG-based emotion recognition using regularized graph neural networks. IEEE Transact Affect Comput

  • Zhong P, Wang D, Miao C (2020) EEG-based emotion recognition using regularized graph neural networks. IEEE Trans Affect Comput

Download references

Acknowledgements

This work was supported by National Key R &D Program of China for Intergovernmental International Science and Technology Innovation Cooperation Project (2017YFE0116800), National Natural Science Foundation of China (U20B2074, U1909202), supported by Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province (2020E10010), Key R &D Project of Zhejiang Province(2021C03003).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, 310018, China

    Menghang Li, Min Qiu, Li Zhu & Wanzeng Kong

  2. Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou, 310018, China

    Menghang Li, Min Qiu, Li Zhu & Wanzeng Kong

Authors
  1. Menghang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Min Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Li Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wanzeng Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wanzeng Kong.

Ethics declarations

Conflicts of interest

All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, M., Qiu, M., Zhu, L. et al. Feature hypergraph representation learning on spatial-temporal correlations for EEG emotion recognition. Cogn Neurodyn 17, 1271–1281 (2023). https://doi.org/10.1007/s11571-022-09890-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11571-022-09890-3

Keywords

Search

Navigation