Skip to main content
SpringerLink
Log in

STSNet: a novel spatio-temporal-spectral network for subject-independent EEG-based emotion recognition

  • Research
  • Published:
Health Information Science and Systems Aims and scope Submit manuscript

Abstract

How to use the characteristics of EEG signals to obtain more complementary and discriminative data representation is an issue in EEG-based emotion recognition. Many studies have tried spatio-temporal or spatio-spectral feature fusion to obtain higher-level representations of EEG data. However, these studies ignored the complementarity between spatial, temporal and spectral domains of EEG signals, thus limiting the classification ability of models. This study proposed an end-to-end network based on ManifoldNet and BiLSTM networks, named STSNet. The STSNet first constructed a 4-D spatio-temporal-spectral data representation and a spatio-temporal data representation based on EEG signals in manifold space. After that, they were fed into the ManifoldNet network and the BiLSTM network respectively to calculate higher-level features and achieve spatio-temporal-spectral feature fusion. Finally, extensive comparative experiments were performed on two public datasets, DEAP and DREAMER, using the subject-independent leave-one-subject-out cross-validation strategy. On the DEAP dataset, the average accuracy of the valence and arousal are 69.38% and 71.88%, respectively; on the DREAMER dataset, the average accuracy of the valence and arousal are 78.26% and 82.37%, respectively. Experimental results show that the STSNet model has good emotion recognition performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

Not applicable.

Code availability

Not applicable.

References

  1. Ali M, Mosa AH, Al Machot F, Kyamakya K, EEG-based emotion recognition approach for e-healthcare applications. In: 2016 eighth international conference on ubiquitous and future networks (ICUFN). IEEE; 2016. p. 946–50.

  2. Vij A, Pruthi J. An automated psychometric analyzer based on sentiment analysis and emotion recognition for healthcare. Procedia Comput Sci. 2018;132:1184–91.

    Google Scholar 

  3. Rivas JJ, del Carmen Lara M, Castrejon L, Hernandez-Franco J, Orihuela-Espina F, Palafox L, Williams A, Berthouze N, Sucar E. Multi-label and multimodal classifier for affective states recognition in virtual rehabilitation. IEEE Trans Affect Comput. 2021. https://doi.org/10.1109/TAFFC.2021.3055790.

    Article  Google Scholar 

  4. Deniz E, Sobahi N, Omar N, Sengur A, Acharya UR. Automated robust human emotion classification system using hybrid EEG features with ICBrainDB dataset. Health Inf Sci Syst. 2022;10(31):1–14. https://doi.org/10.1007/s13755-022-00201-y.

    Article  Google Scholar 

  5. Roy Y, Banville H, Albuquerque I, Gramfort A, Falk TH, Faubert J. Deep learning-based electroencephalography analysis: a systematic review. J Neural Eng. 2019;16(5): 051001.

    Google Scholar 

  6. Hamann S, Canli T. Individual differences in emotion processing. Curr Opin Neurobiol. 2004;14(2):233–8.

    Google Scholar 

  7. Song T, Zheng W, Song P, Cui Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput. 2018;11(3):532–41.

    Google Scholar 

  8. Li Y, Zheng W, Wang L, Zong Y, Cui Z. From regional to global brain: a novel hierarchical spatial-temporal neural network model for EEG emotion recognition. IEEE Trans Affect Comput. 2022;13(02):568–78.

    Google Scholar 

  9. Zhang X, Liu J, Shen J, Li S, Hou K, Hu B, Gao J, Zhang T. Emotion recognition from multimodal physiological signals using a regularized deep fusion of kernel machine. IEEE Trans Cybern. 2020;51(9):4386–99.

    Google Scholar 

  10. Bhosale S, Chakraborty R, Kopparapu SK. Calibration free meta learning based approach for subject independent EEG emotion recognition. Biomed Signal Process Control. 2022;72: 103289.

    Google Scholar 

  11. Cui H, Liu A, Zhang X, Chen X, Liu J, Chen X. EEG-based subject-independent emotion recognition using gated recurrent unit and minimum class confusion. IEEE Trans Affect Comput. 2022. https://doi.org/10.1109/TAFFC.2022.3179717.

    Article  Google Scholar 

  12. Cohen MX. Where does EEG come from and what does it mean? Trends Neurosci. 2017;40(4):208–18.

    Google Scholar 

  13. Lindquist KA, Barrett LF. A functional architecture of the human brain: emerging insights from the science of emotion. Trends Cognit Sci. 2012;16(11):533–40.

    Google Scholar 

  14. Zheng W-L, Lu B-L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev. 2015;7(3):162–75.

    Google Scholar 

  15. Li P, Liu H, Si Y, Li C, Li F, Zhu X, Huang X, Zeng Y, Yao D, Zhang Y, et al. EEG based emotion recognition by combining functional connectivity network and local activations. IEEE Trans Biomed Eng. 2019;66(10):2869–81.

    Google Scholar 

  16. Jiang D, Yu M, Yuanyuan W. Sleep stage classification using covariance features of multi-channel physiological signals on Riemannian manifolds. Comput Methods Programs Biomed. 2019;178:19–30.

    Google Scholar 

  17. Arsigny V, Fillard P, Pennec X, Ayache N. Log-Euclidean metrics for fast and simple calculus on diffusion tensors. Magn Reson Med. 2006;56(2):411–21.

    Google Scholar 

  18. Yger F, Berar M, Lotte F. Riemannian approaches in brain-computer interfaces: a review. IEEE Trans Neural Syst Rehabil Eng. 2016;25(10):1753–62.

    Google Scholar 

  19. Lotte F, Bougrain L, Cichocki A, Clerc M, Congedo M, Rakotomamonjy A, Yger F. A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update. J Neural Eng. 2018;15(3): 031005.

    Google Scholar 

  20. Abdel-Ghaffar EA, Wu Y, Daoudi M. Subject-dependent emotion recognition system based on multidimensional electroencephalographic signals: a riemannian geometry approach. IEEE Access. 2022;10:14993–5006.

    Google Scholar 

  21. Gao Y, Sun X, Meng M, Zhang Y. EEG emotion recognition based on enhanced SPD matrix and manifold dimensionality reduction. Comput Biol Med. 2022;146: 105606.

    Google Scholar 

  22. Chakraborty R, Bouza J, Manton J, Vemuri BC. Manifoldnet: a deep neural network for manifold-valued data with applications. IEEE Trans Pattern Anal Mach Intell. 2020;44(2):799–810.

    Google Scholar 

  23. Yu Z, Ramanarayanan V, Suendermann-Oeft D, Wang X, Zechner K, Chen L, Tao J, Ivanou A, Qian Y: Using bidirectional LSTM recurrent neural networks to learn high-level abstractions of sequential features for automated scoring of non-native spontaneous speech. In: 2015 IEEE workshop on automatic speech recognition and understanding (ASRU). IEEE; 2015. p. 338–45.

  24. Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I. DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput. 2011;3(1):18–31.

    Google Scholar 

  25. Katsigiannis S, Ramzan N. DREAMER: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J Biomed Health Inform. 2017;22(1):98–107.

    Google Scholar 

  26. Jenke R, Peer A, Buss M. Feature extraction and selection for emotion recognition from EEG. IEEE Trans Affect Comput. 2014;5(3):327–39.

    Google Scholar 

  27. Li X, Zhang Y, Tiwari P, Song D, Hu B, Yang M, Zhao Z, Kumar N, Marttinen P. EEG based emotion recognition: a tutorial and review. ACM Comput Surv. 2022. https://doi.org/10.1145/3524499.

    Article  Google Scholar 

  28. Li X, Song D, Zhang P, Zhang Y, Hou Y, Hu B. Exploring EEG features in cross-subject emotion recognition. Front Neurosci. 2018;12:162.

    Google Scholar 

  29. Khateeb M, Anwar SM, Alnowami M. Multi-domain feature fusion for emotion classification using DEAP dataset. IEEE Access. 2021;9:12134–42.

    Google Scholar 

  30. Mohammadi Z, Frounchi J, Amiri M. Wavelet-based emotion recognition system using EEG signal. Neural Comput Appl. 2017;28(8):1985–90.

    Google Scholar 

  31. Rahman MM, Sarkar AK, Hossain MA, Moni MA. EEG-based emotion analysis using non-linear features and ensemble learning approaches. Expert Syst Appl. 2022;207: 118025.

    Google Scholar 

  32. Liang Z, Zhou R, Zhang L, Li L, Huang G, Zhang Z, Ishii S. EEGFuseNet: hybrid unsupervised deep feature characterization and fusion for high-dimensional EEG with an application to emotion recognition. IEEE Trans Neural Syst Rehabil Eng. 2021;29:1913–25.

    Google Scholar 

  33. Wang H, Zhu X, Chen P, Yang Y, Ma C, Gao Z. A gradient-based automatic optimization CNN framework for EEG state recognition. J Neural Eng. 2022;19(1): 016009.

    Google Scholar 

  34. Li C, Wang B, Zhang S, Liu Y, Song R, Cheng J, Chen X. Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism. Comput Biol Med. 2022;143: 105303.

    Google Scholar 

  35. Zhao H, Liu J, Shen Z, Yan J. SCC-MPGCN: self-attention coherence clustering based on multi-pooling graph convolutional network for EEG emotion recognition. J Neural Eng. 2022;19(2): 026051.

    Google Scholar 

  36. Wang Z, Wang Y, Zhang J, Hu C, Yin Z, Song Y. Spatial-temporal feature fusion neural network for EEG-based emotion recognition. IEEE Trans Instrum Meas. 2022;71:1–12.

    Google Scholar 

  37. Ding Y, Robinson N, Zhang S, Zeng Q, Guan C. TSception: capturing temporal dynamics and spatial asymmetry from EEG for emotion recognition. IEEE Trans Affect Comput. 2022. https://doi.org/10.1109/TAFFC.2022.3169001.

    Article  Google Scholar 

  38. Liu S, Wang X, Zhao L, Li B, Hu W, Yu J, Zhang Y-D. 3DCANN: a spatio-temporal convolution attention neural network for EEG emotion recognition. IEEE J Biomed Health Inform. 2021;26(11):5321–31.

    Google Scholar 

  39. Li J, Wu X, Zhang Y, Yang H, Wu X. DRS-Net: a spatial-temporal affective computing model based on multichannel EEG data. Biomed Signal Process Control. 2022;76: 103660.

    Google Scholar 

  40. Mithra U, Aravinth, J.: Spatial spectral based 3D feature map for EEG emotion recognition. In: 2022 3rd international conference on electronics and sustainable communication systems (ICESC). IEEE; 2022. p. 247–52.

  41. Li R, Ren C, Li C, Zhao N, Lu D, Zhang X. SSTD: a novel spatio-temporal demographic network for EEG-based emotion recognition. IEEE Trans Comput Soc Syst. 2023;10(1):376–87.

    Google Scholar 

  42. Sabbagh D, Ablin P, Varoquaux G, Gramfort A, Engemann DA. Predictive regression modeling with MEG/EEG: from source power to signals and cognitive states. Neuroimage. 2020;222: 116893.

    Google Scholar 

  43. Schuster M, Paliwal KK. Bidirectional recurrent neural networks. IEEE Trans Signal Process. 1997;45(11):2673–81.

    Google Scholar 

  44. Walter S, Kim J, Hrabal D, Crawcour SC, Kessler H, Traue HC. Transsituational individual-specific biopsychological classification of emotions. IEEE Trans Syst Man Cybern Syst. 2013;43(4):988–95.

    Google Scholar 

  45. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–30.

    MathSciNet  Google Scholar 

  46. Saha, P., Fels, S., Abdul-Mageed, M.: Deep learning the EEG manifold for phonological categorization from active thoughts. In: ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE; 2019. p. 2762–6.

  47. Pan B, Zheng W. Emotion recognition based on EEG using generative adversarial nets and convolutional neural network. Comput Math Methods Med. 2021;2021:2520394.

    Google Scholar 

  48. Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng. 2018;15(5): 056013.

    Google Scholar 

  49. Huang D, Chen S, Liu C, Zheng L, Tian Z, Jiang D. Differences first in asymmetric brain: a bi-hemisphere discrepancy convolutional neural network for EEG emotion recognition. Neurocomputing. 2021;448:140–51.

    Google Scholar 

  50. Arjun, Rajpoot AS, Panicker MR. Subject independent emotion recognition using EEG signals employing attention driven neural networks. Biomed Signal Process Control 2022;75:103547

  51. Zhang G, Yu M, Liu Y, Zhao G, Zhang D, Zheng W. SparseDGCNN: recognizing emotion from multichannel EEG signals. IEEE Trans Affect Comput. 2021. https://doi.org/10.1109/TAFFC.2021.3051332.

    Article  Google Scholar 

  52. Wang Y, Qiu S, Ma X, He H. A prototype-based SPD matrix network for domain adaptation EEG emotion recognition. Pattern Recogn. 2021;110: 107626.

    Google Scholar 

  53. Priyasad D, Fernando T, Denman S, Sridharan S, Fookes C. Affect recognition from scalp-EEG using channel-wise encoder networks coupled with geometric deep learning and multi-channel feature fusion. Knowl-Based Syst. 2022;250: 109038.

    Google Scholar 

  54. He Z, Zhong Y, Pan J. Joint temporal convolutional networks and adversarial discriminative domain adaptation for EEG-based cross-subject emotion recognition. In: ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE; 2022. p. 3214–8.

Download references

Acknowledgements

This work was supported in part by the National Key Research and Development Program of China (Grant No. 2019YFA0706200), in part by the National Natural Science Foundation of China (Grant No.62072219, No.61632014), in part by the Natural Science Foundation of Gansu Province, China (Grant No. 22JR5RA401), and in part by the Fundamental Research Funds for the Central Universities (No. lzujbky-2022-ey13).

Funding

This work was supported in part by the National Key Research and Development Program of China (Grant No. 2019YFA0706200), in part by the National Natural Science Foundation of China (Grant No.62072219, No.61632014), in part by the Natural Science Foundation of Gansu Province, China (Grant No. 22JR5RA401), and in part by the Fundamental Research Funds for the Central Universities (No. lzujbky-2022-ey13).

Author information

Authors and Affiliations

  1. Gansu Provincial Key Laboratory of Wearable Computing, School of Information Science and Engineering, Lanzhou University, Lanzhou, 730000, Gansu, China

    Rui Li, Chao Ren, Sipo Zhang, Yikun Yang, Qiqi Zhao, Kechen Hou, Wenjie Yuan, Xiaowei Zhang & Bin Hu

Authors
  1. Rui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chao Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sipo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yikun Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qiqi Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kechen Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Wenjie Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xiaowei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Bin Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by CR, SZ, YY, QZ, KH, WY, XZ, BH; The first draft of the manuscript was written by RL and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Chao Ren or Bin Hu.

Ethics declarations

Conflict of interest

The authors have no relevant fnancial or non-fnancial interests to disclose.

Research involving human and animal participants

Not applicable.

Ethical approval

The used datasets are publically available. So no need to get a further ethical approval.

Consent to participate

Not applicable.

Consent for publication

All authors have checked the manuscript and have agreed to the submission.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, R., Ren, C., Zhang, S. et al. STSNet: a novel spatio-temporal-spectral network for subject-independent EEG-based emotion recognition. Health Inf Sci Syst 11, 25 (2023). https://doi.org/10.1007/s13755-023-00226-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13755-023-00226-x

Keywords

Search

Navigation