Skip to main content
SpringerLink
Log in

From gram to attention matrices: a monotonicity constrained method for eeg-based emotion classification

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In this work, a parameter efficient attention module is developed for the task of emotion classification as well as improved model interpretability based on EEG source data. Inspired by the self-attention mechanism used in transformers, we propose a Monotonicity Constrained Attention Module (MCAM) that can help incorporate different priors easily on the monotonicity when converting Gram matrices from deep features into attention matrices for better feature refinement. In the subject-dependent classification task, MCAM achieves 95.0% mean prediction accuracy on four classification task with DEAP and 91.1% mean prediction accuracy on three classification task with SEED. On both datasets, MCAM is shown comparable to state-of-the-art attention modules in terms of boosting the backbone network’s predictive performance while requiring significantly fewer parameters. A thorough analysis is also performed on tracking the different effects inserted modules have on the backbone model’s behavior. For example, visualization and analysis techniques are presented to examine changes in spatial attention patterns reflected via kernel weights, change in prediction performance when different frequency information is filtered out, or changes that occur when different amplitude information is suppressed; as well as how different models change their predictions along linear morphisms between two samples belonging to different emotion categories. The results help to reveal what different modules learn and use during prediction, and can also provide guidance when applying them to specific applications.

Graphical Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data Availability

SEED dataset can be downloaded upon request at https://bcmi.sjtu.edu.cn/home/seed/DEAP dataset can be downloaded upon request at http://www.eecs.qmul.ac.uk/mmv/datasets/deap/

Notes

  1. We choose a Gram matrix representation over the batch covariance matrix because of the fact that the batch mean fluctuates during stochastic training. These fluctuating means have the effect of causing feature vectors to be centered differently in each batch, causing the resulting attention matrix to have an unnecessary dependency (and thus potential sensitivity) to the fluctuating stochastic batch centers.

  2. The shape of the weight matrix is 1 by C, where C is the number of EEG channels of input samples.

  3. The +QKV case is not as pronounced in this regard as others around the P4 area.

  4. We put a note here that more meaningful interpretations are likely to emerge from this method when the samples selected represent specific kinds of signal templates associated with clinically admissible subsets.

  5. Though we only show this result on one subject, this is the case for all 14 other subjects from SEED as well.

References

  1. Kuang D, Michoski C, Li W, Guo R (2022) A monotonicity constrained attention module for emotion classification with limited EEG data. In: Zamzmi G, Antani S, Bagci U, Linguraru MG, Rajaraman S, Xue Z (eds) Medical image learning with limited and noisy data, Springer, pp 218–228

  2. Yamawaki S, Okada G, Okamoto Y, Liberzon I (2012) Mood dysregulation and stabilization: perspectives from emotional cognitive neuroscience. Int J Neuropsychopharmacol 15(5):681–694

    Article  Google Scholar 

  3. Berridge KC (2004) Motivation concepts in behavioral neuroscience. Physiol Behav 81(2):179–209

    Article  Google Scholar 

  4. DeYoung CG (2010) Personality neuroscience and the biology of traits. Soc Personal Psychol Compass 4(12):1165–1180

    Article  Google Scholar 

  5. Dietrich A (2004) The cognitive neuroscience of creativity. Psychon Bull Rev 11(6):1011–1026

    Article  Google Scholar 

  6. Immordino-Yang MH, Damasio A (2007) We feel, therefore we learn: the relevance of affective and social neuroscience to education. Mind Brain Educ 1(1):3–10

    Article  Google Scholar 

  7. Reiss J, Sprenger J (2020) Scientific objectivity. In: Zalta EN (ed) The Stanford encyclopedia of philosophy, winter. 2020 edn. Metaphysics Research Lab, Stanford University. https://plato.stanford.edu/archives/win2020/entries/scientific-objectivity/

  8. Lane RD, Nadel L (2002) Cognitive neuroscience of emotion. Oxford University Press

  9. Durán JI, Reisenzein R, Fernández-Dols J-M (2017) Coherence between emotions and facial expressions. The Science of Facial Expression 107–129

  10. Grimm M, Kroschel K, Mower E, Narayanan S (2007) Primitives-based evaluation and estimation of emotions in speech. Speech Comm 49(10-11):787–800

    Article  Google Scholar 

  11. Chatterjee A, Gupta U, Chinnakotla MK, Srikanth R, Galley M, Agrawal P (2019) Understanding emotions in text using deep learning and big data. Comput Hum Behav 93:309–317

    Article  Google Scholar 

  12. Gentile V, Milazzo F, Sorce S, Gentile A, Augello A, Pilato G (2017) Body gestures and spoken sentences: a novel approach for revealing user’s emotions. In: 2017 IEEE 11th international conference on semantic computing (ICSC), IEEE, pp 69–72

  13. Kuang D, Michoski C (2020) Dual stream neural networks for brain signal classification. Journal of Neural Engineering

  14. Ekman P (1992) An argument for basic emotions. Cognit Emot 6(3-4):169–200

    Article  Google Scholar 

  15. Spinoza B (2006) The essential spinoza: ethics and related writings hackett publishing

  16. Russell JA (1980) A circumplex model of affect. J Personal Soc Psychol 39(6):1161

    Article  Google Scholar 

  17. Mehrabian A (1996) Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in temperament. Curr Psychol 14:261–292

    Article  MathSciNet  Google Scholar 

  18. Subramanian R, Wache J, Abadi MK, Vieriu RL, Winkler S, Sebe N (2016) Ascertain: emotion and personality recognition using commercial sensors. IEEE Trans Affect Comput 9(2):147–160

    Article  Google Scholar 

  19. Wang X-W, Nie D, Lu B-L (2014) Emotional state classification from EEG data using machine learning approach. Neurocomputing 129:94–106

    Article  Google Scholar 

  20. Asghar MA, Khan MJ, Amin Y, Rizwan M, Rahman M, Badnava S, Mirjavadi SS et al (2019) EEG-based multi-modal emotion recognition using bag of deep features: an optimal feature selection approach. Sensors 19(23):5218

    Article  Google Scholar 

  21. Subasi A, Tuncer T, Dogan S, Tanko D, Sakoglu U (2021) EEG-based emotion recognition using tunable q wavelet transform and rotation forest ensemble classifier. Biomed Sig Process Control 68:102648

    Article  Google Scholar 

  22. Tuncer T, Dogan S, Subasi A (2021) A new fractal pattern feature generation function based emotion recognition method using EEG. Chaos, Solitons Fractals 144:110671

    Article  MathSciNet  Google Scholar 

  23. Tuncer T, Dogan S, Baygin M, Acharya UR (2022) Tetromino pattern based accurate EEG emotion classification model. Artif Intell Med 123:102210

    Article  Google Scholar 

  24. Kılıç B, Aydın S (2022) Classification of contrasting discrete emotional states indicated by EEG based graph theoretical network measures. Neuroinformatics 1–15

  25. Aydın S, Demirtaş S, Tunga MA, Ateş K (2018) Comparison of hemispheric asymmetry measurements for emotional recordings from controls. Neural Comput Appl 30(4):1341–1351

    Article  Google Scholar 

  26. Gao J, Yang C, Liu F, Qi J (2021) Emotion prediction of EEG signals based on 1d convolutional neural network. J Neural Eng 2024

  27. Liu S, Wang X, Zhao L, Li B, Hu W, Yu J, Zhang Y (2021) 3dcann: a spatio-temporal convolution attention neural network for EEG emotion recognition. IEEE Journal of Biomedical and Health Informatics

  28. Aydın S (2019) Deep learning classification of neuro-emotional phase domain complexity levels induced by affective video film clips. IEEE J Biomed Health Inf 24(6):1695–1702

    Article  Google Scholar 

  29. Zhang Y, Chen J, Tan JH, Chen Y, Chen Y, Li D, Yang L, Su J, Huang X, Che W (2020) An investigation of deep learning models for EEG-based emotion recognition. Front Neurosci 14:622759

    Article  Google Scholar 

  30. Feng L, Cheng C, Zhao M, Deng H, Zhang Y (2022) EEG-Based emotion recognition using spatial-temporal graph convolutional lstm with attention mechanism. IEEE J Biomed Health Inf

  31. Ahmed MZI, Sinha N (2021) EEG-Based emotion classification using lstm under new paradigm. J Neural Eng 7

  32. Li G, Chen N, Jin J (2022) Semi-supervised EEG emotion recognition model based on enhanced graph fusion and gcn. J Neural Eng 19

  33. Song T, Zheng W, Liu S, Zong Y, Cui Z, Li Y (2021) Graph-embedded convolutional neural network for image-based EEG emotion recognition. IEEE Transactions on Emerging Topics in Computing

  34. Asadzadeh S, Yousefi Rezaii T, Beheshti S, Meshgini S (2022) Accurate emotion recognition using bayesian model based EEG sources as dynamic graph convolutional neural network nodes. Sci Rep 12 (1):1–14

    Article  Google Scholar 

  35. Luo Y, Zhu L-Z, Wan Z-Y, Lu B-L (2020) Data augmentation for enhancing EEG-based emotion recognition with deep generative models. J Neural Eng 17

  36. Zhang A, Su L, Zhang Y, Fu Y, Wu L, Liang S (2022) EEG data augmentation for emotion recognition with a multiple generator conditional wasserstein gan. Compl Intell Syst 8(4):3059–3071

    Article  Google Scholar 

  37. Li X, Zhao Z, Song D, Zhang Y, Pan J, Wu L, Huo J, Niu C, Wang D (2020) Latent factor decoding of multi-channel eeg for emotion recognition through autoencoder-like neural networks. Front Neurosci 14:87

    Article  Google Scholar 

  38. Salankar N, Mishra P, Garg L (2021) Emotion recognition from EEG signals using empirical mode decomposition and second-order difference plot. Biomed Sig Process Control 65:102389

    Article  Google Scholar 

  39. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  40. Woo S, Park J, Lee J-Y, Kweon IS (2018) CBAM: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19

  41. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Advances in neural information processing systems 30

  42. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N (2021) An image is worth 16x16 words: transformers for image recognition at scale. In: International conference on learning representations. https://openreview.net/forum?id=YicbFdNTTy

  43. Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) Deap: a database for emotion analysis ;using physiological signals. IEEE Trans Affect Comput 3(1):18–31. https://doi.org/10.1109/T-AFFC.2011.15

    Article  Google Scholar 

  44. Duan R-N, Zhu J-Y, Lu B-L (2013) Differential entropy feature for EEG-based emotion classification. In: 6th international IEEE/EMBS conference on neural engineering (NER), IEEE, pp 81–84

  45. Zheng W-L, Lu B-L (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev 7:162–175

    Article  Google Scholar 

  46. Lee J, Lee I, Kang J (2019) Self-attention graph pooling. In: International conference on machine learning, PMLR, pp 3734–3743

  47. Sankar A, Wu Y, Gou L, Zhang W, Yang H (2020) Dysat: deep neural representation learning on dynamic graphs via self-attention networks. In: Proceedings of the 13th international conference on web search and data mining

  48. Mittag G, Naderi B, Chehadi A, Möller S (2021) Nisqa: a deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets. In: Interspeech

  49. Koizumi Y, Yatabe K, Delcroix M, Masuyama Y, Takeuchi D (2020) Speech enhancement using self-adaptation and multi-head self-attention. In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pp 181–185

  50. Li Y, Fu B, Li F, Shi G, Zheng W (2021) A novel transferability attention neural network model for EEG emotion recognition. Neurocomputing 447:92–101. https://doi.org/10.1016/j.neucom.2021.02.048

    Article  Google Scholar 

  51. Li D, Xie L, Chai B, Wang Z, Yang H (2022) Spatial-frequency convolutional self-attention network for EEG emotion recognition. Appl Soft Comput 122:108740

    Article  Google Scholar 

  52. Rajpoot AS, Panicker MR, et al. (2022) Subject independent emotion recognition using EEG signals employing attention driven neural networks. Biomed Sig Process Control 75:103547

    Article  Google Scholar 

  53. Xie J, Wang Z, Yu Z, Guo B (2022) Enabling timely medical intervention by exploring health-related multivariate time series with a hybrid attentive model. Sensors 22(16):6104

    Article  Google Scholar 

  54. Sreeram V, Agathoklis P (1994) On the properties of gram matrix. IEEE Trans Circ Syst I: Fundam Theory Appl 41(3):234–237

    Article  MathSciNet  MATH  Google Scholar 

  55. Gilmer J, Schoenholz SS, Riley PF, Vinyals O, Dahl GE (2017) Neural message passing for quantum chemistry. In: International conference on machine learning, PMLR, pp 1263–1272

  56. Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ (2018) Eegnet: a compact convolutional neural network for EEG-based brain–computer interfaces. J Neural Eng 15(5):056013

    Article  Google Scholar 

  57. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1251–1258

  58. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, PMLR, pp 448–456

  59. Khateeb M, Anwar SM, Alnowami MR (2021) Multi-domain feature fusion for emotion classification using deap dataset. IEEE Access 9:12134–12142

    Article  Google Scholar 

  60. Stajić T, Jovanović J, Jovanović N, Jankovic MM (2021) Emotion recognition based on deap database physiological signals. In: 2021 29th telecommunications forum (TELFOR), pp 1–4

  61. Wang Z-M, Zhang J-W, He Y, Zhang J (2022) EEG Emotion recognition using multichannel weighted multiscale permutation entropy. Appl Intell 1–13

  62. Li Y, Huang J, Zhou H, Zhong N (2017) Human emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks. Appl Sci 7(10):1060

    Article  Google Scholar 

  63. Asghar MA, Khan MJ, Rizwan M, Shorfuzzaman M, Mehmood RM (2022) AI inspired EEG-based spatial feature selection method using multivariate empirical mode decomposition for emotion classification. Multimed Syst 28(4):1275–1288

    Article  Google Scholar 

  64. Sharma R, Pachori RB, Sircar P (2020) Automated emotion recognition based on higher order statistics and deep learning algorithm. Biomed Sig Process Control 58:101867

    Article  Google Scholar 

  65. Al Machot F, Elmachot A, Ali M, Al Machot E, Kyamakya K (2019) A deep-learning model for subject-independent human emotion recognition using electrodermal activity sensors. Sensors 19 (7):1659

    Article  Google Scholar 

  66. Salama ES, El-Khoribi RA, Shoman ME, Shalaby MAW (2018) EEG-based emotion recognition using 3d convolutional neural networks. International Journal of Advanced Computer Science and Applications 9(8)

  67. Wei Y, Liu Y, Li C, Cheng J, Song R, Chen X (2022) Tc-net: a transformer capsule network for EEG-based emotion recognition. Computers in Biology and Medicine 106463

  68. Wu Y, Xia M, Nie L, Zhang Y, Fan A (2022) Simultaneously exploring multi-scale and asymmetric EEG features for emotion recognition. Comput Biol Med 149:106002

    Article  Google Scholar 

  69. Zhong X, Gu Y, Luo Y, Zeng X, Liu G (2022) Bi-hemisphere asymmetric attention network: recognizing emotion from EEG signals based on the transformer. Appl Intell 1–17

  70. Zhang T, Zheng W, Cui Z, Zong Y, Li Y (2018) Spatial–temporal recurrent neural network for emotion recognition. IEEE Trans Cybernet 49(3):839–847

    Article  Google Scholar 

  71. Song T, Zheng W, Song P, Cui Z (2018) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing

  72. Li Y, Zheng W, Cui Z, Zhang T, Zong Y (2018) A novel neural network model based on cerebral hemispheric asymmetry for EEG emotion recognition. In: IJCAI, pp 1561–1567

  73. Li Y, Zheng W, Wang L, Zong Y, Cui Z (2019) From regional to global brain: a novel hierarchical spatial-temporal neural network model for EEG emotion recognition. IEEE Transactions on Affective Computing

  74. Joshi VM, Ghongade RB (2021) EEG based emotion detection using fourth order spectral moment and deep learning. Biomed Sig Process Control 68:102755

    Article  Google Scholar 

  75. Miao M, Zheng L, Xu B, Yang Z, Hu W (2023) A multiple frequency bands parallel spatial–temporal 3d deep residual learning framework for EEG-based emotion recognition. Biomed Sig Process Control 79:104141

    Article  Google Scholar 

  76. Zheng W-L, Lu B-L (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev 7(3):162–175

    Article  Google Scholar 

  77. Kuang D, Michoski C (2022) KAM - A kernel attention module for emotion classification with EEG data. In: Reyes M, Henriques Abreu P, Cardoso J (eds) Interpretability of machine intelligence in medical image computing, Springer, pp 93–103

Download references

Acknowledgements

This work was supported in part by the Fundamental Research Funds for the Central Universities, Sun Yat-sen University, CHINA, under Grant 22qntd2901.

Author information

Authors and Affiliations

  1. School of Mathematics (Zhuhai), Sun Yat-Sen University, No.2 Daxue Road, Zhuhai, 519082, Guangdong, China

    Dongyang Kuang, Wenting Li & Rui Guo

  2. Oden Institute for Computational Engineering and Sciences, University of Texas at Austin, 201 E 24th St, Austin, 78712, Texas, USA

    Craig Michoski

Authors
  1. Dongyang Kuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Craig Michoski
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenting Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rui Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Dongyang Kuang: Conceptualization, Methodology, Validation, Formal analysis, Writing – original draft, Writing – review & editing, Visualization. Craig Michoski: Formal analysis, Validation, Visualization, Writing – review & editing. Wenting Li: Data Curation, Visualization, Writing – original draft, Writing – review & editing. Rui Guo: Data Curation, Validation, Writing – review & editing

Corresponding author

Correspondence to Dongyang Kuang.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This submission is a reworked and thoroughly extended version of [1] presented in The Workshop on Medical Image Learning with Noisy and Limited Data at MICCAI 2022.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kuang, D., Michoski, C., Li, W. et al. From gram to attention matrices: a monotonicity constrained method for eeg-based emotion classification. Appl Intell 53, 20690–20709 (2023). https://doi.org/10.1007/s10489-023-04561-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04561-0

Keywords

Search

Navigation