Skip to main content

Enhanced deep capsule network for EEG-based emotion recognition

Signal, Image and Video Processing volume 17, pages 463–469 (2023)Cite this article

Abstract

Recently, it has become very popular to use electroencephalogram (EEG) signals in emotion recognition studies. But, EEG signals are much more complex than image and audio signals. There may be inconsistencies even in signals recorded from the same person. Therefore, EEG signals obtained from the human brain must be analyzed and processed accurately and consistently. In addition, traditional algorithms used to classify emotion ignore the neighborhood relationship and hierarchical order within the EEG signals. In this paper, a method including selection of suitable channels from EEG data, feature extraction by Welch power spectral density estimation of selected channels and enhanced capsule network-based classification model is presented. The most important innovation of the method is to adjust the architecture of the capsule network to adapt to the EEG signals. Thanks to the proposed method, 99.51% training and 98.21% test accuracy on positive, negative and neutral emotions were achieved in the Seed EEG dataset. The obtained results were also compared and evaluated with other state-of-the-art methods. Finally, the method was tested with Dreamer and Deap EEG datasets.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

References

  1. Aldemir, R.: Evaluation of drug treatment processes of children with attention deficit and hyperactivity by EEG analysis. Thesis, Erciyes University (2019)

  2. Basar, M.D., Duru, A.D., Akan, A.: Emotional state detection based on common spatial patterns of EEG. In: Signal, Image and Video Processing, pp. 1–9 (2019)

  3. Bos, D.O., et al.: EEG-based emotion recognition. Influ. Vis. Audit. Stimuli 56(3), 1–17 (2006)

    Google Scholar 

  4. Côté-Allard, U., Fall, C.L., Drouin, A., Campeau-Lecours, A., Gosselin, C., Glette, K., Laviolette, F., Gosselin, B.: Deep learning for electromyographic hand gesture signal classification using transfer learning. IEEE Trans. Neural Syst. Rehabil. Eng. 27(4), 760–771 (2019)

    Article  Google Scholar 

  5. Duan, R.N., Zhu, J.Y., Lu, B.L.: Differential entropy feature for EEG-based emotion classification. In: 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER), IEEE, pp. 81–84 (2013)

  6. Fei, H., Ji, D., Zhang, Y., Ren, Y.: Topic-enhanced capsule network for multi-label emotion classification. IEEE/ACM Trans. Audio Speech Lang. Process. 28, 1839–1848 (2020)

    Article  Google Scholar 

  7. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  8. Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  9. Hinton, G.E., Krizhevsky, A., Wang, S.D.: Transforming auto-encoders. In: International Conference on Artificial Neural Networks. Springer, pp. 44–51 (2011)

  10. Islam, M.R., Ahmad, M.: Wavelet analysis based classification of emotion from EEG signal. In: 2019 International Conference on Electrical, Computer and Communication Engineering (ECCE), IEEE, pp. 1–6 (2019)

  11. Katsigiannis, S., Ramzan, N.: Dreamer: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J. Biomed. Health Inform. 22(1), 98–107 (2017)

    Article  Google Scholar 

  12. Koelstra, S., Muhl, C., Soleymani, M., Lee, J.S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: Deap: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2011)

    Article  Google Scholar 

  13. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

  14. Lai, D., Heyat, M.B.B., Khan, F.I., Zhang, Y.: Prognosis of sleep bruxism using power spectral density approach applied on EEG signal of both EMG1–EMG2 and ECG1–ECG2 channels. IEEE Access 7, 82553–82562 (2019)

    Article  Google Scholar 

  15. LeCun, Y., Bengio, Y., et al.: Convolutional networks for images, speech, and time series. Handb. Brain Theory Neural Netw. 3361(10), 1995 (1995)

    Google Scholar 

  16. Li, H.C., Wang, W.Y., Pan, L., Li, W., Du, Q., Tao, R.: Robust capsule network based on maximum correntropy criterion for hyperspectral image classification. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 13, 738–751 (2020)

    Article  Google Scholar 

  17. Li, M., Lu, B.L.: Emotion classification based on gamma-band EEG. In: 2009 Annual International Conference of the IEEE Engineering in medicine and biology society, IEEE, pp. 1223–1226 (2009)

  18. Liu, Y., Ding, Y., Li, C., Cheng, J., Song, R., Wan, F., Chen, X.: Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Comput. Biol. Med. 123, 103927 (2020)

    Article  Google Scholar 

  19. Luo, Y., Wu, G., Qiu, S., Yang, S., Li, W., Bi, Y.: EEG-based emotion classification using deep neural network and sparse autoencoder. Front. Syst. Neurosci. 14, 43 (2020)

    Article  Google Scholar 

  20. Martinez, H.P., Bengio, Y., Yannakakis, G.N.: Learning deep physiological models of affect. IEEE Comput. Intell. Mag. 8(2), 20–33 (2013)

    Article  Google Scholar 

  21. NiederMeyer, E.: Niedermeyer’s Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Lippincott Williams & Wilkins, Philadelphia (2011)

    Google Scholar 

  22. Ozcan, C., Cizmeci, H.: EEG based emotion recognition with convolutional neural networks. In: 2020 28th Signal Processing and Communications Applications Conference (SIU), IEEE, pp. 1–4 (2020). https://doi.org/10.1109/SIU49456.2020.9302498

  23. Paluš, M.: Nonlinearity in normal human EEG: cycles, temporal asymmetry, nonstationarity and randomness, not chaos. Biol. Cybern. 75(5), 389–396 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  24. Patrick, M.K., Adekoya, A.F., Mighty, A.A., Edward, B.Y.: Capsule networks—a survey. J. King Saud Univ. Comput. Inform. Sci. 34, 1295–1310 (2019)

    Google Scholar 

  25. Rifai, S., Vincent, P., Muller, X., Glorot, X., Bengio, Y.: Contractive auto-encoders: explicit invariance during feature extraction. In: ICML (2011)

  26. Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: Advances in Neural Information Processing Systems, pp. 3856–3866 (2017)

  27. Sankisa, A., Punjabi, A., Katsaggelos, A.K.: Temporal capsule networks for video motion estimation and error concealment. SIViP 14(7), 1369–1377 (2020)

    Article  Google Scholar 

  28. Shao, H.M., Wang, J.G., Wang, Y., Yao, Y., Liu, J.: EEG-based emotion recognition with deep convolution neural network. In: 2019 IEEE 8th Data Driven Control and Learning Systems Conference (DDCLS), IEEE, pp. 1225–1229 (2019)

  29. Valenzi, S., Islam, T., Jurica, P., Cichocki, A.: Individual classification of emotions using EEG. J. Biomed. Sci. Eng. 7, 1–17 (2014)

    Article  Google Scholar 

  30. Welch, P.: The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Trans. Audio Electroacoust. 15(2), 70–73 (1967)

    Article  Google Scholar 

  31. Wu, J.: Introduction to convolutional neural networks. Natl. Key Lab Novel Softw. Technol. 5, 23 (2017)

    Google Scholar 

  32. Zheng, W.L., Lu, B.L.: Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans. Auton. Ment. Dev. 7(3), 162–175 (2015)

    Article  Google Scholar 

  33. Zheng, W.L., Guo, H.T., Lu, B.L.: Revealing critical channels and frequency bands for emotion recognition from EEG with deep belief network. In: 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER), IEEE, pp. 154–157 (2015)

  34. Zubair, M., Kim, J., Yoon, C.: An automated ecg beat classification system using convolutional neural networks. In: 2016 6th International Conference on IT Convergence and Security (ICITCS), IEEE, pp. 1–5 (2016)

Download references

Acknowledgements

This work was supported by Scientific Research Projects Unit of Karabuk University under project number FDK-2020-2309. The authors appreciate the financial and scientific support.

Author information

Authors and Affiliations

  1. Computer Technology Department, Vocational School Of Technical Sciences, Hitit University, Corum, Turkey

    Huseyin Cizmeci

  2. Software Engineering Department, Engineering Faculty, Karabuk University, Karabuk, Turkey

    Caner Ozcan

Authors
  1. Huseyin Cizmeci
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Caner Ozcan
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All the authors contributed to the article.

Corresponding author

Correspondence to Huseyin Cizmeci.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Human and animal rights

No human or animal participants were used in this study. Data used in this work were publicly available and granted by the original database owner.

Informed consent

All the authors are well informed about this submission.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Cizmeci, H., Ozcan, C. Enhanced deep capsule network for EEG-based emotion recognition. SIViP 17, 463–469 (2023). https://doi.org/10.1007/s11760-022-02251-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02251-x

Keywords

  • Emotion recognition
  • EEG
  • Feature extraction
  • Deep learning
  • Capsule network