Recently, it has become very popular to use electroencephalogram (EEG) signals in emotion recognition studies. But, EEG signals are much more complex than image and audio signals. There may be inconsistencies even in signals recorded from the same person. Therefore, EEG signals obtained from the human brain must be analyzed and processed accurately and consistently. In addition, traditional algorithms used to classify emotion ignore the neighborhood relationship and hierarchical order within the EEG signals. In this paper, a method including selection of suitable channels from EEG data, feature extraction by Welch power spectral density estimation of selected channels and enhanced capsule network-based classification model is presented. The most important innovation of the method is to adjust the architecture of the capsule network to adapt to the EEG signals. Thanks to the proposed method, 99.51% training and 98.21% test accuracy on positive, negative and neutral emotions were achieved in the Seed EEG dataset. The obtained results were also compared and evaluated with other state-of-the-art methods. Finally, the method was tested with Dreamer and Deap EEG datasets.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Aldemir, R.: Evaluation of drug treatment processes of children with attention deficit and hyperactivity by EEG analysis. Thesis, Erciyes University (2019)
Basar, M.D., Duru, A.D., Akan, A.: Emotional state detection based on common spatial patterns of EEG. In: Signal, Image and Video Processing, pp. 1–9 (2019)
Bos, D.O., et al.: EEG-based emotion recognition. Influ. Vis. Audit. Stimuli 56(3), 1–17 (2006)
Côté-Allard, U., Fall, C.L., Drouin, A., Campeau-Lecours, A., Gosselin, C., Glette, K., Laviolette, F., Gosselin, B.: Deep learning for electromyographic hand gesture signal classification using transfer learning. IEEE Trans. Neural Syst. Rehabil. Eng. 27(4), 760–771 (2019)
Duan, R.N., Zhu, J.Y., Lu, B.L.: Differential entropy feature for EEG-based emotion classification. In: 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER), IEEE, pp. 81–84 (2013)
Fei, H., Ji, D., Zhang, Y., Ren, Y.: Topic-enhanced capsule network for multi-label emotion classification. IEEE/ACM Trans. Audio Speech Lang. Process. 28, 1839–1848 (2020)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)
Hinton, G.E., Krizhevsky, A., Wang, S.D.: Transforming auto-encoders. In: International Conference on Artificial Neural Networks. Springer, pp. 44–51 (2011)
Islam, M.R., Ahmad, M.: Wavelet analysis based classification of emotion from EEG signal. In: 2019 International Conference on Electrical, Computer and Communication Engineering (ECCE), IEEE, pp. 1–6 (2019)
Katsigiannis, S., Ramzan, N.: Dreamer: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J. Biomed. Health Inform. 22(1), 98–107 (2017)
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: Deap: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2011)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Lai, D., Heyat, M.B.B., Khan, F.I., Zhang, Y.: Prognosis of sleep bruxism using power spectral density approach applied on EEG signal of both EMG1–EMG2 and ECG1–ECG2 channels. IEEE Access 7, 82553–82562 (2019)
LeCun, Y., Bengio, Y., et al.: Convolutional networks for images, speech, and time series. Handb. Brain Theory Neural Netw. 3361(10), 1995 (1995)
Li, H.C., Wang, W.Y., Pan, L., Li, W., Du, Q., Tao, R.: Robust capsule network based on maximum correntropy criterion for hyperspectral image classification. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 13, 738–751 (2020)
Li, M., Lu, B.L.: Emotion classification based on gamma-band EEG. In: 2009 Annual International Conference of the IEEE Engineering in medicine and biology society, IEEE, pp. 1223–1226 (2009)
Liu, Y., Ding, Y., Li, C., Cheng, J., Song, R., Wan, F., Chen, X.: Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Comput. Biol. Med. 123, 103927 (2020)
Luo, Y., Wu, G., Qiu, S., Yang, S., Li, W., Bi, Y.: EEG-based emotion classification using deep neural network and sparse autoencoder. Front. Syst. Neurosci. 14, 43 (2020)
Martinez, H.P., Bengio, Y., Yannakakis, G.N.: Learning deep physiological models of affect. IEEE Comput. Intell. Mag. 8(2), 20–33 (2013)
NiederMeyer, E.: Niedermeyer’s Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Lippincott Williams & Wilkins, Philadelphia (2011)
Ozcan, C., Cizmeci, H.: EEG based emotion recognition with convolutional neural networks. In: 2020 28th Signal Processing and Communications Applications Conference (SIU), IEEE, pp. 1–4 (2020). https://doi.org/10.1109/SIU49456.2020.9302498
Paluš, M.: Nonlinearity in normal human EEG: cycles, temporal asymmetry, nonstationarity and randomness, not chaos. Biol. Cybern. 75(5), 389–396 (1996)
Patrick, M.K., Adekoya, A.F., Mighty, A.A., Edward, B.Y.: Capsule networks—a survey. J. King Saud Univ. Comput. Inform. Sci. 34, 1295–1310 (2019)
Rifai, S., Vincent, P., Muller, X., Glorot, X., Bengio, Y.: Contractive auto-encoders: explicit invariance during feature extraction. In: ICML (2011)
Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: Advances in Neural Information Processing Systems, pp. 3856–3866 (2017)
Sankisa, A., Punjabi, A., Katsaggelos, A.K.: Temporal capsule networks for video motion estimation and error concealment. SIViP 14(7), 1369–1377 (2020)
Shao, H.M., Wang, J.G., Wang, Y., Yao, Y., Liu, J.: EEG-based emotion recognition with deep convolution neural network. In: 2019 IEEE 8th Data Driven Control and Learning Systems Conference (DDCLS), IEEE, pp. 1225–1229 (2019)
Valenzi, S., Islam, T., Jurica, P., Cichocki, A.: Individual classification of emotions using EEG. J. Biomed. Sci. Eng. 7, 1–17 (2014)
Welch, P.: The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Trans. Audio Electroacoust. 15(2), 70–73 (1967)
Wu, J.: Introduction to convolutional neural networks. Natl. Key Lab Novel Softw. Technol. 5, 23 (2017)
Zheng, W.L., Lu, B.L.: Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans. Auton. Ment. Dev. 7(3), 162–175 (2015)
Zheng, W.L., Guo, H.T., Lu, B.L.: Revealing critical channels and frequency bands for emotion recognition from EEG with deep belief network. In: 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER), IEEE, pp. 154–157 (2015)
Zubair, M., Kim, J., Yoon, C.: An automated ecg beat classification system using convolutional neural networks. In: 2016 6th International Conference on IT Convergence and Security (ICITCS), IEEE, pp. 1–5 (2016)
This work was supported by Scientific Research Projects Unit of Karabuk University under project number FDK-2020-2309. The authors appreciate the financial and scientific support.
Conflict of interest
The authors declare that they have no conflict of interest.
Human and animal rights
No human or animal participants were used in this study. Data used in this work were publicly available and granted by the original database owner.
All the authors are well informed about this submission.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Cizmeci, H., Ozcan, C. Enhanced deep capsule network for EEG-based emotion recognition. SIViP 17, 463–469 (2023). https://doi.org/10.1007/s11760-022-02251-x
- Emotion recognition
- Feature extraction
- Deep learning
- Capsule network