Visualizing Emotional States: A Method Based on Human Brain Activity

  • Yanfang Long
  • Wanzeng KongEmail author
  • Xuanyu Jin
  • Jili Shang
  • Can Yang
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1072)


Recently, how to infer one’s emotional state by nonverbal components has attracted great attention from the scientific community. If we can decode and visualize the emotional states from human brain activity, what an amazing thing that would be? The research in this paper found a way to decode and visualize different emotional states from human brain activity. In our experiments, at first, the power spectral density (PSD) was extracted from EEG signals evoked by visual stimulation of different emotional facial images. PSD can be viewed as a clue containing specific emotional states in human brain. After that, we use the conditional variational auto-encoder (VAE) to decode and visualize the emotional state, which takes the extracted PSD feature as input and generates the corresponding image. Specifically, VAE is a framework consisting of an encoder and a decoder. The former is used to learn low-dimension potential features of specific emotional state from the input PSD, and the later outputs an image containing the corresponding emotional state. Finally, our method was trained and tested on the EEG data from six subjects while they were looking at images from the Chinese Facial Affective Picture System (CFAPS) and obtained some promising results.


Human brain activity Emotional states PSD VAE 



This work was supported by National Key R&D Program of China for Intergovernmental International Science and Technology Innovation Cooperation Project (2017YFE0116800), National Natural Science Foundation of China (61671193), Key Science and Technology Program of Zhejiang Province (2018C04012), Science and technology platform construction project of Fujian science and Technology Department (2015Y2001).


  1. 1.
  2. 2.
    Bao, F.S., Liu, X., Zhang, C.: PyEEG: an open source python module for EEG/MEG feature extraction. Comput. Intell. Neurosci. (2011)Google Scholar
  3. 3.
    DeYoe, E.A., et al.: Mapping striate and extrastriate visual areas in human cerebral cortex. Proc. Natl. Acad. Sci. 93(6), 2382–2386 (1996)CrossRefGoogle Scholar
  4. 4.
    Dieleman, S., van den Oord, A., Simonyan, K.: The challenge of realistic music generation: modelling raw audio at scale. In: Advances in Neural Information Processing Systems, pp. 7989–7999 (2018)Google Scholar
  5. 5.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  6. 6.
    Gupta, A., Agarwal, A., Singh, P., Rai, P.: A deep generative framework for paraphrase generation. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)Google Scholar
  7. 7.
    Haynes, J.D., Rees, G.: Neuroimaging: decoding mental states from brain activity in humans. Nat. Rev. Neurosci. 7(7), 523 (2006)CrossRefGoogle Scholar
  8. 8.
    Huth, A.G., de Heer, W.A., Griffiths, T.L., Theunissen, F.E., Gallant, J.L.: Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532(7600), 453 (2016)CrossRefGoogle Scholar
  9. 9.
    Huth, A.G., Lee, T., Nishimoto, S., Bilenko, N.Y., Vu, A.T., Gallant, J.L.: Decoding the semantic content of natural movies from human brain activity. Front. Syst. Neurosci. 10, 81 (2016)CrossRefGoogle Scholar
  10. 10.
    Kavasidis, I., Palazzo, S., Spampinato, C., Giordano, D., Shah, M.: Brain2image: converting brain signals into images. In: Proceedings of the 25th ACM International Conference on Multimedia, pp. 1809–1817. ACM (2017)Google Scholar
  11. 11.
    Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  12. 12.
    Lin, Y.P., et al.: EEG-based emotion recognition in music listening. IEEE Trans. Biomed. Eng. 57(7), 1798–1806 (2010)CrossRefGoogle Scholar
  13. 13.
    Miller, E.K., Cohen, J.D.: An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 24(1), 167–202 (2001)CrossRefGoogle Scholar
  14. 14.
    Naselaris, T., Prenger, R.J., Kay, K.N., Oliver, M., Gallant, J.L.: Bayesian reconstruction of natural images from human brain activity. Neuron 63(6), 902–915 (2009)CrossRefGoogle Scholar
  15. 15.
    Ray, W.J., Cole, H.W.: EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science 228(4700), 750–752 (1985)CrossRefGoogle Scholar
  16. 16.
    Razavi, A., van den Oord, A., Vinyals, O.: Generating diverse high-resolution images with VQ-VAE (2019)Google Scholar
  17. 17.
    Rozgić, V., Vitaladevuni, S.N., Prasad, R.: Robust EEG emotion classification using segment level decision fusion. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1286–1290. IEEE (2013)Google Scholar
  18. 18.
    Serban, I.V., et al.: A hierarchical latent variable encoder-decoder model for generating dialogues. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)Google Scholar
  19. 19.
    Spampinato, C., Palazzo, S., Kavasidis, I., Giordano, D., Souly, N., Shah, M.: Deep learning human mind for automated visual classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6809–6817 (2017)Google Scholar
  20. 20.
    Tirupattur, P., Rawat, Y.S., Spampinato, C., Shah, M.: Thoughtviz: visualizing human thoughts using generative adversarial network. In: 2018 ACM Multimedia Conference on Multimedia Conference, pp. 950–958. ACM (2018)Google Scholar
  21. 21.
    Vandenhende, S., De Brabandere, B., Neven, D., Van Gool, L.: A three-player GAN: generating hard samples to improve classification networks. arXiv preprint arXiv:1903.03496 (2019)
  22. 22.
    Wen, H., Shi, J., Zhang, Y., Lu, K.H., Cao, J., Liu, Z.: Neural encoding and decoding with deep learning for dynamic natural vision. Cereb. Cortex 28(12), 4136–4160 (2017)CrossRefGoogle Scholar
  23. 23.
    Xu, H., Plataniotis, K.N.: Affective states classification using EEG and semi-supervised deep learning approaches. In: 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP), pp. 1–6. IEEE (2016)Google Scholar
  24. 24.
    Yan, X., Yang, J., Sohn, K., Lee, H.: Attribute2Image: conditional image generation from visual attributes. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 776–791. Springer, Cham (2016). Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  • Yanfang Long
    • 1
  • Wanzeng Kong
    • 1
    • 2
    Email author
  • Xuanyu Jin
    • 1
  • Jili Shang
    • 1
  • Can Yang
    • 3
  1. 1.Hangzhou Dianzi UniversityHangzhouChina
  2. 2.Fujian Key Laboratory of Rehabilitation TechnologyFuzhouChina
  3. 3.Hong Kong University of Science and TechnologyKowloonHong Kong, China

Personalised recommendations