Multimodal Classification with Deep Convolutional-Recurrent Neural Networks for Electroencephalography

  • Chuanqi Tan
  • Fuchun SunEmail author
  • Wenchang Zhang
  • Jianhua Chen
  • Chunfang Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10635)


Electroencephalography (EEG) has become the most significant input signal for brain computer interface (BCI) based systems. However, it is very difficult to obtain satisfactory classification accuracy due to traditional methods can not fully exploit multimodal information. Herein, we propose a novel approach to modeling cognitive events from EEG data by reducing it to a video classification problem, which is designed to preserve the multimodal information of EEG. In addition, optical flow is introduced to represent the variant information of EEG. We train a deep neural network (DNN) with convolutional neural network (CNN) and recurrent neural network (RNN) for the EEG classification task by using EEG video and optical flow. The experiments demonstrate that our approach has many advantages, such as more robustness and more accuracy in EEG classification tasks. According to our approach, we designed a mixed BCI-based rehabilitation support system to help stroke patients perform some basic operations.


Multimodal EEG classification Optical flow Deep learning CNN RNN 



This work was supported by the National Natural Fund: 91420302 and 91520201. Thanks to the contributors of the open source software used in our system.


  1. 1.
    Amiri, S., Fazel-Rezai, R., Asadpour, V.: A review of hybrid brain-computer interface systems. Adv. Hum. Comput. Interact. 2013, 1 (2013)CrossRefGoogle Scholar
  2. 2.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  3. 3.
    Min, S., Lee, B., Yoon, S.: Deep learning in bioinformatics. Briefings in Bioinformatics, p. bbw068 (2016)Google Scholar
  4. 4.
    Schmidhuber, J.: Deep learning in neural networks: an overview. Neural networks 61, 85–117 (2015)CrossRefGoogle Scholar
  5. 5.
    Ng, Y.H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., Toderici, G.: Beyond short snippets: deep networks for video classification. In: Computer Vision and Pattern Recognition, pp. 4694–4702 (2015)Google Scholar
  6. 6.
    Yu, D., Deng, L.: Automatic Speech Recognition: A Deep Learning Approach. Springer, London (2014). doi: 10.1007/978-1-4471-5779-3 zbMATHGoogle Scholar
  7. 7.
    Kaiser, V., Kreilinger, A., Müller-Putz, G.R., Neuper, C.: First steps toward a motor imagery based stroke bci: new strategy to set up a classifier. Front Neurosci. 5, 86 (2011)CrossRefGoogle Scholar
  8. 8.
    Waldert, S., Pistohl, T., Braun, C., Ball, T., Aertsen, A., Mehring, C.: A review on directional information in neural signals for brain-machine interfaces. J. Physiol. Paris 103(3), 244–254 (2009)CrossRefGoogle Scholar
  9. 9.
    Ramoser, H., Muller-Gerking, J., Pfurtscheller, G.: Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans. Rehabil. Eng. 8(4), 441–446 (2000)CrossRefGoogle Scholar
  10. 10.
    Aghaei, A.S., Mahanta, M.S., Plataniotis, K.N.: Separable common spatio-spectral patterns for motor imagery bci systems. IEEE Trans. Biomed. Eng. 63(1), 15–29 (2016)CrossRefGoogle Scholar
  11. 11.
    Verma, G.K., Tiwary, U.S.: Multimodal fusion framework: a multiresolution approach for emotion classification and recognition from physiological signals. NeuroImage 102, 162–172 (2014)CrossRefGoogle Scholar
  12. 12.
    Bashivan, P., Rish, I., Yeasin, M., Codella, N.: Learning representations from EEG with deep recurrent-convolutional neural networks. Computer Science (2015)Google Scholar
  13. 13.
    Tan, C., Sun, F., Zhang, W., Liu, S., Liu, C.: Spatial and spectral features fusion for EEG classification during motor imagery in bci. In: 2017 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), pp. 309–312. IEEE (2017)Google Scholar
  14. 14.
    Mamoshina, P., Vieira, A., Putin, E., Zhavoronkov, A.: Applications of deep learning in biomedicine. Mol. Pharm. 13(5), 1445 (2016)CrossRefGoogle Scholar
  15. 15.
    Greenspan, H., Ginneken, B.V., Summers, R.M.: Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging 35(5), 1153–1159 (2016)CrossRefGoogle Scholar
  16. 16.
    An, X., Kuang, D., Guo, X., Zhao, Y., He, L.: A deep learning method for classification of eeg data based on motor imagery. In: International Conference on Intelligent Computing, pp. 203–210 (2014)Google Scholar
  17. 17.
    Cecotti, H., Graser, A.: Convolutional neural networks for p300 detection with application to brain-computer interfaces. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 433 (2011)CrossRefGoogle Scholar
  18. 18.
    Stober, S., Cameron, D.J., Grahn, J.A.: Using convolutional neural networks to recognize rhythm stimuli from electroencephalography recordings. In: Advances in Neural Information Processing Systems, pp. 1449–1457 (2014)Google Scholar
  19. 19.
    Soleymani, M., Asghariesfeden, S., Pantic, M., Fu, Y.: Continuous emotion detection using EEG signals and facial expressions. In: IEEE International Conference on Multimedia and Expo, pp. 1–6 (2014)Google Scholar
  20. 20.
    Li, J., Struzik, Z., Zhang, L., Cichocki, A.: Feature learning from incomplete EEG with denoising autoencoder. Neurocomputing 165, 23–31 (2015)CrossRefGoogle Scholar
  21. 21.
    Farneback, G.: Two-frame motion estimation based on polynomial expansion. In: Scandinavian Conference on Image Analysis, pp. 363–370 (2003)Google Scholar
  22. 22.
    Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: continual prediction with lstm. neural computation. Neural Comput. 12(10), 2451–2471 (2000)CrossRefGoogle Scholar
  23. 23.
    Cho, K., Merrienboer, B.V., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using rnn encoder-decoder for statistical machine translation. Computer Science (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Chuanqi Tan
    • 1
  • Fuchun Sun
    • 1
    Email author
  • Wenchang Zhang
    • 1
  • Jianhua Chen
    • 1
  • Chunfang Liu
    • 1
  1. 1.State Key Laboratory of Intelligent Technology and Systems, Tsinghua National Laboratory for Information Science and Technology (TNList), Department of Computer Science and TechnologyTsinghua UniversityBeijingChina

Personalised recommendations