Multimodal Classification with Deep Convolutional-Recurrent Neural Networks for Electroencephalography
Electroencephalography (EEG) has become the most significant input signal for brain computer interface (BCI) based systems. However, it is very difficult to obtain satisfactory classification accuracy due to traditional methods can not fully exploit multimodal information. Herein, we propose a novel approach to modeling cognitive events from EEG data by reducing it to a video classification problem, which is designed to preserve the multimodal information of EEG. In addition, optical flow is introduced to represent the variant information of EEG. We train a deep neural network (DNN) with convolutional neural network (CNN) and recurrent neural network (RNN) for the EEG classification task by using EEG video and optical flow. The experiments demonstrate that our approach has many advantages, such as more robustness and more accuracy in EEG classification tasks. According to our approach, we designed a mixed BCI-based rehabilitation support system to help stroke patients perform some basic operations.
KeywordsMultimodal EEG classification Optical flow Deep learning CNN RNN
This work was supported by the National Natural Fund: 91420302 and 91520201. Thanks to the contributors of the open source software used in our system.
- 3.Min, S., Lee, B., Yoon, S.: Deep learning in bioinformatics. Briefings in Bioinformatics, p. bbw068 (2016)Google Scholar
- 5.Ng, Y.H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., Toderici, G.: Beyond short snippets: deep networks for video classification. In: Computer Vision and Pattern Recognition, pp. 4694–4702 (2015)Google Scholar
- 12.Bashivan, P., Rish, I., Yeasin, M., Codella, N.: Learning representations from EEG with deep recurrent-convolutional neural networks. Computer Science (2015)Google Scholar
- 13.Tan, C., Sun, F., Zhang, W., Liu, S., Liu, C.: Spatial and spectral features fusion for EEG classification during motor imagery in bci. In: 2017 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), pp. 309–312. IEEE (2017)Google Scholar
- 16.An, X., Kuang, D., Guo, X., Zhao, Y., He, L.: A deep learning method for classification of eeg data based on motor imagery. In: International Conference on Intelligent Computing, pp. 203–210 (2014)Google Scholar
- 18.Stober, S., Cameron, D.J., Grahn, J.A.: Using convolutional neural networks to recognize rhythm stimuli from electroencephalography recordings. In: Advances in Neural Information Processing Systems, pp. 1449–1457 (2014)Google Scholar
- 19.Soleymani, M., Asghariesfeden, S., Pantic, M., Fu, Y.: Continuous emotion detection using EEG signals and facial expressions. In: IEEE International Conference on Multimedia and Expo, pp. 1–6 (2014)Google Scholar
- 21.Farneback, G.: Two-frame motion estimation based on polynomial expansion. In: Scandinavian Conference on Image Analysis, pp. 363–370 (2003)Google Scholar
- 23.Cho, K., Merrienboer, B.V., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using rnn encoder-decoder for statistical machine translation. Computer Science (2014)Google Scholar