Skip to main content
SpringerLink
Log in

Recognize enhanced temporal-spatial-spectral features with a parallel multi-branch CNN and GRU

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Deep learning has been applied to the recognition of motor imagery electroencephalograms (MI-EEG) in brain-computer interface, and the performance results depend on data representation as well as neural network structure. Especially, MI-EEG is so complex with the characteristics of non-stationarity, specific rhythms, and uneven distribution; however, its multidimensional feature information is difficult to be fused and enhanced simultaneously in the existing recognition methods. In this paper, a novel channel importance (NCI) based on time–frequency analysis is proposed to develop an image sequence generation method (NCI-ISG) for enhancing the integrity of data representation and highlighting the contribution inequalities of different channels as well. Each electrode of MI-EEG is converted to a time–frequency spectrum by utilizing short-time Fourier transform; the corresponding part to 8–30 Hz is combined with random forest algorithm for computing NCI; and it is further divided into three sub-images covered by α (8–13 Hz), β1 (13–21 Hz), and β2 (21–30 Hz) bands; their spectral powers are further weighted by NCI and interpolated to 2-dimensional electrode coordinates, producing three main sub-band image sequences. Then, a parallel multi-branch convolutional neural network and gate recurrent unit (PMBCG) is designed to successively extract and identify the spatial-spectral and temporal features from the image sequences. Two public four-class MI-EEG datasets are adopted; the proposed classification method respectively achieves the average accuracies of 98.26% and 80.62% by 10-fold cross-validation experiment; and its statistical performance is also evaluated by multi-indexes, such as Kappa value, confusion matrix, and ROC curve. Extensive experiment results show that NCI-ISG + PMBCG can yield great performance on MI-EEG classification compared to state-of-the-art methods. The proposed NCI-ISG can enhance the feature representation of time–frequency-space domains and match well with PMBCG, which improves the recognition accuracies of MI tasks and demonstrates the preferable reliability and distinguishable ability.

Graphical Abstract

This paper proposes a novel channel importance (NCI) based on time–frequency analysis to develop an image sequences generation method (NCI-ISG) for enhancing the integrity of data representation and highlighting the contribution inequalities of different channels as well. Then, a parallel multi-branch convolutional neural network and gate recurrent unit (PMBCG) is designed to successively extract and identify the spatial-spectral and temporal features from the image sequences.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1:
Algorithm 2:
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References 

  1. Kevric J, Subasi A (2017) Comparison of signal decomposition methods in classification of EEG signals for motor-imagery BCI system. Biomed Signal Process Control 31:398–406. https://doi.org/10.1016/j.bspc.2016.09.007

    Article  Google Scholar 

  2. Santoro S, Buono VL, Corallo F, Cartella E, Micchia K, Palmeri R, Arcadi FA, Bramanti A, Marino S (2019) Motor imagery in stroke patients: a descriptive review on a multidimensional ability. Int J Neurosci 129(8):821–832. https://doi.org/10.1080/00207454.2019.1567509

    Article  PubMed  Google Scholar 

  3. Qiu S, Li Z, He W, Zhang L, Yang C, Su C (2017) Brain-machine interface and visual compressive sensing-based teleoperation control of an exoskeleton robot. IEEE Trans Fuzzy Syst 25(1):58–69. https://doi.org/10.1109/TFUZZ.2016.2566676

    Article  Google Scholar 

  4. Yadav D, Yadav S, Veer K (2020) A comprehensive assessment of brain computer interfaces: recent trends and challenges. J Neurosci Methods 346:108918. https://doi.org/10.1016/j.jneumeth.2020.108918

    Article  PubMed  Google Scholar 

  5. Rashid M, Sulaiman N, Majeed APPA, Musa RM, Nasir AFA, Bari BS, Khatun S (2020) Current status, challenges, and possible solutions of EEG-based brain-bomputer interface: a comprehensive review. Front Neurorobot 14:25. https://doi.org/10.3389/fnbot.2020.00025

  6. Ang KK, Chin ZY, Wang C, Guan C, Zhang H (2012) Filter bank common spatial pattern algorithm on bci competition iv datasets 2a and 2b. Front Neurosci 6:39. https://doi.org/10.3389/fnins.2012.00039

    Article  PubMed  PubMed Central  Google Scholar 

  7. Riyad M, Khalil M, Adib A (2021) A novel multi-scale convolutional neural network for motor imagery classification. Biomed Signal Process Control 68:102747. https://doi.org/10.1016/j.bspc.2021.102747

    Article  Google Scholar 

  8. Blanco-Diaz CF, Antelis JM, Ruiz-Olaya AF (2022) Comparative analysis of spectral and temporal combinations in CSP-based methods for decoding hand motor imagery tasks. J Neurosci Methods 371:109495. https://doi.org/10.1016/j.jneumeth.2022.109495

    Article  PubMed  Google Scholar 

  9. Kim Y, Ryu J, Kim KK, Took CC, Mandic DP, Park C (2016) Motor imagery classification using mu and beta rhythms of EEG with strong uncorrelating transform based complex common spatial patterns. Comput Intell Neurosci 2016:1489692. https://doi.org/10.1155/2016/1489692

    Article  PubMed  PubMed Central  Google Scholar 

  10. Tiwari S, Goel S, Bhardwaj A (2022) MIDNN-a classification approach for the EEG based motor imagery tasks using deep neural network. Appl Intell 52(2):4824–4843. https://doi.org/10.1007/s10489-021-02622-w

    Article  Google Scholar 

  11. Liu C, Jin J, Daly I, Sun H, Huang Y, Wang X, Cichocki A (2022) Bispectrum-based hybrid neural network for motor imagery classification. J Neurosci Methods 375:109593. https://doi.org/10.1016/j.jneumeth.2022.109593

    Article  PubMed  Google Scholar 

  12. Saeidi M, Karwowski W, Farahani FV, Fiok K, Taiar R, Hancock PA, Al-Juaid A (2021) Neural decoding of EEG signals with machine learning: a systematic review. Brain Sci 11(11):1525. https://doi.org/10.3390/brainsci11111525

    Article  PubMed  PubMed Central  Google Scholar 

  13. Zhu H, Forenzo D, He B (2022) On the deep learning models for EEG-based brain-computer interface using motor imagery. IEEE Trans Neural Syst Rehabil Eng 30:2283–2291. https://doi.org/10.1109/TNSRE.2022.3198041

    Article  PubMed  PubMed Central  Google Scholar 

  14. Tabar YR, Halici U (2017) A novel deep learning approach for classification of EEG motor imagery signals. J Neural Eng 14(1):016003. https://doi.org/10.1088/1741-2560/14/1/016003

    Article  PubMed  Google Scholar 

  15. Zhang K, Xu G, Han Z, Ma K, Zheng X, Chen L, Duan N, Zhang S (2020) Data augmentation for motor imagery signal classification based on a hybrid neural network. Sensors (Basel) 20(16):4485. https://doi.org/10.3390/s20164485

    Article  PubMed  Google Scholar 

  16. Xie J, Zhang J, Sun J, Ma Z, Qin L, Li G, Zhou H, Zhan Y (2022) A transformer-based approach combining deep learning network and spatial-temporal information for raw EEG classification. IEEE Trans Neural Syst Rehabil Eng 30:2126–2136. https://doi.org/10.1109/TNSRE.2022.3194600

    Article  PubMed  Google Scholar 

  17. Sun B, Liu Z, Wu Z, Mu C, Li T (2022) Graph convolution neural network based end-to-end channel selection and classification for motor imagery brain-computer interfaces, IEEE Trans. Ind Inform Online 1–10. https://doi.org/10.1109/TII.2022.3227736

  18. Sakhavi S, Guan C, Yan S (2018) Learning temporal information for brain computer interface using convolutional neural networks. IEEE Trans Neural Netw Learn Syst 29(11):5619–5629. https://doi.org/10.1109/TNNLS.2018.2789927

    Article  PubMed  Google Scholar 

  19. Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ (2018) EEGNet: a compact convolutional network for EEG-based brain-computer interfaces. J Neural Eng 15(5):056013.1–056013.17. https://doi.org/10.1088/1741-2552/aace8c

  20. Dose H, Moller JS, Iversen HK, Puthusserypady S (2018) An end-to-end deep learning approach to MI-EEG signal classification for BCIs. Expert Syst Appl 114:532–542. https://doi.org/10.1016/j.eswa.2018.08.031

    Article  Google Scholar 

  21. Li H, Ding M, Zhang R, Xiu C (2022) Motor imagery EEG classification algorithm based on CNN-LSTM feature fusion network. Biomed Signal Process Control 72:103342. https://doi.org/10.1016/j.bspc.2021.103342

    Article  Google Scholar 

  22. Liu X, Xiong S, Wang X, Liang T, Wang H, Liu X (2023) A compact multi-branch 1D convolutional neural network for EEG-based motor imagery classification. Biomed Signal Process Control 81:104456. https://doi.org/10.1016/j.bspc.2022.104456

    Article  Google Scholar 

  23. Zhang D, Yao L, Chen K, Wang S, Chang X, Liu Y (2020) Making sense of spatio-temporal preserving representations for EEG-based human intention recognition. IEEE T Cybern 50(7):3033–3044. https://doi.org/10.1109/TCYB.2019.2905157

    Article  Google Scholar 

  24. Li Y, Yang H, Li J, Chen D, Du M (2020) EEG-based intention recognition with deep recurrent-convolution neural network: performance and channel selection by Grad-CAM. Neurocomputing 415:225–233. https://doi.org/10.1016/j.neucom.2020.07.072

    Article  Google Scholar 

  25. Zhao X, Zhang H, Zhu G, You F, Kuang S, Sun L (2019) A multi-branch 3d convolutional neural network for eeg-based motor imagery classication. IEEE Trans Neural Syst Rehabil Eng 27(10):2164–2177. https://doi.org/10.1109/TNSRE.2019.2938295

    Article  PubMed  Google Scholar 

  26. Chen J, Yu ZL, Gu Z, Li Y (2020) Deep temporal-spatial feature learning for motor imagery-based brain-computer interfaces. IEEE Trans Neural Syst Rehabil Eng 28(11):2356–2366. https://doi.org/10.1109/TNSRE.2020.3023417

    Article  PubMed  Google Scholar 

  27. Yang L, Song Y, Jia X, Ma K, Xie L (2021) Two-branch 3D convolutional neural network for motor imagery EEG decoding. J Neural Eng 18(4):0460c7. https://doi.org/10.1088/1741-2552/ac17d6

  28. Huang W, Chang W, Yan G, Zhang Y, Yuan Y (2023) Spatio-spectral feature classification combining 3D-convolutional neural networks with long short-term memory for motor movement/imagery. Eng Appl Artif Intell 120:105862. https://doi.org/10.1016/j.engappai.2023.105862

    Article  Google Scholar 

  29. Lu N, Li T, Ren X (2016) Miao H (2016) A deep learning scheme for motor imagery classification based on restricted boltzmann machines. IEEE Trans Neural Syst Rehabil Eng 25(6):566–576. https://doi.org/10.1109/TNSRE.2016.2601240

    Article  PubMed  Google Scholar 

  30. Xu B, Zhang L, Song A, Wu C, Li W, Zhang D, Xu G, Li H, Zeng H (2019) Wavelet transform time-frequency image and convolutional network-based motor imagery EEG classification. IEEE Access 7:6084–6093. https://doi.org/10.1109/ACCESS.2018.2889093

    Article  Google Scholar 

  31. Chaudhary S, Taran S, Bajaj V, Sengur A (2019) Convolutional neural network based approach towards motor imagery tasks EEG signals classification. IEEE Sens J 19(12):4494–4500. https://doi.org/10.1109/JSEN.2019.2899645

    Article  Google Scholar 

  32. Qiao W, Bi X (2019) Deep spatial-temporal neural network for classification of eeg-based motor imagery. In: Proceedings of the 2019 international conference on artificial intelligence and computer science, AICS 2019, New York, NY, USA 265–272. https://doi.org/10.1145/3349341.3349414

  33. Li MA, Han JF, Duan LJ (2019) A novel MI-EEG imaging with the location information of electrodes. IEEE Access 8:3197–3211. https://doi.org/10.1109/ACCESS.2019.2962740

    Article  Google Scholar 

  34. Rammy SA, Abbas W, Mahmood SS, Riaz H, Rehman HU, Abideen RZU, Aqeel M, Zhang W (2021) Sequence-to-sequence deep neural network with spatio-spectro and temporal features for motor imagery classification. Biocybern Biomed Eng 41(1):97–110. https://doi.org/10.1016/j.bbe.2020.12.004

    Article  Google Scholar 

  35. Li MA, Peng WM, Yang JF (2021) Key band image sequences and a hybrid deep neural network for recognition of motor imagery EEG. IEEE Access 99:1–1. https://doi.org/10.1109/ACCESS.2021.3085865

    Article  Google Scholar 

  36. Wang L, Li M (2022) The quantitative application of channel importance in movement intention decoding. Biocybern Biomed Eng 42:630–645. https://doi.org/10.1016/j.bbe.2022.05.002

    Article  Google Scholar 

  37. Franke R (1982) Scattered data interpolation: tests of some methods. Math Comput 38:181–200. https://doi.org/10.1090/S0025-5718-1982-0637296-4

    Article  Google Scholar 

  38. Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng CK (2000) Stanley H E (2000) Physiobank, physiotoolkit, and physionet. Circulation 101(23):e215–e220. https://doi.org/10.1161/01.CIR.101.23.e215

    Article  CAS  PubMed  Google Scholar 

  39. Maaten LVD, Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res 9(86):2579–2605

    Google Scholar 

  40. Li M, Wang R, Xu D (2020) An improved composite multiscale fuzzy entropy for feature extraction of MI-EEG. Entropy (Basel) 22(12):1356. https://doi.org/10.3390/e22121356

  41. Tangermann M, Müller KR, Aertsen A, Birbaumer N, Braun C, Brunner C, Leeb R, Mehring C, Miller KJ, Müller-Putz GR, Nolte G, Pfurtscheller G, Preissl H, Schalk G, Schlögl A, Vidaurre C, Waldert S, Blankertz B (2012) Review of the BCI competition IV. Front Neurosci 6:55. https://doi.org/10.3389/fnins.2012.00055

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We would like to thank the provider of the dataset and all the people who have given us helpful suggestions and advice. The authors are obliged to the anonymous reviewers and the editor for carefully looking over the details and for useful comments that improved this paper.

Funding

The research was financially supported by the National Key Research and Development Program of China (Grant No. 2021YFA1000200) and the National Natural Science Foundation of China (Nos. 62173010, No. 11832003).

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Beijing University of Technology, Beijing, 100124, China

    Linlin Wang & Mingai Li

  2. Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing, 100124, China

    Mingai Li

  3. Engineering Research Center of Digital Community, Ministry of Education, Beijing, 100124, China

    Mingai Li

  4. Faculty of Environment and Life, Beijing University of Technology, Beijing, 100124, China

    Liyuan Zhang

Authors
  1. Linlin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingai Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Liyuan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingai Li.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Highlights

1. A novel channel importance (NCI) is measured with the time-frequency spectrums to highlight the contribution inequalities of different channels.

2. An NCI-based image sequences generation method, called NCI-ISG, is proposed to generate three main band image sequences for motor imagery (MI) EEG.

3. A parallel multi-branch convolutional neural network and gate recurrent unit is designed to match with the characteristics of three main band image sequences for decoding MI tasks.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, L., Li, M. & Zhang, L. Recognize enhanced temporal-spatial-spectral features with a parallel multi-branch CNN and GRU. Med Biol Eng Comput 61, 2013–2032 (2023). https://doi.org/10.1007/s11517-023-02857-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-023-02857-4

Keywords

Search

Navigation