Abstract
Motor imagery (MI) based brain-computer interfaces (BCIs) decode the users’ intentions from electroencephalography (EEG) to achieve information control and interaction between the brain and external devices. In this paper, firstly, we apply Riemannian geometry to the covariance matrix extracted by spatial filtering to obtain robust and distinct features. Then, a multiscale temporal-spectral segmentation scheme is developed to enrich the feature dimensionality. In order to determine the optimal feature configurations, we utilize a linear learning-based temporal window and spectral band (TWSB) selection method to evaluate the feature contributions, which efficiently reduces the redundant features and improves the decoding efficiency without excessive loss of accuracy. Finally, support vector machines are used to predict the classification labels based on the selected MI features. To evaluate the performance of our model, we test it on the publicly available BCI Competition IV dataset 2a and 2b. The results show that the method has an average accuracy of 79.1% and 83.1%, which outperforms other existing methods. Using TWSB feature selection instead of selecting all features improves the accuracy by up to about 6%. Moreover, the TWSB selection method can effectively reduce the computational burden. We believe that the framework reveals more interpretable feature information of motor imagery EEG signals, provides neural responses discriminative with high accuracy, and facilitates the performance of real-time MI-BCI.
Graphical Abstract
Similar content being viewed by others
Data availability
Data is publicly available online.
References
Ramadan RA, Vasilakos AV (2017) Brain computer interface: control signals review. Neurocomputing 223:26–44. https://doi.org/10.1016/j.neucom.2016.10.024
Baniqued PDE, Stanyer EC, Awais M, Alazmani A, Jackson AE, Mon-Williams MA, Mushtaq F, Holt RJ (2021) Brain-computer interface robotics for hand rehabilitation after stroke: a systematic review. J NeuroEng Rehabil 18(1):15. https://doi.org/10.1186/s12984-021-00820-8
Abiri R, Borhani S, Sellers EW, Jiang Y, Zhao X (2019) A comprehensive review of EEG-based brain-computer interface paradigms. J Neural Eng 16(1):011001. https://doi.org/10.1088/1741-2552/aaf12e
Phadikar S, Sinha N, Ghosh R (2023) Unsupervised feature extraction with autoencoders for EEG based multiclass motor imagery BCI. Expert Syst Appl 213:118901. https://doi.org/10.1016/j.eswa.2022.118901
Kumar S, Sharma A (2018) A new parameter tuning approach for enhanced motor imagery EEG signal classification. Med Biol Eng Comput 56(10):1861–1874. https://doi.org/10.1007/s11517-018-1821-4
Pichiorri F, Morone G, Petti M, Toppi J, Pisotta I, Molinari M, Paolucci S, Inghilleri M, Astolfi L, Cincotti F, Mattia D (2015) Brain-computer interface boosts motor imagery practice during stroke recovery. Ann Neurol 77(5):851–865. https://doi.org/10.1002/ana.24390
Li H, Zhang D, Xie J (2023) MI-DABAN: a dual-attention-based adversarial network for motor imagery classification. Comput Biol Med 152:106420. https://doi.org/10.1016/j.compbiomed.2022.106420
Kappenman ES, Farrens JL, Zhang W, Stewart AX, Luck SJ (2021) ERP CORE: an open resource for human event-related potential research. Neuroimage 225:117465. https://doi.org/10.1016/j.neuroimage.2020.117465
Liu X, Xie Q, Lv J, Huang H, Wang W (2021) P300 event-related potential detection using one-dimensional convolutional capsule networks. Expert Syst Appl 174:114701. https://doi.org/10.1016/j.eswa.2021.114701
Nakanishi M, Wang Y, Wang Y-T, Mitsukura Y, Jung T-P (2014) A high-speed brain speller using steady-state visual evoked potentials. Int J Neural Syst 24(6):1450019. https://doi.org/10.1142/s0129065714500191
Nakanishi M, Wang Y, Chen X, Wang Y-T, Gao X, Jung T-P (2018) Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis. IEEE Trans Biomed Eng 65(1):104–112. https://doi.org/10.1109/tbme.2017.2694818
Al-Saegh A, Dawwd SA, Abdul-Jabbar JM (2021) Deep learning for motor imagery EEG-based classification: a review. Biomed Signal Process Control 63:102172. https://doi.org/10.1016/j.bspc.2020.102172
Sadiq MT, Yu X, Yuan Z, Aziz MZ, Siuly S, Ding W (2022) A matrix determinant feature extraction approach for decoding motor and mental imagery EEG in subject-specific tasks. IEEE Trans Cognit Dev Syst 14(2):375–387. https://doi.org/10.1109/tcds.2020.3040438
Zheng M, Yang B, Xie Y (2020) EEG classification across sessions and across subjects through transfer learning in motor imagery-based brain-machine interface system. Med Biol Eng Comput 58(7):1515–1528. https://doi.org/10.1007/s11517-020-02176-y
Liu GY, Tian L, Zhou WD (2022) Multiscale time-frequency method for multiclass motor imagery brain computer interface. Comput Biol Med 143:105299. https://doi.org/10.1016/j.compbiomed.2022.105299
Liu S, Fu W, Wei C, Ma F, Cui N, Shan X, Zhang Y (2022) Interference of unilateral lower limb amputation on motor imagery rhythm and remodeling of sensorimotor areas. Front Hum Neurosci 16:1011463. https://doi.org/10.3389/fnhum.2022.1011463
Gao L, Wang J, Chen L (2013) Event-related desynchronization and synchronization quantification in motor-related EEG by Kolmogorov entropy. J Neural Eng 10(3):036023. https://doi.org/10.1088/1741-2560/10/3/036023
Guggenberger R, Kraus D, Naros G, Leao MT, Ziemann U, Gharabaghi A (2018) Extended enhancement of corticospinal connectivity with concurrent cortical and peripheral stimulation controlled by sensorimotor desynchronization. Brain Stimulat 11(6):1331–1335. https://doi.org/10.1016/j.brs.2018.08.012
Wu Y, Ge Y (2013) A novel method for motor imagery EEG adaptive classification based biomimetic pattern recognition. Neurocomputing 116:280–290. https://doi.org/10.1016/j.neucom.2012.03.030
Fang H, Jin J, Daly I, Wang XY (2022) Feature extraction method based on filter banks and Riemannian tangent space in motor-imagery BCI. IEEE J Biomed Health Inf 26(6):2504–2514. https://doi.org/10.1109/jbhi.2022.3146274
Talukdar U, Hazarika SM, Gan JQ (2020) Adaptive feature extraction in EEG-based motor imagery BCI: tracking mental fatigue. J Neural Eng 17(1):016020. https://doi.org/10.1088/1741-2552/ab53f1
Zhang Y, Nam CS, Zhou G, Jin J, Wang X, Cichocki A (2019) Temporally constrained sparse group spatial patterns for motor imagery BCI. IEEE Trans Cybern 49(9):3322–3332. https://doi.org/10.1109/tcyb.2018.2841847
Chen J, Leong YC, Honey CJ, Yong CH, Norman KA, Hasson U (2017) Shared memories reveal shared structure in neural activity across individuals. Nat Neurosci 20(1):115–125. https://doi.org/10.1038/nn.4450
Zhang J, Wang X, Xu B, Wu Y, Lou X, Shen X (2023) An overview of methods of left and right foot motor imagery based on Tikhonov regularisation common spatial pattern. Med Biol Eng Comput. https://doi.org/10.1007/s11517-023-02780-8
Wu S-L, Liu Y-T, Hsieh T-Y, Lin Y-Y, Chen C-Y, Chuang C-H, Lin C-T (2017) Fuzzy integral with particle swarm optimization for a motor-imagery-based brain-computer interface. IEEE Trans Fuzzy Syst 25(1):21–28. https://doi.org/10.1109/tfuzz.2016.2598362
Ang KK, Chin ZY, Wang C, Guan C, Zhang H (2012) Filter bank common spatial pattern algorithm on BCI competition IV datasets 2a and 2b. Front Neurosci 6:39. https://doi.org/10.3389/fnins.2012.00039
Sakhavi S, Guan C, Yan S (2018) Learning temporal information for brain-computer interface using convolutional neural networks. IEEE Trans Neural Networks Learn Syst 29(11):5619–5629. https://doi.org/10.1109/tnnls.2018.2789927
McGinley MJ, Vinck M, Reimer J, Batista-Brito R, Zagha E, Cadwell CR, Tolias AS, Cardin JA, McCormick DA (2015) Waking state: rapid variations modulate neural and behavioral responses. Neuron 87(6):1143–1161. https://doi.org/10.1016/j.neuron.2015.09.012
Peterson V, Wyser D, Lambercy O, Spies R, Gassert R (2019) A penalized time-frequency band feature selection and classification procedure for improved motor intention decoding in multichannel EEG. J Neural Eng 16(1):016019. https://doi.org/10.1088/1741-2552/aaf046
Dennis EL, Thompson PM (2014) Functional brain connectivity using fMRI in aging and Alzheimerʼs disease. Neuropsychol Rev 24(1):49–62. https://doi.org/10.1007/s11065-014-9249-6
Rivet B, Souloumiac A, Attina V, Gibert G (2009) xDAWN algorithm to enhance evoked potentials: application to brain-computer interface. IEEE Trans Biomed Eng 56(8):2035–2043. https://doi.org/10.1109/tbme.2009.2012869
Yang B, Zhang T, Zhang Y, Liu W, Wang J, Duan K (2017) Removal of electrooculogram artifacts from electroencephalogram using canonical correlation analysis with ensemble empirical mode decomposition. Cogn Comput 9(5):626–633. https://doi.org/10.1007/s12559-017-9478-0
Lotte F, Bougrain L, Cichocki A, Clerc M, Congedo M, Rakotomamonjy A, Yger F (2018) A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update. J Neural Eng 15(3):031005. https://doi.org/10.1088/1741-2552/aab2f2
Yger F, Berar M, Lotte F (2017) Riemannian approaches in brain-computer interfaces: a review. IEEE Trans Neural Syst Rehabil Eng 25(10):1753–1762. https://doi.org/10.1109/tnsre.2016.2627016
Barachant A, Bonnet S, Congedo M, Jutten C (2013) Classification of covariance matrices using a Riemannian-based kernel for BCI applications. Neurocomputing 112:172–178. https://doi.org/10.1016/j.neucom.2012.12.039
He H, Wu D (2020) Transfer learning for brain-computer interfaces: a Euclidean space data alignment approach. IEEE Trans Biomed Eng 67(2):399–410. https://doi.org/10.1109/tbme.2019.2913914
Barachant A, Bonnet S, Congedo M, Jutten C (2012) Multiclass brain-computer interface classification by Riemannian geometry. IEEE Trans Biomed Eng 59(4):920–928. https://doi.org/10.1109/tbme.2011.2172210
Nguyen CH, Artemiadis P (2018) EEG feature descriptors and discriminant analysis under Riemannian Manifold perspective. Neurocomputing 275:1871–1883. https://doi.org/10.1016/j.neucom.2017.10.013
Tangermann M, Mueller K-R, Aertsen A, Birbaumer N, Braun C, Brunner C, Leeb R, Mehring C, Miller KJ, Mueller-Putz GR, Nolte G, Pfurtscheller G, Preissl H, Schalk G, Schoegl A, Vidaurre C, Waldert S, Blankertz B (2012) Review of the BCI competition IV. Front Neurosci 6:55. https://doi.org/10.3389/fnins.2012.00055
Brunner C, Leeb R, Müller-Putz G, Schlögl A, Pfurtscheller G (2008) BCI Competition 2008–Graz data set A. Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces), Graz University of Technology 16:1-6
Bauer R, Fels M, Vukelic M, Ziemann U, Gharabaghi A (2015) Bridging the gap between motor imagery and motor execution with a brain-robot interface. Neuroimage 108:319–327. https://doi.org/10.1016/j.neuroimage.2014.12.026
Yang Y, Chevallier S, Wiart J, Bloch I (2017) Subject-specific time-frequency selection for multi-class motor imagery-based BCIs using few Laplacian EEG channels. Biomed Signal Process Control 38:302–311. https://doi.org/10.1016/j.bspc.2017.06.016
Malan NS, Sharma S (2021) Time window and frequency band optimization using regularized neighbourhood component analysis for Multi-View Motor Imagery EEG classification. Biomed Signal Process Control 67:102550. https://doi.org/10.1016/j.bspc.2021.102550
Ma X, Chen W, Pei Z, Liu J, Huang B, Chen J (2023) A temporal dependency learning CNN with attention mechanism for MI-EEG decoding. IEEE Trans Neural Syst Rehabil Eng 31:3188–3200. https://doi.org/10.1109/tnsre.2023.3299355
Bennett JD, John SE, Grayden DB, Burkitt AN (2021) A neurophysiological approach to spatial filter selection for adaptive brain-computer interfaces. J Neural Eng 18(2):026017. https://doi.org/10.1088/1741-2552/abd51f
Miao M, Wang A, Liu F (2018) Application of artificial bee colony algorithm in feature optimization for motor imagery EEG classification. Neural Comput Appl 30(12):3677–3691. https://doi.org/10.1007/s00521-017-2950-7
Tugin S, Hernandez-Pavon JC, Ilmoniemi RJ, Nikulin VV (2016) Visual deviant stimuli produce mismatch responses in the amplitude dynamics of neuronal oscillations. Neuroimage 142:645–655. https://doi.org/10.1016/j.neuroimage.2016.07.024
Hassani A, Bertrand A, Moonen M (2016) GEVD-based low-rank approximation for distributed adaptive node-specific signal estimation in wireless sensor networks. IEEE Trans Signal Process 64(10):2557–2572. https://doi.org/10.1109/tsp.2015.2510973
Evert E, Vandecappelle M, De Lathauwer L (2022) Canonical polyadic decomposition via the generalized Schur Decomposition. IEEE Signal Process Lett 29:937–941. https://doi.org/10.1109/lsp.2022.3156870
Kalaganis FP, Laskaris NA, Oikonomou VP, Nikopolopoulos S, Kompatsiaris I (2022) Revisiting Riemannian geometry-based EEG decoding through approximate joint diagonalization. J Neural Eng 19(6):066030. https://doi.org/10.1088/1741-2552/aca4fc
Hersche M, Rellstab T, Schiavone P D, Cavigelli L, Benini L, Rahimi A, Ieee 2018 Fast and accurate multiclass inference for MI-BCIs using large multiscale temporal and spectral features. In: European Signal Processing Conference (EUSIPCO), ed M Hersche, et al. (Rome, ITALY pp 1690–1694
Moakher M (2005) A differential geometric approach to the geometric mean of symmetric positive-definite matrices. SIAM J Matrix Anal Appl 26(3):735–747. https://doi.org/10.1137/s0895479803436937
Navickas Z, Telksnys T, Timofejeva I, Marcinkevicius R, Ragulskis M (2018) An operator-based approach for the construction of closed-form solutions to fractional differential equations. Math Model Anal 23(4):665–685. https://doi.org/10.3846/mma.2018.040
Jin J, Miao Y, Daly I, Zuo C, Hu D, Cichocki A (2019) Correlation-based channel selection and regularized feature optimization for MI-based BCI. Neural Netw 118:262–270. https://doi.org/10.1016/j.neunet.2019.07.008
Li X, Zhu C, Xu C, Zhu J, Li Y, Wu S (2020) VR motion sickness recognition by using EEG rhythm energy ratio based on wavelet packet transform. Comput Methods Programs Biomed 188:105266. https://doi.org/10.1016/j.cmpb.2019.105266
Yan W, Zhao M, Fu Z, Pearlson GD, Sui J, Calhoun VD (2022) Mapping relationships among schizophrenia, bipolar and schizoaffective disorders: a deep classification and clustering framework using fMRI time series. Schizophr Res 245:141–150. https://doi.org/10.1016/j.schres.2021.02.007
Meyer BH, Ramirez Pozo AT, Nunan Zola WM (2022) Global and local structure preserving GPU t-SNE methods for large-scale applications. Expert Syst Appl 201:116918. https://doi.org/10.1016/j.eswa.2022.116918
Izzuddin TA, Safri NM, Othman MA (2021) Compact convolutional neural network (CNN) based on SincNet for end-to-end motor imagery decoding and analysis. Biocybern Biomed Eng 41(4):1629–1645. https://doi.org/10.1016/j.bbe.2021.10.0010168-8227
She Q, Hu B, Luo Z, Nguyen T, Zhang Y (2019) A hierarchical semi-supervised extreme learning machine method for EEG recognition. Med Biol Eng Comput 57:147–157. https://doi.org/10.1007/s11517-018-1875-3
Wang J, Yao L, Wang Y (2023) IFNet: an interactive frequency convolutional neural network for enhancing motor imagery decoding from EEG. IEEE Trans Neural Syst Rehabil Eng 31:1900–1911. https://doi.org/10.1109/TNSRE.2023.3257319
Luo T-j (2023) Parallel genetic algorithm based common spatial patterns selection on time-frequency decomposed EEG signals for motor imagery brain-computer interface. Biomed Signal Process Control 80:104397. https://doi.org/10.1016/j.bspc.2022.104397
Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ (2018) EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng 15(5):056013. https://doi.org/10.1088/1741-2552/aace8c
Ha K-W, Jeong J-W, Ieee 2019 Decoding two-class motor imagery EEG with capsule networks. In: IEEE International Conference on Big Data and Smart Computing (BigComp), (Kyoto, JAPAN pp 387–390
Zheng Q, Zhu F, Heng P-A (2018) Robust support matrix machine for single trial EEG classification. IEEE Trans Neural Syst Rehabil Eng 26(3):551–562. https://doi.org/10.1109/tnsre.2018.2794534
Molla MKI, Al Shiam A, Islam MR, Tanaka T (2020) Discriminative feature selection-based motor imagery classification using EEG signal. IEEE Access 8:98255–98265. https://doi.org/10.1109/access.2020.2996685
Zhang W, Song A, Zeng H, Xu B, Miao M (2022) The effects of bilateral phase-dependent closed-loop vibration stimulation with motor imagery paradigm. IEEE Trans Neural Syst Rehabil Eng 30:2732–2742. https://doi.org/10.1109/tnsre.2022.3208312
Rim B, Sung N-J, Min S, Hong M (2020) Deep learning in physiological signal data: a survey. Sensors 20(4):969. https://doi.org/10.3390/s20040969
Janani A, Sasikala M, Chhabra H, Shajil N, Venkatasubramanian G (2020) Investigation of deep convolutional neural network for classification of motor imagery fNIRS signals for BCI applications. Biomed Signal Process Control 62:102133. https://doi.org/10.1016/j.bspc.2020.102133
Schirrmeister RT, Springenberg JT, Fiederer LDJ, Glasstetter M, Eggensperger K, Tangermann M, Hutter F, Burgard W, Ball T (2017) Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp 38(11):5391–5420. https://doi.org/10.1002/hbm.23730
Zhao H, Zheng Q, Ma K, Li H, Zheng Y (2021) Deep representation-based domain adaptation for nonstationary EEG classification. IEEE Trans Neural Networks Learn Syst 32(2):535–545. https://doi.org/10.1109/tnnls.2020.3010780
Al-Saegh A, Dawwd SA, Abdul-Jabbar JM (2021) CutCat: an augmentation method for EEG classification. Neural Netw 141:433–443. https://doi.org/10.1016/j.neunet.2021.05.032
Zhang S, Yao L, Sun A, Tay Y (2019) Deep learning based recommender system: a survey and new perspectives. ACM Comput Surv 52(1):5. https://doi.org/10.1145/3285029
Tang X, Zhang N, Zhou J, Liu Q (2017) Hidden-layer visible deep stacking network optimized by PSO for motor imagery EEG recognition. Neurocomputing 234:1–10. https://doi.org/10.1016/j.neucom.2016.12.039
Zhang L, Wen D, Li C, Zhu R (2020) Ensemble classifier based on optimized extreme learning machine for motor imagery classification. J Neural Eng 17(2):026004. https://doi.org/10.1088/1741-2552/ab7264
Funding
We acknowledge the support of the National Natural Science Foundation of China (grant number 61971374, 62073280), the Key Research and Development Project of Hebei Province (grant number 23372006D), and the Hebei Key Laboratory Project (grant number 202250701010046).
Author information
Authors and Affiliations
Contributions
Zhaohui Li: conceptualization, formal analysis, investigation, writing — original draft, writing — review and editing. Xiaohui Tan: formal analysis, methodology, software, writing — original draft. Xinyu Li: validation, visualization, writing — review and editing. Liyong Yin: conceptualization, writing — review and editing.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Li, Z., Tan, X., Li, X. et al. Multiclass motor imagery classification with Riemannian geometry and temporal-spectral selection. Med Biol Eng Comput (2024). https://doi.org/10.1007/s11517-024-03103-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11517-024-03103-1