Abstract
Brain Computer Interface (BCI) systems have the potential to provide a communication tool using non-invasive signals which can be applied to various fields including neuro-rehabilitation and entertainment. Interpreting multi-class movement intentions in a real time setting to control external devices such as robotic arms remains to be one of the main challenges in the BCI field. We propose a learning framework to decode upper limb movement intentions before and during the movement execution (ME) with the inclusion of motor imagery (MI) trials. The design of the framework allows the system to evaluate the uncertainty of the classification output and respond accordingly. The EEG signals collected during MI and ME trials are fed into a hybrid architecture consisting of Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) with limited pre-processing. Outcome of the proposed approach shows the potential to anticipate the intended movement direction before the onset of the movement, while waiting to reach a certainty level by potentially observing more EEG data from the beginning of the actual movement before sending control commands to the robot to avoid undesired outcomes. Presented results indicate that both the accuracy and the confidence level of the model improves with the introduction of MI trials right before the movement execution. Our results confirm the possibility of the proposed model to contribute to real-time and continuous decoding of movement directions for robotic applications.
This work was supported in part by the ERC (European Research Council), Swedish Research Council and EnTimeMent (EU Horizon 2020 FET PROACTIVE project).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Barsotti, M., Leonardis, D., Vanello, N., Bergamasco, M., Frisoli, A.: Effects of continuous kinaesthetic feedback based on tendon vibration on motor imagery BCI performance. IEEE Trans. Neural Syst. Rehabil. Eng. 26(1), 105–114 (2018). https://doi.org/10.1109/TNSRE.2017.2739244
Buerkle, A., Eaton, W., Lohse, N., Bamber, T., Ferreira, P.: EEG based arm movement intention recognition towards enhanced safety in symbiotic human-robot collaboration. Robot. Comput.-Integr. Manuf. 70, 102137 (2021). https://doi.org/10.1016/j.rcim.2021.102137, https://www.sciencedirect.com/science/article/pii/S0736584521000223
Dose, H., Møller, J.S., Iversen, H.K., Puthusserypady, S.: An end-to-end deep learning approach to MI-EEG signal classification for BCIS. Expert Syst. Appl. 114, 532–542 (2018). https://doi.org/10.1016/j.eswa.2018.08.031
Edelman, B.J., et al.: Noninvasive neuroimaging enhances continuous neural tracking for robotic device control. Sci. Robot. 4(31), eaaw6844 (2019). https://doi.org/10.1126/scirobotics.aaw6844
Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: International Conference on Machine Learning, pp. 1050–1059. PMLR (2016)
Ieracitano, C., Mammone, N., Hussain, A.E.A.: A novel explainable machine learning approach for EEG-based brain-computer interface systems. Neural Comput. Appl. (2021). https://doi.org/10.1007/s00521-020-05624-w
Lawhern, V.J., Solon, A.J., Waytowich, N.R., Gordon, S.M., Hung, C.P., Lance, B.J.: EEGNET: a compact convolutional neural network for EEG-based brain-computer interfaces. J. Neural Eng. 15(5), 056013 (2018). https://doi.org/10.1088/1741-2552/aace8c
Jeong, J.H., et al.: Multimodal signal dataset for 11 intuitive movement tasks from single upper extremity during multiple recording sessions. GigaScience 9(10) (2020). https://doi.org/10.1093/gigascience/giaa098
Jeong, J.H., Shim, K.H., Kim, D.J., Lee, S.W.: Brain-controlled robotic arm system based on multi-directional CNN-BILSTM network using EEG signals. IEEE Trans. Neural Syst. Rehabil. Eng. 28(5), 1226–1238 (2020). https://doi.org/10.1109/TNSRE.2020.2981659
Kahn, G., Villaflor, A., Pong, V., Abbeel, P., Levine, S.: Uncertainty-aware reinforcement learning for collision avoidance. arXiv preprint arXiv:1702.01182 (2017)
Lee, D.Y., Jeong, J.H., Shim, K.H., Lee, S.W.: Decoding movement imagination and execution from EEG signals using BCI-transfer learning method based on relation network. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1354–1358 (2020). https://doi.org/10.1109/ICASSP40776.2020.9052997
Livezey, J.A., Glaser, J.I.: Deep learning approaches for neural decoding: from CNNs to lSTMS and spikes to fMRI. arXiv (2020). https://doi.org/10.48550/ARXIV.2005.09687
Lotte, F.E.A.: A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update. J. Neural Eng. 15, 031005 (2018). https://doi.org/10.1088/1741-2552/aab2f2
Mammone, N., Ieracitano, C., Morabito, F.C.: A deep CNN approach to decode motor preparation of upper limbs from time–frequency maps of EEG signals at source level. Neural Networks 124, 357–372 (2020). https://doi.org/10.1016/j.neunet.2020.01.027, https://www.sciencedirect.com/science/article/pii/S089360802030037X
Mammone, N., Ieracitano, C., Morabito, F.C.: MPnnet: a motion planning decoding convolutional neural network for EEG-based brain computer interfaces. In: 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2021). https://doi.org/10.1109/IJCNN52387.2021.9534028
Mercado, L., Quiroz-Compean, G., Azorín, J.M.: Analyzing the performance of segmented trajectory reconstruction of lower limb movements from EEG signals with combinations of electrodes, gaps, and delays. Biomed. Sig. Process. Control 68, 102783 (2021). https://doi.org/10.1016/j.bspc.2021.102783, https://www.sciencedirect.com/science/article/pii/S1746809421003803
Miller, K.J., Schalk, G., Fetz, E.E., den Nijs, M., Ojemann, J.G., Rao, R.P.N.: Cortical activity during motor execution, motor imagery, and imagery-based online feedback. Proc. Natl. Acad. Sci. 107(9), 4430–4435 (2010). https://doi.org/10.1073/pnas.0913697107, https://www.pnas.org/content/107/9/4430
Müller-Putz, G., Schwarz, A., Pereira, J., Ofner, P.: Chapter 2 - from classic motor imagery to complex movement intention decoding: the noninvasive GRAZ-BCI approach. In: Coyle, D. (ed.) Brain-Computer Interfaces: Lab Experiments to Real-World Applications, Progress in Brain Research, vol. 228, pp. 39–70. Elsevier (2016). https://doi.org/10.1016/bs.pbr.2016.04.017, https://www.sciencedirect.com/science/article/pii/S0079612316300437
Ofner, P., Schwarz, A., Pereira, J., Müller-Putz, G.R.: Upper limb movements can be decoded from the time-domain of low-frequency EEG. PLoS ONE 12(8), 1–24 (2017). https://doi.org/10.1371/journal.pone.0182578
Pereira, J., Ofner, P., Schwarz, A., Sburlea, A.I., Müller-Putz, G.R.: EEG neural correlates of goal-directed movement intention. NeuroImage 149, 129–140 (2017). https://doi.org/10.1016/j.neuroimage.2017.01.030, https://www.sciencedirect.com/science/article/pii/S1053811917300368
Pfurtscheller, G., Brunner, C., Schlögl, A., Lopes da Silva, F.: Mu rhythm (de)synchronization and eeg single-trial classification of different motor imagery tasks. NeuroImage 31(1), 153–159 (2006). https://doi.org/10.1016/j.neuroimage.2005.12.003, https://www.sciencedirect.com/science/article/pii/S1053811905025140
Pfurtscheller, G., Neuper, C.: Motor imagery activates primary sensorimotor area in humans. Neurosci. Lett. 239(2), 65–68 (1997). https://doi.org/10.1016/S0304-3940(97)00889-6, https://www.sciencedirect.com/science/article/pii/S0304394097008896
Sobierajewicz, J., Szarkiewicz, S., Przekoracka-Krawczyk, A., Jaśkowski, W., van der Lubbe, R.: To what extent can motor imagery replace motor execution while learning a fine motor skill? Adv. Cogn. Psychol. 12(4), 179–192 (2016). https://doi.org/10.5709/acp-0197-1
Valenti, A., Barsotti, M., Bacciu, D., Ascari, L.: A deep classifier for upper-limbs motor anticipation tasks in an online BCI setting. Bioengineering 8(2) (2021). https://doi.org/10.3390/bioengineering8020021, https://www.mdpi.com/2306-5354/8/2/21
Wang, H., et al.: Brain-controlled wheelchair review: From wet electrode to dry electrode, from single modal to hybrid modal, from synchronous to asynchronous. IEEE Access 9, 55920–55938 (2021). https://doi.org/10.1109/ACCESS.2021.3071599
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Demir Kanik, S.U., Yin, W., Guneysu Ozgur, A., Ghadirzadeh, A., Björkman, M., Kragic, D. (2022). Improving EEG-based Motor Execution Classification for Robot Control. In: Meiselwitz, G. (eds) Social Computing and Social Media: Design, User Experience and Impact. HCII 2022. Lecture Notes in Computer Science, vol 13315. Springer, Cham. https://doi.org/10.1007/978-3-031-05061-9_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-05061-9_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-05060-2
Online ISBN: 978-3-031-05061-9
eBook Packages: Computer ScienceComputer Science (R0)