Modeling and classification of voluntary and imagery movements for brain–computer interface from fNIR and EEG signals through convolutional neural network
- 3 Downloads
Practical brain–computer interface (BCI) demands the learning-based adaptive model that can handle diverse problems. To implement a BCI, usually functional near-infrared spectroscopy (fNIR) is used for measuring functional changes in brain oxygenation and electroencephalography (EEG) for evaluating the neuronal electric potential regarding the psychophysiological activity. Since the fNIR modality has an issue of temporal resolution, fNIR alone is not enough to achieve satisfactory classification accuracy as multiple neural stimuli are produced by voluntary and imagery movements. This leads us to make a combination of fNIR and EEG with a view to developing a BCI model for the classification of the brain signals of the voluntary and imagery movements. This work proposes a novel approach to prepare functional neuroimages from the fNIR and EEG using eight different movement-related stimuli. The neuroimages are used to train a convolutional neural network (CNN) to formulate a predictive model for classifying the combined fNIR–EEG data. The results reveal that the combined fNIR–EEG modality approach along with a CNN provides improved classification accuracy compared to a single modality and conventional classifiers. So, the outcomes of the proposed research work will be very helpful in the implementation of the finer BCI system.
KeywordsVoluntary and imagery movements Functional near-infrared spectroscopy (fNIR) Electroencephalography (EEG) Modeling and classification Convolutional neural network (CNN) Brain–computer interface (BCI)
This work was partially supported by the Higher Education Quality Enhancement Project (HEQEP), UGC, Bangladesh; under Subproject “Postgraduate Research in BME”, CP#3472, KUET, Bangladesh.
Compliance with ethical standards
Conflict of interest
This research work has no conflict of interest to anyone.
- 1.Fantini S. Dynamic model for the tissue concentration and oxygen saturation of hemoglobin in relation to blood volume, flow velocity, and oxygen consumption: implications for functional neuroimaging and coherent hemodynamics spectroscopy (CHS). NeuroImage. 2014;85:202–21. https://doi.org/10.1016/j.neuroimage.2013.03.065.CrossRefGoogle Scholar
- 8.Basic principles of magnetoencephalography. MIT Class Notes. http://web.mit.edu/kitmitmeg/whatis.html. Accessed 24 Oct 2006.
- 10.Ayaz H, Onaral B, Izzetoglu K, Shewokis PA, McKendrick R, Parasuraman R. Continuous monitoring of brain dynamics with functional near infrared spectroscopy as a tool for neuro ergonomic research: empirical examples and a technological development. Front Hum Neurosci. 2013;7(871):1–13. https://doi.org/10.3389/fnhum.2013.00871.CrossRefGoogle Scholar
- 11.Ernst LH, Plichta MM, Lutz E, Zesewitz AK, Tupak SV, Dresler T, Ehlis AC, Fallgatter AJ. Prefrontal activation patterns of automatic and regulated approach avoidance reactions: a functional near-infrared spectroscopy (fNIRS) study. Cortex. 2013;49(1):131–42. https://doi.org/10.1016/j.cortex.2011.09.013.CrossRefGoogle Scholar
- 14.Ferrari M, Giannini I, Carpi A, Fasella P, Fieschi C, Zanette E. Non-invasive infrared monitoring of tissue oxygenation and circulatory parameters. In: XII world congress of angiology 1980, Athens, September 7–12.Google Scholar
- 15.Giannini I, Ferrari M, Carpi A, Fasella P. Rat brain monitoring by near-infrared spectroscopy: an assessment of possible clinical significance. Physiol Chem Phys. 1982;14(3):295–305.Google Scholar
- 21.Lee MH, Fazli S, Mehnert J, Lee SW. Improving the performance of brain-computer interface using multi-modal neuroimaging. In: 2nd IAPR Asian conference on pattern recognition, Naha, 2013, pp. 511–15. https://doi.org/10.1109/acpr.2013.132.
- 24.World Bank and WHO. World report on disability. World Bank and WHO; 2015. http://www.who.int/disabilities/world_report/2011/report.pdf.
- 26.Batula AM, Ayaz H, Kim YE. Evaluating a four-class motor-imagery-based optical brain-computer interface. In: 36th annual international conference of the IEEE engineering in medicine and biology society, Chicago, IL, 2014, pp. 2000–03. https://doi.org/10.1109/embc.2014.6944007.
- 27.Abbas W, Khan NA. FBCSP-based multi-class motor imagery classification using BP and TDP features. In: 2018 40th annual international conference of the IEEE engineering in medicine and biology society (EMBC), Honolulu, HI, 2018, pp. 215–18.Google Scholar
- 28.Mahmood A, Zainab R, Ahmad RB, Saeed M, Kamboh AM. Classification of multi-class motor imagery EEG using four band common spatial pattern. In: Annual international conference of the IEEE engineering in medicine and biology society (EMBC), Seogwipo, 2017, pp. 1034–37. https://doi.org/10.1109/embc.2017.8037003.
- 29.Mishra PK, Jagadish B, Kiran MPRS, Rajalakshmi P, Reddy DS. A novel classification for EEG based four class motor imagery using kullback-leibler regularized Riemannian manifold. In: 2018 IEEE 20th international conference on e-Health networking, applications and services (Healthcom), Ostrava, 2018, pp. 1–5. https://doi.org/10.1109/healthcom.2018.8531086.
- 31.León CL. Multilabel classification of EEG-based combined motor imageries implemented for the 3D control of a robotic arm. PhD thesis, Université de Lorraine, 2017.Google Scholar
- 32.Rahman MA. Matlab based graphical protocol. MATLAB Central File Exchange. https://www.mathworks.com/matlabcentral/fileexchange/69162-matlab-based-graphical-protocol. Accessed 20 Oct 2018.
- 33.World medical association declaration of Helsinki-ethical principles for medical research involving human subjects. Adopted by 64th WMA General Assembly, Fortaleza, Brazil, Special Communication: Clinical Review & Education, 2013.Google Scholar
- 37.Nyhof L. Biomedical signal filtering for noisy environments. PhD thesis, Deakin University, Australia, 2014. http://hdl.handle.net/10536/DRO/DU:30079016.
- 38.Chavan MS, Agarwala R, Uplane MD. Digital elliptic filter application for noise reduction in ECG signal. WSEAS Trans Electron. 2006;3(1):210–6.Google Scholar
- 39.Lutovac MD, Tosic DV, Evans BL. Filter design for signal processing. Upper Saddle River: Prentice Hall; 2001.Google Scholar
- 41.Orfanidis SJ. Introduction to signal processing. Upper Saddle River: Prentice Hall; 1996.Google Scholar
- 43.Khanam F, Rahman MA, Ahmad M. Evaluating alpha relative power of EEG signal during psychophysiological activities in Salat. In: International conference on innovations in science, engineering and technology (ICISET), 2018, Bangladesh, pp. 1–6. https://doi.org/10.1109/iciset.2018.8745614.
- 44.Ifeachor EC, Jervis BW. Digital signal processing: a practical approach. Boston: Addison Wesley; 1993.Google Scholar
- 49.Rahman MA, Ahmad M. Lie detection from single feature of functional near infrared spectroscopic (fNIRS) signals. In: 2nd international conference on electrical & electronic engineering (ICEEE 2017), 27–29 December, Rajshahi University of Engineering & Technology (RUET), Rajshahi, Bangladesh. https://doi.org/10.1109/ceee.2017.8412900.
- 51.Rahman MA, Khanam F, Ahmad M. Detection of effective temporal window for classification of motor imagery events from prefrontal hemodynamics. In: International conference on electrical, computer and communication engineering (ECCE), Cox’s Bazar, Bangladesh, 2019. https://doi.org/10.1109/ecace.2019.8679317.
- 52.Vakkuri A, Yli-Hankala A, Talja P, Mustola S, Tolvanen-Laakso H, Sampson T, Viertiö-Oja H. Time-frequency balanced spectral entropy as a measure of anesthetic drug effect in central nervous system during sevoflurane, propofol, and thiopental anesthesia. Acta Anaesthesiol Scand. 2004;48(2):145–53. https://doi.org/10.1111/j.0001-5172.2004.00323.x.CrossRefGoogle Scholar
- 53.Rahman MA. Topoplot for B-Alert X-10 9-channel EEG signal. MATLAB Central File Exchange. https://www.mathworks.com/matlabcentral/fileexchange/69991-topoplot-for-b-alert-x-10-9-channel-eeg-signal. Accessed 2 Apr 2019.