Multi frequency band fusion method for EEG signal classification

This paper proposes a novel convolutional neural network (CNN) fusion method for electroencephalography (EEG) motor imagery (MI) signal classification. The method is named MFBF, which stands for multifrequency band fusion. The MFBF method relies on filtering the input signal with different frequency bands and feeding each band signal to a duplicate of a CNN model; then, all duplicates are concatenated to form a fusion model. This paper also introduces the second release of Coleeg software, which is used for evaluation. The MFBF method has the advantage of the flexibility of choosing any model and any number of frequency bands. In the experimental evaluation, the CNN1D model and three frequency bands were used to form the CNN1D_MFBF model, and it was evaluated against the EEGNet_fusion model on three different datasets, which are: Physionet, BCI competition IV-2a, and a dataset from the Hungarian Academy of Sciences Research Centre for Natural Sciences (MTA-TTK). The CNN1D_MFBF model had comparable or better accuracy results with less than one-fifth of the training time, which is a significant advantage for the proposed method.


Introduction
Electroencephalography (EEG) signal analysis [1,2] has been used intensively in many applications such as diagnosis of brain-related diseases and rehabilitation of neuromuscular disorders [3][4][5], emotion recognition [6][7][8], brain-computer interface (BCI) [9][10][11][12], and many others. BCI has played an essential role in helping disabled people to communi-  [13][14][15], especially convolutional neural networks (CNNs) [16,17], have been extensively used in the field of BCI motor imagery (MI) signal analysis for their high classification accuracy and simple construction procedure. One successful approach that has been proposed in the literature is fusion CNN, which enhances classification accuracy by concatenating (i.e., fusing) the output of different CNNs. The following paragraph lists some of the recent work that has been carried out in the field of fusion CNNs. Yang et al. [18] proposed a combination of a long short-term memory (LSTM) network and a spatial CNN, where the spatial and temporal features are simultaneously learned from the EEG signals. A discrete wavelet transform was used to represent the EEG signals. Amin et al. [19] showed the benefits of fusing multilevel convolutional features from various CNN layers, their proposed model can learn spectral and temporal features from the EEG data, and it outperforms the models where no fusion is used, their proposed method achieved a 74.5% accuracy for the BCI competition IV-2a dataset. In another work, Amin et al. [20] proposed improved fusion CNN methods (MCNN) and CCNN that fuse different characteristics and architectures. Their methods capture spatial and temporal features from raw EEG data using different convolutional features. The MCNN and CCNN methods achieved 75.7% and 73.8%, respectively, for the BCI Competition IV-2a dataset, and 95.4% and 93.2%, respectively, for the High Gamma dataset. Roots et al. [21] proposed a multibranch CNN, namely EEGNet_fusion, that utilizes different hyperparameter values for each branch, and it achieved an accuracy of 84.1% for two classes in the Physionet dataset. Their model achieved higher accuracy than EEGNet [22], ShallowCon-vNet, and DeepConvNet [23]; however, it has four times more computational cost. Li et al. [24] proposed a multiscale fusion CNN based on an attention mechanism, where spatiotemporal features from different brain areas are extracted and fed to a fusion network to increase classification accuracy. The added attention mechanism improved the sensitivity of the network. Their proposed method achieved 79.9% for the BCI Competition IV-2a dataset. Li et al. [25] proposed a feature fusion algorithm for neural networks that combines CNN and LSTM networks and connects them in parallel. The spatial and temporal features are extracted by the CNN and the LSTM networks, respectively. They achieved 87.68% accuracy for the BCI Competition IV-2a dataset. Musallam et al. [26] proposed TCNet-Fusion, a fixed hyperparameter-based CNN model. In addition to fusion layers, different techniques were used in this model, such as separable convolution, depth-wise convolution, and temporal convolution networks (TCNs). The EEG signal undergoes two successive 1D convolutions along the time domain and channel-wise, and then an image-like representation is fed to the main TCN. Their model achieved 83.73% and 94.41% for the BCI Competition IV-2a and the High Gamma datasets, respectively. The rest of this paper is organized as follows: Sect. 2 describes the proposed method. Section 3 presents the second release of Coleeg software, and describes the used datasets. The evaluation setup and the experimental results are presented in Sect. 4. Finally, the paper is concluded in Sect. 5.

The proposed method
The proposed method is named multifrequency band fusion (MFBF). Its main idea is to divide the signal spectrum into multiple frequency bands and feed each band into a duplicate of the selected model. All the model duplicates are then concatenated to give the required classification. The proposed method, which is illustrated in Fig. 1, has the flexibility of choosing any neural network model and any number of frequency bands. This paper uses Coleeg software [27] for the evaluation process; the CNN1D model was selected to build the CNN1D_MFBF model. A block diagram that describes the CNN1D model is shown in Fig. 2. The spectrum of the signal is divided into the following frequency bands: 0.5-8.

Introduction to the second release of Coleeg software
Coleeg [27,29] is an open-source initiative for EEG MI signal classification. Its goal is to deal with different neural network models and datasets in a unified and simplified manner. Coleeg uses Google T M Colaboratory (or Colab), so there is no need to install any software on the computer. Coleeg consists of an application file (Coleeg2_App.ipynb) and a python module file (coleeg.py). The application file calls the utility   functions in the module file, which facilitates the evaluation process. In the second release of Coleeg, two models were added: CNN1D_MFBF, and EEGNet_fusion [21]. A new dataset from the Hungarian Academy of Sciences Research Centre for Natural Sciences (MTA-TTK) is also added to the two datasets that Coleeg already have, which are: Physionet [30] and BCI Competition IV-2a [31]. New utility functions were also added, which are: (1) divide_time: used to divide the time of a trial event into multiple portions, and that could be used for data augmentation.
(2) evaluate_model: model evaluation is now done using this function instead of in the application file (as in the first release). (3) get_data_ttk: used to load the data from the MTA-TTK dataset. (4) plot_results: used to plot the evaluation accuracy versus training epochs. (5) save_results: save the evaluation results to Google Drive. The first used dataset is Physionet, which has 109 subjects, 64 EEG sensor channels, and a 160 Hz sampling frequency. Five classes were considered, which are: rest and the imag-  ined opening and closing of the left fist, right fist, both fists, and both feet. Subjects 88, 89, 92, 100, and 104 were excluded due to data annotation errors. The second dataset is BCI Competition IV-2a, which has 18 subjects, 22 EEG sensor channels, and a 250 Hz sampling frequency. Four classes were considered: the imagined movements of the left hand, right hand, both feet, and tongue. The third dataset is MTA-TTK, with 25 subjects, 63 EEG sensor channels, and a 500 Hz sampling frequency. Five classes were considered: rest, imagined movements of the left hand, right hand, left leg, and right leg. No filtering was applied to the original raw signals; however, a 0.5-Hz low-pass filter is used in this paper in order to remove the DC component from the signal and enhance its accuracy.

Evaluation setup and experimental results
The signals from the three datasets were resampled at 100 Hz, and the loaded data were balanced so that each subject had the same number of trial events for each class. The data are normalized to have zero mean and unity standard deviation. In this way, the batch normalization stage is not used in CNN1D and CNN1D_MFBF models. The evaluation process was carried out considering two scenarios. The first one is where no multiband filtering is used, and it was applied to the CNN1D and the EEGNet_fusion models. The second scenario is where the following frequency bands are used to filter the signal: 0.5-8.0 Hz, 8.0-13.0 Hz, and 13.0-40.0 Hz. The second scenario is applied to the CNN1D and CNN1D_MFBF models. It should be noted that the CNN1D model can accept single or multi frequency bands at its input. Cross-subject validation was carried out for each dataset by dividing the subjects into five groups and performing the evaluation for each group for 500 epochs. Figures 3, 4, and 5 show the mean accuracy versus epochs for both scenarios for Physionet, BCI Competition IV-2a, and MTA-TTK datasets, respectively. The evaluation time is also shown in Table 1. Figures 3, 4, and 5 show that when the CNN1D_MFBF model is compared to the EEGNet_fusion model, it gives comparable accuracy results for the BCI Competition IV-2a dataset and better accuracy results for Physionet and MTA-TTK datasets. The significant advantage is apparent when the evaluation time is considered, where the time taken by the CNN1D_MFBF is less than one-fifth of the time taken by the EEGNet_fusion model. The experimental results also show that feeding multiple bands for the CNN1D model increases the accuracy, even though no fusion is used.
In order to get a numerical insight regarding the accuracy, in Table 2  It is worth mentioning that the accuracy results are affected by the parameters used in the evaluation process. Some of the parameters used in this paper, such as cross-subject instead of within-subject validation, data balancing, unprocessed raw signal, and a higher number of classes, generally result in reduced accuracy values.

Conclusion
In this paper, a novel fusion method for CNNs is proposed. The method is named multifrequency band fusion (MFBF) and is based on dividing the input signal spectrum into multiple frequency bands and feeding each band into a duplicate of a CNN model. The output of the model duplicates is concatenated to form a fusion model. The method shows comparable or better results when compared to EEGNet_fusion model, with less than one-fifth of the training time. The evaluation was conducted using the second release of Coleeg software, which is an open-source initiative for dealing with different models and datasets for EEG MI signal classification in a unified and simplified manner.
Author Contributions All authors made substantial contributions to the concept, design, and revision of the paper. Methodology by Wahdow and Alnaanah, software development by Alnaanah, and project administration/supervision by Ulbert.
Funding Open access funding provided by Pázmány Péter Catholic University.

Availability of data and materials
The MTA-TTK dataset is private and not publicly available. The links for the Physionet and the BCI Competition IV-2a datasets are provided in the manuscript.

Code availability
The code is made freely available under GNU Public Licence (GPL), and its download link is provided in the manuscript.

Conflict of interest
The authors have no competing interests that might be perceived to influence the results and/or discussion reported in this paper.
Ethical approval Not applicable.

Consent to participate Not applicable.
Consent for publication All authors agreed on the final approval of the version to be published.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecomm ons.org/licenses/by/4.0/.