Abstract
Under certain task conditions, error-related potential (ErrP) will be elicited, meaning that the subject is perceiving an error, responding to an external error, or engaging in a cognitive process of reinforcement learning. The detection of ErrP on a single trial basis has been studied and applied to improve all kinds of brain–computer interfaces (BCIs). However, the performance of this kind of detection is not currently good enough. In the paper, we proposed a novel method, called window-adjusted common spatial pattern (WACSP), for detecting ErrP in P300 BCI. In this method, the coefficient of determination was introduced to measure the difference of Electroencephalogram (EEG) signals on a channel at a moment and to guide the search of time windows in which EEG differences are significant, and common spatial pattern (CSP) was further used to capture the stable spatial patterns of EEG differences between correct and incorrect responses in each time window. WACSP and the commonly used methods were tested on the data sets that were built using the EEG signals acquired during the P300 BCI experiments with different feedback. The comparisons of accuracy, area under receiver operating characteristics curve (AUC) and F-measure show that WACSP significantly outperforms the commonly used methods. The proposed method can improve ErrP detection based on a single trial.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
The electrical activities in the brain are measured by the electrodes placed on the scalp as Electroencephalogram (EEG). In the late 1980s and early 1990s, Michael Falkensttein’s group [1] and William J. Gehring’s group [2] respectively observed the error-related negativity (ERN) in EEG. Twenty-five years later, Gehring et al. looked back on the events of discovering ERN in the paper [3]. ERN is a component of event-related potential in EEG, which is characterized by a negative deflection (Ne) at approximately 50–200 ms and a positive deflection (Pe) at approximately 200–500 ms following the error over the frontocentral and centroparietal areas of the brain [1,2,3,4]. On the other hand, the correct-related negativity (CRN) following the correct responses was also observed in EEG, showing a morphology similar to the ERN, but with a smaller amplitude. Generally, the difference waveform (error minus correct) between the ERN and CRN is considered to be error-related potential (ErrP) [4, 5]. The researchers on ErrP suggest that ErrP is elicited under certain task conditions [3, 4]. According to the different task conditions, ErrP is called response ErrP, feedback ErrP, interaction ErrP, observation ErrP, outcome ErrP, execution ErrP, and so on. For example, response ErrP occurs when a subject responds to external events as fast as possible, and feedback ErrP occurs when a subject perceives that given feedback on a task is incorrect.
The reinforcement learning theory of ERN suggests that error signals originate from the basal ganglia, spread to the anterior cingulate cortex and then to the cortex, and ERN is essential to the reinforcement learning of the brain [6]. The errors that are immediately correctable in control are believed to be represented by positive deflections originating from the posterior parietal cortex, which is hypothesized by many researchers to be involved with action conflict monitoring and movement correction [5, 6]. However, although we do know that people will have ERN when they make mistakes, a consensus view of ErrP is still elusive [3]. Many variations of ErrP that have more than two peaks with dissimilar latencies have been observed [4]. Additionally, some researchers reported that ErrP could also be observed in the motor, somatosensory, parietal, temporal, and pre-frontal areas [6].
Brain–computer interface (BCI) is a promising technology, providing a direct information pathway from the brain to external environments without depending on the peripheral nervous system and muscles. It is believed that BCI can be used to replace, restore, enhance, supplement or improve the natural output of the central nervous system. In recent years, the advancements in the field are inspiring people’s interest in BCI. More and more researchers devote themselves to the study of BCI. Various methods have been introduced to improve BCI. One of them is based on ErrP.
In 2010, Dal Seno et al. studied the use of ErrP in P300 BCI but hardly found any gain due to the low accuracy of recognizing ErrP [7]. In 2012, a series of research on using ErrP in P300 BCI were reported. Combaz et al. studied nine subjects’ EEG responses to correct and incorrect feedback of BCI and explored the possibility of detecting ErrP in a single trial and integrating the detector into the P300 Speller system [8]. Schmidt et al. carried out a performance assessment on incorporating ErrP into P300 BCI. Margaux et al. implemented a P300 BCI, for the first time, including an automatic correction function based on online ErrP detection [9]. The special P300 BCI would correct the selected character with the second best guess of a probabilistic classifier whenever ErrP was detected and was evaluated in 16 healthy subjects. Spüler et al. studied the approach to recognizing ErrP by an offline analysis on EEG data of six amyotrophic lateral sclerosis [10]. Thereafter, they developed an error correction system (ECS) that could recognize ErrP online and delete the letter that their P300 BCI responds wrong. Their conclusion is that ErrP as a secondary information source can be utilized to improve the performance of a P300 BCI. In 2015, Mainsah et al. compared the correction based on ErrP with the rectification on the basis of language models in P300 BCI [11]. In 2016, Zeyl et al. designed a two-step P300 speller including ECS and achieved an increase in the selection accuracy of P300 BCI [12]. In 2018, Cruz et al. implemented a P300 BCI including ECS [13]. Their P300 BCI selected the character with the second-highest probability as a candidate if ErrP was detected. The P300 BCI re-selected the first selected character if the second character elicited ErrP again when it was presented to the subject, otherwise accepted the candidate.
Besides P300 BCI, another kind of BCI was also improved by the means of recognizing ErrP [14]. In 2019, a review [4] summarized various studies of applying ErrP in BCIs, including robot control, wheelchair control, prosthetics, exoskeletons, gesture-enabled BCI, and so on. After that, some new similar research was still reported in succession. Yokota et al. studied the ERN of the players who were engaging in competitive video games and found that ERN could predict failures [15]. In order to facilitate using ErrP in BCI, Keyl et al. systematically examined the ErrP in the subjects with spinal cord injury (SCI) and compared the characteristics of ErrP in individuals with SCI and those of healthy subjects [16]. Chou et al. used ErrP to evaluate neural activities underlying action-monitoring dysfunction in patients with obstructive sleep apnea [17]. Ehrlich et al. studied the decoding of ErrP of persons who were interacting with a robot and the feasibility of validating robot actions by online detection of ErrP [18]. Lopes-Dias et al. investigated the feasibility of online asynchronous ErrP detection [19]. Kim et al. used ErrP in their investigation on how the subjects responded to the delayed cursor control [20].
One of the key problems of detecting ErrP is the feature extraction of ErrP. Many ways have been tried in the field. Some researchers extracted features of ErrP by computing the powers of EEG signals in certain frequency bands [21, 22]. The time-frequency analysis using wavelet transform is a common method in signal processing and was also utilized to extract the features of ErrP [21, 23]. When the EEG amplitudes of multi-channels at a moment were viewed as a sample vector, principal component analysis (PCA) and independent component analysis (ICA), which are the classical methods in the area of pattern recognition, were applied in the feature extraction of ErrP [22, 24, 25]. In the early 1990s, Koles et al. proposed the common spatial pattern (CSP) to strengthen the difference between the EEG signals of two kinds of trials [26, 27]. So far, CSP has been widely applied in the feature extraction of EEG signals, including extracting the features of ErrP [22]. In 2009, an unsupervised algorithm, xDAWN, was proposed to enhance evoked potentials in P300 BCI by estimating spatial filters, by which EEG signals were projected into a subspace [28]. In the ErrP detection, xDAWN was often used to intensify the features of ErrP [9, 22]. Additionally, windowed mean (WM), which simply averages EEG signals every time window, was also widely used [29,30,31].
As for ErrP, we do know that ErrPs occur when people make mistakes. However, we still lack a thorough understanding of ErrP [3]. Many variations of ErrP with dissimilar latencies or on different brain areas were observed [4, 6]. It is necessary to develop a feature extraction of ErrP that can adapt to the variety of ErrP across subjects. This paper presents our work towards acheiving this task. We developed an approach based on the coefficient of determination and CSP. The proposed approach is able to search several effective time windows for a subject, construct a spatial filter for each time window and extract the features of ErrP using the time windows and spatial filters.
2 Methods
The EEG differences between correct and incorrect responses occur only in certain time segments that vary across subjects [4], and these kinds of differences are likely to appear in different brain areas for subjects [6]. Our approach is based on the idea of tailoring the feature extraction process for ErrP to suit the given situation.
2.1 Difference Measure
In order to build our method, we need to identify a good difference measure of EEG signals between correct and incorrect responses. In this study, we used the coefficient of determination. Here, a course of T duration beginning with a feedback onset is called a trial. A trial is labelled +1 if the feedback is correct, or \(-1\) if the feedback is incorrect. Let \(X^i\in \mathbb {R}^{N\times T},\, i\in \{1,\ldots ,n\}\) represent the EEG signals of one of n trials, where N indicates the number of channels and T is the number of sampling points during a trial. For any \(k\in \{1,\ldots ,N\},\, l\in \{1,\ldots ,T\}\), we view \(X^i_{k,l}\) as an independent variable value and the label (i.e. +1 or \(-1\)) of the i th trial as a dependent variable value, to construct a linear regression model, thereafter compute a coefficient of determination and assign it to \(R^2_{k,l}\), where \(R^2\in \mathbb {P}^{N\times T},\, \mathbb {P}\) represents all coefficients of determination of T sampling points on N channels.
2.2 Adjusted Time Window
Obviously, \(R^2_{k,l}\) measures the EEG difference of the l th time point on the k th channel. The matrix \(R^2\) is an objective basis for going ahead. We need to search several time windows for a subject in accordance with \(R^2\). Our method to handle this problem is formally depicted as Algorithm 1, called GetRange.
GetRange aims at getting M time windows in which the EEG differences between correct and incorrect responses are more significant than in other time ranges. Intuitively speaking, the purpose of GetRange is to find the time windows of the M biggest peaks of N curves of a subject. The input of GetRange, \(R^2\in \mathbb {P}^{N\times T}\), is the matrix of coefficients of determination for a subject, which is obtained in the way described in Sect. 2.1. The output of GetRange, \(\mathcal {R}_r, r\in \{1,\ldots ,M\}\), represents the M time windows of concern. Algorithm 1 includes three procedures: Search, Merge and Select.
The task of Search is to produce a group of time windows, denoted as \(W_k\), for the k th channel according to \(R^2_k\). The procedure Search first cuts the time range of 1-T apart into several initial time windows in the light of a fixed length and obtains a time window from each initial time window by a series of iterative updates of setting the peak of the previous window as the center of the new time window. Then it removes the time window if its peak coefficient of determination is less than the counterpart of one of its neighbor time windows and the time gap between the two peaks (TGBP) is less than a given threshold. Finally, it denotes the group of remained time windows as \(W_k\). For N channels, N groups of time windows, \(W_k, k\in \{1,\ldots ,N\}\), will be produced by the Search procedure.
The Merge procedure merges the N groups of time windows of a subject into one group, in which, among the adjacent or overlapped time windows with TGBP less than a given threshold, only the time window with the biggest peak coefficient remains. The Select procedure selects the time windows with the M biggest peak coefficients from the merged result and denotes them as \(\mathcal {R}_r, r\in \{1,\ldots ,M\}\).
2.3 Window-Adjusted Common Spatial Pattern
The EEG differences appear possibly in different brain areas for subjects [6]. In order to effectively extract the features of ErrP, we apply the common spatial pattern(CSP) [26, 27] in each of the selected time windows and combine the results of several CSPs for the feature extraction. We call the method window-adjusted common spatial pattern (WACSP), as presented in Algorithm 2.
The input of Algorithm 2, \(X^i\in \mathbb {R}^{N\times T},\, i\in \{1,\ldots ,n\}\), is the EEG signals of n trials in the training dataset of a subject. Its outputs \(\mathcal {R}_r, P_r, r\in \{1,\ldots ,M\}\) are respectively M selected time windows and their corresponding transformation matrices for the subject. For feature extraction, \(\mathcal {R}_r, P_r, r\in \{1,\ldots ,M\}\) can be used to cut out the EEG signals and transform them to feature vectors. Algorithm 2 contains three procedures: GetMatrix, GetRange and GetTrans.
The GetMatrix procedure obtains a matrix of coefficients of determination, \(R^2\), for a subject by computing the EEG data of n trials in the training dataset of the subject in the way described in Sect. 2.1. The GetRange procedure has been depicted in Sect. 2.2. The GetTrans procedure is constructed on the basis of the combination of the adjusted time windows and common spatial pattern. For \(\mathcal {R}_r\), GetTrans firstly cut out the signal segments of \(X^i\) in the time window into \(Y^i\in \mathbb {R}^{N\times T\!r}, i\in \{1,\ldots ,n\}\), where \(T\!r\) is the duration of \(\mathcal {R}_r\); Then, GetTrans conducts a series of computations on \(Y^i\) to obtain \(P_r\).
The first step of the computations is to compute the covariance matrices of n trials according to Eq. 1, and then obtain \(\Sigma _c\): the mean covariance matrix of the correct response, \(\Sigma _e\): the mean covariance matrix of the incorrect response and \(\Sigma _s\): the sum of \(\Sigma _c\) and \(\Sigma _e\).
The second step of the computations is to perform the Eigendecomposition of \(\Sigma _s\) depicted as Eq. 2, where \(\lambda \) is the diagonal matrix of eigenvalues of \(\Sigma _s\) and B a matrix being comprised of normalized eigenvectors of \(\Sigma _s\), get a matrix W by \(W=\lambda ^{-1/2}B^T\), and transform \(\Sigma _c\) and \(\Sigma _e\) to \(S_c\) and \(S_e\) by \(S_c=W \Sigma _c W^T\) and \(S_e=W \Sigma _e W^T\).
According to the references [26, 27], it can be inferred that \(S_c\) and \(S_e\) have the same eigenvectors. This can be described as Eqs. 3 and 4, where U is the common eigenvector matrix of \(S_c\) and \(S_e\) and \(\psi _c\) and \(\psi _e\) respectively diagonal matrices of eigenvalues of \(S_c\) and \(S_e\). Additionally \(\psi _c\) and \(\psi _e\) meet the relation expressed by Eq. 5, where I is an identity matrix.
The final step of the computations is to obtain the matrix \(P_r\) by Eq. 6, where \(\hbar (\cdot )\) means selecting the first and last a few columns of U after sorting the eigenvectors in ascending order of the eigenvalues.
For feature extraction, the EEG signals of a trial are cut into \(Y_r\) in the light of \(\mathcal {R}_r, r\in \{1,\ldots ,M\}\). The feature vector of the trial can be obtained by Eq. 7.
where \(\mathcal {F}(\cdot )\) represents the computation that transforms \(P_1 Y_1,\ldots , P_M Y_M\) into a feature vector. \(\mathcal {F}(\cdot )\) may be implemented in various ways. One of the usual implementations is to concatenate all rows of \(P_1 Y_1,\ldots , P_M Y_M\) into a vector, and, sometimes, a downsampling is further applied. Another one is to compute the variance of each row of \(P_rY_r, r\in \{1,\ldots ,M\}\) and concatenate all variances into a vector.
In this study, the sample of a trial was the EEG signals of 1000 ms on 32 channels, that is, \(X^i\in \mathbb {R}^{32\times 1000}\); the fixed length of the time window was 101 ms and the length would be adjusted if the time window went beyond the range of 1-T; the threshold of TGBP was set 50 ms. The implementation of WACSP produced four time windows, that is, \(M=4\). It set \(\hbar (\cdot )\) to select the first and last four columns of U, meaning that every time window produced a \(P_r\in \mathbb {R}^{8\times 32}\), which projected the 32-channel EEG signals in the time window into eight virtual channels by \(P_rY_r\). Finally, it computed the variances of the four time windows on each virtual channel to obtained the 32-dimension feature vectors.
2.4 Other Methods
Besides WACSP, we also implemented the commonly used approaches in the field and compared them with WACSP. All implemented methods transformed one sample into a 32-dimensional vector in their respective ways. Here, we briefly introduce these methods, including Windowed Means (WM), Band Power (BP), Time-frequency (TF), Principal components analysis (PCA), independent component analysis (ICA), xDAWN and common spatial pattern (CSP).
WM, a simple but widely used technique, obtains the feature vectors by averaging EEG signals in every 250-ms time window [29,30,31]. BP means using the EEG powers of a few frequency bands of concern as the features [21, 22]. To carry out BP in this study, we calculated the power in the frequency band of 1–30 Hz every 250-ms time window, and connected the band powers into a feature vector. To obtain 32-dimensional feature vectors, we had to confine the calculations of the WM and BP on the channels of F3, Fz, F4, FC3, FCz, FC4, Cz and CPz, which are most closely related to the ErrP. TF approach usually uses wavelet transformation to process EEG signals and selects the time-frequency points that significantly differ under different conditions as features according to Fisher Criterion. The detail is described in Algorithm 1 in [23]. The implementation of TF selected, as a feature vector, the 32 time-frequency points whose differences under different conditions were at the top level.
PCA and ICA both are classical methods in the area of Pattern Recognition. When applied to feature extraction of EEG signals, they view the EEG amplitude vector of all channels at a time point as one vector and, by a series of computations on the vector collections, obtain the matrices that are able to transform the raw EEG signals to the signals on a few virtual channels [22]. Certainly, the principles and results of the two methods are different from each other. Originally, xDAWN was proposed to enhance P300 evoked potentials in BCI [28]. Afterwards, it was extensively exploited in similar tasks [9, 22]. By an unsupervised algorithm, xDAWN estimates a spatial filter to project the raw EEG signals to a signal subspace [28]. Similarly, CSP, which was proposed for the purpose of processing EEG signals [26, 27], also constructs a spatial filter, but its core is a supervised algorithm. A lot of research shows that CSP is a very good method in the field of processing EEG signals [22]. In this study, PCA, ICA, xDAWN, and CSP all projected the EEG signals of one sample to the signals of eight virtual channels in their respective ways, computed a variance every 250 milliseconds on each virtual channel and connected all variances into a feature vector.
3 Experiment and Data
In this study, we recruited 20 right-handed BCI-naive subjects (10 males, 10 females), whose age ranged from 19 to 28 years, with a mean age of 23 years and a standard deviation of 2.35, to participate in the experiment. All subjects with a history of visual or neurological disorders, head trauma or any drug use that would affect nervous system function were excluded, and the subjects were asked to wash their hair before the experiment. This experiment was approved by the Institutional Review Board at Fuzhou University. In accordance with the Helsinki Declaration of Human Rights, informed consent for the experimentation was obtained from all subjects after a detailed explanation of the study.
As shown in Fig. 1, all subjects carried out two sessions respectively at two different times. The first session used a pseudo-detector to detect the symbols. A pseudo-detector means that the subjects were provided with the outcomes that were generated by the BCI platform according to the configured error rates, and they were told that a real detector was working. In the second session, a real classifier trained on the data of the first session by stepwise linear discriminant analysis (SWLDA) [32] was adopted to recognize P300 and the results of recognizing P300 corresponding to a symbol were synthesized to detect the symbol. Each session included 14 runs. In a run, the subjects selected 18 symbols of Chinese pinyin through the interaction of brain–computer on the platform. For each symbol, six sequences of flashes were presented. Every time a symbol was selected by the platform, it was presented to the subject. The selected symbol was possibly correct or incorrect. For a subject, the ratio of the incorrect number to the total number of symbol detection is called Error Rate. The mean error rate of the 20 subjects was 24.9% and the standard deviation was 5.6%.
Based on BCI2000 [33], we developed a BCI platform for Chinese pinyin. In nature, the BCI platform is a P300 BCI system [34]. The BCI platform uses a standard \(6\times 6\) matrix of symbols [34] to present stimuli in the way proposed by Townsend et al. [35], which is called checkerboard stimulus paradigm. According to the checkerboard stimulus paradigm, the 36 symbols are randomly rearranged in an inner matrix of \(6\times 6\). In a sequence, the six groups of symbols in the presentation matrix corresponding to six rows of the inner matrix and another six groups of symbols in the presentation matrix corresponding to six columns of the inner matrix are flashed one time in random order. Normally, a few sequences are essential for the detection of a symbol. The main differences between the BCI platform and traditional P300 BCIs lie in that the symbols in the presentation matrices are not English characters but the initial consonants, vowels or tones of Chinese pinyin and the BCI platform includes a formal procedure of result presentation after P300 detection, allowing subjects to judge whether the result is correct or not while observing it.
Figure 2 shows an experiment course of selecting a symbol on the BCI platform. This course includes 3 steps. In Step 1, the current target is cued for 2.4s using an ellipse frame, as shown in the first screen of Fig. 2. Step 2 contains six sequences of flashes. In each sequence, each of the 12 groups of symbols is flashed one time. There are no breaks in-between sequences. The intensification duration of the flash is equal to 80 ms and the interval between the successive intensification onsets is 120 ms. Step 2 is shown in the section from the second screen to the third screen of Fig. 2. In Step 2, the subjects are instructed to silently count how many times the target has been flashed to keep their attentions. In Step 3, the result of P300 detection is presented, as shown in the fourth screen of Fig. 2, and the BCI platform records the EEG signals while the subjects are observing the feedback. In the context of this paper, Step 3 of the course is called a trial.
A 64-channel Neuroscan system, including the EEG cap, the amplifier, and the signal acquisition software, was used to acquire EEG signals. For convenience, the EEG signals of only 32 channels were recorded. The 32 channels were: FP1 FP2 F7 F3 Fz F4 F8 FT7 FC3 FCz FC4 FT8 T7 C3 Cz C4 T8 TP7 CP3 CPz CP4 TP8 P7 P3 Pz P4 P8 PO7 PO8 O1 Oz O2. The sampling rate was set to 1000 Hz. The procedure of constructing the EEG datasets is included in Fig. 1. The raw EEG signals were firstly preprocessed by common average reference and finite impulse response with the order at 64 and the frequency range of 0.1-30Hz. Next, the EEG signals of 1000 ms beginning with the feedback onsets were segmented after the baseline correction of subtracting the mean of 200 ms before the feedback onset. Then, Savitzky-Golay filter [36] with the order at 3 and the window length at 101 was applied to smoothen the EEG segments. Finally, every EEG segment, as the sample of a trial, was labelled according to the feedback result, +1 if correct or \(-1\) if incorrect, and was added to the datasets of the subjects. In the dataset of a subject, there were 504 samples, including about 399 samples corresponding to correct feedbacks (labelled as +1) and about 125 samples corresponding to incorrect feedback (labelled as \(-1\)).
4 Results
In this study, every method transformed the sample set of one subject into a feature vector set. The feature vector set of each subject was randomly split into five subsets. Each subset of a subject was in turn selected as a test set and the other four subsets were merged as the corresponding training set. Here, each test set contains the feature vectors of 101 or 100 trials, so the chance level of classification accuracy is 0.58. Since our focus was on feature extraction, only shrinkage linear discriminant analysis (sLDA) [29, 37], a way that performed very well in EEG classification [22, 29], was further used to train the classifiers on the training sets and classify the feature vectors in the corresponding test sets. According to [38], the three performance indexes of classification were used to compare our approach with the commonly used methods. They were Accuracy (Acc), Area Under the Receiver Operating Characteristics Curve (AUC) and F-measure (F1). Due to the 5-fold procedure, every method obtained the three performance values five times for a subject. In Fig. 3, the mean values are presented for each performance.
Figure 3 graphically shows the comparisons of Acc, AUC and F1 between WACSP and one of the other methods on all subjects. In Fig. 3, (a) is the comparison of accuracy, (b) presents the comparison of AUC, (c) exhibits the comparison of F1, the Y value of each point is the performance value of WACSP on a subject, its X value represents the performance value of one of the other methods on the subject. Other methods include CSP, PCA, BP, ICA, xDAWN, TF and WM. A legend represents a specific method. It means that WACSP outperforms the specific method on the subject when a point lies above the diagonal. According to Fig. 3, we can roughly conclude that WACSP performed better than the other methods. We ranked the eight methods respectively in accordance with the means of Acc, AUC and F1. The ranking result is shown in Table 1. Friedman test (p value < 0.01) shows that significant performance difference exists among the eight methods.
Furthermore, repeated measures analysis of variance (ANOVA) with least significant difference (LSD) adjustment (p value < 0.05) was used to test the statistical significance of the differences of Acc, AUC and F1 among these methods. Table 2 shows the means and standard deviations of accuracies of WACSP, CSP, PCA, BP, ICA, xDAWN, TF and WM and the results of their pair-wise comparisons. The accuracies of WACSP are significantly higher than those of the other methods. The statistical inference is consistent with the intuition from the left subgraph of Fig. 3. No significant difference of accuracy exists among CSP, ICA, PCA, BP, xDAWN and WM. The accuracies of CSP, ICA, PCA, BP, xDAWN and WM are significantly higher than those of TF.
Table 3 presents the means and standard deviations of AUCs of the eight methods and the results of their pairwise comparisons. The AUCs of WACSP significantly exceed those of the other methods. The rough conclusion from the middle subgraph of Fig. 3 is further verified by the statistical inference. No significant difference in AUC is discovered among CSP,PCA, xDAWN and WM. The AUCs of CSP, PCA, xDAWN and WM are significantly higher than those of BP, ICA, and TF. The AUCs of BP are significantly higher than those of ICA and TF. The AUCs of ICA are significantly higher than those of TF.
Likewise, Table 4 exhibits the means and standard deviations of F-measures of the eight methods and the results of their pair-wise comparisons. The F-measures of WACSP are significantly higher than those of the other methods. The statistical inference further verifies the intuitive conclusion from the right subgraph of Fig. 3. No significant differences in F-measure appear among CSP, PCA, xDAWN and WM. The F-measures of CSP, PCA, xDAWN and WM are significantly higher than those of BP, ICA and TF. No significant differences of F-measure are discovered among BP, ICA and TF.
The ErrP waveforms of a subject on Fz, FCz and Cz are presented in Fig. 4. Compared with the ErrP waveforms in [10], the negative peak at 670 ms and the positive peak at 840 ms in Fig. 4 are extra. However, each peak in the ErrP waveforms has its counterparts in Fig. 1 in [39], except that the latencies in Fig. 4 are about 100 ms later. As for the latency difference, it is known that the latencies vary across subjects. On the other hand, the onsets of stimuli in the different tasks are specified in different ways, probably leading to the latency difference. In summary, the comparisons show that similarity and dissimilarity in the ErrP waveforms both exist in the related literature.
Figure 5 shows \(R^2\) of the subject graphically. The first row of Fig. 5 presents the curves representing the changes of coefficients of determination over time. The two peak regions of the curves are marked in red color. The second row presents the brain maps drawn using the coefficients of determination on all channels at the first peak moments. The brain maps in the third row correspond to the second peak moments. Before drawing the pictures, we divided the training set of the subject into five parts, randomly and equably, and obtained five training subsets by deleting one part from the training set every time. The five columns of Fig. 5 respectively correspond to the five training subsets. In the first row of this figure, the two peak regions of all five training subsets, which are marked in red color, are very similar. Likewise, only very small differences exist among the brain maps in the second row, and among the brain maps in the third row as well. This reveals that the spatio-temporal differences in EEG between correct and incorrect responses are stable for a subject. Additionally, in Fig. 5, the spatial distribution patterns presented by the brain maps in the third row are significantly different from the counterparts of the second row, indicating the change of the spatial pattern of ErrP over time. To adapt to this situation, our method intends to search several time windows for a subject and build respective spatial filters in each time window.
5 Discussion
As for the feature extraction of ErrP, many methods, such as WM, CSP, PCA, BP, ICA, xDAWN and TF, have been investigated [9, 21,22,23, 26,27,28,29,30,31]. However, ErrPs with dissimilar latencies on different brain areas have been discovered [4, 6] and we haven’t totally understood ErrP as far [3]. As a result, the methods, which were developed for other tasks of pattern recognition, probably miss the point for the feature extraction of ErrP, even though they seemingly work.
Based on extensive observations, we think that, for a subject, the spatial pattern of ErrP is stable in a time window although it changes from one time window to another. The point of the feature extraction of ErrP is to find the time windows, in which the EEG differences between correct and incorrect responses are significant, and to capture the spatial pattern of EEG difference in each time window. Our method WACSP was developed on the basis of this idea. We introduced the coefficient of determination to measure the EEG differences and guide the search of the time windows, and further used CSP [26, 27] for each time window to obtain a matrix projecting the EEG signals in the time window to a few virtual channels.
Among the methods of WM, CSP, PCA, BP, ICA, xDAWN and TF, CSP and xDAWN are more similar to WACSP. The difference between WACSP, CSP and xDAWN deserves further investigation. CSP and xDAWN estimate their spatial filters directly from the given EEG signals, while the proposed WACSP estimates the spatial filters in several time windows, which are mined using the guidance of the determination coefficients that reflects EEG dissimilarities under different conditions.
We tested WACSP and the commonly used methods on the data sets that were built using the EEG signals acquired during the P300 BCI experiments with feedback. The result presented graphically in Fig. 3 shows that WACSP obviously outperforms such commonly used methods as WM, CSP, PCA, BP, ICA, xDAWN and TF. A series of statistical inferences shown in Tables 1, 2, 3 and 4 further confirm the advantage of WACSP over the alternative methods. Conversely, the superiority of WACSP verifies our idea that, for a subject, the spatial pattern of EEG difference between correct and incorrect responses in a time window keeps stable even though it varies from one time window to another.
Additionally, we also compared the performances of WM, CSP, PCA, BP, ICA, xDAWN and TF. In Table 1, the average ranks of WM, CSP and PCA (2.67, 3 and 3.33) are approximate and they are followed by that of xDAWN (5). According to Tables 2, 3 and 4, no significant differences in Acc, AUC and F1 values among WM, CSP, PCA and xDAWN are observed. The Acc, AUC and F1 values of WM, CSP, PCA and xDAWN are significantly higher than their counterparts of BP and TF. A commonality of CSP, PCA and xDAWN is that they all consider the spatial distribution of EEG differences between correct and incorrect responses during the feature extraction of ErrP. Thus, we infer that the spatial pattern of EEG differences is very important for detecting ErrP. On the other hand, the cognitive fundamentals implied by the feature extractions of ErrP are also our concern.
The improvement of P300 BCI involves many aspects. In this task, using ErrP detection is one of the directions of exploration. We acquired the data in the experiments of P300 BCI as shown in Fig. 2 and studied the performance of ErrP detection in the data sets. The results show that the proposed method, WACSP, is a good option for feature extraction in single-trial ErrP detection and it is feasible to integrate ErrP detection into online P300 BCIs to improve the information transfer rate. We will go ahead along this road in the future.
6 Conclusions
In summary, there are several time windows for a subject, in which the spatial difference patterns of EEG between correct and incorrect responses are significant and stable but the patterns vary from one time window to another. Based on the idea, WACSP for detecting ErrP in P300 BCI was developed by combining coefficient of determination and CSP. We tested WACSP and compared it with the commonly used methods on the data sets that were constructed using the EEG signals acquired during the P300 BCI experiments with feedback. The intuitive comparisons and statistical analysis of accuracy, AUC and F-measure between WACSP and the commonly used methods both show that WACSP significantly outperforms the commonly used methods. The results show the superiority of WACSP and meanwhile verify the idea underlying WACSP.
References
Falkenstein M, Hohnsbein J, Hoormann J, Blanke L (1991) Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. Electroencephalogr Clin Neurophysiol 78(6):447–455
Gehring WJ, Goss B, Coles MGH, Meyer DE, Donchin E (1993) A neural system for error detection and compensation. Psychol Sci 4(6):385–390
Gehring WJ, Goss B, Coles MGH, Meyer DE, Donchin E (2018) The error-related negativity. Perspect Psychol Sci 13(2):200–204
Kumar A, Gao L, Pirogova E, Fang Q (2019) A review of error-related potential-based brain–computer interfaces for motor impaired people. IEEE Access 7:142451–142466
Falkenstein M, Hoormann J, Christ S, Hohnsbein J (2000) ERP components on reaction errors and their functional significance: a tutorial. Biol Psychol 51(2–3):87–107
Wilson NR, Sarma D, Wander JD, Weaver KE, Ojemann JG, Rao RPN (2019) Cortical topography of error-related high-frequency potentials during erroneous control in a continuous control brain–computer interface. Front Neurosci 13:502
Dal Seno B, Matteucci M, Mainardi L (2010) Online detection of P300 and error potentials in a BCI speller. Comput Intell Neurosci 2010:307254
Combaz A, Chumerin N, Manyakov NV, Robben A, Suykens JAK, Van Hulle MM (2012) Towards the detection of error-related potentials and its integration in the context of a P300 speller brain–computer interface. Neurocomputing 80(C):73–82
Margaux P, Emmanuel M, Sébastien D, Olivier B, Jérémie M (2012) Objective and subjective evaluation of online error correction during P300-based spelling. Adv Hum Comput Interact 2012:578295
Spüler M, Bensch M, Kleih S, Rosenstiel W, Bogdan M, Kübler A (2012) Online use of error-related potentials in healthy users and people with severe motor impairment increases performance of a P300-BCI. Clin Neurophysiol 123(7):1328–1337
Mainsah B, Morton K, Collins L, Sellers E, Throckmorton C (2015) Moving away from error-related potentials to achieve spelling correction in P300 spellers. IEEE Trans Neural Syst Rehabil Eng 23(5):737–743
Zeyl T, Yin E, Keightley M, Chau T (2016) Adding real-time Bayesian ranks to error-related potential scores improves error detection and auto-correction in a P300 speller. IEEE Trans Neural Syst Rehabil Eng 24(1):46–56
Cruz A, Pires G, Nunes UJ (2018) Double ErrP detection for automatic error correction in an ERP-based BCI speller. IEEE Trans Neural Syst Rehabil Eng 26(1):26–36
Mousavi M, Koerner AS, Zhang Q, Noh E, de Sa VR (2017) Improving motor imagery BCI with user response to feedback. Brain Comput Interfaces 4(1–2):74–86
Yokota Y, Soshi T, Naruse Y (2019) Error-related negativity predicts failure in competitive dual-player video games. Plos One 14(2):1–19
Keyl P, Schneiders M, Schuld C, Franz S, Hommelsen M, Weidner N, Rupp R (2019) Differences in characteristics of error-related potentials between individuals with spinal cord injury and age- and sex-matched able-bodied controls. Front Neurol 9:1192
Chou P-S, Sharon SC-J, Hsu C-Y, Liou L-M, Wu M-N, Liu C-K, Lai C-L (2019) Compensatory neural recruitment for error-related cerebral activity in patients with moderate-to-severe obstructive sleep apnea. J Clin Med 8(7):1077
Ehrlich SK, Cheng G (2019) A feasibility study for validating robot actions using EEG-based error-related potentials. Int J Soc Robot 11(2):271–283
Lopes-Dias C, Sburlea AI, Müller-Putz GR (2019) Online asynchronous decoding of error-related potentials during the continuous control of a robot. Sci Rep 9(1):1–9
Kim H, Yoshimura N, Koike Y (2020) Investigation of delayed response during real-time cursor control using electroencephalography. J Healthc Eng 2020:1418437
Lotte F, Congedo M, Lécuyer A, Lamarche F, Arnaldi B (2007) A review of classification algorithms for EEG-based brain-computer interfaces. J Neural Eng 4:R1
Lotte F, Bougrain L, Cichocki A, Clerc M, Congedo M, Rakotomamonjy A, Yger F (2018) A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update. J Neural Eng 15(3):031005
Huang Z, Li M, Ma Y (2018) Parallel computing sparse wavelet feature extraction for P300 speller BCI. Comput Math Methods Med 2018:4089021
Iturrate I, Montesano L, Chavarriaga R, del R Millán J, Minguez J (2011) Spatio-temporal filtering for EEG error related potentials. In: 5th International brain–computer interface conference
Kachenoura A, Albera L, Senhadji L, Comon P (2008) ICA: a potential tool for BCI systems. IEEE Signal Process Mag 25(1):57–68
Koles ZJ (1991) The quantitative extraction and topographic mapping of the abnormal components in the clinical EEG. Electroencephalogr Clin Neurophysiol 79(6):440–447
Koles ZJ, Lind JC, Soong AC (1995) Spatio-temporal decomposition of the EEG: a general approach to the isolation and localization of sources. Electroencephalogr Clin Neurophysiol 95(4):219–230
Rivet B, Souloumiac A, Attina V, Gibert G (2009) xDAWN algorithm to enhance evoked potentials: application to brain–computer interface. IEEE Trans Biomed Eng 56(8):2035–2043
Blankertz B, Lemm S, Treder M, Haufe S, Mller KR (2011) Single-trial analysis and classification of ERP components—a tutorial. Neuroimage 56(2):814–825
Zander TO, Krol LR, Birbaumer NP, Gramann K (2016) Neuroadaptive technology enables implicit cursor control based on medial prefrontal cortex activity. Proc Natl Acad Sci USA 113(52):14898–14903
Mousavi M, de Sa VR (2019) Spatio-temporal analysis of error-related brain activity in active and passive brain–computer interfaces. Brain Comput Interfaces 6(4):118–127
Krusienski DJ, Sellers EW, Cabestaing F, Bayoudh S, Mcfarland DJ, Vaughan TM, Wolpaw JR (2006) A comparison of classification techniques for the p300 speller. J Neural Eng 3(4):299–305
Schalk G, McFarland DJ, Hinterberger T, Birbaumer N, Wolpaw JR (2004) BCI2000: a general-purpose brain–computer interface (BCI) system. IEEE Trans Biomed Eng 51(6):1034–1043
Farwell LA, Donchin E (1988) Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr Clin Neurophysiol 70(6):510–523
Townsend G, LaPallo BK, Boulay CB, Krusienski DJ, Frye GE, Hauser CK, Schwartz NE, Vaughan TM, Wolpaw JR, Sellers EW (2010) A novel P300-based brain–computer interface stimulus presentation paradigm: moving beyond rows and columns. Clin Neurophysiol 121(7):1109–1120
Schafer RW (2011) What is a Savitzky-Golay filter? IEEE Signal Process Mag 28(4):111–117
Friedman JH (1989) Regularized discriminant analysis. J Am Stat Assoc 84(405):165–175
Fawcett T (2005) An introduction to ROC analysis. Pattern Recognit Lett 27(8):861–874
Buttfield A, Ferrez PW, Millan JR (2006) Towards a robust BCI: error potentials and online learning. IEEE Trans Neural Syst Rehabil Eng 14(2):164–168
Acknowledgements
This work is supported by the Science and Technology Plan Project of Fujian Province, China (2023Y0009). Ting Li, Zhilei Lv and Faqiang Peng participated in this work when they studied at Fuzhou University. We thank them for their contributions.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Huang, Z., Li, M., Zheng, W. et al. Window-Adjusted Common Spatial Pattern for Detecting Error-Related Potentials in P300 BCI. Neural Process Lett 55, 10829–10844 (2023). https://doi.org/10.1007/s11063-023-11353-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-023-11353-7