The dorsolateral pre-frontal cortex bi-polar error-related potential in a locked-in patient implanted with a daily use brain–computer interface

Abstract

While brain computer interfaces (BCIs) offer the potential of allowing those suffering from loss of muscle control to once again fully engage with their environment by bypassing the affected motor system and decoding user intentions directly from brain activity, they are prone to errors. One possible avenue for BCI performance improvement is to detect when the BCI user perceives the BCI to have made an unintended action and thus take corrective actions. Error-related potentials (ErrPs) are neural correlates of error awareness and as such can provide an indication of when a BCI system is not performing according to the user’s intentions. Here, we investigate the brain signals of an implanted BCI user suffering from locked-in syndrome (LIS) due to late-stage ALS that prevents her from being able to speak or move but not from using her BCI at home on a daily basis to communicate, for the presence of error-related signals. We first establish the presence of an ErrP originating from the dorsolateral pre-frontal cortex (dLPFC) in response to errors made during a discrete feedback task that mimics the click-based spelling software she uses to communicate. Then, we show that this ErrP can also be elicited by cursor movement errors in a continuous BCI cursor control task. This work represents a first step toward detecting ErrPs during the daily home use of a communications BCI.

Introduction

Brain computer interfaces (BCIs) offer the hope of being able to once again fully engage with their environment to those who suffer from a loss of motor control, by bypassing the affected motor system and decoding user intentions directly from brain activity. While the challenge of sensing, interpreting and controlling computer and robotic devices with brain signals was already being posed in the 1970s [1], the potential of BCIs to become systems that can be used at home to restore lost motor function is only now starting to be realized [2].

One prime example of such a system is the Utrecht Neural Prosthesis (UNP), which provides an individual with locked-in syndrome (LIS) due to late-stage ALS with a means of spelling words on a computer screen, despite an inability to speak or move [3]. The fully implanted system is able to detect the brain activity generated during attempted hand movements and translate them into a 'click' which is used to select letters from a matrix being scanned through on a screen. While there is no decreasing or increasing trend in UNP performance over time and the performance is high enough (89%) for daily home use, errors (such as unintended clicks) are still made.

In fact, the main challenge faced by the neurologists, neurosurgeons, computer scientists, systems engineers and many other specialists working on developing BCIs over the past decades has been to gain robust and reliable control [4,5,6]. There are numerous factors, both fundamental and technical, that disrupt the mapping of user intentions to system performance and for the foreseeable future BCIs are likely to remain error prone. This presents two challenges to BCI usability at home. First, during home use, the user’s true intentions are not known and performance errors may not be apparent. For example, the UNP system allows free communication and it may not be clear when the typed charters do not match the user’s intention. This is especially the case when performance enhancing features, such as word completion and auto-correction are employed. Second, it is generally the case that the number of successful control events needed to correct for erroneous BCI performance is much greater than that which caused the performance error.

One possible solution to this problem is the fact that brain signals not only encode users’ intentions, they can also reflect the users’ perception of errors. Error-related potentials (ErrPs) are a neural correlate of error awareness [7]. Furthermore, it has been shown that ErrPs are generated in response to errors made by a BCI system during both discrete control [8], such as selecting letters, and continuous control [9], such as controlling computer mouse movement. By detecting ErrPs a BCI system can take fast corrective actions, such as deleting the previously selected character, before further actions are made.

Problem formulation

Proof of concept ErrP use has been shown in a multitude of BCI systems (see [10] for a review of ErrPs in BCI). In fact, ErrPs have even been used as the sole control signal in BCIs that can infer which letter a user intends to choose by simply having them observe a cursor moving through a letter matrix and measuring which cursor movements elicit an ErrP, and are thus incorrectly moving away from the target letter [11]. However, to the best of our knowledge, no studies have reported the use or presence of ErrPs in an implanted home-use BCI.

Traditionally, ErrPs are defined as deflections in the electric voltage potential measured using non-invasive electroencephalogram (EEG) electrodes located centrally at the top of the head [7,8,9,10,11] and studied using healthy volunteers. Error-related neural responses (ERNRs) in the spectral decomposition of the signal from electrocorticography (ECoG) electrodes [12], as well as error-related changes in the spike rate and local motor potential (also a spectral features) of needle arrays [13], implanted on the primary motor cortex (M1) have also been reported.

Here, we expand upon this work by searching for the presence of perceived error-related signals after known errors during both discrete feedback and continuous feedback UNP BCI control training games. Because the UNP user presented in this work has sensors implanted over both M1 and dorsolateral pre-frontal cortex (dLPFC), we can investigate both areas for error responses. While the dLPFC has been associated with error processing in the brain [14], to the best of our knowledge, this work provides the first investigation of ErrPs in dLPFC.

Methods

The UNP brain signal

The data reported in this study was generated by the first UNP participant who was implanted with ECoG electrode over her left hemisphere hand knob area of M1 and her left dLPFC. Each cortical area is covered by a strip of four electrodes with dimensions matching those of most clinically used ECoG electrodes (circular with 4 mm of exposed surface contact spaced with a 1 cm center-to-center distance). The electrode strips are attached via wires that travel through a burr hole in the skull and under the skin to an Activa™ PC + S (Medtronic plc; investigational devices) implanted subcutaneously beneath the left clavicle. From each of the implanted strips a single bi-polar electrode is created measuring the difference in electric potential between two electrodes on the strip. The electric potential signal is amplified and filtered with the Activa™ PC + S and transmitted wirelessly to a Nexus-1 (Medtronic plc; investigational devices) receiver worn on the chest, which is connected via a cable to a tablet. Further preprocessing and translation of the signal into the ‘click’ signal is done on the tablet using custom software, which then interfaces with the user’s existing click-based assistive communication system. For further description of the UNP system and a visualization of the implanted electrodes and hardware, see [3].

For this work, the Activa™ PC + S bi-polar referenced voltage potential signals from both the M1 and dLPFC were recorded at a sampling rate of 200 Hz (see [3] for location of M1 bi-polar electrode and [13] for location of dLPFC bi-polar electrode). M1 is a popular target cortical area for BCI in general and especially ECoG-based BCI because measured signal changes in this area are highly correlated to a wide range of movements, including attempted movements in people with motor disabilities. For this reason, the primary UNP control signal is derived from the M1 electrodes [3]. Specifically, the two main functional response features used for control in the UNP are a decrease in power in the 4–40 Hz low-frequency band (LFB) and an increase in power in the 50–100 Hz high-frequency band (HFB) (see 3 for more details of computing the control signal). These features match nicely with those commonly reported in the ECoG literature focusing on M1 [16].

It has been shown that the dLPFC can be a target area for BCI control using mental calculation [15] and this provides an alternative UNP control strategy [17]. In this work, the BCI feedback is generated from the M1 electrodes, but signals are also being simultaneously measured from the dLPFC.

Tasks

Since ErrPs have been observed in both discrete and continuous control tasks and our UNP user performed both discrete and continuous BCI control tasks (respectively, the ‘click’ and ‘target’ tasks described in [3]), we analyzed the M1 and dLPFC signals during both types of tasks for evidence of a signal responses to errors. ECoG data was time locked to the visual task feedback which was presented on a screen mounted to her wheelchair. The visual feedback had a refresh rate of 5 Hz.

Discrete feedback ‘click’ task

The UNP user played the ‘whack-a-mole’ BCI training game (‘click task’) in which the objective was to make clicks using the UNP system to select holes that contain a mole (see Fig. 1a). This task uses the same row-then-column scanning as the UNP user’s spelling software and requires her to make brain clicks (using attempted hand movements) during specific time intervals when the scanning box contains a mole. For the data presented here a scan rate of 2 s was used, meaning that each row or column was highlighted for 2 s before the scanning box moved to the next position. The UNP user needed to raise the control signal above a threshold for at least 1 s to produce a click.

Fig. 1

UNP BCI control training games. a The ‘whack-a-mole’ or ‘click task’ game with four types of discrete feedback moments (described in Sect. 2.2.1) depicted from left to right. b Depiction of hit (top) and miss (bottom) trials for the ‘target task’ game (described in Sect. 2.2.2). c Example of a ‘target task’ trial with a cursor movement error during a miss (left) and hit (right) trial

As depicted in Fig. 1a from left to right, there are four types of discrete feedback moments: (1) when a scanning selection box changes position (true negatives; TNs), (2) when a click was made during a correct click period (true positives; TPs), (3) when the feedback indicated that a click failed to be made during an intended click period (false negatives; FNs), and (4) when a click was made outside a correct click period (false positives; FPs). Since the user reports that she never intends to make such FP clicks, these clicks are perceived to be system errors. It should be noted that clicks made in a scan period just before or after a target mole was highlighted have been removed from the analysis because they are likely do to timing errors in actual attempts to make a click and not spontaneously made clicks when no target was present.

Over 40 sessions of the UNP user playing the discrete feedback game, 722 TN, 461 TP, 174 FN, and 68 FP feedback moments were recorded (resulting in an overall click accuracy of ~ 83%).

Continuous feedback ‘target’ task

While training to control her brain signal, the UNP user also played the ‘target task’, a 1D cursor control task that provided continuous feedback of the control signal during attempted hand movements. In this task a yellow cursor (see Fig. 1b) moves at a continuous speed from left to right across the screen. The control signal was used to control the y-velocity (up and down) of the cursor with a positive deflection in the control signal causing the cursor to move up and a negative deflection causing the cursor to move down. The goal of the task was to hit one of two red targets at either the top or bottom half of the left edge of the screen by attempting hand movements to push the cursor up or relaxing to let the cursor move down.

Each trial of the task lasted 5 s during which the cursor first remained still at the center of the left edge of the screen and the target was presented at the right for 0.5 s, then the cursor moved across the screen for a period of 2.5 s, and finally the screen was highlighted with either a green or red border to indicate whether the target was respectively hit or missed, for 1 s.

Over ten runs of this task, a total of 108 trials, were completed with 90 targets being correctly hit and 18 missed, resulting in total target accuracy of ~ 83% that matches the click accuracy of the discrete task.

Marking movement errors made in the continuous feedback task

Given that the refresh rate of both tasks was 5 Hz and the cursor movement period of the continuous feedback task was 2.5 s, each trial provided 12 cursor position updates. Hence, the continuous feedback was not smooth and distinct moments when the cursor moves in the y-position toward the non-target half of the screen can be perceived as system errors by the user. For this reason, in addition to the end of trial feedback moments that indicate either a hit or miss, feedback moments of incorrect cursor movement were also marked for the continuous feedback task.

Marking was done by screening for periods (consecutive screen refresh samples) when the y-position of the cursor moved away from the target (see Fig. 1c). If the y-position change during these periods was > 5% of the height of the screen, then the beginning of the period was marked as an ‘error movement’ feedback moment. Note that error movement feedback moments can occur in both miss and hit trials. In total 48 error movements’ feedback moments were found.

Analysis of the error responses

Our search for bi-polar ECoG error responses during UNP BCI use started with the discrete feedback task since this task was repeated many times, providing the most data, and ErrP signals are most prominent in response to discrete feedback moments that clearly indicate that an error has occurred. Based on these findings we narrowed our search in the continuous feedback task. All analysis was done using the Matlab (www.mathworks.com) software package.

The recorded bi-polar potential signals were smoothed by taking the mean over a 0.2 s sliding window. Thus, temporal fluctuations faster than the 5 Hz task feedback refresh rate were removed. This smoothed signal is referred to as the ‘potential response’ in this work.

In addition to being used to derive the UNP control signal, frequency power features are commonly used for M1-based EEG [6] and ECoG [16] BCI control and have also been used in dLPFC BCI control [15, 17]. For these reasons, we also performed a frequency decomposition on the unsmoothed potential signals to investigate error-related signals in the frequency domain. The spectral power response of the frequencies 1–100 Hz was computed using the square of the real component of the convolution of the potential signal with a complex Gabor wavelet dictionary [18] (span 4 cycles at full width half max). The spectral response of each frequency was then divided by the mean power to remove the 1/frequency power law seen in electrophysiological data. These 100 power signals are referred to as the ‘spectral response’ in this work.

Potential response to feedback analysis

For each type of feedback moment described above the mean potential response over periods’ time locked to each moment (trials) was computed. Similar to the method of Canolty et al. [19] the mean of the ‘baseline’ period (the period of each trace prior to the feedback moment) was subtracted from each trial.

To statistically quantify the mean traces, for each value in the mean traces a non-parametric p-value (q-value) was computed following the method proposed for by Maris and Oostenveld for the evaluation of neurophysiological signals [20].

First, 1000 pseudo-mean traces were created by randomly shifting the potential response data relative to the feedback moments and recomputing the mean baseline-corrected response traces for each feedback type using the new trials of data, which are no longer time locked to true feedback moments.

Then, each value was compared to the distribution of the absolute value of the values in the corresponding pseudo-mean traces. The absolute value was taken because we were interested in both positive peaks and negative troughs in the normalized mean traces. The q-value of each value in the mean traces was computed as the percentage of pseudo-values > the absolute value of the true mean trace value. Since the q-values are computed per trace sample point, a multiple comparison correction was applied. The correct multiple comparisons factor to use in the context of neural potential signals is hard to know since they contain oscillations (hence the commonly used spectral features used in BCI) and thus neighboring samples in time are not independent from each other. The multiple comparison factor used in this work was 10, such that mean trace values with q-values < 0.005 were considered to be outside the chance range and thus represent a potential response to the task feedback.

For the TP, TN, FN, and FP feedback moments in the discrete feedback task, a period of 4 s before and 2 s after the feedback was used.

For the continuous feedback task, a period of 3 s before and 2 s after the target hit and target miss feedback moments was used. Note that the 3 s before feedback includes the 0.5 s target cue period and 2.5 s cursor movement period. The trial period for the movement error feedback moments is discussed below (Sect. 2.3.3).

Analysis of single trial error detection in the discrete feedback task

To further investigate the presence of an ErrP in the UNP potential signal, a single trial detection was done on the TP and FP click trials. The 2 s period after feedback of the mean potential response for the FP trials was correlated with the 2 s period after each of the TP and FP click feedback moments. For a range of thresholds, starting from the lowest correlation value and moving through all computed correlation values until the highest, threshold crossing error detection was then performed. Each trial with a correlation above the threshold was counted as a detected ErrP and each trial with a correlation below was counted as having no ErrP. In context, TP click trials with a detected ErrP are FPs and FP click trials with a detected ErrP are TPs. The area under the receiver-operator curve (AUROC) was computed over the range of thresholds.

The AUROC was statistically quantified by randomly shuffling the labels of the TP and FP correlation values, repeating the thresholding detection, and recomputing the AUROC 1000 times. This procedure serves to compensate for the unequal distribution on TP and FP trials and allows for the computation (following the same procedure described above) of a q-value. Note that 1000 shuffles means that q-values < 0.001 cannot be reached.

Analysis of movement error potential responses

The speed at which a movement error occurs will likely affect the moment at which the error is perceived by the user. If the cursor quickly makes a jump in the wrong direct the error will likely be perceived shortly after the beginning of the error movement. However, if the cursor slowly drifts in the wrong direction the error may be perceived with a longer delay relative to the beginning of the error movement. For this reason, a four-step analysis was developed to allow for jitter in the timing of the potential error response relative to the start of the error movement (see Fig. 2).

Fig. 2

The four-step procedure (described in Sect. 2.3.3.) for computing significant responses to movement errors in the continuous feedback task while allowing for variations in time

First, the mean smoothed response trace (over all error movement trials) of the dLPFC bi-polar electrode for the range of − 0.2 to 1.2 s time locked to the beginning of the error movements was computed (Fig. 2: Step 1). Next, this mean was shifted for the range of − 0.1 to 0.6 s over each trial until the shift with the highest correlation to the original mean was found, and a new mean over the shifted trials was computed (Fig. 2: Step 2). This process of shifting to fit the mean response traces and computing a new mean trace was repeated 20 times (at with point no trial needed to be further shifted to optimally correlate to the mean). Then (Fig. 2: Step 3) 1000 trial-mean traces were computed by randomly shifting the actual error movement start times such that the − 0.2 to 1.2 s period relative to these times will no longer be time-lock to actual movement errors and repeating steps 1 and 2. This step thus produced 1000 mean traces that were not time locked to movement errors. For each of these 1000 pseudo-trial-mean-traces the maximum of the absolute value of the maximum and minimum of the trace was computed (max(abs(max(trace)), abs(min(trace))). Finally, this distribution of 1000 peak/trough values from the pseudo-trial-mean-traces was used to quantify the likelihood that that any peaks or troughs of the actual mean trace shifted around the actual error movements onsets indicate a significant feedback response (Fig. 2: Step 4). In this context significance was defined to be trace values whose absolute values were > 95% of the 1000 pseudo-trial-peak/trough values.

Steps 3 and 4 were done because the shifting in Steps 1 and 2 serves to sharpen or increase any peak or trough in the mean and thus prevents a direct comparison to non-shifted pseudo-trial-mean trace values.

Results

The UNP data shows a dLPFC bi-polar ErrP (dLPFC-bErrP) in response to FP clicks made during the discrete feedback task

Both the M1 and dLPFC potential response and spectral response signals were screened for responses to all four feedback types of the discrete feedback task. The results of this screening are shown in Fig. 3. Given that the spectral response of M1 was used to produce clicks through attempted hand movements, the mean spectral signal patterns time locked to TN and TP click moments seen in the first two columns of Fig. 3a are as expected. A clear increase in the 50–100 Hz HFB 1 s prior to TP clicks (the minimum time need to make a click) coupled with a decrease in the 5–40 Hz LFB fits the expect response from ECoG literature and the definition of the UNP control signal. In contrast to this, for TN moments, when the scanning box moves from one empty mole hole to the next and no attempted movement is made, a relatively high LFB power and low HFB power can be seen. Looking at the mean spectral response for FN moments (3rd column, Fig. 3a), it is clear from the high HFB power and low LFB power that the UNP user indeed made attempted hand movements, but failed to exceed the control signal threshold for the 1 s needed to make a click. The fact that she was attempting hand movements is also supported by the sharp increase in LFB power 0.5 s after the feedback moment (the ‘beta rebound’ phenomenon), which is also seen to a lesser extent for the TP clicks. This increase in the 13–30 Hz frequency band after the cessation of movement is another common feature of M1 [21]. Interestingly, for the FP clicks (4th column, Fig. 3a), a HFB increase and LFB decrease 1 s prior to the click feedback is also seen. However, the post-click feedback period does not contain a sharp LFB increase. This, along with the user reporting that she did not intentionally try to click for these trials, supports the notion that these FP clicks were indeed due to spontaneous activation of M1 and were not related to attempted movements. Thus, while Fig. 3a shows the expected patterns of M1 activation due to attempted movements before feedback, the post-feedback M1 spectral responses to the error FN and FP clicks also match those expected and no clear error response is seen.

Fig. 3

Spectral and potential discrete feedback task response patterns. a Mean spectral response patterns for the M1 bi-polar electrode time locked to TN, TP, FN, and FP (respectively from left to right) discrete feedback task moments. b Mean potential response patterns of the M1 electrode. c Mean spectral response patterns of the dLPFC electrode. d Mean potential response patterns of the dLPFC electrode. In all plots the vertical black lines at 0 s on the y-axis indicate the feedback moment. The colors in the plots of a and c indicate spectral power with yellow and blue corresponding to minimum and maximum power, respectively. The blue traces in the plots of b and d show the normalized potential with red points indicating value with q-values < 0.005. The red, blue and cyan, and green circles in b and d indicate the dLPFC bErrP, M1 feedback negative troughs, and M1 readiness potential (respectively) reported in Results Sect. 3.1. The yellow bars in the two right most plots of d indicate the period over which dLPFC bErrP detection was done (see Sect. 2.3.2)

The mean M1 potential response shown in Fig. 3b contains clear negative deflections, or troughs (blue circles), after all four types of feedback moments. In fact, even the additional troughs seen prior to TN feedback and later after TP and FP clicks (cyan circles) can be attributed to feedback moments. Since the scan rate is 2 s and TN shifts of the scanning box to the next position are often preceded by other TN scanning box shifts, the troughs just after 2 s and 4 s (the 1st and 3rd cyan circle from the left) prior to TN feedback can also be attributed to TN scanning box shifts. In addition, scanning box shifts also occur 1 s after clicks (either TP or FP) are made, since the click feedback last 1 s and then scanning resumes. Hence, the troughs that occur 1 s after the initial post TP and FP feedback troughs (cyan circles in the Fig. 3c TP and TP plots) can also be attributed to feedback moments. Also, while the 2 s period before a TN scanning box shift does not contain any visual changes on the screen, due to the fact that the user first needs to select the correct row before the scanning box scans through the columns of that row, there are many moments in the task when scanning is resumed after a 1 s click feedback pause. Hence, there was often a click feedback moment 1 s before the last TN scanning shift feedback moment and the trough after 3 s prior to TN feedback (2nd cyan circle from the right in the rightmost plot of Fig. 3c) can also be attributed to a feedback moment. One final feature in the M1 potential signal that can be reported is the presence of a gradual increase in potential leading to a peak just prior to the beginning of the 1 s period in which an attempted hand movement is made to produce a TP click (green circle in the second plot from the left in Fig. 3c). Such a buildup can be attributed to the ‘Readiness Potential’ reported in M1 prior to executed movements [22]. Taken together, these results indicate a clear M1 potential response to all forms of visual changes on the screen and in preparation of attempted movement, but no post feedback response that could indicate that an error has been made.

When looking at the dLPFC, the lack of spectral response changes, seen in Fig. 3c, prior to discrete feedback moments is to be expected since dLPFC is not reported to have a functional response related to attempted hand movements. However, the lack of spectral changes after the feedback moments also indicates that no spectral response to correct TP, missed FP, or incorrect FP clicks is seen in dLPFC either.

In contrast, while Fig. 3d shows no distinct deflection in the mean dLPFC potential response prior to any discrete feedback moment, a clear peak in the dLPFC potential response ~ 600–800 ms after an FP click is made (red circle in rightmost plot of Fig. 3d) is evident. A response trough is also seen ~ 900–1200 ms after a TP click, and to a lesser degree after a FN feedback moment. Hence, the dLPFC potential response does not show a distinct response to FN errors, but it does show a distinct response after an erroneous FP click has been made. Because this response is seen in the bi-polar response from dLPFC, we will refer to it as the dLFPC bErrP.

Single trial detection of the dLPFC bErrP

While the dLPFC bErrP is observed in the mean potential response after FP clicks are made, more analysis is needed to establish that it is truly a consistent and unique response to unintended clicks made during UNP use. Thus, we performed single trial ErrP detection using the dLPFC potential response and found an AUROC value of 0.795. As can be seen in Fig. 4, the AUROC value was not only well above the theoretical chance level of 0.5 (for a 2-class discrimination task), it was highly significant (q-value < 0.001). Thus, dLPFC bErrP represents a distinct pattern that can be used to discriminate intended UNP clicks from unintended UNP clicks.

Fig. 4

dLPFC single trial detection results. a AUROC over the full range of computed correlation values (see Sect. 2.3.2). b AUROC value for the true TP and FP labels compared to the distributions of values created by randomly shuffling the labels

The UNP ErrP in the continuous feedback task

After finding an ErrP response in dLPFC to FP clicks, it could be expected that missed targets in the continuous feedback task would also induce an ErrP response. However, as Fig. 5a shows, this is not the case. No deflection in potential signals is seen after moments when a target is missed or hit. A sharp positive peak followed by a negative trough is seen at the point when the cursor starts to move in response to the appearance of the target 0.5 s earlier.

However, in Fig. 5b it can be seen that there is a significant peak in the dLPFC potential response when time locked to the onset of erroneous cursor movements that can occur both in trials where the target is eventually hit or missed. Thus, moments of unintended movement during the continuous feedback task are also perceived as control errors by the UNP user and, as such, elicit a dLPFC bErrP very similar that that seen in the discrete feedback task.

As shown by the gray box plot in Fig. 5b depicting the distribution of shifts relative to the movement onset, the is an ~ 400 ms variance in the timing of when these errors are perceived. Interestingly, as Fig. 5c illustrates, even given this range the movement errors tend induce an even faster dLPFC bErrP than the discrete FP click errors.

Fig. 5

The dLPFC potential response to the continuous feedback task. a Mean dLPFC potential response time locked to the beginning and end of the continuous feedback task trials. The leftmost vertical black line indicates the moment when the cursor reaches the right side of the screen and the target is either hit of missed. The leftmost vertical black line indicates the moment when the cursor starts to move 0.5 s after the target appears. The plotted black trace is the mean potential response over all trials and the green and red traces are means of the hit and miss trials respectively. Trace points with a q-value < 0.005 are marked with dots. b Shifted mean dLPFC potential response time locked to the onset of error movements. The dark red trace and light red shaded region plot the mean and standard error of the potential response. The dark yellow shaded area indicated the part of the trace with a q-value < 0.05. The vertical black line marks error movement onset with shaded grey box and whiskers plot showing the distribution of shifts relative to error movement onset. c The relative timing of the dLPFC bErrP found in response errors made in the continuous (light red) and discrete (dark red) feedback tasks

Discussion

In our study, we showed a significant potential peak of the bi-polar dLPFC potential response related exclusively to feedback indicating UNP system closed-loop BCI control errors, which we call the dLFPC bErrP. The dLPFC bErrP was seen in response to both unintended clicks made during a discrete feedback letter grid spelling task and unintended cursor movements during a continuous feedback 1D cursor control task. To the best of our knowledge, this is the first study to demonstrate an ErrP response to error movements made during continuous BCI control in a user of a fully implanted home-use BCI. This work also represents the first demonstration of an ErrP in dLPFC.

ErrPs have been reported in many studies during online discrete and continuous control and even BCI control [10]. In fact, Schalk et al. even reported ErrPs in BCI users playing the same two-target 1D cursor control game analyzed in this work [9]. However, in their study ErrPs were seen in response to the feedback moments when the cursor missed the target. In the present study, no dLFPC bErrP was seen in response to these moments and was only seen at moments when the cursor moved in the unintended direction regardless of whether the target was eventually hit or missed. In addition, no dLFPC bErrP was seen when the UNP system failed to make an intended click and only in response to clicks that were unintended. This suggests that the UNP user perceives missed click moments and targets as joint user and system failures, while unintended clicks or sudden cursor movements are perceived as unexpected erratic system behavior. This discrepancy could also be due to the dLPFC source of the ErrP (see discussion below).

Application of the dLFPC bErrP to improve UNP performance

When looking into the feasibility of using the discrete feedback dLPFC bErrP signal to directly correct for FP clicks, we faced the previously described problem of needing a very high detection rate and very low false detection rate for ErrP-based corrections to be helpful [10]. Because the UNP system click performance is very good there are many more TP clicks than FP clicks. When attempting to detect ErrPs after a click has been made, the ~ 80 AUROC means that 80% of FP clicks (54 of 68) would be corrected while 20% of TP clicks (92 of 461) would be falsely ‘corrected’. This would likely not improve the UNP performance in the user’s eyes.

However, as the recent work of Lopes-Dias et al. demonstrates, ErrP signals can also be used for controller adaptation [23]. In this context the single trial detection of an ErrP can be less reliable and still inform the system as to when performance is generally good or bad. For example, in the cursor control task, the detection of a dLPFC ErrP could be used to slow down the cursor y-movement and its absence used to speed it up. This use of an ErrP is similar to its use in ErrP-driven BCI spellers where letters are selected by observing which scanner movements induce an ErrP because they move away from the intended letter and which do not [11]. In the context of UNP click-based spelling, the dLPFC bErrP could be used to offer the user the choice to delete a letter if a dLPFC bErrP is detected, offering a double check as to whether the click was really unintended. This basic approach is similar to a two-step ErrP-based error correction shown to improve the performance of a P300 BCI spelling system [24]. While these strategies could make good use of the dLPFC bErrP, further investigation is needed to verify the utility the dLPFC bErrP in improving UNP performance.

Neural origins of the UNP bErrP signal

The dLPFC bErrP is likely related to the ErrPs reported for EEG signals [7,8,9,10,11], which are believed to originate in the anterior cingulate cortex (ACC). While the local nature of the ECoG signal makes it unlikely that the dLPFC bErrP is a direct measurement of the ErrPs generated by the ACC, the fact that both the dLPFC and the ACC are part of the corticolimbic system [13] suggests that it is closely related. The difference in timing of the dLPFC bErrP (+ 700 ms in the discrete case and a range covering + 400 to + 700 ms in the continuous case) compared to the 320 ms delay of the large positive peak in the EEG signal could be due to the differing roles of ACC and dLPFC within that system. The ACC is more engaged in processing emotional experiences, while the dLPFC is more engaged in regulating motivational responses [25]. Hence, an error signal originating from dLPFC could reflect awareness of the error (as opposed to a pre-awareness response in ACC). In light of this hypothesis, FN feedback moments in the discrete click task did not induce the dLPFC bErrP could be because they do not require corrective action and FP clicks do. Also, the slightly earlier timing of dLPFC bErrPs in response to movement errors in the continuous control task could relate to the fact that quick action is required in these situations to correct for the error and hit the final target.

One factor to consider when interpreting our results is our use of bi-polar referenced signals. Bi-polar referencing makes the actual direction and temporal shape of the potential change under the individual electrodes unclear.

Finally, because the previously reported ERNRs using ECoG [12] and local motor potential and spike-rate ‘putative task outcome’ signals using needle arrays [14] were over sensorimotor cortex and also not potential response features, it is difficult to relate the dLPFC bErrP to this work. It is noteworthy that we saw no marked difference in spectral response to FPs in the M1 bi-polar electrode, despite the fact its spectral response was highly correlated to the feedback (i.e., it was used to produce clicks and overall task performance was 83%). While unlikely given the fact that a robust spectral control signal was shown in M1 in this work and has been seen in dLPFC for mental calculation-based BCI control [17], this lack of spectral response could be due to the bi-polar referencing.

Significance

There are two fundamental challenges to BCIs that could lead to ErrP detection being a crucial part of BCI systems.

First, the brain is always active and may never be fully functionally mapped. Thus, even as the brain signal features that are defined to control BCIs become ever more specific to certain cognitive events, the mapping of unique brain signal features to specific BCI outcomes is very challenging. In this context, it is likely that any brain signal feature that highly correlates to a cognitive task, such as attempted hand movements, will also be modulated by many other cognitive tasks and processes. Since the true intentions of the user are not known, such modulation can result in FPs. For example, in the UNP brain activity associated with the processing of sensory input from the touch of the hand causes FP clicks. Hence, there will likely continue to be room for improvement in BCI performance by identifying unintended FP BCI events.

Second, the brain is always changing and adapting. Thus, BCIs will need to adapt with them and this need becomes greater as the recorded brains signals used for control become more complex. One of the keys to success that allows the UNP system to be used daily at home is the remarkable stability of the simple M1 bi-polar ECoG electrode control signal, which measures the collective activity of millions of neurons. However, even the UNP system requires recalibration at times. On the other extreme of implanted BCIs are systems that can detect the activity of multiple single neurons. These systems can provide more complex BCI control, but need frequent recalibration and are not yet suitable for daily home use. As BCIs move forward they strive to get the best of both worlds by proving complex control that is stable over time in the daily home-use setting. In this context, being able to continuously monitor whether a BCI is performing according to the user’s intentions by detecting ErrPs could go a long way toward being able to adapt to changes in the control signal.

We feel that this work helps establish the feasibility of detecting ErrPs in BCI systems that are suited for long-term home use.