The UNP brain signal
The data reported in this study was generated by the first UNP participant who was implanted with ECoG electrode over her left hemisphere hand knob area of M1 and her left dLPFC. Each cortical area is covered by a strip of four electrodes with dimensions matching those of most clinically used ECoG electrodes (circular with 4 mm of exposed surface contact spaced with a 1 cm center-to-center distance). The electrode strips are attached via wires that travel through a burr hole in the skull and under the skin to an Activa™ PC + S (Medtronic plc; investigational devices) implanted subcutaneously beneath the left clavicle. From each of the implanted strips a single bi-polar electrode is created measuring the difference in electric potential between two electrodes on the strip. The electric potential signal is amplified and filtered with the Activa™ PC + S and transmitted wirelessly to a Nexus-1 (Medtronic plc; investigational devices) receiver worn on the chest, which is connected via a cable to a tablet. Further preprocessing and translation of the signal into the ‘click’ signal is done on the tablet using custom software, which then interfaces with the user’s existing click-based assistive communication system. For further description of the UNP system and a visualization of the implanted electrodes and hardware, see [3].
For this work, the Activa™ PC + S bi-polar referenced voltage potential signals from both the M1 and dLPFC were recorded at a sampling rate of 200 Hz (see [3] for location of M1 bi-polar electrode and [13] for location of dLPFC bi-polar electrode). M1 is a popular target cortical area for BCI in general and especially ECoG-based BCI because measured signal changes in this area are highly correlated to a wide range of movements, including attempted movements in people with motor disabilities. For this reason, the primary UNP control signal is derived from the M1 electrodes [3]. Specifically, the two main functional response features used for control in the UNP are a decrease in power in the 4–40 Hz low-frequency band (LFB) and an increase in power in the 50–100 Hz high-frequency band (HFB) (see 3 for more details of computing the control signal). These features match nicely with those commonly reported in the ECoG literature focusing on M1 [16].
It has been shown that the dLPFC can be a target area for BCI control using mental calculation [15] and this provides an alternative UNP control strategy [17]. In this work, the BCI feedback is generated from the M1 electrodes, but signals are also being simultaneously measured from the dLPFC.
Tasks
Since ErrPs have been observed in both discrete and continuous control tasks and our UNP user performed both discrete and continuous BCI control tasks (respectively, the ‘click’ and ‘target’ tasks described in [3]), we analyzed the M1 and dLPFC signals during both types of tasks for evidence of a signal responses to errors. ECoG data was time locked to the visual task feedback which was presented on a screen mounted to her wheelchair. The visual feedback had a refresh rate of 5 Hz.
Discrete feedback ‘click’ task
The UNP user played the ‘whack-a-mole’ BCI training game (‘click task’) in which the objective was to make clicks using the UNP system to select holes that contain a mole (see Fig. 1a). This task uses the same row-then-column scanning as the UNP user’s spelling software and requires her to make brain clicks (using attempted hand movements) during specific time intervals when the scanning box contains a mole. For the data presented here a scan rate of 2 s was used, meaning that each row or column was highlighted for 2 s before the scanning box moved to the next position. The UNP user needed to raise the control signal above a threshold for at least 1 s to produce a click.
As depicted in Fig. 1a from left to right, there are four types of discrete feedback moments: (1) when a scanning selection box changes position (true negatives; TNs), (2) when a click was made during a correct click period (true positives; TPs), (3) when the feedback indicated that a click failed to be made during an intended click period (false negatives; FNs), and (4) when a click was made outside a correct click period (false positives; FPs). Since the user reports that she never intends to make such FP clicks, these clicks are perceived to be system errors. It should be noted that clicks made in a scan period just before or after a target mole was highlighted have been removed from the analysis because they are likely do to timing errors in actual attempts to make a click and not spontaneously made clicks when no target was present.
Over 40 sessions of the UNP user playing the discrete feedback game, 722 TN, 461 TP, 174 FN, and 68 FP feedback moments were recorded (resulting in an overall click accuracy of ~ 83%).
Continuous feedback ‘target’ task
While training to control her brain signal, the UNP user also played the ‘target task’, a 1D cursor control task that provided continuous feedback of the control signal during attempted hand movements. In this task a yellow cursor (see Fig. 1b) moves at a continuous speed from left to right across the screen. The control signal was used to control the y-velocity (up and down) of the cursor with a positive deflection in the control signal causing the cursor to move up and a negative deflection causing the cursor to move down. The goal of the task was to hit one of two red targets at either the top or bottom half of the left edge of the screen by attempting hand movements to push the cursor up or relaxing to let the cursor move down.
Each trial of the task lasted 5 s during which the cursor first remained still at the center of the left edge of the screen and the target was presented at the right for 0.5 s, then the cursor moved across the screen for a period of 2.5 s, and finally the screen was highlighted with either a green or red border to indicate whether the target was respectively hit or missed, for 1 s.
Over ten runs of this task, a total of 108 trials, were completed with 90 targets being correctly hit and 18 missed, resulting in total target accuracy of ~ 83% that matches the click accuracy of the discrete task.
Marking movement errors made in the continuous feedback task
Given that the refresh rate of both tasks was 5 Hz and the cursor movement period of the continuous feedback task was 2.5 s, each trial provided 12 cursor position updates. Hence, the continuous feedback was not smooth and distinct moments when the cursor moves in the y-position toward the non-target half of the screen can be perceived as system errors by the user. For this reason, in addition to the end of trial feedback moments that indicate either a hit or miss, feedback moments of incorrect cursor movement were also marked for the continuous feedback task.
Marking was done by screening for periods (consecutive screen refresh samples) when the y-position of the cursor moved away from the target (see Fig. 1c). If the y-position change during these periods was > 5% of the height of the screen, then the beginning of the period was marked as an ‘error movement’ feedback moment. Note that error movement feedback moments can occur in both miss and hit trials. In total 48 error movements’ feedback moments were found.
Analysis of the error responses
Our search for bi-polar ECoG error responses during UNP BCI use started with the discrete feedback task since this task was repeated many times, providing the most data, and ErrP signals are most prominent in response to discrete feedback moments that clearly indicate that an error has occurred. Based on these findings we narrowed our search in the continuous feedback task. All analysis was done using the Matlab (www.mathworks.com) software package.
The recorded bi-polar potential signals were smoothed by taking the mean over a 0.2 s sliding window. Thus, temporal fluctuations faster than the 5 Hz task feedback refresh rate were removed. This smoothed signal is referred to as the ‘potential response’ in this work.
In addition to being used to derive the UNP control signal, frequency power features are commonly used for M1-based EEG [6] and ECoG [16] BCI control and have also been used in dLPFC BCI control [15, 17]. For these reasons, we also performed a frequency decomposition on the unsmoothed potential signals to investigate error-related signals in the frequency domain. The spectral power response of the frequencies 1–100 Hz was computed using the square of the real component of the convolution of the potential signal with a complex Gabor wavelet dictionary [18] (span 4 cycles at full width half max). The spectral response of each frequency was then divided by the mean power to remove the 1/frequency power law seen in electrophysiological data. These 100 power signals are referred to as the ‘spectral response’ in this work.
Potential response to feedback analysis
For each type of feedback moment described above the mean potential response over periods’ time locked to each moment (trials) was computed. Similar to the method of Canolty et al. [19] the mean of the ‘baseline’ period (the period of each trace prior to the feedback moment) was subtracted from each trial.
To statistically quantify the mean traces, for each value in the mean traces a non-parametric p-value (q-value) was computed following the method proposed for by Maris and Oostenveld for the evaluation of neurophysiological signals [20].
First, 1000 pseudo-mean traces were created by randomly shifting the potential response data relative to the feedback moments and recomputing the mean baseline-corrected response traces for each feedback type using the new trials of data, which are no longer time locked to true feedback moments.
Then, each value was compared to the distribution of the absolute value of the values in the corresponding pseudo-mean traces. The absolute value was taken because we were interested in both positive peaks and negative troughs in the normalized mean traces. The q-value of each value in the mean traces was computed as the percentage of pseudo-values > the absolute value of the true mean trace value. Since the q-values are computed per trace sample point, a multiple comparison correction was applied. The correct multiple comparisons factor to use in the context of neural potential signals is hard to know since they contain oscillations (hence the commonly used spectral features used in BCI) and thus neighboring samples in time are not independent from each other. The multiple comparison factor used in this work was 10, such that mean trace values with q-values < 0.005 were considered to be outside the chance range and thus represent a potential response to the task feedback.
For the TP, TN, FN, and FP feedback moments in the discrete feedback task, a period of 4 s before and 2 s after the feedback was used.
For the continuous feedback task, a period of 3 s before and 2 s after the target hit and target miss feedback moments was used. Note that the 3 s before feedback includes the 0.5 s target cue period and 2.5 s cursor movement period. The trial period for the movement error feedback moments is discussed below (Sect. 2.3.3).
Analysis of single trial error detection in the discrete feedback task
To further investigate the presence of an ErrP in the UNP potential signal, a single trial detection was done on the TP and FP click trials. The 2 s period after feedback of the mean potential response for the FP trials was correlated with the 2 s period after each of the TP and FP click feedback moments. For a range of thresholds, starting from the lowest correlation value and moving through all computed correlation values until the highest, threshold crossing error detection was then performed. Each trial with a correlation above the threshold was counted as a detected ErrP and each trial with a correlation below was counted as having no ErrP. In context, TP click trials with a detected ErrP are FPs and FP click trials with a detected ErrP are TPs. The area under the receiver-operator curve (AUROC) was computed over the range of thresholds.
The AUROC was statistically quantified by randomly shuffling the labels of the TP and FP correlation values, repeating the thresholding detection, and recomputing the AUROC 1000 times. This procedure serves to compensate for the unequal distribution on TP and FP trials and allows for the computation (following the same procedure described above) of a q-value. Note that 1000 shuffles means that q-values < 0.001 cannot be reached.
Analysis of movement error potential responses
The speed at which a movement error occurs will likely affect the moment at which the error is perceived by the user. If the cursor quickly makes a jump in the wrong direct the error will likely be perceived shortly after the beginning of the error movement. However, if the cursor slowly drifts in the wrong direction the error may be perceived with a longer delay relative to the beginning of the error movement. For this reason, a four-step analysis was developed to allow for jitter in the timing of the potential error response relative to the start of the error movement (see Fig. 2).
First, the mean smoothed response trace (over all error movement trials) of the dLPFC bi-polar electrode for the range of − 0.2 to 1.2 s time locked to the beginning of the error movements was computed (Fig. 2: Step 1). Next, this mean was shifted for the range of − 0.1 to 0.6 s over each trial until the shift with the highest correlation to the original mean was found, and a new mean over the shifted trials was computed (Fig. 2: Step 2). This process of shifting to fit the mean response traces and computing a new mean trace was repeated 20 times (at with point no trial needed to be further shifted to optimally correlate to the mean). Then (Fig. 2: Step 3) 1000 trial-mean traces were computed by randomly shifting the actual error movement start times such that the − 0.2 to 1.2 s period relative to these times will no longer be time-lock to actual movement errors and repeating steps 1 and 2. This step thus produced 1000 mean traces that were not time locked to movement errors. For each of these 1000 pseudo-trial-mean-traces the maximum of the absolute value of the maximum and minimum of the trace was computed (max(abs(max(trace)), abs(min(trace))). Finally, this distribution of 1000 peak/trough values from the pseudo-trial-mean-traces was used to quantify the likelihood that that any peaks or troughs of the actual mean trace shifted around the actual error movements onsets indicate a significant feedback response (Fig. 2: Step 4). In this context significance was defined to be trace values whose absolute values were > 95% of the 1000 pseudo-trial-peak/trough values.
Steps 3 and 4 were done because the shifting in Steps 1 and 2 serves to sharpen or increase any peak or trough in the mean and thus prevents a direct comparison to non-shifted pseudo-trial-mean trace values.