Keywords

1 Introduction

Until a few years ago, the brain-computer interface (BCI) research community was strongly focused on providing communication for persons with severe motor disabilities who could not reliably communicate otherwise. Many such BCIs relied on motor imagery (MI) [1, 2, 11, 14]. While these efforts have met with some success, they have yielded little benefit to people outside of this relatively small patient group. Recent scientific and commentary articles have presented new directions with BCI technology, including new goals with new patient groups.

One of the most promising new directions entails using BCIs based on MI to provide new options for stroke patients [3–9, 12, 13]. Rather than providing communication, the MI is used to introduce closed-loop feedback within conventional motor rehabilitation therapy. This approach is called paired associative stimulation (PAS) because it pairs each user’s motor imagery with stimulation and feedback, such as activation of a functional electrical stimulator (FES), avatar movement, and/or auditory feedback indicating successful task completion.

PAS is a crucial element of the feedback cycle because of Hebbian learning, which is widely recognized as a fundamental principle of learning. Hebbian learning states that neural connections are strengthened only when the presynaptic and postsynaptic neurons are both active. Conventional rehabilitation therapy is unpaired, because feedback often occurs when users are not performing the required MI. Users would thus receive rewarding feedback even though they are not producing the necessary MI and concordant neural activation [11]. This is a major reason why conventional movement rehabilitation therapy tends to produce, at best, moderate improvement during the acute phase. Improvement after the acute phase is even rarer.

Figure 1 illustrates the concept underlying RecoveriX, a complete hardware and software platform that can record, analyze, and utilize EEG activity in real-time to “close the loop” in stroke rehabilitation. The user (top left) imagines or performs specific movements, such as wrist dorsiflexion. The resulting EEG activity is detected through electrodes positioned in an electrode cap, then sent to an amplifier. In this figure, the amplification occurs through a small purple box on the back of the cap, and the resulting EEG signals are then transmitted wirelessly to a laptop. The laptop manages data analysis and presentation of feedback.

Fig. 1.
figure 1

A schematic illustration of the conceptual approach used in RecoveriX.

Full size image

Like conventional therapy, RecoveriX users are instructed to imagine motor activity and receive feedback (specifically, through an avatar and FES). Unlike conventional therapy, RecoveriX users also wear an EEG cap that monitors MI that influences the feedback. The key element of PAS is the real-time connection between brain activity and feedback. In Fig. 1, this feedback is provided through an FES and a third-person view of an avatar’s hands. The feedback only occurs when the user imagines movement. Thus, unlike conventional therapy, the feedback is always paired with brain activity.

This paper presents further details about our system, experimental procedures, and other methods, results from two patients, and future directions. Many of our future directions will be addressed within the new RecoveriX project, an SME Instrument active from 2016–2018. Since PAS is a new research direction, and RecoveriX is the first system to implement it, we currently have only initial results available.

2 Methods

2.1 Subjects

We present data from two subjects, both of whom signed a consent form. Subject 1 is a right-handed man, born in 1953, who suffered a stroke in 2014 that left him with some difficulty moving his right hand. Shortly after his stroke, he participated in 24 RecoveriX training sessions. Subject 2 is a right-handed woman, born in 1974, who had a stroke in 2010. After her stroke, her left hand was completely paralyzed. For two years, she participated in conventional therapy, which produced no improvement. In 2014, she participated in 10 RecoveriX training sessions.

Both patients were recorded in an open room at the Rehabilitation Hospital of Iasi. The patients were not placed in an anechoic chamber to reduce noise that might affect the EEG, and none of the equipment was placed in a shielded area.

2.2 Data Acquisition

Data were recorded from a 45 channel electrode cap using g.Ladybird active electrodes (Fig. 2). We recorded from electrode sites overlying the sensorimotor area. The ground electrode was placed on the forehead and the reference on the right ear lobe. Data were transmitted via cables to a g.HIamp, which then relayed the data to a laptop that was running the RecoveriX software.

Fig. 2.
figure 2

The electrode montage, with sites used shown in red (Color figure online).

Full size image

The method of CSP yields a set of spatial filters that are designed to minimize the variance of one class while maximizing it for the other class. Given N channels of EEG for each left and right trial, the CSP method provides an N × N projection matrix. This matrix is a set of subject-dependent spatial patterns, which reflect the specific activation of cortical areas during hand movement imagination. With the projection matrix W, the decomposition of a trial X is described by

$$ Z = WX $$
(1)

This transformation projects the variance of X onto the rows of Z and results in N new time series. The columns of W−1 are a set of CSPs and can be considered as time-invariant EEG source distributions. Due to the definition of W, the variance for a left hand movement imagination is largest in the first row of Z and decreases in each subsequent row. The opposite occurs for a trial with right hand motor imagery. To classify the left and right trials, the variances have to be extracted as reliable features of the newly designed N time series. However, it is not necessary to calculate the variances of all N time series. The method provides a dimensionality reduction of the EEG. Mueller-Gerking et al. [15] showed that the optimal number of CSPs is four. Based on their results, after building the projection matrix W from an artifact corrected training set XT, only the first and last two rows (p = 4) of W are used to process new input data X. Then the variance (VARp) of the resulting four time series is calculated for a time window T. These values are normalized and log transformed according to the formula:

$$ f_{p} = \log_{10} \left( {\frac{{VAR_{p} }}{{\mathop \sum \nolimits_{P = 1}^{4} VAR_{p} }}} \right) $$
(2)

Where fp (p = 1.4) are the normalized feature vectors and VARp is the variance of the p-th spatially filtered signal. These four features can be classified with a linear discriminant analysis (LDA) classifier.

2.3 Stimuli and Procedure

After informed consent and being prepared for EEG recording, each patient was seated in a comfortable chair, about one meter in front of a monitor that presented cues and feedback (see Fig. 3). FES pads were placed over the dorsal side of the forearm with the affected hand. Figure 3 shows Patient 1, who had FES pads over the right forearm. Each trial began with a cue presented on a monitor in the form of a red arrow pointing to the left or right, which instructed the subject to imagine left or right hand movement. After a delay of 0.5 s, the user began to receive two types of feedback. First, a blue bar extended to the left or right, indicating both the direction and magnitude of the motor imagery. Second, the FES would activate if the user was imagining movement in the affected hand. This FES was sufficiently strong to cause movement in the affected hand. Feedback continued for 4 s, after which the screen went blank and any FES activation ended. There was a 2 s break before the next trial began (Fig. 4).

Fig. 3.
figure 3

A photograph of Patient 1 using the system.

Full size image
Fig. 4.
figure 4

Timing within one trial. A fixation cross appears when the trial begins A short beep cues the user after two seconds. One second later, a visual cue is presented. From 4.25 s until the end of the trial (8 s), online feedback is presented.

Full size image

3 Results

3.1 Patient 1

Table 1 and Figs. 5, 6 and 7 present results from Patient 1.

Table 1. This table presents results from the 9-HPT with Patient 1. These results indicate steady improvement over about three months. At the end of training, the subject could perform the test with his right hand about as well as his unaffected left hand. The times indicate the time required to successfully complete the task. “Falls” reflects the number of times that a peg was dropped, which did not occur with either hand.
Full size table
Fig. 5.
figure 5

Patient 1’s BCI classification accuracy across twenty four RecoveriX usage sessions. Chance accuracy was 50%.

Full size image
Fig. 6.
figure 6

Patient 1’s LDA values in session 1 (Color figure online).

Full size image
Fig. 7.
figure 7

Patient 1’s LDA values in session 24 (Color figure online).

Full size image

3.2 Patient 2

We did not attempt the 9-HPT with Patient 2, because her left hand movement impairment was too severe. However, Fig. 8 below shows that she regained limited left hand control after only ten training sessions.

Fig. 8.
figure 8

Three images showing Patient 2 performing left wrist dorsiflexion.

Full size image

Figure 9 presents BCI classification accuracy across Patient 2’s ten sessions of BCI use. Since she was effectively participating in a two-choice task, accuracy in the first two sessions is only slightly better than chance accuracy of 50%. The remaining eight sessions exhibit a substantial, though not monotonic, improvement (Figs. 9, 10 and 11).

Fig. 9.
figure 9

Patient 2’s BCI classification accuracy across 10 RecoveriX usage sessions.

Full size image
Fig. 10.
figure 10

Patient 2’s LDA values in session 1 (Color figure online).

Full size image
Fig. 11.
figure 11

Patient 2’s LDA values in session 10 (Color figure online).

Full size image

4 Discussion

4.1 Discussion of Results

Our results include both electrophysiological and behavioral analyses, including BCI accuracy, scalp maps, and performance on standardized tests such as the Nine-Hole Peg Test (9-HPT). These results support our view that RecoveriX can effectively incorporate EEG activity within a closed loop neurofeedback paradigm, and lead to improved functional outcomes. Notably, we present promising results even during the post-acute stage, after conventional therapy was ineffective.

4.2 Future Directions

We emphasize that these are still preliminary results. We do not have a direct comparison between conventional and PAS therapy. This is a major future direction that is clearly needed, and is currently our highest research priority, but is well beyond the scope of this preliminary report. Patient 2 did regain limited movement with RecoveriX after conventional therapy was unsuccessful, but this is only one result without a properly controlled experiment. This paper also validates the RecoveriX approach from a technical perspective by demonstrating that, at least with the two patients using the prototype system presented here, our system can function in real-world settings.

Another future direction involves improved immersive software to present feedback and maximize users’ engagement. The patients described above saw feedback in the form of a blue bar, which has been used in many MI BCI studies but it graphically simple. We are exploring more advanced environments, such as different views of an avatar whose movements attempt to mimic the movements that the user imagines. In addition to providing feedback, these avatars could also help instruct users regarding which movements to perform. Figure 12 presents three examples of avatar feedback for different types of movements.

Fig. 12.
figure 12

Examples of immersive environments with avatars to provide instructions and visual feedback. Left: imagine left wrist dorsiflexion; Middle: a different view for imagining left elbow flexion; Right: imagine knee extension.

Full size image

Figure 12 also reflects our effort to expand the movements that RecoveriX can detect and support. The work presented here only addressed wrist and grasp function. In future work, we hope to support a broader range of movement types, including more upper-limb activities and lower limb movements. This will allow us to help a broader range of patients who are seeking rehabilitation for different areas impaired by stroke. This effort partly leverages our activity in the EU-funded BETTER project, which explored BCIs for lower-limb and gait rehabilitation.

We are also exploring improved hardware. The patients presented here used a 16 channel wired electrode cap with gel electrodes. However, we recently developed a wireless version with a smaller montage focused over motor areas. This system (see Figs. 1 and 8) has a much smaller amplifier that is weighs only 70 g, and should be more practical in field settings. Furthermore, based on analyses with 16 channel data, we identified the most useful electrode sites for PAS. Figure 13 presents the new montage that we plan to use for future research.

Fig. 13.
figure 13

The new montage that we plan to use for future research. The electrode sites are centered around C3, C4 and Cz. All electrodes are placed near the sensorimotor cortex (Color figure online).

Full size image

Another future direction involves incorporation of other signals to complement EEG activity. EMG (electromyogram) measures and position sensors are of obvious value, and other signals such as heart rate and eye activity could help identify (at least) if users are confused, frustrated, or overloaded. This idea leverages the “hybrid” BCI concept [10], essentially creating a system called a brain-neuronal computer interface (BNCI) that could be more informative than EEG alone [11, 12].

Finally, beyond influencing the PAS feedback loop, real-time detection of MI could be helpful for assessing compliance. Physiotherapists recognize that patients often do not comply with task instructions. Noncompliance may occur because patients are unmotivated, inattentive, unable to maintain the required MI throughout the trial, uncomfortable with the advanced technologies in rehabilitation, or misunderstand task instructions. BCI research has also shown that differences in task instructions can affect BCI classification accuracy [14]. By determining when users are performing the correct MI, and assessing the intensity of this MI, RecoveriX and other tools could help patients and therapist recognize noncompliance. Therapists could then take corrective measures such as providing revised instructions, counseling patients to improve motivation, and/or modifying the task and feedback environment.